A model fusion method and related apparatus

Abstract

The application discloses a model fusion method applied to the technical field of computer vision. The method comprises the following steps: acquiring a first model and a second model, wherein the first model and the second model both belong to part of human body models; respectively registering the first model and the second model with a preset human body model to obtain a third model, wherein the third model comprises the registered first model and the registered second model; performing deformation processing on a transition region in the third model to smooth the transition region, and obtaining a fourth model; wherein the transition region comprises a part of the registered first model close to the registered second model and a part of the registered second model close to the registered first model. Based on the scheme, different head models and body models can be fused to generate a new human body model, and the generation efficiency of the human body model is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a model fusion method and related apparatus. Background Technology

[0002] With the rapid development of computer graphics and computer vision technologies, virtual human models have entered people's daily lives extensively, such as a large number of virtual characters in the gaming industry, and an increasing number of virtual anchor images and other virtual human models.

[0003] Currently, the mainstream technologies for generating virtual human models include equipment scanning and manual design by computer graphics (CG) designers. Specifically, after acquiring raw human body data using laser equipment or SLR camera arrays, the data is processed by algorithms to generate a raw model, which is then further refined by designers into a virtual human model that meets the required specifications. In addition to the relatively high investment in equipment, generating virtual human models requires a significant amount of manpower and time. Generally, it takes 2-5 months to complete a high-precision virtual human model, resulting in extremely low production efficiency. The processed virtual human models are then applied to various scenarios, such as consumer entertainment scenarios with lower precision requirements, large-scale game character generation scenarios with higher precision requirements, and even more demanding film and television scenarios.

[0004] Generally, virtual human models created by different manufacturers or designers often employ different production standards. For example, models of the same category created by the same manufacturer typically require a unified geometric topology and texture mapping relationship, making it impossible to share and interchange model materials from different sources. Developers often face a large number of virtual human models without a unified standard, yet they cannot effectively utilize existing models and often need to invest significant resources in additional modeling or standardization, resulting in low efficiency in virtual human model generation. Summary of the Invention

[0005] This application provides a model fusion method that can fuse head models and body models from different sources to generate new human body models, thereby improving the generation efficiency of human body models.

[0006] This application provides a model fusion method, comprising: acquiring a first model and a second model, both of which are partial models within a human body model. For example, the first model is a target head model and the second model is a target body model; or, the first model is a target torso model and the second model is a target limb model. Specifically, the first model and the second model can be models from different sources, that is, the geometric topology and texture of the first model and the second model can be different.

[0007] The first model and the second model are registered with a preset human body model to obtain a third model, which includes the registered first model and the registered second model. Model registration refers to the process of matching and superimposing two models of the same type but different shapes and sizes. In the registered two models, corresponding spatial coordinate points are located at the same position.

[0008] A deformation process is performed on the transition region in the third model to smooth the transition region, resulting in a fourth model. The transition region includes a portion of the registered first model that is close to the registered second model, and a portion of the registered second model that is close to the registered first model. For example, the transition region could be the neck region in the third model.

[0009] In this scheme, by registering a first model and a second model from different sources with the same preset human body model, a preliminary fusion of the first and second models based on a unified body model is achieved. Then, by smoothing the transition region between the first and second models, a precise fusion of the first and second models is achieved, thereby generating a new human body model. This improves the generation efficiency of the human body model while maintaining the accuracy of the original model.

[0010] In one possible implementation, deformation processing is performed on the transition region in the third model to smooth the transition region and obtain a fourth model. This includes: determining a second region in the third model corresponding to the first region based on a first region in the preset human body model, wherein the second region includes the transition region. Specifically, when the first model is a target head model and the second model is a target body model, the first region may be located in the neck region of the preset human body model. Since the transition region between the target head model and the target body model is usually located in the neck region, the first region in the preset human body model is located in the neck region of the preset human body model. Thus, the second region determined based on the first region is also located in the neck region of the third model.

[0011] Based on the first and second control regions in the second region, the deformation coefficient of the transition region is determined. The first and second control regions are located on different target models in the third model, and the first and second control regions correspond to the same region in the preset human body model. That is, the first control region may be located on the target head model, and the second control region may be located on the target body model; or, the first control region may be located on the target body model, and the second control region may be located on the target head model.

[0012] Based on the deformation coefficient, deformation processing is performed on the transition region in the third model, that is, coordinate transformation is performed on the coordinate points in the transition region so that the target head model and the target body model can be smoothly connected, and finally the fourth model is obtained.

[0013] In one possible implementation, determining the deformation coefficient of the transition region based on the first control region and the second control region in the second region includes:

[0014] Multiple first sampling points in the first control region, multiple second sampling points in the second control region, and multiple third sampling points in the third control region are acquired, wherein the multiple first sampling points and the multiple second sampling points have a corresponding relationship. The second region includes the third control region, meaning the third control region is also mapped from the first region in a preset human body model, and the third control region and the first control region are located on the same target model. Specifically, the third control region and the first control region are both located on the target head model, while the second control region is located on the target body model; or, the third control region and the first control region are both located on the target body model, while the second control region is located on the target head model.

[0015] The deformation coefficient of the transition region is determined based on the coordinate transformation relationship between the plurality of first sampling points and the plurality of second sampling points, and the coordinate transformation relationship between the plurality of third sampling points.

[0016] In this scheme, the deformation coefficient is obtained by using the coordinate transformation relationship between the first control region and the second control region, as well as the coordinate transformation relationship between the third control region and itself. This enables the deformation processing based on the deformation coefficient to gradually increase the deformation amplitude in the transition region along the vertical direction, thus achieving a smooth transition in the transition region.

[0017] In one possible implementation, the region in the transition region where deformation processing is performed includes the first control region, the third control region, and the deformation region located between the first control region and the third control region.

[0018] In one possible implementation, to achieve a smooth transition of the transition region as much as possible, the areas of both the first control region and the third control region are smaller than the area of ​​the deformed region. For example, the area ratio of the first control region, the third control region, and the deformed region can be 1:1:3 or 1:1:2.

[0019] In one possible implementation, when the first model is a target head model and the second model is a target body model, the first control region, the third control region, and the deformation region are located on the target head model, and the first control region is close to the target body model, while the second control region is located on the target body model; or, the first control region, the third control region, and the deformation region are located on the target body model, and the first control region is close to the target head model, while the second control region is located on the target head model.

[0020] In one possible implementation, when the first model is a target head model and the second model is a target body model, determining the second region in the third model corresponding to the first region based on the first region in the preset human body model includes: obtaining the central axis of the neck in the preset human body model; determining the second region corresponding to the first region based on the central axis and the first region; wherein any two corresponding sampling points in the first region and the second region are located on a straight line perpendicular to the central axis.

[0021] In this solution, the neck region in the preset human body model is fitted as a cylinder, and the second region corresponding to the first region in the preset human body model is obtained in the third model based on vertical mapping, thereby achieving rapid determination of the second region and improving the feasibility of the solution.

[0022] In one possible implementation, when the second model is a target body model, the registration of the second model with a preset human body model includes: determining a plurality of first key feature points in the target body model; obtaining a plurality of pre-marked second key feature points in the body model of the preset human body model, wherein the plurality of first key feature points and the plurality of second key feature points have a corresponding relationship; adjusting the shape and / or posture of the body model of the preset human body model according to the plurality of first key feature points and the plurality of second key feature points, and registering the adjusted preset human body model onto the target body model.

[0023] In one possible implementation, the plurality of first key feature points include one or more of the following: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0024] In one possible implementation, when the first model is a target head model, the registration of the first model with a preset human body model includes: determining multiple third key feature points in the target head model; obtaining multiple pre-labeled fourth key feature points in the head model of the preset human body model, wherein the multiple third key feature points and the multiple fourth key feature points have a corresponding relationship; and registering the target head model to the head model of the preset human body model according to the multiple third key feature points and the multiple fourth key feature points.

[0025] In one possible implementation, the plurality of third key feature points include one or more of the left eye, right eye, tip of the nose, left corner of the mouth, and right corner of the mouth.

[0026] In one possible implementation, determining multiple third key feature points in the target head model includes: acquiring multiple head projection images of the target head model in different directions; performing face detection on the multiple projection images to determine the orientation of the face in the target head model; determining multiple face projection images of the face in the target head model at different projection angles based on the orientation of the face in the target head model; detecting key facial feature points in the multiple face projection images respectively, and determining the spatial coordinates of multiple third key feature points in the target head model based on the key facial feature points. The category of the key facial feature points can be the same as the category of the third key feature points, i.e., the key facial feature points can include the left eye, right eye, nose tip, left corner of mouth, and right corner of mouth.

[0027] In this scheme, the spatial coordinates of the third key feature point in the target head model are determined by projecting the target head model into two-dimensional images at different angles and detecting key facial feature points in multiple two-dimensional images, thereby improving the feasibility of the scheme.

[0028] A second aspect of this application provides a model fusion apparatus, comprising: an acquisition unit for acquiring a first model and a second model, both of which are partial models within a human body model; a registration unit for registering the first model and the second model with a preset human body model respectively to obtain a third model, the third model including the registered first model and the registered second model; and a processing unit for performing deformation processing on a transition region in the third model to smooth the transition region and obtain a fourth model; wherein the transition region includes a portion of the registered first model that is close to the registered second model, and a portion of the registered second model that is close to the registered first model.

[0029] In one possible implementation, the first model is a target head model, and the second model is a target body model;

[0030] Alternatively, the first model may be a target torso model, and the second model may be a target limb model.

[0031] In one possible implementation, the processing unit is specifically configured to: determine a second region in the third model corresponding to the first region based on a first region in the preset human body model, wherein the first region is located in the neck region of the preset human body model, and the second region includes the transition region; determine a deformation coefficient of the transition region based on a first control region and a second control region in the second region, wherein the first control region and the second control region are located on different target models in the third model, and the first control region and the second control region correspond to the same region in the preset human body model; and perform deformation processing on the transition region in the third model based on the deformation coefficient to obtain the fourth model.

[0032] In one possible implementation, the processing unit is specifically configured to: acquire a plurality of first sampling points in the first control region, a plurality of second sampling points in the second control region, and a plurality of third sampling points in the third control region, wherein the plurality of first sampling points correspond to the plurality of second sampling points, the second region includes the third control region, and the third control region and the first control region are located on the same target model; and determine the deformation coefficient of the transition region based on the coordinate transformation relationship between the plurality of first sampling points and the plurality of second sampling points and the coordinate transformation relationship between the plurality of third sampling points.

[0033] In one possible implementation, the region in the transition region where deformation processing is performed includes the first control region, the third control region, and the deformation region located between the first control region and the third control region.

[0034] In one possible implementation, the areas of both the first control region and the third control region are smaller than the area of ​​the deformable region.

[0035] In one possible implementation, when the first model is a target head model and the second model is a target body model, the first control region, the third control region, and the deformation region are located on the target head model, and the first control region is close to the target body model, while the second control region is located on the target body model; or, the first control region, the third control region, and the deformation region are located on the target body model, and the first control region is close to the target head model, while the second control region is located on the target head model.

[0036] In one possible implementation, when the first model is a target head model and the second model is a target body model, the processing unit is specifically used to: obtain the central axis of the neck in the preset human body model; determine the second region corresponding to the first region based on the central axis and the first region; wherein any two corresponding sampling points in the first region and the second region are located on a straight line perpendicular to the central axis.

[0037] In one possible implementation, when the second model is a target body model, the registration unit is specifically used to: determine a plurality of first key feature points in the target body model; obtain a plurality of pre-marked second key feature points in the body model of the preset human model, wherein the plurality of first key feature points and the plurality of second key feature points have a corresponding relationship; adjust the shape and / or posture of the body model of the preset human model according to the plurality of first key feature points and the plurality of second key feature points, and register the adjusted preset human model to the target body model.

[0038] In one possible implementation, the plurality of first key feature points include one or more of the following: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0039] In one possible implementation, when the first model is a target head model, the registration unit is specifically used to: determine multiple third key feature points in the target head model; obtain multiple pre-marked fourth key feature points in the head model of the preset human body model, wherein the multiple third key feature points and the multiple fourth key feature points have a corresponding relationship; and register the target head model to the head model of the preset human body model according to the multiple third key feature points and the multiple fourth key feature points.

[0040] In one possible implementation, the plurality of third key feature points include one or more of the left eye, right eye, tip of the nose, left corner of the mouth, and right corner of the mouth.

[0041] In one possible implementation, the registration unit is specifically configured to: acquire multiple head projection images of the target head model in different directions; perform face detection on the multiple projection images to determine the orientation of the face in the target head model; determine multiple face projection images of the face in the target head model at different projection angles based on the orientation of the face in the target head model; detect key facial feature points in the multiple face projection images respectively, and determine the spatial coordinates of multiple third key feature points in the target head model based on the key facial feature points. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application;

[0043] Figure 2 A schematic flowchart of a model fusion method 200 provided in an embodiment of this application;

[0044] Figure 3 A schematic diagram illustrating model registration as provided in an embodiment of this application;

[0045] Figure 4 A schematic diagram of the vertical mapping of a region provided in an embodiment of this application;

[0046] Figure 5 This is a schematic diagram illustrating the division of a first region in a preset human body model, as provided in an embodiment of this application.

[0047] Figure 6 A schematic diagram of a target head model and a target body model provided for embodiments of this application;

[0048] Figure 7 This application provides a schematic diagram of obtaining a second region in a third model based on mapping a first region in a preset human body model.

[0049] Figure 8 A schematic diagram of a fourth model obtained after deformation processing, provided in an embodiment of this application;

[0050] Figure 9 A schematic diagram of multiple third key feature points provided in the embodiments of this application;

[0051] Figure 10 This is a schematic diagram of the structure of a model fusion device provided in an embodiment of this application;

[0052] Figure 11 This is a schematic diagram of the structure of a computer-readable storage medium device provided in an embodiment of this application. Detailed Implementation

[0053] The embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. As those skilled in the art will recognize, with the development of technology and the emergence of new scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0055] Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to those processes, methods, products, or apparatuses. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect can be achieved.

[0056] For ease of understanding, the technical terms involved in the embodiments of this application will be introduced below.

[0057] Model registration is the process of matching and superimposing two models of the same type but different shapes and sizes. After registration, corresponding spatial coordinate points in the two models are located at the same position. For example, for two registered head models, corresponding feature points such as the mouth, eyes, and nose are located at the same position. That is, the mouth in head model A is in the same position as the mouth in head model B, the eyes in head model A are in the same position as the eyes in head model B, and the nose in head model A is in the same position as the nose in head model B.

[0058] Structure from Motion (SfM): A self-calibration technique that automatically performs camera tracking and motion matching. The goal is to automatically reconstruct camera motion and scene structure using two or more scenes.

[0059] Multi-view Stereo (MVS): A method for generating dense point cloud data from 2D images to construct 3D models. MVS obtains dense point cloud data by matching a large number of pixels in an image and reconstructing the 3D coordinates of each pixel.

[0060] Neural Radiance Field (NERF): A method that uses a two-dimensional image as input to a neural network to construct a static three-dimensional model. Typically, to train the neural network, a large number of two-dimensional images with known camera parameters are needed for a static scene. The neural network trained based on these two-dimensional images can then render a three-dimensional model corresponding to the two-dimensional image from any angle.

[0061] In the gaming industry, the generation of virtual human models mainly involves two stages: initial design by designers and subsequent editing by users. In the first stage, designers create a certain number of basic models and materials, such as several virtual human models of different races, as well as various hairstyles, facial features, skin tones, and clothing. Models and materials of the same category are required to have consistent geometric topology and texture mapping relationships. In the second stage, when users enter the character system, they can select and combine models from a database according to their preferences to create their desired virtual human model. This method of generating virtual human models effectively enriches the styles of digital human models and meets the preferences of different users, but it still has certain limitations. Specifically, this method still requires designers to complete a large amount of design work in the early stages, often making it only suitable for high-budget game projects; secondly, most model materials must meet a strict set of standards, such as consistent geometric topology and consistent texture mapping relationships.

[0062] Generally, virtual human models created by different manufacturers or designers often employ different production standards. For example, models of the same category created by the same manufacturer typically require a unified geometric topology and texture mapping relationship, making it impossible to share and interchange model materials from different sources. Developers often face a large number of virtual human models without a unified standard, yet they cannot effectively utilize existing models and often need to invest significant resources in additional modeling or standardization, resulting in low efficiency in virtual human model generation.

[0063] In simple terms, when faced with virtual human body models from different sources, developers cannot create new models by merging these existing models because the production standards of these models are different.

[0064] In view of this, embodiments of this application provide a model fusion method. By registering target head models and target body models from different sources with the same preset human body model, a preliminary fusion of the target head model and target body model based on a unified body model is achieved. Then, by smoothing the transition region between the target head model and the target body model, a precise fusion of the target head model and target body model is achieved, thereby generating a new human body model. Based on this solution, head models and body models from different sources can be fused, such as head models and body models with inconsistent geometric topologies, thereby generating a new human body model and improving the efficiency of human body model generation.

[0065] The above describes the application scenarios of the methods provided in the embodiments of this application. The following will describe the devices used in the methods provided in the embodiments of this application.

[0066] Specifically, the model fusion method provided in this application embodiment can be applied to electronic devices. For example, the electronic device may be a server, smartphone, personal computer (PC), laptop, tablet, smart TV, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless electronic device in industrial control, wireless electronic device in remote medical surgery, wireless electronic device in intelligent driving, wireless electronic device in smart grid, wireless electronic device in transportation safety, wireless electronic device in smart city, wireless electronic device in smart home, etc.

[0067] For ease of description, the following will take the application of the method provided in the embodiments of this application to a server as an example to introduce the method provided in the embodiments of this application.

[0068] To facilitate understanding of this solution, the embodiments of this application first combine... Figure 1 The structure of the electronic device provided in this application is described.

[0069] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the structure of an electronic device 101 provided in an embodiment of this application. Figure 1As shown, electronic device 101 includes processor 103, which is coupled to system bus 105. Processor 103 can be one or more processors, each of which can include one or more processor cores. A video adapter 107 drives a display 109, which is coupled to system bus 105. System bus 105 is coupled to input / output (I / O) bus via bus bridge 111. I / O interface 115 is coupled to the I / O bus. I / O interface 115 communicates with various I / O devices, such as input device 117 (e.g., touchscreen), external storage 121 (e.g., hard disk, floppy disk, optical disk, or USB flash drive), multimedia interface, etc. A transceiver 123 (capable of sending and / or receiving radio communication signals), a camera 155 (capable of capturing still and moving digital video images), and an external USB port 125. Optionally, the interface connected to I / O interface 115 can be a USB interface.

[0070] The processor 103 can be any conventional processor, including a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, or a combination thereof. Optionally, the processor can be a special-purpose device such as an ASIC.

[0071] Electronic device 101 can communicate with software deployment server 149 via network interface 129. Exemplarily, network interface 129 is a hardware network interface, such as a network interface card (NIC). Network 127 can be an external network, such as the Internet, or an internal network, such as Ethernet or a virtual private network (VPN). Optionally, network 127 can also be a wireless network, such as a WiFi network or a cellular network.

[0072] Hard disk drive interface 131 is coupled to system bus 105. Hardware driver interface is connected to hard disk drive 133. Internal memory 135 is coupled to system bus 105. Data running in internal memory 135 may include operating system (OS) 137, applications 143, and schedules of electronic device 101.

[0073] The processor 103 can communicate with the internal memory 135 via the system bus 105, and retrieve instructions and data from the application program 143 from the internal memory 135 to execute the program.

[0074] An operating system consists of the Shell 139 and the kernel 141. The Shell 139 is an interface between the user and the operating system kernel. The shell is the outermost layer of the operating system. The shell manages the interaction between the user and the operating system: waiting for user input, interpreting user input for the operating system, and processing various operating system outputs.

[0075] Kernel 141 consists of the parts of the operating system used to manage memory, files, peripherals, and system resources. Kernel 141 interacts directly with the hardware. The operating system kernel typically runs processes and provides inter-process communication, CPU time-slice management, interrupts, memory management, I / O management, and so on.

[0076] For example, in the case where the electronic device 101 is a smartphone, the application 143 includes instant messaging related programs. In one embodiment, when the application 143 needs to be executed, the electronic device 101 can download the application 143 from the software deployment server 149.

[0077] The above describes the scenarios and devices used in the embodiments of this application. The model fusion method provided in the embodiments of this application will be described in detail below.

[0078] Please refer to Figure 2 , Figure 2 This is a schematic flowchart of a model fusion method 200 provided in an embodiment of this application. Figure 2 As shown, the model fusion method 200 includes the following steps 201-203.

[0079] Step 201: Obtain the first model and the second model, both of which are partial models in the human body model.

[0080] In this embodiment, the first model and the second model are two models to be fused, and both the first model and the second model are partial models within a human body model. The first model may be, for example, a target head model, and the second model may be, for example, a target body model; or, the first model may be, for example, a target torso model, and the second model may be, for example, a target limb model. For ease of description, the model fusion method provided in this embodiment will be described in detail below using the first model as the target head model and the second model as the target body model as an example.

[0081] The target head model and the target body model are the two models to be fused. By fusing the target head model and the target body model, a new human body model can be obtained. Specifically, the target head model and the target body model can be models from different sources, that is, the geometric topology and texture of the target head model and the target body model can be different.

[0082] There are multiple ways to obtain both the target head model and the target body model.

[0083] Method 1: Obtain the target head model or target body model based on the already created human body model.

[0084] In Method 1, given a pre-made complete human body model, the target head model and target body model can be obtained separately from different human body models. For example, given pre-made human body models A and B, head model 'a' from human body model A can be obtained as the target head model; and body model 'b' from human body model B can be obtained as the target body model. Simply put, in this scenario of Method 1, the user already has two or more human body models and does not need to generate additional models using other methods; instead, the target body model and target head model can be selected from the existing models.

[0085] It should be noted that, when the user inputs two complete human body models, this embodiment can automatically separate the head model and body model from the input human body model to obtain the target head model and target body model. Alternatively, the user can manually separate the head and body models from the human body model to obtain the target head model and target body model.

[0086] Method 2 involves the designer manually designing and creating the target head model or target body model.

[0087] In method 2, the model is designed and manufactured by hand by a designer, and the target body model and target head model can be made with different levels of detail according to actual needs. The target head model and target body model can be designed by different designers, and the production standards of the target head model and target body model can also be different, that is, the geometric topology and texture of the target head model and target body model can be different.

[0088] Method 3 involves collecting and processing data using a light stage device to generate a target head model or target body model.

[0089] Lightstage equipment consists of multiple cameras and multiple light sources, specifically designed for high-precision modeling and data acquisition for relighting. Generally, models built using lightstage equipment have high accuracy but are also more expensive. Data acquisition methods using lightstage equipment can typically be divided into full-body acquisition and half-body acquisition (i.e., acquiring data from the head or above the chest). After obtaining the raw data using lightstage equipment, appropriate algorithms can be used to generate models.

[0090] Data collected and processed using the LightStage device can generate target head or body models. In some applications, users are typically more concerned with the accuracy of the head model, hoping to obtain a medium-quality body and a high-quality head. In such cases, since the model generated by the LightStage device has high accuracy, the target head model can be generated using the LightStage device, while other low-cost methods can be used to obtain the target body model.

[0091] Method 4 involves acquiring multi-view data using an RGB camera and combining it with methods such as Structure from Motion (SfM), Multi-view Stereo (MVS), and Neural Radiance Field (NERF) to generate a target head model or target body model.

[0092] In method 4, 2D image data of the same human body is acquired from multiple perspectives using an RGB camera. This acquired 2D image data is used as input, and SFM or MVS is applied. Through processes such as camera pose calculation, sparse point cloud generation, dense point cloud generation, model meshing, and texture mapping, the target head model and / or target body model are reconstructed. Alternatively, a neural network trained using the NERF method can be used. The 2D image data is input into the trained neural network to obtain the target head model and / or target body model.

[0093] Method 5 involves acquiring multi-view data using a depth camera or laser device and generating a target head model or target body model using a matching algorithm based on non-rigid deformation.

[0094] In method 5, depth information of the human object can be directly acquired using a depth camera or laser device, and a corresponding model can be generated based on the depth and color information of the human object. For example, in the case of using the Dynamic Fusion scheme, human data acquired by a Kinect device is used as input, and human-related color and depth information is acquired simultaneously. This input human information is then used to continuously deform and update the reconstructed object, ultimately resulting in a complete model. This type of method typically supports dynamic model reconstruction, allowing the scanned human object to have a certain degree of motion.

[0095] It is understood that, in addition to obtaining the target head model and target body model based on the above methods, the target head model and target body model can also be obtained by combining the above methods or by other methods. This embodiment does not specifically limit this.

[0096] Step 202: Register the first model and the second model with the preset human body model respectively to obtain a third model, which includes the registered first model and the registered second model.

[0097] Specifically, the target body model can be registered with the body model of the preset human body model, and the target head model can be registered with the head model of the preset human body model to obtain a third model. The third model is obtained by splicing the registered target head model and the registered target body model.

[0098] In this embodiment, the preset human body model is a complete human body model, including a head model and a body model, which can be used to achieve registration with the target head model and the target body model.

[0099] During the registration process between the target body model and the preset human body model, specific feature points in both models can be identified first, such as one or more feature points including the left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees, and left and right ankles. Based on the identified feature points in both models, the target body model is matched onto the preset human body model so that the corresponding feature points in both models are located in the same position, i.e., their spatial coordinates are consistent.

[0100] Similarly, in the process of registering the target head model with the head model of the preset human body model, specific feature points in both models can be determined first, such as one or more feature points including the eyes, the tip of the nose, and the corners of the mouth. Based on the determined feature points in both models, the target head model is registered onto the head model of the preset human body model so that the corresponding feature points in the two models are located in the same position, that is, the spatial coordinates of the corresponding feature points in the two models are consistent.

[0101] Since the head model and body model in the preset human body model are integrated, after registering the target head model and target body model with the preset human body model respectively, the initial fusion of the target head model and target body model can be achieved, resulting in a third model.

[0102] For example, see Figure 3 , Figure 3 This is a schematic diagram illustrating a model registration method provided in an embodiment of this application. Figure 3 As shown, by registering the target head model with the head model in the preset human body model, and registering the target body model with the body model in the preset human body model, the initial stitching of the target head model and the target body model can be achieved, resulting in the third model.

[0103] Step 203: Perform deformation processing on the transition region in the third model to smooth the transition region and obtain the fourth model.

[0104] The transition region includes a portion of the first registered model that is close to the second registered model, and a portion of the second registered model that is close to the first registered model. For example, the transition region could be the neck region in a third model.

[0105] In the third model, the target head model and the target body model are already in relatively reasonable positions. However, because the geometric topology of the target head model and the target body model may be different, the transition region between the target head model and the target body model may not be able to achieve a smooth transition. For example, if the neck of the target head model is thinner and the neck of the target body model is thicker, the target head model may not be able to be well spliced ​​with the target body model, resulting in a phenomenon in the third model where the neck area near the head model is thinner and the neck area near the body model is thicker.

[0106] Based on this, in this embodiment, deformation processing can be performed on the transition region in the third model so that the area connecting the target head model and the target body model can transition smoothly, thereby obtaining a fourth model after further fine fusion.

[0107] For example, when the neck of the target head model is thinner and the neck of the target body model is thicker, the neck of the target head model can be deformed by gradually thickening along the direction of the target body model so that the transition area between the target head model and the target body model can be as smooth as possible.

[0108] In this scheme, by registering target head models and target body models from different sources with the same preset human body model, a preliminary fusion of target head and body models based on a unified body model is achieved. Since the geometric topology of the head and body models in the preset human body model is consistent, registering the target head and body models separately to the same human body model enables the fusion of target head and body models with different geometric topologies. Then, by smoothing the transition region between the target head and body models, precise fusion of the target head and body models is achieved, thereby generating a new human body model and improving the efficiency of human body model generation.

[0109] To facilitate understanding, the process of performing deformation processing on the transition region in the third model will be described in detail below.

[0110] In one possible embodiment, step 203 described above specifically includes the following steps 2031-2033.

[0111] Step 2031: Based on the first region in the preset human body model, determine the second region in the third model corresponding to the first region, wherein the first region is located in the neck region of the preset human body model, and the second region includes the transition region.

[0112] In this embodiment, a first region can be pre-divided in a preset human body model to facilitate the determination of a second region in the third model corresponding to the first region. This second region is used for subsequent deformation algorithms to enable deformation of transition regions in the third model.

[0113] Since the transition area between the target head model and the target body model is usually located in the neck region, the first region in the preset human body model is located in the neck region of the preset human body model. Thus, the second region, determined based on the first region, is also located in the neck region of the third model. Furthermore, the second region includes the transition area to be deformed.

[0114] In this embodiment, the second region corresponding to the first region in the third model can be determined by migration sampling. Specifically, after determining the first region in the preset human body model, the first region can be migrated to the corresponding regions of the target head model and the target body model through vertical mapping and sampling is completed. Finally, the sampling points are connected to form the second region.

[0115] For example, since the neck region in the preset human body model can be approximated as a cylinder, the neck region of the preset human body model can be fitted as a cylinder, and the central axis of the cylinder can be used as the central axis of the neck in the preset human body model.

[0116] Then, based on the central axis and the first region in the preset human body model, the second region corresponding to the first region is determined, wherein any two corresponding sampling points in the first and second regions lie on a straight line perpendicular to the central axis. Specifically, multiple sampling points in the first region can be predetermined, and then a straight line perpendicular to the central axis can be drawn for each sampling point in the first region. The intersection of this straight line with the target head model or target body model (hereinafter referred to as the target model) is then obtained, thereby obtaining the corresponding sampling point in the target model. By obtaining the straight line perpendicular to the central axis for each sampling point in the first region, the sampling points corresponding to each sampling point in the first region in the third model can be obtained. Finally, based on the sampling points in the third model, the second region corresponding to the first region in the third model can be constructed.

[0117] Please refer to Figure 4 , Figure 4 This is a schematic diagram of a vertical mapping of a region provided in an embodiment of this application. For example... Figure 4 As shown in (a), for any sampling point P in the first region, a straight line can be drawn through sampling point P, perpendicular to the central axis m in the preset human body model. Furthermore, the direction from the central axis m to sampling point P is n. Figure 4 As shown in (b), the straight line passing through sampling point P intersects the target model along direction n at point P'. Point P' is the sampling point in the second region corresponding to sampling point P in the first region. This is achieved through... Figure 4 The vertical mapping method shown can obtain the corresponding sampling points on the target model for each sampling point in the first region, and finally the second region is formed by the corresponding sampling points on the target model.

[0118] Understandably, calculating the sampling point P' in the second region essentially involves calculating the intersection of line n and the target model. Since 3D models are generally represented as meshes, finding the intersection point P' only requires determining the intersection relationship between line n and the triangular facets in the target model; that is, finding which triangular facet contains the intersection point of line n and the target model. Specifically, this can be achieved by determining the relationship between a spatial ray and a spatial triangle to obtain the intersection point from the point to the plane, thus obtaining the intersection point P'.

[0119] Step 2032: Determine the deformation coefficient of the transition region based on the first control region and the second control region in the second region. The first control region and the second control region are located on different target models in the third model, and the first control region and the second control region correspond to the same region in the preset human body model.

[0120] In this embodiment, during the process of determining the second region based on the first region in the preset human body model, since some regions in the target head model and the target body model are on the same horizontal line, the first control region and the second control region located on different target models can be determined. These first and second control regions correspond to the same region in the preset human body model. That is, the first control region can be located in the target head model, and the second control region can be located in the target body model; or, the first control region can be located in the target body model, and the second control region can be located in the target head model.

[0121] Since the first and second control regions correspond to the same areas in the preset human body model, the sampling points in the first and second control regions also correspond to the same sampling points in the first region. Based on this, a one-to-one correspondence can be established between the sampling points in the first and second control regions, resulting in multiple pairs of sampling points. With multiple pairs of sampling points, the coordinate transformation relationship between the sampling points in the first and second control regions can be calculated based on the spatial coordinates of the sampling points in the first and second control regions, thus obtaining the deformation coefficient. This deformation coefficient is used for subsequent region deformation processing.

[0122] Optionally, in determining the deformation coefficient, multiple first sampling points in the first control region, multiple second sampling points in the second control region, and multiple third sampling points in the third control region can be obtained, wherein the multiple first sampling points and the multiple second sampling points have a corresponding relationship. The second region includes the third control region, meaning the third control region is also mapped from the first region in the preset human body model, and the third control region and the first control region are located on the same target model. Specifically, the third control region and the first control region are both located on the target head model, while the second control region is located on the target body model; or, the third control region and the first control region are both located on the target body model, while the second control region is located on the target head model.

[0123] Then, based on the coordinate transformation relationship between the plurality of first sampling points and the plurality of second sampling points, and the coordinate transformation relationship between the plurality of third sampling points, the deformation coefficient of the transition region is determined.

[0124] In simple terms, multiple first sampling points and multiple second sampling points form a control point pair, each pair including one first sampling point and one corresponding second sampling point. Conversely, multiple third sampling points form another control point pair, each pair including two identical third sampling points. Based on these two sets of control point pairs, the total deformation coefficient can be calculated. For the third sampling point, after coordinate transformation using the deformation coefficient, the coordinates of the third sampling point remain unchanged. For the first sampling point, after coordinate transformation using the deformation coefficient, the coordinates of the first sampling point are transformed to the coordinates of the corresponding second sampling point.

[0125] The transition region of the third model includes the first control region, the third control region, and the deformation region located between the first and third control regions. Since the deformation coefficients are calculated based on the coordinate transformation relationship between the third control region and itself, and the coordinate transformation relationship between the first and second control regions, when deforming the transition region according to the deformation coefficients, the deformation amplitude of the third control region is relatively small, the deformation amplitude of the first control region is relatively large, and the deformation amplitude of the deformation region located between the first and third control regions is moderate. This ensures that the deformation amplitude of the transition region gradually increases along the vertical direction, achieving a smooth transition.

[0126] In this scheme, the deformation coefficient is obtained by using the coordinate transformation relationship between the first control region and the second control region, as well as the coordinate transformation relationship between the third control region and itself. This enables the deformation processing based on the deformation coefficient to gradually increase the deformation amplitude in the transition region along the vertical direction, thus achieving a smooth transition in the transition region.

[0127] Optionally, to achieve a smooth transition in the transition region as much as possible, the areas of both the first control region and the third control region are smaller than the area of ​​the deformed region. For example, the area ratio of the first control region, the third control region, and the deformed region can be 1:1:3 or 1:1:2.

[0128] Optionally, the first control region, the third control region, and the deformation region may be located on the target head model, with the first control region close to the target body model and the second control region located on the target body model. Simply put, on the neck region of the target head model, from top to bottom, are the third control region, the deformation region, and the first control region.

[0129] Alternatively, the first control region, the third control region, and the deformation region are located on the target body model, with the first control region close to the target head model and the second control region located on the target head model. In simpler terms, on the neck region of the target body model, from top to bottom, are the first control region, the deformation region, and the third control region.

[0130] Step 2033: Based on the deformation coefficient, perform deformation processing on the transition region in the third model to obtain the fourth model.

[0131] After obtaining the deformation coefficients, deformation processing can be performed on the transition region in the third model based on the deformation coefficients, that is, coordinate transformation can be performed on the coordinate points in the transition region so that the target head model and the target body model can be smoothly connected, and finally the fourth model is obtained.

[0132] In simple terms, assuming the neck region of the target head model is thinner while the neck region of the target body model is thicker, after stitching together to obtain the third model, the target human body model and the target body model in the third model do not connect well in the neck region. Based on the first region in the preset human body model, a first control region can be determined in the target head model, and a second control region can be determined in the target body model. The first and second control regions are located on the same horizontal plane, and both can be considered as partial annular regions of the neck region. The inner diameter of the first control region is smaller, and the inner diameter of the second control region is larger. Based on the corresponding sampling points in the first and second control regions, as well as the sampling points in the third control region, a deformation coefficient can be obtained. This deformation coefficient indicates the coordinate transformation relationship between the sampling points in the first and second control regions, and the coordinate transformation relationship between the sampling points in the third control region and itself. Based on this deformation coefficient, a portion of the neck region of the target head model is deformed to gradually thicken the neck region of the target head model, achieving a smooth connection with the neck region of the target body model, resulting in a smoothly connected fourth model.

[0133] To facilitate understanding, the following will describe in detail the process of performing deformation processing on the transition region, with reference to the accompanying drawings.

[0134] Please refer to Figure 5 , Figure 5 This is a schematic diagram illustrating the division of a first region within a preset human body model, as provided in an embodiment of this application. Figure 5 As shown, the neck area of ​​the preset human body model is divided into three regions in order from top to bottom: region 1, region 2 and region 3. These three regions constitute the first region of the preset human body model.

[0135] Please refer to Figure 6 , Figure 6 This is a schematic diagram of a target head model and a target body model provided in an embodiment of this application. Figure 6 As shown, the neck region of the target head model is thinner, while the neck region of the target body model is thicker. After the target head model and the target body model are initially fused to obtain the third model through registration with the preset human body model, the neck region in the target head model cannot be connected with the neck region in the target body model, and part of the neck region in the target head model is located within the neck region of the target body model.

[0136] Please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating how a second region in a third model is obtained by mapping a first region in a preset human body model, as provided in an embodiment of this application. Figure 7As shown, based on the first region in the preset human body model, a vertical mapping is performed on the first region to obtain the second region in the third model corresponding to the first region. The second region comprises four areas: the third control region located on the target head model (i.e.,... Figure 7 Region A in the middle), deformed region (i.e. Figure 7 Area B) and the first control area (i.e. Figure 7 Region C in the model, and the second control region located on the target body model (i.e., Figure 7 Region C' in the model. The first control region and the second control region correspond to the same region in the preset human body model (i.e., region 3 in the first region).

[0137] After determining the second region in the third model, the corresponding deformation coefficients can be obtained based on the first, second, and third control regions within the second region. In this embodiment, an interpolation method based on radial basis functions can be used to implement the deformation processing of the transition region.

[0138] Specifically, such as Figure 7 As shown, sampling points in regions A, C, and C' are obtained respectively, resulting in point sets for regions A, C, and C'. Then, a set of control points is constructed using the point set of region A, where each sampling point in region A corresponds to itself. Another set of control points is constructed using the point sets of regions C and C', where sampling points in regions C and C' also correspond. Based on these two sets of control points, control points for the radial basis function are constructed. This involves using the distance between corresponding discrete point pairs in the two sets of control points as the independent variable, and introducing a basis function to calculate a deformable basis, which is the aforementioned deformation coefficient. After obtaining the deformable basis, coordinate transformations can be performed on the sampling points in regions A, B, and C, thereby deforming the transition region formed by regions A, B, and C to obtain the fourth model.

[0139] Please refer to Figure 8 , Figure 8 This is a schematic diagram of a fourth model obtained after deformation processing, as provided in an embodiment of this application. Figure 8 As shown, after the deformation process is performed, the neck region in the target head model and the neck region in the target body model are well connected, and the transition area between the target head model and the target body model is smooth without any protruding or recessed areas.

[0140] The above describes the process of performing deformation processing on the transition region in the third model. The following will describe in detail the process of registering the target body model and the target head model onto the preset human body model.

[0141] Understandably, for the acquired target body model and the preset human body model, the shape and movement of the body model in the preset human body model may differ from those of the target body model. To register the target body model with the body model in the preset human body model, it is often necessary to adjust the parameters of the body model in the preset human body model, thereby registering the body model in the preset human body model onto the target body model. Furthermore, for head models, head models typically do not involve non-rigid changes; that is, the shape of the model does not change. Rigid registration can be performed directly, meaning that only the size, spatial position, and pose of the head model need to be adjusted to achieve registration between the two head models.

[0142] Based on this, in this embodiment, the target body model and the body model of the preset human model can be registered first. That is, by adjusting the parameters in the body model of the preset human model and performing a spatial transformation operation on the coordinates of the body model of the preset human model, the preset human model can be registered to the target body model. Then, keeping the spatial coordinates of the preset human model and the target body model unchanged, the target head model is registered to the head model of the preset human model through coordinate transformation. If the registration between the target head model and the preset human model is performed first, then when the registration between the target body model and the preset human model is performed subsequently, both the preset human model and the registered target head model need to be transformed simultaneously.

[0143] Therefore, in this scheme, by registering the target body model first and then the target head model, the coordinate transformation of the target head model can be repeatedly processed, thus improving the efficiency of model registration.

[0144] Specifically, the process of registering the target body model with the preset human body model may include the following steps a-c.

[0145] Step a: Identify multiple first key feature points in the target body model.

[0146] Specifically, the plurality of first key feature points may include the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0147] Since the target body model is a three-dimensional model, multiple primary key feature points in the target body model can be determined by acquiring a two-dimensional image of the target body model and then performing feature point detection on the two-dimensional image.

[0148] For example, the target body model can be normalized to a preset unit space by calculating the bounding box of the target body model. That is, any coordinate in the normalized target body model will be within the range of [0,1] in all three dimensions. The bounding box is an algorithm for solving the optimal bounding space of a discrete point set. Its basic idea is to use a geometric object with a slightly larger volume and simpler characteristics (called a bounding box) to approximate a complex geometric object, thereby achieving fast normalization of the target body model.

[0149] Then, the target body model is projected onto the six faces of a square bounding box, resulting in six 2D projection images. After obtaining these six projection images, human pose detection is performed on each image. The projection images are then filtered based on the presence of a human figure and the orientation angle of the figure, resulting in the image that includes the human figure and has the largest visible range. Based on the determined projection image, the frontal orientation of the target body model is calculated. Further, based on the frontal orientation, the body is rotated to a right-handed coordinate system facing the positive z-axis and upward along the y-axis. After this rotation, the target body model is projected onto multiple predefined projection viewpoints, resulting in 2D images of the target body model at different angles. Generally, the predefined projection viewpoints can be three: the positive z-axis direction and the directions obtained by rotating the model ±30° around the y-axis from the z-axis.

[0150] Finally, based on the human keypoint detection algorithm, body feature points are detected in the two-dimensional images of the target body model from different angles, obtaining the positions of the body feature points in each two-dimensional image from different angles. Having obtained the positions of the body feature points in multiple two-dimensional images from different angles, a triangulation method is used to calculate the three-dimensional spatial position of each body feature point, thereby determining the positions of multiple first key feature points in the target body model.

[0151] Step b: Obtain multiple pre-marked second key feature points in the body model of the preset human body model, wherein the multiple first key feature points and the multiple second key feature points have a corresponding relationship.

[0152] Since the geometric topology of the preset human body model is known, multiple pre-marked second key feature points are identified within the body model of the preset human body model. Specifically, these multiple second key feature points may include the left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. Therefore, each of these multiple second key feature points has a corresponding first key feature point.

[0153] Step c: Based on the plurality of first key feature points and the plurality of second key feature points, adjust the shape and / or posture of the body model of the preset human body model, and register the adjusted preset human body model onto the target body model.

[0154] In this embodiment, the preset human body model can change the shape and movement of the body model within the preset human body model by varying a set of control parameters. Therefore, during the specific registration process, by changing the control parameters of the preset human body model, the shape and movement of the preset human body model are adjusted and registered onto the target body model, thereby completing the registration between the target body model and the preset human body model. There are two ways to adjust the control parameters of the preset human body model.

[0155] Method 1 uses the first key feature point in the target body model and the pre-marked corresponding second key feature point on the preset human body model as constraints, and uses gradient descent and other methods to iteratively adjust the control parameters until the coordinate error between the second key feature point in the preset human body model and the first key feature point in the target body model is less than the threshold or the number of times the control parameters are adjusted reaches the maximum number of iterations.

[0156] Method two involves using a neural network estimation approach. This method requires pre-training a network model to estimate the parameters of a pre-defined human body model. For example, a parameter estimation model can be pre-trained to estimate the control parameters of the human body model based on a single image. When used, a projection image of the target body model can be input, and the parameter estimation model will output the pre-defined control parameters of the human body model based on the projection image.

[0157] After registering the target body model with the preset human body model, the target head model can be registered with the head model of the preset human body model.

[0158] Specifically, the process of registering the target head model with the body model of the preset human body model may include the following steps A-C.

[0159] Step A: Identify multiple third key feature points in the target head model.

[0160] For details, please refer to Figure 9 , Figure 9 This is a schematic diagram illustrating multiple third key feature points provided in embodiments of this application. For example... Figure 9 As shown, the plurality of third key feature points include the left eye, right eye, tip of the nose, left corner of the mouth, and right corner of the mouth. Optionally, the plurality of third key feature points may further include feature points such as eyebrows, chin, and cheeks.

[0161] Since the target head model is a three-dimensional model, multiple third key feature points in the target head model can be determined by obtaining the corresponding two-dimensional image of the target head model and then performing feature point detection on the two-dimensional image.

[0162] For example, the target head model can be normalized to a preset unit space by calculating the bounding box of the target head model. That is, the coordinate range of any coordinate in the normalized target head model in the three dimensions is between [0,1].

[0163] Then, the target head model is projected onto the six faces of a square bounding box, resulting in six 2D projection images. After obtaining these six projection images, face detection and face pose detection are performed on each image. The projection images are then filtered based on the presence and orientation of faces to obtain the one with the largest visible face area. Based on the determined projection images, the frontal orientation of the target head model is calculated. Further, the head is rotated to a right-handed coordinate system facing the positive z-axis and upward along the y-axis. After this rotation, the target head model is projected onto multiple predefined projection viewpoints, resulting in 2D images of the target head model at different angles. Generally, the predefined projection viewpoints can be three: the positive z-axis direction and the directions obtained by rotating the z-axis ±30° around the y-axis.

[0164] Finally, based on the facial landmark detection algorithm, head feature points are detected in the two-dimensional images of the target head model from different angles, obtaining the positions of the head feature points in each two-dimensional image from different angles. Having obtained the positions of the head feature points in multiple two-dimensional images from different angles, a triangulation method is used to calculate the three-dimensional spatial position of each head feature point, thereby determining the positions of multiple first key feature points in the target head model.

[0165] Step B: Obtain multiple pre-marked fourth key feature points in the head model of the preset human body model, wherein the multiple third key feature points and the multiple fourth key feature points have a corresponding relationship.

[0166] Since the geometric topology of the preset human body model is known, multiple pre-marked fourth key feature points are identified in the head model of the preset human body model. Specifically, the multiple fourth key feature points may include the left eye, right eye, nose tip, left corner of the mouth, and right corner of the mouth. Therefore, each of the multiple fourth key feature points has a corresponding third key feature point.

[0167] Step C: Based on the plurality of third key feature points and the plurality of fourth key feature points, register the target head model onto the head model of the preset human body model.

[0168] In this step, multiple third key feature points of the target head model and multiple corresponding fourth key feature points in the preset human body model can be used as constraints. The Iterative Closest Point (ICP) algorithm is then used to adjust the size, spatial position, and pose of the target head model and register it onto the preset human body model.

[0169] It should be noted that this step does not require adjusting the control parameters of the preset human body model, because the head model does not involve non-rigid changes, that is, the shape of the model does not change, so rigid registration can be performed directly, that is, only the size, spatial position and posture of the target head model are adjusted.

[0170] It is understandable that for a 3D model described by vertices and meshes, its topology can be expressed as the order and dependencies of model vertices and triangular faces. For head models with the same topology, corresponding point pairs between models can be found directly according to the semantic relationships of vertices. For example, assuming the vertex describing the tip of the nose is numbered 8086, the position of vertex number 8086 in two face models with the same topology constitutes a pair of corresponding points, which can be used as a set of constraints between the original position and the target position during deformation. Therefore, for two models with the same topology, their spatial coordinates can be directly read using pre-labeled keypoint information to construct control point pairs and complete the deformation. However, in practical applications, topological consistency is often a difficult condition to meet. Therefore, in this embodiment, the corresponding feature point pairs are automatically constructed based on the above-described method, thereby registering the target head model and the target body model with the same preset human body model, and completing the fusion of the target head model and the target body model.

[0171] In some possible embodiments, after obtaining the fourth model with a smooth transition region, the transition region between the target head model and the target body model in the fourth model may be separated. In some applications, it may be necessary to obtain a watertight model, i.e., a fully enclosed model. Therefore, the mesh of the fourth model can be recalculated to further optimize the transition region in the fourth model. For example, the deformed transition region in the fourth model can be re-meshed using Poisson meshing to obtain a fully enclosed model. In addition, in scenarios where high texture quality is required, algorithms such as Poisson fusion of images can also be applied to optimize the texture of the fourth model to obtain a smooth transition texture effect.

[0172] exist Figures 1 to 9Based on the corresponding embodiments, in order to better implement the above-described solutions of the embodiments of this application, related equipment for implementing the above-described solutions is also provided below.

[0173] For details, please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of a model fusion device provided in an embodiment of this application. The model fusion device includes: an acquisition unit 1001, used to acquire a first model and a second model, both of which are partial models in a human body model; a registration unit 1002, used to register the first model and the second model with a preset human body model respectively to obtain a third model, the third model including the registered first model and the registered second model; and a processing unit 1003, used to perform deformation processing on the transition region in the third model to smooth the transition region and obtain a fourth model; wherein the transition region includes a portion of the registered first model that is close to the registered second model, and a portion of the registered second model that is close to the registered first model.

[0174] In one possible implementation, the first model is a target head model, and the second model is a target body model;

[0175] Alternatively, the first model may be a target torso model, and the second model may be a target limb model.

[0176] In one possible implementation, the processing unit 1003 is specifically configured to: determine a second region in the third model corresponding to the first region based on a first region in the preset human body model, wherein the first region is located in the neck region of the preset human body model and the second region includes the transition region; determine a deformation coefficient of the transition region based on a first control region and a second control region in the second region, wherein the first control region and the second control region are located on different target models in the third model and correspond to the same region in the preset human body model; and perform deformation processing on the transition region in the third model based on the deformation coefficient to obtain the fourth model.

[0177] In one possible implementation, the processing unit 1003 is specifically configured to: acquire a plurality of first sampling points in the first control region, a plurality of second sampling points in the second control region, and a plurality of third sampling points in the third control region, wherein the plurality of first sampling points correspond to the plurality of second sampling points, the second region includes the third control region, and the third control region and the first control region are located on the same target model; and determine the deformation coefficient of the transition region based on the coordinate transformation relationship between the plurality of first sampling points and the plurality of second sampling points and the coordinate transformation relationship between the plurality of third sampling points.

[0178] In one possible implementation, the region in the transition region where deformation processing is performed includes the first control region, the third control region, and the deformation region located between the first control region and the third control region.

[0179] In one possible implementation, the areas of both the first control region and the third control region are smaller than the area of ​​the deformable region.

[0180] In one possible implementation, when the first model is a target head model and the second model is a target body model, the first control region, the third control region, and the deformation region are located on the target head model, and the first control region is close to the target body model, while the second control region is located on the target body model; or, the first control region, the third control region, and the deformation region are located on the target body model, and the first control region is close to the target head model, while the second control region is located on the target head model.

[0181] In one possible implementation, when the first model is a target head model and the second model is a target body model, the processing unit 1003 is specifically used to: obtain the central axis of the neck in the preset human body model; determine the second region corresponding to the first region based on the central axis and the first region; wherein any two corresponding sampling points in the first region and the second region are located on a straight line perpendicular to the central axis.

[0182] In one possible implementation, when the second model is a target body model, the registration unit 1002 is specifically used to: determine a plurality of first key feature points in the target body model; obtain a plurality of pre-marked second key feature points in the body model of the preset human body model, wherein the plurality of first key feature points and the plurality of second key feature points have a corresponding relationship; adjust the shape and / or posture of the body model of the preset human body model according to the plurality of first key feature points and the plurality of second key feature points, and register the adjusted preset human body model to the target body model.

[0183] In one possible implementation, the plurality of first key feature points include one or more of the following: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

[0184] In one possible implementation, when the first model is a target head model, the registration unit 1002 is specifically used to: determine multiple third key feature points in the target head model; obtain multiple pre-marked fourth key feature points in the head model of the preset human body model, wherein the multiple third key feature points and the multiple fourth key feature points have a corresponding relationship; and register the target head model to the head model of the preset human body model according to the multiple third key feature points and the multiple fourth key feature points.

[0185] In one possible implementation, the plurality of third key feature points include one or more of the left eye, right eye, tip of the nose, left corner of the mouth, and right corner of the mouth.

[0186] In one possible implementation, the registration unit 1002 is specifically configured to: acquire multiple head projection images of the target head model in different directions; perform face detection on the multiple projection images to determine the orientation of the face in the target head model; determine multiple face projection images of the face in the target head model at different projection angles based on the orientation of the face in the target head model; detect key facial feature points in the multiple face projection images respectively, and determine the spatial coordinates of multiple third key feature points in the target head model based on the key facial feature points.

[0187] The model fusion method provided in this application embodiment can be executed by a chip in an electronic device. This chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuitry. The processing unit can execute computer execution instructions stored in a storage unit, causing the chip within the electronic device to perform the aforementioned operations. Figures 1 to 9The model fusion method described in the illustrated embodiment. Optionally, the storage unit is an on-chip storage unit, such as a register or cache. Alternatively, the storage unit can be an external storage unit located within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0188] Please refer to Figure 11 This application also provides a computer-readable storage medium, in some embodiments of which the above-described... Figure 2 The disclosed method can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0189] Figure 11 A conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein is illustrated schematically. The example computer-readable storage medium includes a computer program for executing computer processes on a computing device.

[0190] In one embodiment, the computer-readable storage medium 1100 is provided using a signal bearer medium 1101. The signal bearer medium 1101 may include one or more program instructions 1102, which, when executed by one or more processors, can provide the above-mentioned... Figure 5 The described function or part of the function. Therefore, for example, refer to... Figure 5 In the embodiment shown, one or more features of steps 501-502 can be fulfilled by one or more instructions associated with the signal carrying medium 1101. Furthermore, Figure 11 The program instruction 1102 in the document also describes example instructions.

[0191] In some examples, the signal carrying medium 1101 may include a computer-readable medium 1103, such as, but not limited to, a hard disk drive, a compact disc (CD), a digital video disc (DVD), a digital magnetic tape, a memory, ROM, or RAM, etc.

[0192] In some embodiments, the signal carrying medium 1101 may include a computer-recordable medium 1104, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal carrying medium 1101 may include a communication medium 1105, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal carrying medium 1101 may be transmitted by a wireless communication medium 1105 (e.g., a wireless communication medium conforming to the IEEE 802.11 standard or other transmission protocols).

[0193] One or more program instructions 1102 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1102 conveyed to the computing device via a computer-readable medium 1103, a computer-recordable medium 1104, and / or a communication medium 1105.

[0194] It should be understood that the arrangements described herein are for illustrative purposes only. Therefore, those skilled in the art will understand that other arrangements and other elements (e.g., machines, interfaces, functions, sequences, and functional groups, etc.) can be used instead, and some elements may be omitted depending on the desired outcome. Furthermore, many of the described elements are functional entities that can be implemented as discrete or distributed components, or in any suitable combination and location with other components.

[0195] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit 1003, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

Claims

1. A model fusion method, characterized in that, include: Obtain a first model and a second model, both of which are partial models in the human body model; The first model and the second model are registered with a preset human body model respectively to obtain a third model. The third model includes the registered first model and the registered second model. The registered first model and the corresponding feature points in the preset human body model are located at the same position. The registered second model and the corresponding feature points in the preset human body model are located at the same position. The transition region in the third model is subjected to deformation processing to smooth the transition region, resulting in the fourth model; The transition region includes a portion of the first registered model that is close to the second registered model, and a portion of the second registered model that is close to the first registered model.

2. The method according to claim 1, characterized in that, The first model is the target head model, and the second model is the target body model; Alternatively, the first model may be a target torso model, and the second model may be a target limb model.

3. The method according to claim 1 or 2, characterized in that, The step of performing deformation processing on the transition region in the third model to smooth the transition region and obtain the fourth model includes: Based on the first region in the preset human body model, a second region in the third model corresponding to the first region is determined, and the second region includes the transition region. Based on the first control region and the second control region in the second region, the deformation coefficient of the transition region is determined. The first control region and the second control region are located on different target models in the third model, and the first control region and the second control region correspond to the same region in the preset human body model. Based on the deformation coefficient, deformation processing is performed on the transition region in the third model to obtain the fourth model.

4. The method according to claim 3, characterized in that, Determining the deformation coefficient of the transition region based on the first control region and the second control region in the second region includes: Acquire multiple first sampling points in the first control region, multiple second sampling points in the second control region, and multiple third sampling points in the third control region. The multiple first sampling points and the multiple second sampling points have a corresponding relationship. The second region includes the third control region. The third control region and the first control region are located on the same target model. The deformation coefficient of the transition region is determined based on the coordinate transformation relationship between the plurality of first sampling points and the plurality of second sampling points, and the coordinate transformation relationship between the plurality of third sampling points.

5. The method according to claim 4, characterized in that, The regions in the transition region where deformation processing is performed include the first control region, the third control region, and the deformation region located between the first control region and the third control region.

6. The method according to claim 5, characterized in that, The areas of both the first control region and the third control region are smaller than the area of ​​the deformable region.

7. The method according to claim 5, characterized in that, When the first model is a target head model and the second model is a target body model, the first control region, the third control region, and the deformation region are located on the target head model, and the first control region is close to the target body model, while the second control region is located on the target body model. Alternatively, the first control region, the third control region, and the deformation region are located on the target body model, with the first control region close to the target head model and the second control region located on the target head model.

8. The method according to claim 3, characterized in that, When the first model is a target head model and the second model is a target body model, determining the second region in the third model corresponding to the first region based on the first region in the preset human body model includes: Obtain the central axis of the neck in the preset human body model; Based on the central axis and the first region, determine the second region corresponding to the first region; Wherein, any two corresponding sampling points in the first region and the second region are located on a straight line perpendicular to the central axis.

9. The method according to claim 1 or 2, characterized in that, When the second model is a target body model, the registration of the second model with the preset human body model includes: Identify multiple first key feature points in the target body model; Obtain multiple pre-marked second key feature points in the body model of the preset human body model, wherein the multiple first key feature points and the multiple second key feature points have a corresponding relationship; Based on the plurality of first key feature points and the plurality of second key feature points, the shape and / or posture of the preset human body model are adjusted, and the adjusted preset human body model is registered onto the target human body model.

10. The method according to claim 9, characterized in that, The plurality of first key feature points include one or more of the following: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

11. The method according to claim 1 or 2, characterized in that, When the first model is a target head model, the registration of the first model with the preset human body model includes: Identify multiple third key feature points in the target head model; Obtain multiple pre-marked fourth key feature points in the head model of the preset human body model, wherein the multiple third key feature points have a corresponding relationship with the multiple fourth key feature points; Based on the plurality of third key feature points and the plurality of fourth key feature points, the target head model is registered to the head model of the preset human body model.

12. The method according to claim 11, characterized in that, The plurality of third key feature points include one or more of the left eye, right eye, tip of nose, left corner of mouth, and right corner of mouth.

13. The method according to claim 11, characterized in that, The determination of multiple third key feature points in the target head model includes: Obtain multiple head projection images of the target head model in different directions; Face detection is performed on the multiple projection images to determine the orientation of the face in the target head model; Based on the orientation of the face in the target head model, determine multiple face projection images of the face in the target head model under different projection angles; Key facial feature points are detected in the multiple face projection images, and the spatial coordinates of multiple third key feature points in the target head model are determined based on the key facial feature points.

14. A model fusion device, characterized in that, include: The acquisition unit is used to acquire a first model and a second model, both of which are partial models in the human body model. A registration unit is used to register the first model and the second model with a preset human body model respectively to obtain a third model. The third model includes the registered first model and the registered second model. The registered first model and the corresponding feature points in the preset human body model are located at the same position. The registered second model and the corresponding feature points in the preset human body model are located at the same position. A processing unit is used to perform deformation processing on the transition region in the third model to smooth the transition region and obtain a fourth model; The transition region includes a portion of the first registered model that is close to the second registered model, and a portion of the second registered model that is close to the first registered model.

15. The apparatus according to claim 14, characterized in that, The first model is the target head model, and the second model is the target body model; Alternatively, the first model may be a target torso model, and the second model may be a target limb model.

16. The apparatus according to claim 14 or 15, characterized in that, The processing unit is specifically used for: Based on the first region in the preset human body model, a second region in the third model corresponding to the first region is determined, and the second region includes the transition region. Based on the first control region and the second control region in the second region, the deformation coefficient of the transition region is determined. The first control region and the second control region are located on different target models in the third model, and the first control region and the second control region correspond to the same region in the preset human body model. Based on the deformation coefficient, deformation processing is performed on the transition region in the third model to obtain the fourth model.

17. The apparatus according to claim 16, characterized in that, The processing unit is specifically used for: Acquire multiple first sampling points in the first control region, multiple second sampling points in the second control region, and multiple third sampling points in the third control region. The multiple first sampling points and the multiple second sampling points have a corresponding relationship. The second region includes the third control region. The third control region and the first control region are located on the same target model. The deformation coefficient of the transition region is determined based on the coordinate transformation relationship between the plurality of first sampling points and the plurality of second sampling points, and the coordinate transformation relationship between the plurality of third sampling points.

18. The apparatus according to claim 17, characterized in that, The regions in the transition region where deformation processing is performed include the first control region, the third control region, and the deformation region located between the first control region and the third control region.

19. The apparatus according to claim 18, characterized in that, The areas of both the first control region and the third control region are smaller than the area of ​​the deformable region.

20. The apparatus according to claim 18, characterized in that, When the first model is a target head model and the second model is a target body model, the first control region, the third control region, and the deformation region are located on the target head model, and the first control region is close to the target body model, while the second control region is located on the target body model. Alternatively, the first control region, the third control region, and the deformation region are located on the target body model, with the first control region close to the target head model and the second control region located on the target head model.

21. The apparatus according to claim 16, characterized in that, When the first model is a target head model and the second model is a target body model, the processing unit is specifically used for: Obtain the central axis of the neck in the preset human body model; Based on the central axis and the first region, determine the second region corresponding to the first region; Wherein, any two corresponding sampling points in the first region and the second region are located on a straight line perpendicular to the central axis.

22. The apparatus according to claim 15, characterized in that, When the second model is a target body model, the registration unit is specifically used for: Identify multiple first key feature points in the target body model; Obtain multiple pre-marked second key feature points in the body model of the preset human body model, wherein the multiple first key feature points and the multiple second key feature points have a corresponding relationship; Based on the plurality of first key feature points and the plurality of second key feature points, the shape and / or posture of the preset human body model are adjusted, and the adjusted preset human body model is registered onto the target human body model.

23. The apparatus according to claim 22, characterized in that, The plurality of first key feature points include one or more of the following: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle.

24. The apparatus according to claim 15, characterized in that, When the first model is a target head model, the registration unit is specifically used for: Identify multiple third key feature points in the target head model; Obtain multiple pre-marked fourth key feature points in the head model of the preset human body model, wherein the multiple third key feature points have a corresponding relationship with the multiple fourth key feature points; Based on the plurality of third key feature points and the plurality of fourth key feature points, the target head model is registered to the head model of the preset human body model.

25. The apparatus according to claim 24, characterized in that, The plurality of third key feature points include one or more of the left eye, right eye, tip of nose, left corner of mouth, and right corner of mouth.

26. The apparatus according to claim 24, characterized in that, The registration unit is specifically used for: Obtain multiple head projection images of the target head model in different directions; Face detection is performed on the multiple projection images to determine the orientation of the face in the target head model; Based on the orientation of the face in the target head model, determine multiple face projection images of the face in the target head model under different projection angles; Key facial feature points are detected in the multiple face projection images, and the spatial coordinates of multiple third key feature points in the target head model are determined based on the key facial feature points.

27. An electronic device, characterized in that, The device includes a memory and a processor; the memory stores code, and the processor is configured to execute the code, wherein when the code is executed, the electronic device performs the method as described in any one of claims 1 to 13.

28. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 13.

29. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 13.

Images