Method, system, and computer-readable medium for implementing customized plastic surgery simulation

Abstract

The present invention relates to a method, a system, and a computer-readable medium for implementing a customized plastic surgery simulation and, more particularly, to a method, a system, and a computer-readable medium for implementing a customized plastic surgery simulation, wherein an appearance image including respective appearance elements for a user is extracted from a plurality of images obtained by photographing the user, 3D modeling data is generated by applying an SfM algorithm and 3D Gaussian splatting to a plurality of appearance images, a synthesized image is generated by changing position coordinates of one or more representative feature points for each appearance element on the basis of modification information of the appearance element determined according to an input of the user, and the synthesized image is provided to the user.

Description

Method, system, and computer-readable medium for implementing custom molding simulation

[0001] The present invention relates to a method, a system and a computer-readable medium for implementing a custom plastic surgery simulation, comprising: extracting an appearance image including each appearance element of a user from a plurality of images taken of the user; applying a SfM algorithm and 3D Gaussian splatting to the plurality of appearance images to generate 3D modeling data; and generating a composite image by changing the position coordinates of at least one representative feature point for each appearance element based on change information of the appearance element determined according to a user input, and providing the composite image to the user.

[0002]

[0003] With the recent improvement in awareness and growing interest in plastic surgery, demand for it is on the rise. Typically, patients seeking plastic surgery communicate with their surgeons to predict the outcome of their surgery. Traditionally, surgeons used Photoshop to predict the shape of the surgical area after surgery. Recently, virtual plastic surgery software has been developed to align the desired shape of the surgical area with the patient's existing overall body image. However, these technologies rely on simple two-dimensional projections and alignment of the composite images, making it difficult for the patient and surgeon to predict the specific outcome. This can result in significantly different results from the actual surgery, resulting in inconvenience for both parties.

[0004] Conventional plastic surgery simulation technologies include plastic surgery simulation service systems and methods, such as those disclosed in Korean Patent No. 10-2533858. These plastic surgery simulation service systems and methods allow users to select desired plastic surgery services (procedures) and, based on desired plastic surgery information regarding the desired plastic surgery area, provide simulation results via 3D or VR technology. Meanwhile, conventional plastic surgery simulation technologies do not disclose or suggest any configuration for precisely modeling the surgeon's plastic surgery area to achieve results that are as similar to reality as possible, in the process of presenting the plastic surgery results to the surgeon in 3D. In other words, there is a high probability that users seeking plastic surgery will not be highly satisfied with simulation services implemented with low quality.

[0005] Therefore, there is a need to develop a technology that can provide the patient with a precise 3D representation of the surgical results for the area they wish to have plastic surgery.

[0006]

[0007] The present invention provides a method, a system and a computer-readable medium for implementing a custom plastic surgery simulation, which extract an appearance image including each appearance element of a user from a plurality of images taken of the user, apply an SfM algorithm and 3D Gaussian splatting to the plurality of appearance images to generate 3D modeling data, and generate a composite image by changing the position coordinates of at least one representative feature point for each appearance element based on change information of the appearance element determined according to a user input, and provide the composite image to the user.

[0008]

[0009] In order to solve the above-described problem, in one embodiment of the present invention, a method for implementing a custom plastic surgery simulation performed in a user terminal and a service server is provided, comprising: an image acquisition step of capturing a user from different angles through a camera by a user terminal, acquiring a plurality of images, and outputting them on a screen; an image transmission step of transmitting, by the user terminal, a representative image determined according to a user input from among the plurality of images and the plurality of images to a service server; an RGB-D image generation step of generating, by the service server, a plurality of RGB-D images including location information, color information, and depth information regarding the user's appearance included in the corresponding images by inputting each of the plurality of images into a first analysis model based on deep learning; an appearance image extraction step of extracting, by the service server, an appearance image including only each of a plurality of appearance elements of the user by inputting the plurality of RGB-D images into a second analysis model based on deep learning; A method for implementing a plastic surgery simulation is provided, comprising: a 3D modeling data derivation step of applying a SfM algorithm and 3D Gaussian splatting to a plurality of appearance images for each appearance element by a service server, and deriving 3D modeling data for each appearance element through a feature map extracted from each of the plurality of appearance images; a representative feature point determination step of determining, by the service server, one or more representative feature points from each of the 3D modeling data for each appearance element; a change information transmission step of transmitting, by a user terminal, change information of appearance elements determined according to a user's input for the representative image to the service server; a composite image generation step of generating a transformed appearance image by changing, by the service server, the position coordinates of one or more representative feature points related to the change information according to the change information, and aligning the transformed appearance image with the representative image to generate a composite image; and a composite image provision step of receiving, by the user terminal, the composite image from the service server and providing it to the user.

[0010] In one embodiment of the present invention, the appearance image extraction step segments only each of a plurality of appearance elements included in the representative image through the second analysis model, and extracts an appearance image including each of the segmented appearance elements, and the plurality of appearance elements may include at least one of the user's eyebrows, eyes, nose, mouth, and facial contour.

[0011] In one embodiment of the present invention, the representative feature point is determined based on a preset reference feature point for each of a plurality of external elements, and the reference feature point is set to allow for size change, position change, and inclination change with respect to the shape of the corresponding external element, and the change information may include information on at least one of size change, position change, and inclination change with respect to the external element determined according to a user's input.

[0012] In one embodiment of the present invention, the change information includes information on an emotional expression determined according to a user's input, and the synthetic image generation step derives an emotional expression determined by the user from a plurality of emotional expressions pre-stored in a service server based on the change information, and changes the position coordinates of one or more representative feature points related to the change information based on a change rate of the position coordinates of one or more representative feature points preset for the emotional expression, thereby generating a transformed appearance image.

[0013] In one embodiment of the present invention, the method for implementing the custom plastic surgery simulation may further include a change information storage step of storing change information received from each of a plurality of user terminals by a service server; and the composite image generation step may further include a recommendation image generation step of generating a recommended appearance image in which the position coordinates of one or more representative feature points for the corresponding appearance element are changed based on the plurality of previously stored change information, and generating a recommended image by aligning the recommended appearance image with the representative image; and the composite image provision step may further include a recommendation image provision step of receiving the recommended image from the service server and providing it to the user.

[0014] In one embodiment of the present invention, the 3D modeling data derivation step includes: an initial 3D data generation step of converting the plurality of appearance images into a point cloud form through an SfM (Structure from Motion) algorithm to generate initial 3D data; a Gaussian 3D data generation step of inputting the initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing covariance values ​​of Gaussian distributions for a plurality of points constituting the point cloud; a feature map extraction step of inputting each of the plurality of appearance images into a deep learning-based feature extraction model to extract a feature map for each of the plurality of appearance images; a voxel 3D data generation step of inputting the Gaussian 3D data and the plurality of feature maps into a deep learning-based first transformation model to generate voxel 3D data in the form of voxels; And it may include a mesh 3D data generation step of inputting the voxel 3D data and a plurality of feature maps into a deep learning-based second transformation model to generate mesh 3D data in the form of a mesh as 3D modeling data.

[0015] In one embodiment of the present invention, each of the plurality of elliptical spheres includes color distribution information including a position coordinate for a center point of the elliptical sphere, an x-axis length, a y-axis length, a z-axis length based on the center point, and an average value and a covariance value of a Gaussian distribution for color values ​​of a plurality of points constituting a point cloud, and a size of the elliptical sphere represents a covariance value of a Gaussian distribution for the position coordinates of each of the plurality of points constituting the point cloud, and the position coordinate for the center point of the elliptical sphere is calculated as an average value of the position coordinates of each of the plurality of points clustered when the elliptical sphere is generated, the position coordinates of each of the plurality of points are determined based on the position information and depth information, and the color value can be determined based on the color information.

[0016] In order to solve the above-described problem, in one embodiment of the present invention, a custom plastic surgery system is provided, which includes a user terminal and a service server, and which performs a method of implementing a custom plastic surgery simulation, the system comprising: an image acquisition unit for capturing a user at different angles through a camera by the user terminal to acquire a plurality of images and outputting them on a screen; an image transmission unit for transmitting, by the user terminal, a representative image determined according to a user input from among the plurality of images and the plurality of images to the service server; an RGB-D image generation unit for generating, by the service server, a plurality of RGB-D images including location information, color information, and depth information regarding the user's appearance included in the corresponding images by inputting, by the service server, each of the plurality of images into a first analysis model based on deep learning, and an appearance image extraction unit for extracting, by the service server, an appearance image including only each of a plurality of appearance elements of the user by inputting, by the service server, the plurality of RGB-D images into a second analysis model based on deep learning, an appearance image; The present invention provides a custom plastic surgery system, comprising: a 3D modeling data extraction unit that applies a SfM algorithm and 3D Gaussian splatting to a plurality of appearance images for each appearance element by a service server, and derives 3D modeling data for each appearance element through a feature map extracted from each of the plurality of appearance images; a representative feature point determination unit that determines, by the service server, one or more representative feature points from each of the 3D modeling data for each appearance element; a change information transmission unit that transmits, by a user terminal, change information of an appearance element determined according to a user's input for the representative image to the service server; a composite image generation unit that generates a transformed appearance image by changing, by the service server, the position coordinates of one or more representative feature points related to the change information according to the change information, and aligns the transformed appearance image with the representative image to generate a composite image; and a composite image provision unit that receives, by the user terminal, the composite image from the service server and provides it to the user.

[0017] In one embodiment of the present invention, the external appearance image extraction unit segments only each of a plurality of external appearance elements included in the representative image through the second analysis model, and extracts an external appearance image including each of the segmented external appearance elements, and the plurality of external appearance elements may include at least one of the user's eyebrows, eyes, nose, mouth, and facial contour.

[0018] In one embodiment of the present invention, the representative feature point is determined based on a preset reference feature point for each of a plurality of external elements, and the reference feature point is set to allow for size change, position change, and inclination change with respect to the shape of the corresponding external element, and the change information may include information on at least one of size change, position change, and inclination change with respect to the external element determined according to a user's input.

[0019] In one embodiment of the present invention, the change information includes information on an emotional expression determined according to a user's input, and the synthetic image generation unit can derive an emotional expression determined by the user from a plurality of emotional expressions pre-stored in a service server based on the change information, and change the position coordinates of one or more representative feature points related to the change information based on a change rate of the position coordinates of one or more representative feature points preset for the emotional expression, thereby generating a transformed appearance image.

[0020] In one embodiment of the present invention, the custom plastic surgery system may further include a change information storage unit that stores change information received from each of a plurality of user terminals by a service server; the composite image generation unit may further include a recommendation image generation unit that generates a recommended appearance image by changing the position coordinates of one or more representative feature points for the corresponding appearance element based on the plurality of pre-stored change information, and generates a recommended image by aligning the recommended appearance image with the representative image; and the composite image provision unit may further include a recommendation image provision unit that receives the recommended image from the service server and provides it to the user.

[0021] In one embodiment of the present invention, the 3D modeling data extraction unit includes: an initial 3D data generation unit that generates initial 3D data by converting the plurality of external images into a point cloud form through an SfM (Structure from Motion) algorithm; a Gaussian 3D data generation unit that inputs the initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing covariance values ​​of Gaussian distributions for a plurality of points constituting the point cloud; a feature map extraction unit that inputs each of the plurality of external images into a deep learning-based feature extraction model to extract a feature map for each of the plurality of external images; and a voxel 3D data generation unit that inputs the Gaussian 3D data and the plurality of feature maps into a deep learning-based first transformation model to generate voxel 3D data in the form of voxels. And it may include a mesh 3D data generation unit that inputs the voxel 3D data and a plurality of feature maps into a deep learning-based second transformation model to generate mesh 3D data in the form of a mesh as 3D modeling data.

[0022] In one embodiment of the present invention, each of the plurality of elliptical spheres includes color distribution information including a position coordinate for a center point of the elliptical sphere, an x-axis length, a y-axis length, a z-axis length based on the center point, and an average value and a covariance value of a Gaussian distribution for color values ​​of a plurality of points constituting a point cloud, and a size of the elliptical sphere represents a covariance value of a Gaussian distribution for the position coordinates of each of the plurality of points constituting the point cloud, and the position coordinate for the center point of the elliptical sphere is calculated as an average value of the position coordinates of each of the plurality of points clustered when the elliptical sphere is generated, the position coordinates of each of the plurality of points are determined based on the position information and depth information, and the color value can be determined based on the color information.

[0023] In order to solve the above problem, in one embodiment of the present invention, a computer-readable medium for implementing a method of implementing a custom plastic surgery simulation performed in a user terminal and a service server, the computer-readable medium including computer-executable instructions causing the user terminal and the service server to perform the following steps, wherein the following steps are: an image acquisition step of photographing a user from different angles through a camera by the user terminal to acquire a plurality of images and outputting them on a screen; an image transmission step of transmitting, by the user terminal, a representative image determined according to a user input among the plurality of images and the plurality of images to the service server; an RGB-D image generation step of inputting, by the service server, each of the plurality of images into a deep learning-based first analysis model to generate a plurality of RGB-D images including location information, color information, and depth information about the user's appearance included in the corresponding image; An appearance image extraction step of inputting the plurality of RGB-D images into a deep learning-based second analysis model by a service server to extract an appearance image including only each of the plurality of appearance elements for the user; A 3D modeling data extraction step of applying an SfM algorithm and 3D Gaussian splatting to the plurality of appearance images by appearance element by the service server and deriving 3D modeling data by appearance element through a feature map extracted from each of the plurality of appearance images; A representative feature point determination step of determining one or more representative feature points from each of the 3D modeling data by appearance element by the service server; A change information transmission step of transmitting, by a user terminal, change information of the appearance elements determined according to the user's input for the representative image to the service server; A synthetic image generation step in which the location coordinates of one or more representative feature points related to the change information are changed according to the change information by the service server to generate a transformed appearance image, and the transformed appearance image is aligned with the representative image to generate a synthetic image;And a computer-readable medium is provided, including a synthetic image providing step of receiving the synthetic image from a service server and providing the synthetic image to a user by a user terminal.

[0024]

[0025] In one embodiment of the present invention, a plastic surgery simulation is provided in which a user can virtually experience plastic surgery by 3D modeling the area the user wants to undergo plastic surgery and synthesizing it with an existing image, thereby providing the user with a high level of satisfaction and a sense of stability when actually undergoing plastic surgery later.

[0026] In one embodiment of the present invention, the user can directly input the degree of change desired for a desired area of ​​plastic surgery into a user terminal, and the changed appearance of the user based on the user's input is provided to the user in real time, thereby providing the user with high satisfaction and convenience.

[0027] In one embodiment of the present invention, by determining one or more representative feature points for a 3D modeled external element, the user can precisely implement a desired molding part by changing the position of the representative feature points based on change information input by the user, thereby providing a molding simulation that can be customized by the user.

[0028] In one embodiment of the present invention, rather than performing 3D modeling based on the values ​​of each of a plurality of points constituting a point cloud, data in the shape of an ellipse is generated by linking overlapping features of points that are related to each other, thereby reducing the amount of computing operations relatively compared to conventional 3D modeling techniques.

[0029] In one embodiment of the present invention, the continuity between a plurality of points constituting a point cloud can be expressed through the covariance value of the Gaussian distribution for the points, thereby achieving the effect of smoothly implementing a desired plastic part by the user.

[0030]

[0031] FIG. 1 schematically illustrates the execution steps of a method for implementing a custom molding simulation according to one embodiment of the present invention.

[0032] FIG. 2 schematically illustrates a plurality of images according to one embodiment of the present invention.

[0033] FIG. 3 schematically illustrates the execution process of the RGB-D image generation step and the external image extraction step according to one embodiment of the present invention.

[0034] FIG. 4 schematically illustrates one or more representative feature points for each external element according to one embodiment of the present invention.

[0035] FIG. 5 schematically illustrates a process for generating a transformed external image based on a user's input according to one embodiment of the present invention.

[0036] Figure 6 schematically illustrates a process for generating a composite image according to one embodiment of the present invention.

[0037] Figure 7 illustrates a process for generating and providing a recommendation image according to another embodiment of the present invention.

[0038] FIG. 8 schematically illustrates components of a computing system that performs a 3D modeling method according to one embodiment of the present invention.

[0039] FIG. 9 schematically illustrates the execution steps of an image-based long-term internal 3D modeling method according to one embodiment of the present invention.

[0040] FIG. 10 schematically illustrates information included in an endoscopic image according to one embodiment of the present invention.

[0041] Figure 11 schematically illustrates a process for generating initial 3D data according to one embodiment of the present invention.

[0042] Figure 12 schematically illustrates a process for generating Gaussian 3D data according to one embodiment of the present invention.

[0043] Figure 13 schematically illustrates a process for adjusting Gaussian 3D data according to one embodiment of the present invention.

[0044] Figure 14 schematically illustrates a process for extracting a feature map according to one embodiment of the present invention.

[0045] Figure 15 schematically illustrates a process for generating 3D modeling data according to one embodiment of the present invention.

[0046] Figure 16 illustrates an example of the internal configuration of a computing device according to one embodiment of the present invention.

[0047] Hereinafter, various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will be apparent to one skilled in the art that such aspects may be practiced without these specific details. The following description and the attached drawings detail specific exemplary aspects of one or more aspects. However, these aspects are exemplary, and it is to be understood that any of the various methods within the principles of the various aspects may be utilized, and the description is intended to encompass all such aspects and their equivalents.

[0048]

[0049] Additionally, various aspects and features will be presented by systems that may include a number of devices, components, and / or modules. It is also to be understood and appreciated that various systems may include additional devices, components, and / or modules, and / or may not include all of the devices, components, and modules discussed in connection with the drawings.

[0050] The terms "embodiment," "example," "aspect," and "example" used herein may not be construed as implying that any aspect or design described is better or advantageous than other aspects or designs. The terms "part," "component," "module," "system," and "interface" used below generally refer to computer-related entities, and may refer to, for example, hardware, a combination of hardware and software, or software.

[0051] Additionally, it should be understood that the terms "comprises" and / or "comprising" imply the presence of the features and / or components, but do not preclude the presence or addition of one or more other features, components and / or groups thereof.

[0052] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any of a plurality of related described items.

[0053] Additionally, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0054]

[0055] 1. Method, system and computer-readable medium for implementing custom molding simulation

[0056]

[0057] The method, system and computer-readable medium for implementing a custom plastic surgery simulation of the present invention described below correspond to the method, system and computer-readable medium for implementing a user's custom plastic surgery simulation using 3D modeling data for each user's external appearance element derived from the method, system and computer-readable medium for image-based organ internal 3D modeling described below.

[0058]

[0059] 2. The endoscopic image described later in the image-based internal 3D modeling method, system, and computer-readable medium is an image to which the SfM algorithm is applied for 3D modeling, and corresponds to an image corresponding to the external image described later in the method, system, and computer-readable medium for implementing custom molding simulation of the present invention.

[0060]

[0061] FIG. 1 schematically illustrates the execution steps of a method for implementing a custom molding simulation according to one embodiment of the present invention.

[0062]

[0063] As illustrated in FIG. 1, a method for implementing a custom plastic surgery simulation performed on a user terminal and a service server comprises: an image acquisition step of capturing a user from different angles through a camera by the user terminal, acquiring a plurality of images, and outputting them on a screen; an image transmission step of transmitting, by the user terminal, a representative image determined according to a user input from among the plurality of images and the plurality of images to the service server; an RGB-D image generation step of generating, by the service server, a plurality of RGB-D images including location information, color information, and depth information regarding the user's appearance included in the corresponding images by inputting each of the plurality of images into a deep learning-based first analysis model; an appearance image extraction step of extracting, by the service server, an appearance image including only each of a plurality of appearance elements of the user by inputting the plurality of RGB-D images into a deep learning-based second analysis model; The present invention comprises: a 3D modeling data derivation step of applying a SfM algorithm and 3D Gaussian splatting to a plurality of appearance images for each appearance element by a service server, and deriving 3D modeling data for each appearance element through a feature map extracted from each of the plurality of appearance images; a representative feature point determination step of determining, by the service server, one or more representative feature points from each of the 3D modeling data for each appearance element; a change information transmission step of transmitting, by a user terminal, change information of an appearance element determined according to a user's input for the representative image to the service server; a composite image generation step of generating a transformed appearance image by changing, by the service server, the position coordinates of one or more representative feature points related to the change information according to the change information, and aligning the transformed appearance image with the representative image to generate a composite image; and a composite image provision step of receiving, by the user terminal, the composite image from the service server and providing it to the user.

[0064]

[0065] Specifically, the present invention provides a user with a custom plastic surgery simulation, and is characterized by performing precise 3D modeling of the user's appearance so that the user does not feel a sense of incongruity while using the simulation. To achieve this, the user terminal performs an image acquisition step, capturing multiple images of the user from different angles using a camera, and outputting them on the screen for the user to view.

[0066] According to an embodiment of the present invention, the plurality of images include an appearance that the user desires to have plastic surgery, and as one embodiment of the present invention, the plurality of images may include the user's facial appearance, and for the convenience of understanding the present invention, the plurality of images described below correspond to images of the user's facial appearance taken from various angles.

[0067] The user can select a representative image from among multiple images displayed on the screen of the user terminal, into which the user wishes to synthesize the external element, by inputting it into the user terminal (e.g., by touch input, etc.).

[0068] Meanwhile, the user terminal and the service server are connected by wire and wireless means to communicate with each other, and after the user decides on a representative image, the user terminal transmits the representative image and the plurality of images to the service server through an image transmission step.

[0069] The service server performs 3D modeling based on the plurality of images received from the user terminal by inputting each of the plurality of images into a deep learning-based first analysis model through an RGB-D image generation step, thereby generating a plurality of RGB-D images including at least one of the location information about the user's appearance included in the image, as well as color information and depth information.

[0070] Thereafter, the service server inputs multiple RGB-D images into a deep learning-based second analysis model through an appearance image extraction step to extract an appearance image that includes only each of multiple appearance elements of the user. In one embodiment of the present invention, the multiple appearance elements may include one or more of the user's eyebrows, eyes, nose, mouth, and facial contour, and the appearance image is extracted for each appearance element.

[0071] The service server applies the SfM algorithm and 3D Gaussian splatting to a plurality of appearance images extracted for each appearance element through a 3D modeling data derivation step, and derives 3D modeling data for each appearance element through a feature map extracted from each of the plurality of appearance images. The 3D modeling data derivation step includes an initial 3D data generation step, a Gaussian 3D data generation step, a feature map extraction step, a voxel 3D data generation step, and a mesh 3D data generation step, and each of the initial 3D data generation step, the Gaussian 3D data generation step, the feature map extraction step, the voxel 3D data generation step, and the mesh 3D data generation step is 2. In the image-based internal organ 3D modeling method, system and computer-readable medium, the same process as each of the initial 3D data generation step (S200), Gaussian 3D data generation step (S300), feature map extraction step (S400), voxel 3D data generation step (S500), and mesh 3D data generation step (S600) described below is performed on an external image instead of an endoscopic image. Therefore, the process of performing each of the initial 3D data generation step, Gaussian 3D data generation step, feature map extraction step, voxel 3D data generation step, and mesh 3D data generation step in the present invention will be described in detail in 2. Image-based internal organ 3D modeling method, system and computer-readable medium.

[0072] Afterwards, the service server determines at least one representative feature point for each 3D modeling data element through the representative feature point determination step. The representative feature point can be determined based on a preset reference feature point for each appearance element. For example, in the case of the mouth, the representative feature point can be determined from the 3D modeling data based on the reference feature point corresponding to the end of the corner of the mouth.

[0073] Meanwhile, the user terminal transmits information on changes in the external appearance elements determined by the user's input for the representative image selected by the user to the service server. It is desirable for the user to experience various external appearances while utilizing the custom molding simulation, and the change information may include information on changes in the size, position, or inclination of the external appearance elements. For example, the change information may include information on the degree to which the user desires a change in the left and right size of the mouth.

[0074] The service server generates a transformed external image by changing the position coordinates of each of one or more representative feature points of 3D modeling data derived from external elements related to change information received from a user terminal through a synthetic image generation step based on the change information, and then aligns the transformed external image with the representative image received from the user terminal to generate a synthetic image.

[0075] Thereafter, the user terminal receives the synthetic image from the service server and performs a synthetic image provision step of displaying the synthetic image on the screen as a plastic surgery prediction result to the user.

[0076]

[0077] The present invention can reduce the risk of plastic surgery by allowing the user to experience a custom plastic surgery simulation before undergoing actual plastic surgery and predict the results after the plastic surgery, can support the user's decision-making regarding plastic surgery by enabling specific communication between the user (the subject of the surgery) and the doctor (the surgeon) based on the composite image, and can provide various effects such as allowing the user to feel psychologically stable by visually experiencing the simulation results that he or she directly manipulated.

[0078]

[0079] Meanwhile, a custom molding system including a user terminal and a service server includes an image acquisition unit that performs an image acquisition step, an image transmission unit that performs an image transmission step, an RGB-D image generation unit that performs an RGB-D image generation step, an appearance image extraction unit that performs an appearance image extraction step, a 3D modeling data extraction unit that performs a 3D modeling data extraction step, a representative feature point determination unit that performs a representative feature point determination step, a change information transmission unit that performs a change information transmission step, a composite image generation unit that performs a composite image generation step, and a composite image provision unit that performs a composite image provision step.

[0080]

[0081] FIG. 2 schematically illustrates a plurality of images according to one embodiment of the present invention.

[0082]

[0083] Specifically, to 3D model a user's appearance, it is necessary to prevent blind spots in the user's appearance. To this end, it is desirable for the user terminal to capture multiple images from various angles and transmit them to the service server. It is more preferable for the multiple images to be captured so as to include the external features desired by the user for shaping.

[0084] In addition, in one embodiment of the present invention, the plurality of images do not correspond to the entire image captured by the camera of the user terminal, but only include images whose resolution exceeds a preset resolution standard, thereby achieving the effect of generating a high-quality composite image thereafter.

[0085]

[0086] FIG. 3 schematically illustrates the execution process of the RGB-D image generation step and the external image extraction step according to one embodiment of the present invention.

[0087]

[0088] As illustrated in FIG. 3, the above-described appearance image extraction step segments only each of a plurality of appearance elements included in the representative image through the second analysis model, and extracts an appearance image including each of the segmented appearance elements, wherein the plurality of appearance elements include at least one of the user's eyebrows, eyes, nose, mouth, and facial contour.

[0089]

[0090] Schematically, Fig. 3 (a) illustrates the execution process of the RGB-D image generation step, and Fig. 3 (b) illustrates the execution process of the external image extraction step.

[0091]

[0092] Specifically, when performing 3D modeling using 2D images, position information, color information, and depth information for each pixel are required so that the information contained in each pixel of the 2D image can be 3D mapped to the correct location. Therefore, as illustrated in (a) of Fig. 3, the service server inputs multiple images of the user into the deep learning-based first analysis model to generate multiple RGB-D images containing position information, color information, and depth information for each pixel of the user's appearance.

[0093]

[0094] In addition, the present invention is characterized by providing a customizable plastic surgery simulation for a desired plastic surgery area by the user. Therefore, after the RGB-D image is generated, the service server inputs the RGB-D image into a deep learning-based second analysis model to segment the user's appearance by appearance element, and extracts an appearance image for each segmented appearance element.

[0095] In one embodiment of the present invention illustrated in (b) of FIG. 3, a plurality of external elements corresponding to eyebrows, eyes, nose, and mouth are segmented for the user's face, and the external elements are not limited thereto, and in other embodiments, the user's facial contour may be further included, and other external elements may be further included as needed.

[0096]

[0097] FIG. 4 schematically illustrates one or more representative feature points for each external element according to one embodiment of the present invention.

[0098]

[0099] As illustrated in FIG. 4, the representative feature points are determined based on preset reference feature points for each of a plurality of external elements, and the reference feature points are set to allow size changes, position changes, and inclination changes for the shape of the corresponding external element, and the change information includes information on at least one of size changes, position changes, and inclination changes for the external element determined based on a user's input.

[0100]

[0101] Specifically, the service server determines one or more representative feature points from the 3D modeling data derived for each external element through the representative feature point determination step. The representative feature points are determined based on preset reference feature points for the corresponding external element, and it is preferable that the reference feature points be preset so that the size, position, or inclination of the external element's shape can be changed.

[0102] For example, (a) of FIG. 4 illustrates one or more representative feature points determined based on one or more preset reference feature points for an external element corresponding to a mouth, and (b) of FIG. 4 illustrates one or more representative feature points determined based on one or more preset reference feature points for an external element corresponding to an eye.

[0103]

[0104] Meanwhile, as the position of the representative feature point changes, the position of the mesh included in the 3D modeling data also changes, and it is desirable that the position of one or more meshes adjacent to the mesh also change appropriately according to the change in the position of the mesh including the representative feature point so that the user does not feel a sense of incongruity with respect to the transformed external element.

[0105]

[0106] FIG. 5 schematically illustrates a process for generating a transformed external image based on user input according to one embodiment of the present invention. FIG. 6 also schematically illustrates a process for generating a composite image according to one embodiment of the present invention.

[0107]

[0108] As illustrated in FIG. 5, the method for implementing the above molding simulation includes a synthetic image generation step of generating a transformed external shape image by changing the position coordinates of one or more representative feature points related to the changed information according to the changed information by a service server, and generating a synthetic image by aligning the transformed external shape image with the representative image.

[0109] In addition, the above change information includes information on an emotional expression determined based on a user's input, and the above synthetic image generation step derives an emotional expression determined by the user from a plurality of emotional expressions pre-stored in a service server based on the change information, and generates a transformed external image by changing the position coordinates of one or more representative feature points related to the change information based on a change rate of the position coordinates of one or more representative feature points preset for the emotional expression.

[0110]

[0111] Schematically, (a) of FIG. 5 illustrates a transformed external image generated in one embodiment of the present invention, and (b) of FIG. 5 illustrates an interface that can be output on a screen of a user terminal in another embodiment of the present invention.

[0112]

[0113] Specifically, the left picture shown in (a) of FIG. 5 is an interface that can be output on the screen of a user terminal in one embodiment of the present invention, and corresponds to an interface that allows a user to customize an external element that he or she wants to shape.

[0114] For example, if a user wants to have their nose plasticized, the service server can output variables that can be plasticized for the nose, such as nose height, nose straightness, and nostril width, through an interface, and can output left and right scrolling on the screen through the interface so that the user can adjust the degree of plasticization desired for each variable.

[0115]

[0116] When a user inputs and saves left and right scrolls for each variable displayed on the screen, the user terminal transmits information about the user's input as change information to the service server through the change information transmission step, and the service server receives the change information and creates a mesh-shaped transformed external image as shown in the right picture of FIG. 5 through the composite image generation step, and creates a composite image by aligning the transformed external image with the representative image received from the user terminal as shown in FIG. 6.

[0117] Thereafter, the user terminal receives the composite image from the service server and outputs it on the screen to visually provide it to the user, thereby enabling the user to predict the desired results after plastic surgery through the composite image, thereby facilitating decision-making regarding plastic surgery.

[0118]

[0119] Meanwhile, since external elements such as eyes, nose, and mouth can be revealed in different shapes depending on the user's emotions, the present invention can predict how the part the user wants to have plastic surgery changed depending on the user's emotional expression and provide the predicted result to the user, thereby enabling the user to experience a custom plastic surgery simulation in various environments.

[0120] As another embodiment of the present invention, as shown in (b) of FIG. 5, a user can select an emotional expression through an interface displayed on the screen of a user terminal, and a service server receives change information including information about an emotional expression determined according to the user's input from the user terminal.

[0121] Preferably, the service server has a preset change ratio of the position coordinates of one or more representative feature points for each of a plurality of emotional expressions, and the service server can generate the transformed appearance image by changing the position coordinates of one or more representative feature points for the appearance elements included in the change information through the synthetic image generation step based on the preset change ratio of the position coordinates of one or more representative feature points for each of the plurality of emotional expressions and the change information.

[0122] For example, if the change information includes a high proportion of emotional expressions corresponding to happiness, the position coordinates of one or more representative feature points may change in the form of the corners of the mouth widening.

[0123]

[0124] Meanwhile, in another embodiment of the present invention, the interface is not limited to the shape of the interface shown in (a) or (b) of FIG. 5, and may be output on the screen of the user terminal in a different form depending on the embodiment.

[0125]

[0126] Figure 7 illustrates a process for generating and providing a recommendation image according to another embodiment of the present invention.

[0127]

[0128] As shown in Fig. 7, the method for implementing the custom molding simulation is as follows:

[0129] The method further includes a change information storage step of storing change information received from each of a plurality of user terminals by a service server; and the composite image generation step further includes a recommendation image generation step of generating a recommended appearance image by changing the position coordinates of one or more representative feature points for the corresponding appearance element based on the plurality of previously stored change information, and generating a recommended image by aligning the recommended appearance image with the representative image; and the composite image provision step further includes a recommendation image provision step of receiving the recommended image from the service server and providing it to the user.

[0130] Additionally, in another embodiment of the present invention, the custom molding system further includes a change information storage unit that performs a change information storage step.

[0131]

[0132] Specifically, in another embodiment of the present invention, the service server can store change information received from each of a plurality of user terminals in a database through a change information storage step. Thereafter, before the service server generates a composite image according to the change information input by the user and the user terminal displays the composite image on the screen, the service server generates a recommended appearance image by changing the position coordinates of one or more representative feature points for the appearance element that the user wants to have plastic surgery based on the plurality of change information pre-stored in the database through the recommended image generation step, and generates a recommended image by aligning the recommended appearance image with the representative image.

[0133] Thereafter, the user terminal outputs the composite image and the recommended image together on the screen through the composite image provision step.

[0134] The above recommended appearance image is an appearance image that has been converted by applying information on the appearance elements that multiple users wanted to have plastic surgery performed on, and allows users to easily use a custom plastic surgery simulation by comparing it with a synthetic image customized by the user.

[0135]

[0136] 2. Image-based internal 3D modeling method, system, and computer-readable medium for organs

[0137]

[0138] 2. The method, system and computer-readable medium for internal organ 3D modeling based on image, which are described below, correspond to the invention that describes in detail the method, system and computer-readable medium for implementing custom plastic surgery simulation of the present invention, the method, system and computer-readable medium for deriving 3D modeling data in 3D form for each external element based on a plurality of external images including each external element of the user so that the user can use the custom plastic surgery simulation without feeling a sense of incongruity.

[0139]

[0140] Specifically, 1. The external image described later in the method, system and computer-readable medium for implementing a custom molding simulation 2. The image to which the SfM algorithm is applied for 3D modeling is the same as the endoscopic image described later in the method, system and computer-readable medium for image-based internal 3D modeling of organs.

[0141]

[0142] FIG. 8 schematically illustrates components of a computing system (1000) that performs a 3D modeling method according to one embodiment of the present invention.

[0143]

[0144] Schematically, (a) of FIG. 8 illustrates components included in a computing system (1000) that performs a 3D modeling method, and (b) of FIG. 8 illustrates a 3D modeling process of a target object.

[0145]

[0146] As in one embodiment of the present invention illustrated in (a) of FIG. 8, a computing system (1000) for performing a 3D modeling method includes: an endoscopic image extraction unit (1100) for extracting a plurality of endoscopic images, each of which includes time information and at least one of position information about the target object and color information and depth information about the target object, from an endoscopic image of a target object captured by an RGB-D camera; an initial 3D data generation unit (1200) for generating initial 3D data by converting the plurality of endoscopic images into a point cloud form through a SfM (Structure from Motion) algorithm; a Gaussian 3D data generation unit (1300) for inputting the initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing covariance values ​​of Gaussian distributions for a plurality of points constituting the point cloud; It may include a feature map extraction unit (1400) that inputs each of the plurality of endoscopic images into a deep learning-based feature extraction model to extract a feature map for each of the plurality of endoscopic images; a voxel 3D data generation unit (1500) that inputs the Gaussian 3D data and the plurality of feature maps into a deep learning-based first transformation model to generate voxel 3D data in the form of voxels; and a mesh 3D data generation unit (1600) that inputs the voxel 3D data and the plurality of feature maps into a deep learning-based second transformation model to generate mesh 3D data in the form of meshes as 3D modeling data.

[0147]

[0148] Specifically, the present invention is a configuration for 3D modeling of a target object based on an endoscopic image captured by an RGB-D camera, wherein the endoscopic image extraction unit (1100) extracts a plurality of endoscopic images included in the endoscopic image in a time series format, as illustrated in (b) of FIG. 8. More specifically, the plurality of endoscopic images extracted from the endoscopic image extraction unit (1100) do not correspond to all image frames included in the endoscopic image, but may correspond to endoscopic images corresponding to preset frame units, or may correspond to endoscopic images having a resolution higher than a preset reference resolution.

[0149] The above multiple endoscopic images are sequentially reconstructed according to the respective execution processes of the initial 3D data generation unit (1200), Gaussian 3D data generation unit (1300), feature map extraction unit (1400), voxel 3D data generation unit (1500), and mesh 3D data generation unit (1600), so that the entire shape of the target object photographed by the RGB-D camera is modeled in a three-dimensional form, as shown in (b) of FIG. 8.

[0150]

[0151] As an example, in the present invention, 3D modeling data reconstructing the inside of an organ as a target object can be implemented in virtual reality or augmented reality, and in the medical field, the implemented 3D internal model of an organ can be used to conduct medical education on the organ, or to provide a surgical simulation for the organ, thereby enabling medical personnel to predict the difficulty of an actual surgery in advance or optimize medical tools and medical procedures required for the surgery.

[0152]

[0153] FIG. 9 schematically illustrates the execution steps of an image-based long-term internal 3D modeling method according to one embodiment of the present invention.

[0154]

[0155] As illustrated in FIG. 9, an image-based internal organ 3D modeling method performed in a computing system (1000) including one or more processors and one or more memories, the method comprising: an endoscopic image extraction step (S100) of extracting a plurality of endoscopic images, each of which includes time information and at least one of positional information about the target object and color information and depth information about the target object, from an endoscopic image captured by an RGB-D camera of the target object; an initial 3D data generation step (S200) of generating initial 3D data by converting the plurality of endoscopic images into a point cloud form using a SfM (Structure from Motion) algorithm; a Gaussian 3D data generation step (S300) of inputting the initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing covariance values ​​of Gaussian distributions for a plurality of points constituting the point cloud; It includes a feature map extraction step (S400) of inputting each of the plurality of endoscopic images into a deep learning-based feature extraction model to extract a feature map for each of the plurality of endoscopic images; a voxel 3D data generation step (S500) of inputting the Gaussian 3D data and the plurality of feature maps into a deep learning-based first transformation model to generate voxel 3D data in the form of voxels; and a mesh 3D data generation step (S600) of inputting the voxel 3D data and the plurality of feature maps into a deep learning-based second transformation model to generate mesh 3D data in the form of meshes as 3D modeling data.

[0156]

[0157] Specifically, the present invention 3D models a target object based on an endoscopic image captured by an RGB-D camera. The endoscopic image extraction step (S100) is performed by an endoscopic image extraction unit (1100), and extracts from the endoscopic image a plurality of endoscopic images, each of which includes at least one of time information and position information for the target object, as well as color information and depth information for the target object.

[0158] In one embodiment of the present invention, the endoscopy image includes a plurality of image frames in a time series, and accordingly, the time information can be determined based on the time at which the endoscopy image was photographed, and the location information includes location coordinates of feature points extracted from the plurality of image frames.

[0159] Meanwhile, as an embodiment of the present invention, when a camera moves and photographs the inside of an organ, such as an endoscope camera, an SfM algorithm that performs 3D modeling using a plurality of 2D images photographed at different locations can be utilized.

[0160] The initial 3D data generation step (S200) is performed by the initial 3D data generation unit (1200), and generates initial 3D data by applying the SfM algorithm to a plurality of extracted endoscopic images to convert the 2D endoscopic images into a 3D point cloud. The initial 3D data includes a plurality of points constituting the point cloud, and each of the plurality of points may include position coordinates and a color value.

[0161] Conventional 3D modeling techniques perform 3D modeling based on data from each of the multiple points that make up a point cloud. However, because the number of points is typically so large, analyzing the data one by one and performing 3D modeling requires a massive amount of computing power. Therefore, the present invention applies 3D Gaussian splatting to the initial 3D data to reduce the amount of computing power during 3D modeling.

[0162] The Gaussian 3D data generation step (S300) is performed by the Gaussian 3D data generation unit (1300), and inputs the generated initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing the covariance value of the Gaussian distribution for a plurality of points constituting a point cloud. Each of the plurality of elliptical spheres includes coordinates of the elliptical sphere, x, y, z sizes of the elliptical sphere, and color distribution information.

[0163] After Gaussian 3D data is generated, the present invention uses feature maps of each of a plurality of endoscopic images to minimize loss of texture information during the 3D modeling process. The feature map extraction step (S400) is performed by a feature map extraction unit (1400), and each of the plurality of endoscopic images is input into a deep learning-based feature extraction model to extract feature maps for each of the plurality of endoscopic images. The feature map can emphasize specific patterns, boundaries, edges, textures, etc. inside an organ, and can be generated in a preset size depending on the number of filters used in the convolution layer included in the feature extraction model.

[0164] The voxel 3D data generation step (S500) is performed by the voxel 3D data generation unit (1500), and inputs Gaussian 3D data and multiple feature maps into a deep learning-based first transformation model to generate voxel 3D data in the form of a voxel, thereby enabling 3D modeling operations to be processed efficiently. Meanwhile, as an embodiment of the present invention, the first transformation model may correspond to a CNN (Convolutional Neural Network) model.

[0165] The mesh 3D data generation step (S600) is performed by the mesh 3D data generation unit (1600), and inputs voxel 3D data and multiple feature maps into a second transformation model based on deep learning to generate mesh 3D data in the form of a mesh, thereby expressing the complex shape inside an organ in detail and reproducing the smooth curvature inside an organ.

[0166]

[0167] The present invention generates mesh 3D data as final 3D modeling data, thereby enabling 3D modeling of a target object with a relatively smaller amount of computing operations than conventional 3D modeling techniques, and by generating 3D modeling data through an SfM algorithm, 3D Gaussian splatting, and a plurality of feature maps, it is possible to prevent a user from feeling a sense of incongruity with respect to the target object in a virtual reality in which the target object is implemented.

[0168]

[0169] FIG. 10 schematically illustrates information included in an endoscopic image according to one embodiment of the present invention.

[0170]

[0171] As illustrated in FIG. 10, the location information includes location coordinates for a portion of the target object, and the time information is included in the endoscopic image as a time series based on the timeline of the endoscopic management, and the location information for the target object is inferred based on the time information, and the time information is inferred based on the location information.

[0172]

[0173] Schematically, Fig. 10 (a) illustrates information included in an endoscopic image, and Fig. 10 (b) illustrates a process for determining location information included in an endoscopic image.

[0174]

[0175] As illustrated in (a) of Fig. 10, an endoscopic image captured by an RGB-D camera includes at least one of time information and location information, color information, and depth information. Since an endoscopic image includes multiple endoscopic images composed of a time series, the endoscopic image may include time information.

[0176] In addition, as shown in (b) of Fig. 10, the pixel coordinates ((x) of multiple reference points of the target object within the endoscopic image 1, y1, z1), (x 2, By analyzing the change in pixel coordinates of reference points (y2, z2, etc.) that change as the RGB-D camera moves, position information for the target object can be determined.

[0177]

[0178] Typically, an endoscopic camera captures images at different times as it moves through the internal organs. Therefore, location information about a target object can be inferred based on the time information, and it is desirable that time information can be inferred based on the location information. In other words, the present invention can easily collate 2D endoscopic images into 3D data based on time information or location information.

[0179]

[0180] Figure 11 schematically illustrates a process for generating initial 3D data according to one embodiment of the present invention.

[0181]

[0182] As described above in Fig. 11, as the RGB-D camera moves through the internal organs to capture images, each of the multiple endoscopic images contains information about different external features of the same target object. In order to prevent confusion of information due to the movement of the camera, the present invention applies the SfM algorithm to the multiple endoscopic images to generate initial 3D data in the form of a 3D point cloud.

[0183] Specifically, the present invention can generate point cloud data in a three-dimensional form by analyzing the change in the position of reference points of a target object through a SfM algorithm to determine the movement of a camera, and reconstructing the position coordinates of a part of the target object captured in each of the plurality of endoscopic images accordingly.

[0184]

[0185] Figure 12 schematically illustrates a process for generating Gaussian 3D data according to one embodiment of the present invention.

[0186]

[0187] As illustrated in FIG. 12, the 3D Gaussian splatting model is a visualization model based on an artificial neural network that is trained to generate Gaussian 3D data by converting initial 3D data in the form of a point cloud into the form of multiple elliptical spheres based on the positional coordinates of each of multiple points constituting the point cloud and the covariance values ​​of Gaussian distributions for the multiple points, and each of the multiple elliptical spheres is formed by clustering the distances between multiple points and the covariance values ​​of Gaussian distributions.

[0188] In addition, each of the plurality of elliptical spheres includes color distribution information including the position coordinates for the center point of the elliptical sphere, the x-axis length, the y-axis length, the z-axis length based on the center point, and the mean value and the covariance value of the Gaussian distribution for the color values ​​of the plurality of points constituting the point cloud, and the size of the elliptical sphere represents the covariance value of the Gaussian distribution for the position coordinates of each of the plurality of points constituting the point cloud, and the position coordinates for the center point of the elliptical sphere are calculated as the mean value of the position coordinates of each of the plurality of points clustered when the elliptical sphere is generated, the position coordinates of each of the plurality of points are determined based on the position information and depth information, and the color value is determined based on the color information.

[0189]

[0190] Schematically, (a) of Fig. 12 shows the covariance values ​​for the position coordinates of the Gaussian distribution for a plurality of points constituting the point cloud, and (b) of Fig. 12 shows an elliptical sphere representing the covariance values ​​for the points.

[0191]

[0192] As described above, when performing 3D modeling for each of a plurality of points constituting a point cloud, a huge amount of computing power is required, and the present invention uses 3D Gaussian splatting to collect points with associated characteristics for a plurality of points and represent them in the shape of an ellipse to solve the above-described problem.

[0193] Specifically, the 3D Gaussian splatting model calculates the covariance value of the Gaussian distribution for the position coordinates of multiple points, as illustrated in (a) of Fig. 12, and expresses the covariance value of the Gaussian distribution calculated for the points in the form of an elliptical sphere, as illustrated in (b) of Fig. 12. More specifically, the elliptical sphere illustrated in (b) of Fig. 12 may correspond to an elliptical sphere representing the covariance value of the Gaussian distribution calculated for a single point, or may correspond to an elliptical sphere representing the covariance value of the Gaussian distribution calculated for multiple points combined.

[0194]

[0195] In one embodiment of the present invention, the distances between a plurality of points and the covariance values ​​of Gaussian distributions can be collected through a clustering algorithm, and the divided clusters can be generated in the form of a single elliptical sphere. In one embodiment of the present invention, the clustering algorithm may include K-Means Clustering, and may further include other algorithms depending on the embodiment. Since the computing system (1000) in the above embodiment does not calculate the data of each of the plurality of points constituting the point cloud one by one, but calculates the data of the elliptical sphere in which the characteristics of the plurality of points are collected, it can have the effect of minimizing the amount of computing operations during 3D modeling work.

[0196]

[0197] In one embodiment of the present invention, the position coordinates of the center point of an elliptical sphere are calculated as an average value of the position coordinates of a plurality of points included in a cluster forming the elliptical sphere, and the x-axis length, y-axis length, and z-axis length based on the center point can be calculated based on the covariance value of the position coordinates of a plurality of points included in the cluster. That is, each of the plurality of elliptical spheres included in Gaussian 3D data can indicate how much the data is spread out around the average, and this can indicate an area in which the data is likely to exist with a certain probability or higher.

[0198] Preferably, a large spherical shape indicates that the data is widely distributed around the mean, and that the data occupy a wide range of space. Meanwhile, it is preferable that the location coordinates of the multiple points are determined based on location and depth information contained in the endoscopic image.

[0199]

[0200] In addition, due to the characteristics of the point cloud, each of the multiple points constituting the point cloud includes a color value, and it is preferable that the color value be determined based on the color information included in the endoscopic image.

[0201] Since Gaussian 3D data is generated based on initial 3D data, each of the plurality of elliptical spheres included in the Gaussian 3D data includes information about color, and more specifically, includes color distribution information including the mean and covariance values ​​of the Gaussian distribution for the color values ​​of the plurality of points. Meanwhile, the center of the elliptical sphere has the highest density of points, and the density of points decreases toward the periphery, and the brightness of the elliptical sphere is expressed differently depending on the data density of the points.

[0202]

[0203] The 3D Gaussian splatting used in the present invention can efficiently perform real-time 3D modeling of a target object because it does not require a relatively complex neural network structure or high computational workload compared to NeRF (Neural Radiance Fields), which corresponds to a conventional 3D modeling technology. In other words, the technical feature of the present invention is that it can easily produce various situations in real time when implementing medical simulations in virtual reality or augmented reality.

[0204]

[0205] Figure 13 schematically illustrates a process for adjusting Gaussian 3D data according to one embodiment of the present invention.

[0206]

[0207] As illustrated in FIG. 13, the 3D Gaussian splatting model is a visualization model based on an artificial neural network trained to generate Gaussian 3D data by removing the corresponding elliptical sphere if the covariance value of the Gaussian distribution for the generated elliptical sphere exceeds a preset covariance reference value and adjusting the number and size of the elliptical spheres according to the rate of change of the loss value according to the change in the position coordinates of the point, and the loss value corresponds to the difference value between the initial 3D data and the information included in the plurality of elliptical spheres.

[0208]

[0209] Specifically, the initial 3D data may include points that are not related to the actual target object due to data distortion caused by noise, etc., as in area A illustrated in Fig. 6. In order to increase the reliability of 3D modeling for the target object, the present invention uses a 3D Gaussian splatting model that is trained not to form an elliptical sphere for points included in the area when the covariance value of the Gaussian distribution for a plurality of points constituting the point cloud is derived to exceed a preset first covariance reference value.

[0210] Through this, the present invention can have the effect of reducing the amount of computing operations by excluding operations on data with low importance during 3D modeling.

[0211]

[0212] Furthermore, the present invention generates Gaussian 3D data through a 3D Gaussian splatting model trained to adjust the number and size of elliptical spheres according to the rate of change in loss values ​​due to changes in the positional coordinates of points constituting a point cloud. Meanwhile, the loss value corresponds to a value determined by the difference between the initial 3D data and the information contained in multiple elliptical spheres.

[0213] More specifically, if the rate of change of the loss value exceeds a preset change standard, the present invention determines that the ellipse was generated incorrectly, and presets the size of the cover area for the ellipse in order to adjust the ellipse. If the size of the ellipse generated based on the covariance value of the Gaussian distribution for a plurality of points exceeds the size of the preset cover area, the Gaussian 3D data generation unit (1300) determines that the ellipse was generated excessively large, and splits the ellipse into several ellipse spheres of relatively smaller sizes.

[0214] Additionally, if the size of the ellipse is smaller than the size of the preset cover area, it is determined that the ellipse was created excessively small, and the ellipse is duplicated to the same size to increase the number of ellipses.

[0215] Meanwhile, it is desirable that the divided or duplicated elliptical sphere be readjusted in a direction in which the rate of change in the loss value according to the change in the position of the point is lowered, and through the above-described process, the present invention can achieve the effect of naturally modeling the texture of the target object through the 3D Gaussian splatting model.

[0216]

[0217] Figure 14 schematically illustrates a process for extracting a feature map according to one embodiment of the present invention.

[0218]

[0219]

[0220] *Generally, when modeling an object in 3D based on a 2D image, complex details about the surface of the target object may be lost during the modeling process, resulting in some parts being modeled distorted.

[0221] In particular, in the case of internal organ models implemented in medical-related virtual reality such as medical surgery simulation, the accuracy of the model is very important. In order to solve the above-described problem, the present invention is characterized by a technical feature of extracting a feature map from each of a plurality of endoscopic images and generating 3D modeling data by reflecting the extracted feature maps.

[0222]

[0223] The above feature map may include color information and depth information that can represent the color and texture characteristics of the target object included in the endoscopic image, and as an embodiment of the present invention, when the target object is inside an organ, it may further include detailed information such as the size, shape, and location of a tumor inside the organ, so that a positive effect can be expected when implementing the target object as a 3D model in a medical-related simulation.

[0224]

[0225] Figure 15 schematically illustrates a process for generating 3D modeling data according to one embodiment of the present invention.

[0226]

[0227] As illustrated in FIG. 15, the first transformation model corresponds to a transformation model learned to generate voxel 3D data through interpolation by applying feature maps of each of a plurality of endoscopic images to the Gaussian 3D data, and the second transformation model corresponds to a transformation model learned to generate mesh 3D data through interpolation by applying feature maps of each of a plurality of endoscopic images to the voxel 3D data.

[0228] In addition, the mesh 3D data generation step (S600) further includes a weight determination step for determining a weight for each of the plurality of feature maps based on the voxel 3D data; and a weight input step for further inputting information about the determined weight into the second transformation model.

[0229]

[0230] The present invention generates voxel 3D data in the form of voxels by inputting Gaussian 3D data and multiple feature maps into a deep learning-based first transformation model through a voxel 3D data generation step (S500). The first transformation model corresponds to a deep learning-based model that performs interpolation using multiple feature maps to minimize information loss during 3D modeling.

[0231] The above voxel 3D data is a standardized voxel form in which Gaussian 3D data is expressed by dividing it into a uniform grid, and since it includes multiple voxels of the same size and shape, the resolution of the data is maintained constant, so that the computing system (1000) can independently perform operations on voxels, and thus 3D modeling operations on a target object can be processed quickly and efficiently.

[0232]

[0233] Meanwhile, in one embodiment of the present invention, when modeling the inside of an organ in 3D, it is necessary to model in the form of a mesh in order to implement a smooth surface inside the organ and to prevent a user viewing the implemented 3D model from feeling a sense of incongruity. Therefore, the present invention generates mesh 3D data in the form of a mesh by inputting voxel 3D data and a plurality of feature maps into a deep learning-based second transformation model through a mesh 3D data generation step (S600). The second transformation model, like the first transformation model, is a model learned to minimize information loss during 3D modeling by performing interpolation through a plurality of feature maps.

[0234]

[0235] Meanwhile, in another embodiment of the present invention, the mesh 3D data generation step (S600) further includes a weight determination step of differently determining weights for each of the plurality of feature maps based on the voxel 3D data, and a weight input step of further inputting information about the determined weights into the second conversion model, thereby generating mesh 3D data in a mesh form through the voxel 3D data, the plurality of feature maps, and the second conversion model into which information about the determined weights is input.

[0236] In addition, in another embodiment of the present invention, the weight determination step determines a weight for each of a plurality of pixels included in each of a plurality of feature maps based on voxel 3D data, and the weight input step can further input information about the weight determined for the pixel into the second transformation model.

[0237]

[0238] Preferably, in one embodiment illustrated in FIG. 15, the first conversion model and the second conversion model are described separately, but depending on the embodiment, the first conversion model and the second conversion model may correspond to the same model.

[0239]

[0240] That is, the present invention can generate a 3D model with higher usability than conventional techniques by 3D modeling a target object that is robust to surrounding environmental materials by reflecting a SfM algorithm, 3D Gaussian splatting, and a feature map based on an RGB-D-based endoscopy image, and can have the effect of minimizing the amount of computing operations required during 3D modeling work.

[0241]

[0242] FIG. 16 exemplarily illustrates the internal configuration of a computing device (11000) according to one embodiment of the present invention.

[0243]

[0244] The custom molding system mentioned in the description of FIG. 1 and the computing system (1000) mentioned in the description of FIG. 8 may include components of the computing device (11000) illustrated in FIG. 16 described below.

[0245]

[0246] As illustrated in FIG. 16, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0247]

[0248] Specifically, the memory (11200) may include, for example, a high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include a software module, a set of instructions, or other various data required for the operation of the computing device (11000).

[0249] At this time, access to the memory (11200) from other components such as the processor (11100) or the peripheral interface (11300) may be controlled by the processor (11100). The processor (11100) may be configured as a single or multiple processors, and may include processors in the form of GPUs and TPUs to improve the processing speed.

[0250] The peripheral interface (11300) may connect input and / or output peripherals of the computing device (11000) to the processor (11100) and the memory (11200). The processor (11100) may execute software modules or instruction sets stored in the memory (11200) to perform various functions for the computing device (11000) and process data.

[0251] The input / output subsystem (11400) can couple various input / output peripheral devices to the peripheral interface (11300). For example, the input / output subsystem (11400) can include a controller for coupling peripheral devices such as a monitor, a keyboard, a mouse, a printer, or, if necessary, a touch screen or a sensor to the peripheral interface (11300). In another aspect, the input / output peripheral devices can be coupled to the peripheral interface (11300) without going through the input / output subsystem (11400).

[0252] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0253] The above communication circuit (11600) may enable communication with another computing device using at least one external port. Alternatively, as described above, the communication circuit (11600) may, if necessary, include an RF circuit to transmit and receive RF signals, also known as electromagnetic signals, thereby enabling communication with another computing device.

[0254]

[0255] This embodiment of FIG. 16 is only an example of the computing device (11000), and the computing device (11000) may have some of the components illustrated in FIG. 16 omitted, may further include additional components not illustrated in FIG. 16, or may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, in addition to the components illustrated in FIG. 16, and the communication circuit (1160) may include a circuit for RF communication of various communication methods (Wi-Fi, 3G, LTE, 5G, 6G, Bluetooth, NFC, Zigbee, etc.). Components that can be included in the computing device (11000) may be implemented as hardware including one or more signal processing or application-specific integrated circuits, software, or a combination of both hardware and software.

[0256] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed by various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application exclusively for mobile terminals. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file at the request of the user terminal.

[0257]

[0258] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0259] The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may be standardized and stored or executed in a standardized manner on a network-connected computing device. The software and data may be stored on one or more computer-readable recording media.

[0260] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0261]

[0262] In one embodiment of the present invention, a plastic surgery simulation is provided in which a user can virtually experience plastic surgery by 3D modeling the area the user wants to undergo plastic surgery and synthesizing it with an existing image, thereby providing the user with a high level of satisfaction and a sense of stability when actually undergoing plastic surgery later.

[0263] In one embodiment of the present invention, the user can directly input the degree of change desired for a desired area of ​​plastic surgery into a user terminal, and the changed appearance of the user based on the user's input is provided to the user in real time, thereby providing the user with high satisfaction and convenience.

[0264] In one embodiment of the present invention, by determining one or more representative feature points for a 3D modeled external element, the user can precisely implement a desired molding part by changing the position of the representative feature points based on change information input by the user, thereby providing a molding simulation that can be customized by the user.

[0265] In one embodiment of the present invention, rather than performing 3D modeling based on the values ​​of each of a plurality of points constituting a point cloud, data in the shape of an ellipse is generated by linking overlapping features of points that are related to each other, thereby reducing the amount of computing operations relatively compared to conventional 3D modeling techniques.

[0266] In one embodiment of the present invention, the continuity between a plurality of points constituting a point cloud can be expressed through the covariance value of the Gaussian distribution for the points, thereby achieving the effect of smoothly implementing a desired plastic part by the user.

[0267]

[0268] Although the embodiments have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents. Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

1. A method for implementing a custom molding simulation performed on a user terminal and a service server, An image acquisition step of capturing multiple images by taking pictures of the user from different angles through a camera using a user terminal and outputting them on a screen; An image transmission step of transmitting a representative image determined according to a user input from among the plurality of images and the plurality of images to a service server by a user terminal; An RGB-D image generation step in which each of the plurality of images is input into a deep learning-based first analysis model by a service server to generate a plurality of RGB-D images including location information, color information, and depth information about the user's appearance included in the corresponding image; An appearance image extraction step for extracting an appearance image including only each of a plurality of appearance elements for a user by inputting the plurality of RGB-D images into a deep learning-based second analysis model by a service server; A 3D modeling data derivation step in which a SfM algorithm and 3D Gaussian splatting are applied to a plurality of appearance images for each appearance element by a service server, and 3D modeling data is derived for each appearance element through a feature map extracted from each of the plurality of appearance images; A representative feature point determination step for determining one or more representative feature points from each of the 3D modeling data for each of the above external elements by the service server; A change information transmission step for transmitting change information of an external element determined according to a user's input for the representative image to a service server by a user terminal; A synthetic image generation step in which the location coordinates of one or more representative feature points related to the change information are changed according to the change information by the service server to generate a transformed external image, and the transformed external image is aligned with the representative image to generate a synthetic image; and A method for implementing a plastic surgery simulation, comprising a synthetic image providing step of receiving the synthetic image from a service server and providing it to a user through a user terminal.

2. In claim 1, The above-mentioned external image extraction step segments only each of the multiple external elements included in the representative image through the second analysis model, and extracts an external image including each of the segmented external elements. A method for implementing a plastic surgery simulation, wherein the plurality of external appearance elements include at least one of the user's eyebrows, eyes, nose, mouth, and facial contour.

3. In claim 1, The above representative feature points are determined based on preset reference feature points for each of the multiple external elements, The above reference feature points are set to allow changes in size, position, and inclination of the shape of the corresponding external element. A method for implementing a molding simulation, wherein the above change information includes information on at least one of a change in size, a change in position, and a change in inclination of an external element determined according to a user's input.

4. In claim 1, The above change information includes information about emotional expressions determined based on user input, The above synthetic image generation step is, A method for implementing a plastic surgery simulation, wherein an emotional expression determined by a user is derived from multiple emotional expressions pre-stored in a service server based on the above change information, and the position coordinates of one or more representative feature points related to the change information are changed based on a change rate of the position coordinates of one or more representative feature points preset for the emotional expression, thereby generating a transformed external image.

5. In claim 1, A method for implementing the above custom molding simulation is as follows: Further comprising a change information storage step for storing change information received from each of a plurality of user terminals by the service server; The above synthetic image generation step is, A recommended appearance image is generated by changing the position coordinates of one or more representative feature points for the corresponding appearance element based on a plurality of previously stored change information, and a recommended image generation step is further included to generate a recommended image by aligning the recommended appearance image with the representative image. The above synthetic image provision step is, A method for implementing a plastic surgery simulation, further comprising a recommendation image providing step of receiving the above recommendation image from a service server and providing it to a user.

6. In claim 1, The above 3D modeling data derivation step is, An initial 3D data generation step that generates initial 3D data by converting the above multiple external images into a point cloud format using the SfM (Structure from Motion) algorithm; A Gaussian 3D data generation step of inputting the above initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing the covariance values ​​of Gaussian distributions for a plurality of points constituting a point cloud; A feature map extraction step of inputting each of the plurality of appearance images into a deep learning-based feature extraction model to extract a feature map for each of the plurality of appearance images; A voxel 3D data generation step for generating voxel 3D data in the form of voxels by inputting the above Gaussian 3D data and multiple feature maps into a first transformation model based on deep learning; and A method for implementing a molding simulation, comprising a mesh 3D data generation step of inputting the above voxel 3D data and a plurality of feature maps into a second transformation model based on deep learning to generate mesh 3D data in the form of a mesh as 3D modeling data.

7. In claim 6, Each of the above plurality of elliptical spheres, The position coordinates of the center point of the ellipse, The x-axis length, y-axis length, z-axis length, and Includes color distribution information including the mean and covariance values ​​of the Gaussian distribution for the color values ​​of multiple points constituting the point cloud, The size of the ellipse represents the covariance value of the Gaussian distribution for the position coordinates of each of the multiple points that make up the point cloud. The position coordinates for the center point of the above elliptical sphere are calculated as the average value of the position coordinates of each of the multiple points clustered when the elliptical sphere is created. A method for implementing a molding simulation, wherein the position coordinates of each of the plurality of points are determined based on the position information and depth information, and the color value is determined based on the color information.

8. A custom molding system that includes a user terminal and a service server and performs a method for implementing a custom molding simulation. An image acquisition unit that captures multiple images by taking pictures of the user from different angles through a camera using a user terminal and outputs them on a screen; An image transmission unit that transmits a representative image determined according to a user input from among the plurality of images and the plurality of images to a service server through a user terminal; An RGB-D image generation unit that inputs each of the plurality of images into a deep learning-based first analysis model by a service server to generate a plurality of RGB-D images including location information, color information, and depth information about the user's appearance included in the corresponding image; An appearance image extraction unit that inputs the plurality of RGB-D images into a deep learning-based second analysis model by a service server to extract an appearance image that includes only each of the plurality of appearance elements for the user; A 3D modeling data extraction unit that applies the SfM algorithm and 3D Gaussian splatting to a plurality of appearance images for each appearance element by a service server and derives 3D modeling data for each appearance element through a feature map extracted from each of the plurality of appearance images; A representative feature point determination unit that determines one or more representative feature points from each of the 3D modeling data for each of the above external elements by the service server; A change information transmission unit that transmits change information of an external element determined according to a user input for the representative image to a service server through a user terminal; A synthetic image generation unit that generates a transformed external image by changing the position coordinates of one or more representative feature points related to the changed information according to the changed information by the service server, and generates a synthetic image by aligning the transformed external image with the representative image; and A custom plastic surgery system, comprising a synthetic image providing unit that receives the synthetic image from a service server and provides it to a user through a user terminal.

9. In claim 8, The above-mentioned external image extraction unit segments only each of the plurality of external elements included in the representative image through the second analysis model, and extracts an external image including each of the segmented external elements. A custom plastic surgery system in which the plurality of external appearance elements include at least one of the user's eyebrows, eyes, nose, mouth, and facial contour.

10. In claim 8, The above representative feature points are determined based on preset reference feature points for each of the multiple external elements, The above reference feature points are set to allow changes in size, position, and inclination of the shape of the corresponding external element. A custom molding system, wherein the above change information includes information on at least one of size change, position change, and inclination change for an external element determined according to a user's input.

11. In claim 8, The above change information includes information about emotional expressions determined based on user input, The above composite image generation unit, A custom plastic surgery system that derives an emotional expression determined by a user from multiple emotional expressions pre-stored in a service server based on the above change information, and generates a transformed external image by changing the position coordinates of one or more representative feature points related to the above change information based on the change rate of the position coordinates of one or more representative feature points pre-set for the emotional expression.

12. In claim 8, The above custom molding system is, Further comprising a change information storage unit that stores change information received from each of a plurality of user terminals by the service server; The above composite image generation unit, It further includes a recommendation image generation unit that generates a recommended appearance image by changing the location coordinates of one or more representative feature points for the corresponding appearance element based on a plurality of previously stored change information, and generates a recommended image by aligning the recommended appearance image with the representative image. The above synthetic image providing unit, A custom plastic surgery system further comprising a recommendation image providing unit that receives the above recommendation image from a service server and provides it to a user.

13. In claim 8, The above 3D modeling data extraction unit is, An initial 3D data generation unit that generates initial 3D data by converting the plurality of external images into a point cloud format using a SfM (Structure from Motion) algorithm; A Gaussian 3D data generation unit that inputs the above initial 3D data into a 3D Gaussian splatting model to generate Gaussian 3D data including a plurality of elliptical spheres representing covariance values ​​of Gaussian distributions for a plurality of points constituting a point cloud; A feature map extraction unit that inputs each of the plurality of appearance images into a deep learning-based feature extraction model and extracts a feature map for each of the plurality of appearance images; A voxel 3D data generation unit that inputs the above Gaussian 3D data and multiple feature maps into a deep learning-based first transformation model to generate voxel 3D data in the form of voxels; and A custom molding system, comprising a mesh 3D data generation unit that inputs the above voxel 3D data and a plurality of feature maps into a deep learning-based second transformation model to generate mesh 3D data in the form of a mesh as 3D modeling data.

14. In claim 13, Each of the above plurality of elliptical spheres, The position coordinates of the center point of the ellipse, The x-axis length, y-axis length, z-axis length, and Includes color distribution information including the mean and covariance values ​​of the Gaussian distribution for the color values ​​of multiple points constituting the point cloud, The size of the ellipse represents the covariance value of the Gaussian distribution for the position coordinates of each of the multiple points that make up the point cloud. The position coordinates for the center point of the above elliptical sphere are calculated as the average value of the position coordinates of each of the multiple points clustered when the elliptical sphere is created. A custom molding system in which the position coordinates of each of the plurality of points are determined based on the position information and depth information, and the color value is determined based on the color information.

15. A computer-readable medium for implementing a method of implementing a custom molding simulation performed in a user terminal and a service server, wherein the computer-readable medium includes computer-executable instructions that cause the user terminal and the service server to perform the following steps: The steps below are: An image acquisition step of capturing multiple images by taking pictures of the user from different angles through a camera using a user terminal and outputting them on a screen; An image transmission step of transmitting a representative image determined according to a user input from among the plurality of images and the plurality of images to a service server by a user terminal; An RGB-D image generation step in which each of the plurality of images is input into a deep learning-based first analysis model by a service server to generate a plurality of RGB-D images including location information, color information, and depth information about the user's appearance included in the corresponding image; An appearance image extraction step for extracting an appearance image including only each of a plurality of appearance elements for a user by inputting the plurality of RGB-D images into a deep learning-based second analysis model by a service server; A 3D modeling data derivation step in which a SfM algorithm and 3D Gaussian splatting are applied to a plurality of appearance images for each appearance element by a service server, and 3D modeling data is derived for each appearance element through a feature map extracted from each of the plurality of appearance images; A representative feature point determination step for determining one or more representative feature points from each of the 3D modeling data for each of the above external elements by the service server; A change information transmission step for transmitting change information of an external element determined according to a user's input for the representative image to a service server by a user terminal; A synthetic image generation step in which the location coordinates of one or more representative feature points related to the change information are changed according to the change information by the service server to generate a transformed external image, and the transformed external image is aligned with the representative image to generate a synthetic image; and A computer-readable medium comprising a step of providing a composite image by receiving the composite image from a service server and providing the composite image to a user through a user terminal.

Images