Texturing method for the generation of three-dimensional virtual models and computing device therefor - Patents.com

Abstract

A texturing method for generating a 3D virtual model according to one technical aspect of the present application may be performed in a computing device that generates a 3D virtual model based on a plurality of data sets, the data sets including a hue image and a depth image, that are generated at a plurality of shooting positions of an indoor space, and may include the steps of: generating a 3D mesh model based on the plurality of data sets, the data sets including a hue image and a depth image, for each of a plurality of faces included in the 3D mesh model, texturing the plurality of faces based on an association between the face and a hue image; identifying a hole face that is indicated as a hole because the association is not set; and identifying a plurality of associated vertices associated with the hole face and setting a hue of the hole face based on the hues of the identified plurality of associated vertices.

Description

[Technical field]

[0001] The present application relates to a texturing method for generating a three-dimensional virtual model for an interior space and a computing device therefor. [Background technology]

[0002] Recently, there has been active development of technology to realize virtual spaces that allow users to experience being in a real space without actually visiting the real space by receiving an online virtual space that corresponds to the real space. Such virtual technology based on real space is technology for realizing digital twins or the metaverse, and a wide variety of developments are being made.

[0003] To realize such a virtual space, it is necessary to take a two-dimensional image of the actual space to be realized, and then generate a three-dimensional virtual image, i.e., a three-dimensional virtual model, based on the two-dimensional image to provide the virtual space.

[0004] Such a 3D virtual model is generated based on data captured from multiple points inside the indoor space. In this case, in order to construct the 3D virtual model, color and distance data acquired from multiple points in the indoor space in 360° is collected, and the 3D virtual model is generated based on the collected color and distance data.

[0005] Such 3D virtual models are generated based on shooting indoor spaces, and therefore have the problem of holes occurring because image data cannot be obtained for blind spots of the camera during shooting. Summary of the Invention [Problem to be solved by the invention]

[0006] One technical aspect of the present application has been devised to solve the problems of the conventional technology described above, and according to one embodiment disclosed in the present application, the objective is to effectively fill in holes caused by blind spots of a camera during shooting.

[0007] According to one embodiment disclosed in the present application, an object is to effectively select an image suitable for the face of a 3D model from among a plurality of images generated at a plurality of points in a room.

[0008] According to one embodiment disclosed in the present application, it is an object of the present invention to more accurately compensate for color unevenness caused by different shooting conditions among a plurality of different points in a room.

[0009] The object of the present application is not limited to the above-mentioned object, and other object not mentioned will be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0010] One technical aspect of the present application proposes a texturing method for generating a 3D virtual model, which may be performed in a computing device that generates a 3D virtual model based on a plurality of data sets, each of which includes a hue image and a depth image, generated at a plurality of shooting positions of an indoor space, and may include the steps of: generating a 3D mesh model based on the plurality of data sets, each of which is generated at a plurality of shooting positions of the indoor space; texturing each of a plurality of faces included in the 3D mesh model based on an association between the face and a hue image; identifying a hole face that is indicated as a hole because the association is not set; and identifying a plurality of associated vertices associated with the hole face and setting a hue of the hole face based on the hues of the identified plurality of associated vertices.

[0011] Another technical aspect of the present application provides a computing device, comprising a memory storing one or more instructions and at least one processor executing the one or more instructions stored in the memory, the at least one processor executing the one or more instructions to prepare a plurality of data sets, each of which includes a color image and a depth image, generated at a plurality of shooting positions of an indoor space, generate a 3D mesh model based on the plurality of data sets, each of which is generated at a plurality of shooting positions of the indoor space, perform texturing on each of a plurality of faces included in the 3D mesh model based on an association between the face and a color image, identify a hole face that is indicated as a hole because the association is not established, identify a plurality of associated vertices associated with the hole face, and set a hue of the hole face based on the hue of the identified plurality of associated vertices.

[0012] Yet another technical aspect of the present application proposes a storage medium having computer readable instructions stored therein. The instructions, when executed by a computing device, may cause the computing device to perform the following operations: preparing a plurality of data sets, each of which is generated at a plurality of shooting positions of an indoor space, the data sets including a hue image and a depth image; generating a 3D mesh model based on the plurality of data sets, each of which is generated at a plurality of shooting positions of the indoor space; texturing each of a plurality of faces included in the 3D mesh model based on an association between the face and a hue image; identifying a hole face, which is indicated as a hole because the association is not established; and identifying a plurality of associated vertices associated with the hole face and setting a hue of the hole face based on the hues of the identified associated vertices.

[0013] The above means for solving the problems are not all of the features of the present application. The various means for solving the problems of the present application can be understood in more detail with reference to the specific embodiments in the following detailed description. Effect of the Invention

[0014] The present application may provide one or more of the following advantages:

[0015] According to one embodiment disclosed in the present application, by filling hole faces using the hues of points in a point cloud, it is possible to accurately fill hole faces in a resource efficient manner.

[0016] According to one embodiment disclosed in the present application, by effectively selecting images suitable for the face of a 3D model, it is possible to provide more accurate texturing even in a 3D generated environment based on images taken from multiple locations in a room that are far apart.

[0017] According to one embodiment disclosed in the present application, it is possible to accurately compensate for color unevenness caused by different shooting conditions between multiple different points in a room, minimize the sense of foreignness on each surface of the virtual indoor space, and provide a texture of the virtual space that is even more similar to the actual space.

[0018] The effects of the present application are not limited to the effects described above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is an exemplary diagram illustrating a system for providing a texturing method for generating a three-dimensional virtual model according to an embodiment disclosed in the present application. [Diagram 2] FIG. 2 is a flow chart illustrating a texturing method for generating a three-dimensional virtual model according to one embodiment disclosed in the present application. [Diagram 3] FIG. 2 is a flow chart illustrating a texturing method for generating a three-dimensional virtual model according to one embodiment disclosed in the present application. [Figure 4] 1A to 1C are diagrams illustrating an example of acquiring a hue image according to an embodiment disclosed in the present application. [Diagram 5] 1A to 1C are diagrams illustrating an example of acquiring a hue image according to an embodiment disclosed in the present application. [Figure 6] 1A to 1C are diagrams illustrating an example of acquiring a hue image according to an embodiment disclosed in the present application. [Figure 7] 1A to 1C are diagrams illustrating an example of acquiring a hue image according to an embodiment disclosed in the present application. [Figure 8]1 is a flowchart illustrating a method for selecting a hue image for generating a 3D virtual model according to an embodiment disclosed in the present application. [Figure 9] 11A and 11B are diagrams illustrating an example of selecting a hue image according to an embodiment disclosed in the present application. [Figure 10] 11A and 11B are diagrams illustrating an example of selecting a hue image according to an embodiment disclosed in the present application. [Figure 11] 11A and 11B are diagrams illustrating an example of selecting a hue image according to an embodiment disclosed in the present application. [Figure 12] 11A and 11B are diagrams illustrating an example of selecting a hue image according to an embodiment disclosed in the present application. [Figure 13] FIG. 2 is a flow chart illustrating a method for setting an unscened face for generating a 3D virtual model according to an embodiment disclosed in the present application. [Figure 14] FIG. 2 is a diagram illustrating an example of setting an unscened face according to an embodiment disclosed in the present application. [Figure 15] FIG. 2 is a diagram illustrating an example of setting an unscened face according to an embodiment disclosed in the present application. [Figure 16] 1 is a flowchart illustrating a hue correction method for generating a 3D virtual model according to an embodiment disclosed in the present application. [Figure 17] 11A to 11C are diagrams illustrating an example of hue correction according to an embodiment disclosed in the present application. [Figure 18] 11A to 11C are diagrams illustrating an example of hue correction according to an embodiment disclosed in the present application. [Figure 19] FIG. 2 is a flow chart illustrating a method for texturing an unscened region for generating a three-dimensional virtual model according to an embodiment disclosed in the present application. [Figure 20] FIG. 1 illustrates an example of texturing an unscened region according to an embodiment disclosed in the present application. [Figure 21] FIG. 1 is a block diagram illustrating a computing device according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, preferred embodiments of the present application will be described with reference to the accompanying drawings.

[0021] However, such embodiments do not represent any of the technical ideas of the present invention, and therefore should be understood to include a wide variety of modifications, equivalents, and / or alternatives of the embodiments of this disclosure.

[0022] In describing this disclosure, if it is deemed that a specific description of related known functions or configurations may unnecessarily obscure the gist of this disclosure, the detailed description will be omitted. The terms used in this disclosure are merely used to describe specific embodiments and are not intended to limit the scope of the present invention. Singular expressions include plural expressions unless otherwise clearly indicated in the context.

[0023] In this disclosure, the words "have," "may have," "include," "may include," "comprise," "may comprise," and the like indicate the presence of a given feature (e.g., a number, function, operation, or component such as a part) and do not exclude the presence of additional features.

[0024] In connection with the description of the drawings, similar or related components may be provided with similar reference numerals. The singular form of a noun corresponding to an item may include one or more of said items, unless otherwise indicated herein or clearly contradicted by the context. In this application, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may cover any one of the items listed together in the corresponding phrase of the phrase, or any possible combination thereof.

[0025] The terms "first," "second," "primary," "secondary," and the like are merely used to distinguish one component from another, and are not intended to limit other aspects of the components (e.g., importance, order, or sequence), etc.

[0026] When a (e.g., first) component is referred to as being "coupled," "connected," or "coupled" to another (e.g., second) component, with or without the terms "functionally" or "communicatively," it means that the component can be coupled, connected, or coupled to the other component directly, or through a third component.

[0027] The phrase "configured to" used in this disclosure may be rephrased as, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of," depending on the context and situation. The phrase "configured to" does not necessarily mean only something that is "specifically designed to" in terms of hardware.

[0028] Instead, in some circumstances, the phrase "a device configured to" may mean that the device, together with other devices or components, is "capable of." For example, the phrase "a processor configured (or set up) to do A, B, and C" may refer to a dedicated processor for performing those operations (e.g., an embedded processor) or a general-purpose processor (e.g., a CPU or application processor) that is capable of performing those operations by executing one or more software programs stored in a memory device.

[0029] In the embodiments, a "module" or a "unit" performs at least one function or operation and may be realized by hardware or software, or may be embodied by a combination of hardware and software. Furthermore, multiple "modules" or multiple "units" may be integrated into at least one module and realized by at least one processor, except for a "module" or "unit" that needs to be realized by specific hardware.

[0030] Various embodiments of the present application may be implemented as software (e.g., a program) including one or more commands stored in a storage medium readable by a machine, such as the user terminal 500 or the computing device 300. For example, the processor 330 may call and execute at least one of the one or more commands stored in the storage medium. This enables the machine to be operated to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" merely means that the storage medium is a tangible device and does not include a signal (e.g., electromagnetic waves), and this phrase does not distinguish between a case where data is stored semi-permanently and a case where data is stored temporarily in the storage medium.

[0031] Although a variety of flow charts are disclosed to explain the embodiments of the present application, this is for ease of explanation of each step or operation, and each step is not necessarily performed in the order depicted in the flow charts, i.e., each step in the flow charts may be performed simultaneously with each other, may be performed in the order depicted in the flow charts, or may be performed in the reverse order to the order depicted in the flow charts.

[0032] FIG. 1 is an exemplary diagram illustrating a system for providing a texturing method for generating a three-dimensional virtual model according to an embodiment disclosed in the present application.

[0033] A system for providing a texturing method for the generation of a three-dimensional virtual model may comprise an image acquisition device 100 , a computing device 300 and a user terminal 500 .

[0034] The image capture device 100 is a device that generates color and depth map images that are used to generate a spherical virtual image.

[0035] In the illustrated example, the image capture device 100 may include a distance measurement device, in the illustrated example a depth scanner, and a camera.

[0036] A camera is a device that provides a photographing function, and generates a hue image that is expressed in hue for a subject area (image capture area).

[0037] In this specification, the term "hue image" refers to any image expressed by a hue, and is not limited to a specific expression method. Therefore, the hue image can be applied to a wide variety of standards, such as an RFG image expressed by red, green, and blue, as well as a CMYK image expressed by cyan, magenta, yellow, and key plate (black).

[0038] As an example, the camera may be used in a mobile phone, a smart phone, a laptop computer, a personal digital assistant (PDA), a tablet PC, an ultrabook, or a wearable device (e.g., smart glass, which is a glasses-type wearable device).

[0039] A depth scanner is a device capable of generating depth information for a subject space, i.e., an imaged area, to generate a depth map image.

[0040] In this specification, a depth map image is an image that includes depth information about a subject space. For example, each pixel in the depth map image may be distance information from an imaging point to each point in the subject space that is photographed, the point corresponding to each pixel.

[0041] The depth scanner may generate such a depth map image and include a certain sensor for measuring distance, such as a LiDAR (Light Detection And Ranging) sensor, an infrared sensor, an ultrasonic sensor, etc. Alternatively, the depth scanner may include a stereo camera, a stereoscopic camera, a 3D depth camera, etc., which can replace the sensor and measure distance information.

[0042] The camera generates a color image for the object space, and the depth scanner generates a depth map image for the object space. The color image generated by the camera and the depth map image generated by the depth scanner can be generated for the same object space under the same conditions (e.g., resolution, etc.), and therefore can be matched 1:1 with each other.

[0043] Such a hue image and a depth map image may be a 360° panoramic image. The depth scanner and the camera may generate a form of a 360° panoramic image for an actually existing indoor space, i.e., a 360° depth map panoramic image and a 360° hue panoramic image, respectively, and provide the image to the computing device 300. Alternatively, depending on the embodiment, the computing device 300 may generate the 360° depth map panoramic image and the 360° hue panoramic image based on data received from the depth scanner and the camera.

[0044] The depth scanner and camera can generate 360° panoramic images of the actual indoor space, i.e., a 360° depth map panoramic image and a 360° color panoramic image, respectively, which can be provided to the computing device 300.

[0045] The depth scanner can generate distance information for each of a plurality of points in the room where such 360° photography was performed. Such distance information may be relative distance information based on the photography point. For example, the depth scanner can have a floor plan for the indoor space and receive an input of an initial starting indoor point in the floor plan according to a user's input. The depth scanner can then generate relative distance movement information based on image analysis and / or a motion detection sensor, such as a three-axis acceleration sensor and / or a gyro sensor. For example, information regarding a second indoor point can be generated based on the relative distance movement information from the starting indoor point, and information regarding a third indoor point can be generated based on the relative distance movement information from the second indoor point. Alternatively, such distance information may be generated by a camera.

[0046] In one embodiment, the depth scanner and the camera can be realized as one image acquisition device. For example, the image acquisition device 100 may be a smartphone equipped with a camera for acquiring images and a LiDAR sensor for measuring distances. Depending on the embodiment, a cradle device on which such a smartphone is placed and which rotates the smartphone 360° under the control of the smartphone and the smartphone may be realized as one image acquisition device.

[0047] The depth scanner or camera may store and provide information regarding the imaging height to the computing device 300. Such imaging height information may be used in the computing device 300 to generate a three-dimensional virtual model.

[0048] The depth map image and the color image may be a panoramic image of a suitable form for providing a 360° image, for example, a panoramic image using equirectangular projection, but for ease of explanation, in the following they will not be distinguished from each other and will be referred to collectively as the depth map image and the color image.

[0049] The user terminal 500 is an electronic device that allows a user to connect to the computing device 300 and experience a virtual three-dimensional model corresponding to an indoor space, and includes, for example, a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, a personal computer (PC), a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, smart glass, a head mounted display (HMD)), and the like. However, in addition to these, the user terminal 500 may include electronic devices used for virtual reality (VR) and augmented reality (AR).

[0050] The computing device 300 can generate a 3D virtual model, which is a 3D virtual space corresponding to the indoor space, using the color images and the depth map images generated at multiple points in the room.

[0051] The computing device 300 can generate a 3D model based on color images and depth map images generated from multiple shooting points in a room, which is a virtual space corresponding to a real space. The 3D model is a virtual model reflecting depth information, and can provide a three-dimensional space equivalent to the real one.

[0052] The computing device 300 can generate a plurality of point sets (e.g., point clouds) in a three-dimensional space based on a plurality of data sets generated at a plurality of shooting points in an indoor space, the data sets including a color image, a depth image, and position information of each point, and generate a 3D mesh model based on the point sets. The 3D mesh model may be a mesh model created by setting a plurality of faces (or polygons) based on a plurality of vertices selected based on the point cloud. As an example, one face may be generated based on three adjacent vertices, and each face may be a flat triangle set as three vertices.

[0053] Once each face is determined in the 3D mesh model, the computing device 300 can set a hue value for each face based on a hue image associated with the face, which may be set with respect to a directional vector perpendicular to the face.

[0054] The computing device 300 may select one hue image to set the hue value of each face, and for this purpose, may calculate a weighting value based on a plurality of weighting value elements for each hue image, and may select one of the hue images based on the weighting value.

[0055] The computing device 300 can fill in and color fill unseen faces. An unseen face is a face that is not displayed in a captured image. For example, a flat surface that is higher than the shooting point, such as the top surface of a refrigerator, is set as an unseen face because it is not captured by the camera. The computing device 300 can fill such unseen faces with color based on the color information of the vertices.

[0056] The computing device 300 can perform hue correction on the 3D model generated after the color filling and color filling of each face is completed. In this application, even if the same camera is used for shooting, the shooting conditions at multiple points in the indoor space are different from each other. Even if the indoor space is the same, the shooting conditions at each point in the indoor space are different, such as the degree of brightness, additional light source, and the hue of the light source. For example, at a shooting point in the room by the window, natural light from the sun is added, and at a shooting point in the room where the lights are turned off, the illuminance is low, so the shooting conditions of the camera may be changed. In this way, since the shooting conditions are different at multiple points in the indoor space, each hue image has a different hue value even for the same object. Therefore, if one object has multiple faces and each face is textured based on a different hue image, there is a risk of unevenness in the expression of the hue of one object. The computing device 300 can perform hue correction to compensate for such unevenness. Such hue correction can be performed by reflecting factors due to differences between multiple shooting points in the indoor space.

[0057] As described above, the 3D model in the present invention has a special environment due to the conditions for generating a virtual space corresponding to an indoor space. That is, it is required to obtain a hue image and a depth image for the indoor space, and therefore, hue images and depth images are obtained at a plurality of indoor shooting points. Meanwhile, the more indoor points from which images are obtained, the more data for the 3D model is obtained, and the better the representation of the 3D model is. In the embodiment of the present application, the representation of such a 3D model, for example, texturing, can be improved with the processing in the computing device 300, so that a high-quality 3D model can be obtained even if the number of indoor shooting points for obtaining indoor images is set to an appropriate number.

[0058] Such a computing device 300 is described in further detail below with reference to FIGS.

[0059] FIG. 2 is a procedure diagram illustrating a texturing method for generating a three-dimensional virtual model.

[0060] 2, the computing device 300 receives a plurality of data sets generated at each indoor point of a plurality of indoor spaces from the image capture device 100 (S201). The plurality of data sets are generated at each indoor point of an actually existing indoor space, and may include, for example, a 360° color image and a 360° depth image.

[0061] The computing device 300 may generate a three-dimensional mesh model based on the multiple data sets (S202).

[0062] The computing device 300 may project a plurality of shooting points in an indoor space onto a three-dimensional coordinate system and set a plurality of three-dimensional reference coordinates corresponding to each of the plurality of shooting points. As an example, the plurality of data sets may include location information in addition to a hue image and a depth image, and the computing device 300 may set a plurality of three-dimensional reference coordinates corresponding to each of the plurality of shooting points using such location information. In one embodiment, such location information may be relative distance information from a previous shooting point to a current shooting point, and a plurality of three-dimensional reference coordinates corresponding to each of the plurality of shooting points may be set by reflecting the relative distance information based on an initial starting indoor point. The computing device 300 may generate a point cloud in a three-dimensional space by reflecting a hue image and a depth image generated at each shooting point based on the three-dimensional reference coordinates for the plurality of shooting points. For example, the computing device 300 may generate point cloud data including a plurality of points set in a three-dimensional space by reflecting data of a hue image and data of a depth image generated at a plurality of shooting points onto the three-dimensional reference coordinates. The computing device 300 may set a plurality of vertices by selecting at least a portion of a plurality of points constituting the point cloud. It becomes possible to construct a 3D mesh model based on the multiple vertices selected in this manner.

[0063] That is, the 3D mesh model can be represented by a plurality of vertices represented as coordinates in a 3D space, and faces or polygons that are polygonal surfaces defined using such vertices. Here, each vertex can have its own 3D coordinates and can have its own hue data. That is, the hue of each vertex can be set based on the data of the hue image associated with each vertex. For example, a point has its own 3D coordinates and hue in the 3D space, reflecting the hue of the hue image and the distance in the depth image, based on the 3D reference coordinates. For example, the hue image and the depth image can be generated according to the same shooting standard and correspond to each other 1:1, and each pixel of the hue image can have a hue value, and each pixel of the depth image can have a depth value. The computing device 300 can form a point cloud by reflecting such hue values ​​and depth values ​​in the reference coordinates. The vertices are selected based on such a point cloud, and can have the same 3D coordinate values ​​and hue values.

[0064] In one embodiment, the computing device 300 divides the 3D point cloud into a predetermined unit space, sets one point that represents a plurality of points included in the unit space, and sets the vertices based on the one point. The computing device 300 divides the 3D point cloud into a predetermined unit space, e.g., 1 cm 3 - may be set as one point, and as an example, one point may be set based on the average hue value and average coordinate value of the plurality of points included in the predetermined unit space. As another example, one point may be set based on the median hue value and median coordinate value of the plurality of points included in the predetermined unit space. In such an example, the computing device 300 may normalize the same points, and then select vertices forming a predetermined surface based on such points to determine them as vertices.

[0065] The computing device 300 may perform texturing for each of a plurality of faces included in the 3D mesh model based on an association between the face and a color image (S203).

[0066] The computing device 300 can identify (S204) any hole faces for which no association was set in step S203 - which are shown as holes because no hue data value was selected, and are referred to as hole faces - and set the hue of the hole face based on the hues of multiple associated vertices associated with the hole face (S205).

[0067] FIG. 3 is a flow chart illustrating a texturing method for generating a three-dimensional virtual model according to another embodiment disclosed in the present application.

[0068] 3, a computing device 300 receives a plurality of data sets generated at each indoor point of a plurality of indoor spaces from an image capture device 100 (S301), where the data sets include color images and depth images captured from the indoor points and position information related to the indoor points.

[0069] The computing device 300 generates a 3D mesh model for generating a 3D model for the interior space based on the multiple data sets (S302).

[0070] A 3D mesh model can be generated by generating a set of points (e.g., a point cloud) for each indoor point based on the color and depth images, and placing these in 3D space based on position information.

[0071] The computing device 300 may generate a 3D mesh model by selecting a plurality of vertices based on the point cloud and setting a plurality of faces based on the selected plurality of vertices. As an example, the computing device 300 may set a face of a triangle based on three adjacent vertices.

[0072] Since no hue values ​​have been set for the faces in such a 3D mesh model, computing device 300 repeats steps S303 to S304 to set hue values ​​for each face, i.e., to perform texturing.

[0073] The computing device 300 can select one (a first) face from among a plurality of faces included in the 3D mesh model, and select one first hue image suitable for the first face from among a plurality of hue images associated with the first face (S303).

[0074] Here, when selecting a hue image associated with a first face, the computing device 300 can calculate a unit vector perpendicular to the first face, and based on this, select at least one hue image having a shooting angle corresponding to the unit vector as a hue image associated with the face. This is because when a hue image is photographed, information on the shooting angle of the hue image is also generated, so that a hue image associated with the first face, i.e., a hue image in which the first face is reflected, can be selected based on information on the shooting height and shooting angle for the hue image. For example, the computing device 300 can select a hue image having a unit vector perpendicular to the first face, i.e., a shooting angle facing the unit vector within a predetermined angle, as a hue image associated with the face.

[0075] The computing device 300 may select one of the hue images associated with a face that is suitable for the face. For example, the computing device 300 may calculate a plurality of weighting value factors for each of the associated hue images, calculate a weighting value based on the weighting value, and select one of the hue images based on the weighting value.

[0076] As an example, a first hue image to be matched with a first face may be selected from a plurality of hue images associated with a 3D mesh model by evaluating the shooting direction, resolution, and color noise relative to the first face as criteria.

[0077] The computing device 300 may select a local area corresponding to the first face from any one of the selected color images and map the local area onto the first face to perform texturing (S304).

[0078] Since the computing device 300 has information about the capture position of each hue image, it can project and map each object in each hue image to each object in the 3D mesh model, and can select a local region in the 2D hue image that corresponds to the face based on the projective mapping between the 2D hue image and the 3D mesh model.

[0079] The computing device 300 repeats the above steps S303 to S304 for all faces of the 3D mesh model to generate hue information for each face and perform texturing (S305). The 3D model generated in this way is in a state where hue correction between each hue image has not been performed, so there is a risk of unevenness occurring even on the same surface. This is because the shooting environment from each shooting point in the room is different, as described above.

[0080] The computing device 300 can perform color adjustment to correct for hue differences due to the shooting environment from each shooting point in the room (S306).

[0081] Although the embodiment shown in FIG. 2 and the embodiment shown in FIG. 3 are described separately from each other, they may be implemented in combination with each other, such as by substituting each step with each other.

[0082] 4 to 7 are diagrams for explaining an example of acquiring a hue image according to an embodiment disclosed in the present application, and a description will be given with reference to these figures.

[0083] Fig. 4 is a perspective view showing, as an example, a hexahedral subject in an indoor space and a first shooting position PP1 and a second shooting position PP2 in relation to the subject, and Fig. 5 is a plan view corresponding to Fig. 4. Fig. 6 shows an example of a hue image shot from the first shooting position PP1, and Fig. 7 shows an example of a hue image shot from the second shooting position PP2.

[0084] 6 and 7 are hue images taken of the same subject, but FIG. 7 shows an example in which a change in hue occurs due to shading.

[0085] Fig. 8 is a flow chart illustrating a method for selecting a hue image for generating a 3D virtual model according to an embodiment disclosed in the present application. The embodiment shown in Fig. 8 relates to an embodiment for determining the association between a face and a hue image performed in a computing device 300. Although three types of weighting values ​​for determining the association are disclosed in Fig. 8, it goes without saying that such three types of weighting values ​​can be used in various combinations depending on the embodiment.

[0086] The procedure diagram shown in Fig. 8 is related to a process of selecting a first hue image to be mapped with a first face from among a plurality of hue images associated with the first face. Figs. 9 to 12 are diagrams for explaining an example of selecting a hue image according to an embodiment disclosed in the present application, and will be further described with reference to these figures.

[0087] Referring to FIG. 8, the computing device 300 may set a reference vector for a first face of the three-dimensional mesh model, i.e., a first direction vector perpendicular to the first face (S801).

[0088] The computing device 300 may calculate a first weighting value element having a directional association with the first direction vector for each of the plurality of color images associated with the first face (S802).

[0089] The computing device 300 can determine the shooting directions of the multiple color images associated with the first face and calculate a first weighting factor based on a directional correlation between a first direction vector of the first face and the shooting direction, for example, the smaller the angle between the first direction vector of the first face and the shooting direction, the higher the weighting factor can be calculated.

[0090] The computing device 300 may calculate second weighting value components for each of the plurality of hue images associated with the first face (S803).

[0091] For example, the computing device 300 may check the resolution of the plurality of color images themselves and calculate the second weighting factor based on the resolution, i.e., the higher the resolution, the higher the second weighting factor.

[0092] As another example, the computing device 300 can identify an object to be textured or a face that is a part of the object, and calculate the second weighting factor based on the resolution of the identified object or face, where the resolution for such an object or face is set inversely proportional to the distance between the objects at the shooting location, so that a high second weighting factor is given to a hue image that is advantageous in terms of distance.

[0093] The computing device 300 may calculate a third weighting value component for color noise for each of the plurality of hue images associated with the first face (S804).

[0094] The computing device 300 can calculate color noise for each hue image. To calculate color noise, various methodologies can be applied, such as unsupervised learning using a deep convolutional generative adversarial network (DCGAN) and a method using Enlighten GAN.

[0095] The computing device 300 may be adapted such that less color noise is given a higher third weighting factor.

[0096] The computing device 300 may calculate a weighting value for each of the plurality of hue images by reflecting the first weighting value factor to the third weighting value factor, and may select the hue image having the highest weighting value as a first image to be mapped to the first face (S805).

[0097] A wide variety of algorithms can be applied to reflect the first to third weighting factors. For example, the computing device 300 can calculate the weighting value in various ways, such as simply adding up the first to third weighting factors or deriving an average thereof.

[0098] In the above example, the first weighting value element to the third weighting value element are all reflected, but the present invention is not limited to this. Therefore, it is possible to perform modifications such as calculating the weighting value based on the first weighting value element and the second weighting value element, or calculating the weighting value based on the first weighting value element and the third weighting value element. However, even in such modifications, including the first weighting value element is an element that provides even higher performance.

[0099] Fig. 9 shows an example of setting a first direction vector perpendicular to a first face Fc1 of a hexahedron. With reference to Fig. 9 and the example shown in Fig. 4, it can be seen that the first photographing point PP1 should have a higher first weighting value than the second photographing point PP2.

[0100] FIG. 10 illustrates a local area P1Fc1 corresponding to a first face in a hue image from a first photographing position PP1, and FIG. 11 illustrates a local area P2Fc1 corresponding to the first face in a hue image from a second photographing position PP2.

[0101] 10 and 11, it can be seen that the hue image from the first shooting point PP1 shown in FIG. 10 has a higher resolution than the hue image in FIG. 11, and therefore the second weighting value factor should be higher.

[0102] Color noise would be set higher in the hue image from the second shooting position PP2 shown in FIG. 11, and therefore the first shooting position PP1 shown in FIG. 10 would have a higher third weighting value factor.

[0103] Therefore, for the first face, the hue image from the first shooting position PP1 should be selected, and Figure 12 shows that texturing has been performed on the first face by matching a local area (P1Fc1) in the hue image from the first shooting position PP1 with the first face.

[0104] FIG. 13 is a flowchart illustrating a method for setting an unscened face for generating a three-dimensional virtual model according to an embodiment disclosed in the present application.

[0105] Although the color image mapping and texturing are performed for each face through the above process, in some cases, no image to be mapped may be selected for some faces. Such faces are usually called unscened faces, and occur in areas that cannot be photographed from the perspective of the image photographing angle.

[0106] Figure 14 shows a 3D model in which such unseen faces are displayed. The unseen faces are displayed in blue, and it can be seen that unseen faces occur behind the bedding of the bed, under the sink, and in areas hidden by the table.

[0107] The computing device 300 can perform color filling on such unscened faces as shown in FIG.

[0108] 13, the computing device 300 may set an unscened face resulting from an unphotographed area (S1301). The computing device 300 may set a face that does not have a color image mapped to the face as an unscened face.

[0109] The computing device 300 determines the hue value of each of a number of vertices associated with the unscene face (S1302).

[0110] For example, in a case where each face is a triangle having three vertices, the computing device 300 can ascertain the hue values ​​of the three vertices that make up the unscened face. As an example, the hue value of a vertex can be determined as a pixel value of a hue image that corresponds to a pixel of the depth image that makes up the vertex. That is, the computing device 300 can select a depth image used to derive position information to determine a vertex, and can select a hue image that constitutes the same data set as the depth image. The computing device 300 can select a vertex-associated depth pixel corresponding to the vertex from the depth image, and can select a vertex-associated hue pixel from the hue image that corresponds to the vertex-associated depth pixel selected from the depth image. The computing device 300 can set the hue value of the vertex-associated hue pixel as the hue value of the vertex.

[0111] The computing device 300 may fill the unscened face based on the hue values ​​of each of a plurality of vertices. For example, the computing device 300 may set each vertex as a starting point, and set the hue value of the unscened face as a gradient with the hue value of an adjacent vertex based on the hue value of each vertex (S1303). Here, the gradient refers to a method of changing a color set by a hue gradation, and various methods may be applied as such a gradation method.

[0112] Fig. 15 shows an example in which an unscened face is filled in for the example of Fig. 14, and in the example shown, it can be seen that the unscened face is filled in based on a hue gradation. Although the texturing of the unscened face by such a gradation may, in itself, produce a somewhat unnatural feeling with the surrounding hues, this can be compensated for more naturally by the hue correction process described below.

[0113] FIG. 16 is a flowchart illustrating a hue correction method for generating a 3D virtual model according to an embodiment disclosed in the present application.

[0114] As described above, since the shooting environment is different between indoor shooting points, the same surface may be displayed in different hues. In particular, when multiple faces are adjacent to each other to form a continuous surface or curved surface, the difference in hue value between each face gives an unnatural feeling.

[0115] Fig. 17 illustrates a portion of a three-dimensional model before hue correction is performed, and in the example of Fig. 17, it can be seen that there is considerable color unevenness on the same wall surface. The computing device 300 provides a process for compensating for such color distortion, i.e., color distortion caused by differences in the shooting environment between shooting points in a room, and will be described with reference to Fig. 16.

[0116] Referring to FIG. 16, the computing device 300 can perform correction between color images and correction between adjacent faces on the first three-dimensional model after texturing has been completed.

[0117] The computing device 300 may set image subsets by associating hue images taken from adjacent shooting positions (S1601). A plurality of such hue image subsets may be set. The computing device 300 may perform a global hue correction for each hue image subset based on a correction weighting value between the hue images associated with the hue image subset (S1602). Such a global hue correction is performed on the entire image.

[0118] As an example, the computing device 300 can determine a dominant hue for the hue images associated with the hue image subset. In the example of FIG. 17, the dominant hue can be gray, and a dominant hue weighting value can be set for the dominant hue gray for each hue image associated with FIG. 17. The dominant hue weighting value can be set from the difference from the average value for the dominant hue of the associated hue images. The more the dominant hue differs from the average value for the hue image, the higher the correction weighting value can be set, and the hue correction can be performed for each hue image based on the correction weighting value. Once the global hue correction has been performed, the correction has been performed on the hue images themselves, and texturing can be performed again based on such corrected images.

[0119] The computing device 300 can then perform local hue correction based on the differences between the faces.

[0120] The computing device 300 may associate adjacent faces to each other to set a plurality of face subsets (S1603).

[0121] The computing device 300 may perform local hue correction for each face subset by setting the face subset to equalize the hue difference between the faces that make up the face subset (S1604). Since various methods can be applied to equalize the hue difference, the method is not limited to any particular method.

[0122] Figure 18 shows an example where global hue correction and local hue correction have been applied to the example of Figure 17. It can be seen that the parts that had large hue differences in Figure 17 have been corrected to fairly similar hues.

[0123] In reality, users will find various hues on the same surface or curved surface to be quite unnatural, so this type of hue correction plays a major role in providing a realistic image of the actual 3D model.

[0124] In the present application, such hue correction is performed by combining global correction and local correction. Since the shooting locations are far apart and the shooting conditions vary greatly in the present invention, this combined hue correction allows the three-dimensional model to be expressed more naturally.

[0125] FIG. 19 is a flow chart illustrating a method for texturing an unscened region for generating a three-dimensional virtual model according to one embodiment disclosed in the present application.

[0126] 19, the computing device 300 may select a hole face that is shown as a hole because a suitable color image is not selected from among a plurality of faces of a 3D mesh model (S1901). Such a hole face is called an unseen hole face caused by an unseen area because it is located in a blind spot during shooting.

[0127] The computing device 300 may cull a number of associated vertices associated with the unscened hole face (S1902).

[0128] The computing device 300 may determine the hue of each of the selected plurality of associated vertices (S1903), and perform texturing by interpolating based on the hues of the identified plurality of associated vertices to set the hue of the unscened hole face (S1902).

[0129] FIG. 20 is a diagram illustrating an example of texturing an unscened area according to one embodiment disclosed in the present application, where (a) of FIG. 20 shows an example in which an unscened area has been created, and (b) of FIG. 20 shows an example in which such an unscened area has been textured.

[0130] Fig. 21 is a block diagram illustrating a computing device according to an embodiment disclosed in the present application. As shown in Fig. 21, a computing device 300 according to an embodiment of the present disclosure may include a communication module 310, a memory 320, and a processor 330. However, such a configuration is merely exemplary, and it goes without saying that in addition to such components, new components may be added or some components may be omitted when implementing this disclosure.

[0131] The communication module 310 includes a circuit and can communicate with an external device (including a server). Specifically, the processor 330 can receive various data or information from the external device connected via the communication module 310, and can transmit various data or information to the external device.

[0132] The communication module 310 may include at least one of a WiFi module, a Bluetooth® module, a wireless communication module, and a Near Field Communication (NFC) module, and may communicate according to a variety of communication standards, such as IEEE, Zigbee®, 3rd Generation (3G), 3rd Generation Partnership Project (3GPP®), Long Term Evolution (LTE), 5th Generation (5G), and the like.

[0133] The memory 320 may store at least one instruction related to the computing device 300. The memory 320 may store an operating system (O / S) for driving the computing device 300. The memory 320 may also store various software programs and applications for operating the computing device 300 according to various embodiments of the present disclosure. The memory 320 may include a semiconductor memory such as a flash memory, a magnetic storage medium such as a hard disk, or the like.

[0134] Specifically, the memory 320 may store various software modules for operating the computing device 300 according to various embodiments of the present disclosure, and the processor 330 may execute the various software modules stored in the memory 320 to control the operation of the computing device 300. That is, the memory 320 may be accessed by the processor 330, and data may be read / recorded / modified / deleted / updated by the processor 330.

[0135] In addition, a wide variety of other information may be stored in memory 320 as needed within the scope of achieving the objectives of this disclosure, and the information stored in memory 320 may be received from an external device or updated by input by the user.

[0136] The processor 330 may be comprised of one or more processors.

[0137] The processor 330 controls the overall operation of the computing device 300. Specifically, the processor 330 is connected to the configuration of the computing device 300 including the communication module 310 and the memory 320 as described above, and can control the overall operation of the computing device 300 by executing at least one instruction stored in the memory 320 as described above.

[0138] The processor 330 may be implemented in a variety of ways. For example, the processor 330 may be implemented by at least one of application specific integrated circuits (ASICs), embedded processors, microprocessors, hardware control logic, hardware finite state machines (FSMs), and digital signal processors (DSPs). Meanwhile, in this disclosure, the term processor 330 may be used to encompass a central processing unit (CPU), a graphic processing unit (GPU), a main processing unit (MPU), and the like.

[0139] Meanwhile, the control method performed in the computing device 300 according to the above-described embodiment may be realized by a program and provided to the computing device 300. For example, the program including the control method of the computing device 300 may be stored in a non-transitory computer readable medium and provided.

[0140] Although the above briefly describes the control method of the computing device 300 and the computer-readable recording medium including a program for executing the control method of the computing device 300, this is merely for the purpose of avoiding redundant description, and it goes without saying that the various embodiments of the electronic device 100 can also be applied to the control method of the computing device 300 and the computer-readable recording medium including a program for executing the control method of the computing device 300.

[0141] On the other hand, the machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" merely means that the storage medium is a tangible device and does not include a signal (e.g., electromagnetic waves), and this phrase does not distinguish between data being stored semi-permanently on the storage medium and data being stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0142] The present application described above is in no way limited by the above-mentioned embodiments and the attached drawings, but is limited by the scope of the claims, and it should be easily understood by a person having ordinary knowledge in the technical field to which the present application pertains that the configuration of the present application can be changed and modified in various ways within the scope not departing from the technical idea of ​​the present application.

[0143] [Acknowledgements]

[0144] The present invention was filed overseas with the support of the following research project supported by the Republic of Korea government.

[0145] Research Project Information Department name: Korea Tourism Organization Research project name: Follow-up support for global tourism leaders Project title: Smartphone-linked automatic human / object recognition and tracking photography device Organizer: Tree Eye Co., Ltd. Research period: 2022.03.04~2022.12.31 [Industrial Applicability]

[0146] The present invention has an effect of filling hole faces accurately with efficient resources by filling the hole faces using the hues of points in a point cloud, and therefore has high industrial applicability.

[0147] Furthermore, the present invention has the effect of being able to provide more accurate texturing even in a 3D generated environment based on images taken from multiple locations far apart indoors by effectively selecting an image suitable for the face of a 3D model, and therefore has high industrial applicability.

[0148] Furthermore, the present invention has the effect of accurately compensating for color unevenness caused by different shooting conditions between multiple different points in a room, minimizing the sense of foreignness on each surface of the virtual indoor space, and providing a texture of a virtual space that has a higher similarity to the actual space, thereby having high industrial applicability. [Explanation of symbols]

[0149] 100 Image acquisition device 300 Computing Equipment 310 Communication Module 320 Memory 330 Processor 500 user terminals

Claims

1. 1. A method operable in a computing device for generating a three-dimensional virtual model based on a plurality of data sets generated at a plurality of capture positions of an indoor space, the data sets including color images and depth images, the method comprising: generating a three-dimensional mesh model based on a plurality of data sets generated at a plurality of shooting positions of the indoor space; texturing each of a plurality of faces included in the three-dimensional mesh model based on an association between the face and a color image; identifying hole faces for which no associations are established, which are denoted as holes; identifying a plurality of associated vertices associated with the hole face and setting a hue of the hole face based on the hues of the identified plurality of associated vertices; Including, The association includes calculating a first weighting factor for directional association for each of a plurality of hue images associated with each face based on a directional vector for the face; calculating, for each of the plurality of hue images associated with each face, a second weighting factor for the resolution of the hue image; Calculating based on the first weighting factor and the second weighting factor; The first weighting value element is set higher as the angle between the first direction vector of the first face and the shooting direction is smaller. Texturing method for the generation of 3D virtual models.

2. The step of generating a three-dimensional mesh model includes: setting the plurality of shooting points as a plurality of three-dimensional reference coordinates; generating point cloud data including a plurality of points set in a three-dimensional space by reflecting the hue image data and the depth image data generated at the plurality of shooting locations on the three-dimensional reference coordinates; A step of selecting at least a portion of the plurality of points to set a plurality of vertices; 2. The texturing method for the generation of a three-dimensional virtual model according to claim 1, comprising:

3. the plurality of vertices constitute the three-dimensional mesh model; Each vertex is 3. The texturing method for generating a three-dimensional virtual model according to claim 2, including color data that is set based on data of a color image associated therewith.

4. The step of texturing each of the plurality of faces included in the three-dimensional mesh model includes: calculating a plurality of weighting factors for each face and a plurality of hue images associated therewith, and determining the association based on the calculated plurality of weighting factors to select one of the hue images; A step of mapping a local region corresponding to the face in any one of the selected hue images onto the face to perform texturing; 2. The texturing method for the generation of a three-dimensional virtual model according to claim 1, comprising:

5. The step of texturing each of the plurality of faces included in the three-dimensional mesh model includes: Selecting the color image with the highest correlation.

2. The texturing method for the generation of a three-dimensional virtual model according to claim 1, comprising:

6. The step of texturing each of the plurality of faces included in the three-dimensional mesh model includes: determining a third weighting factor based on color noise for each of the plurality of hue images associated with each face; calculating the association based on the first weighting factor through the third weighting factor; selecting one hue image having the highest correlation; 2. The texturing method for the generation of a three-dimensional virtual model according to claim 1, comprising:

7. identifying a plurality of associated vertices associated with the hole face and setting a hue of the hole face based on the hues of the identified plurality of associated vertices, setting a plurality of vertices corresponding to the vertices of the hole face as the plurality of associated vertices; determining the hue data of each of the plurality of associated vertices; determining the hue of the hole face by performing hue interpolation on the hue data of the plurality of associated vertices; 4. A texturing method for the generation of a three-dimensional virtual model according to claim 3, comprising:

8. 1. A computing device comprising: a memory for storing one or more instructions; at least one processor for executing the one or more instructions stored in the memory; Equipped with The at least one processor executes the one or more instructions to: Preparing a plurality of data sets generated at a plurality of shooting positions of an indoor space, the data sets including a hue image and a depth image; generating a three-dimensional mesh model based on a plurality of data sets generated at a plurality of shooting points of the indoor space; texturing each of a plurality of faces included in the three-dimensional mesh model based on an association between the face and a hue image; Identifying hole faces for which no associations are established, which are denoted as holes; identifying a plurality of associated vertices associated with the hole face; and setting a hue of the hole face based on the hues of the identified plurality of associated vertices; The association includes calculating a first weighting factor for directional association for each of a plurality of hue images associated with each face based on a directional vector for the face; calculating, for each of the plurality of hue images associated with each face, a second weighting factor for the resolution of the hue image; Calculating based on the first weighting factor and the second weighting factor; The first weighting value element is set higher as the angle between the first direction vector of the first face and the shooting direction is smaller. Computing device.

9. The computing device comprises: The plurality of photographing points are set as a plurality of three-dimensional reference coordinates; generating point cloud data including a plurality of points set in a three-dimensional space by reflecting the hue image data and the depth image data generated at the plurality of shooting locations on the three-dimensional reference coordinates; The computing device of claim 8 , further comprising: selecting at least a portion of the plurality of points to define a plurality of vertices.

10. the plurality of vertices constitute the three-dimensional mesh model; Each vertex is 10. The computing device of claim 9, further comprising hue data that is set based on data of a hue image associated therewith.

11. The computing device comprises: calculating a plurality of weighting factors for each of the plurality of hue images associated with each face, and determining the association based on the calculated plurality of weighting factors to select one of the hue images; The computing device according to claim 8 , further comprising: mapping a local region of one of the selected color images corresponding to the face onto the face to perform texturing.

12. The computing device comprises: Select the hue image with the highest correlation; 9. The computing device of claim 8.

13. The computing device comprises: determining a third weighting factor based on color noise for each of the plurality of hue images associated with each face; Calculating the association based on the first weighting factor through the third weighting factor; Select the hue image with the highest correlation; 9. The computing device of claim 8.

14. The computing device comprises: A plurality of vertices corresponding to the vertices of the hole face are set as the plurality of associated vertices; determining hue data of each of the plurality of associated vertices; The computing device of claim 10 , further comprising: hue interpolating the hue data of the plurality of associated vertices to set the hue of the hole face.

15. A storage medium having computer readable instructions stored thereon, The instructions, when executed by a computing device, cause the computing device to: preparing a plurality of data sets generated at a plurality of capture positions of an indoor space, the data sets including a color image and a depth image; generating a three-dimensional mesh model based on a plurality of data sets generated at a plurality of shooting positions of the indoor space; an operation of texturing each of a plurality of faces included in the three-dimensional mesh model based on an association between the face and a hue image; Identifying a hole face, which is denoted as a hole because the association is not established; identifying a plurality of associated vertices associated with the hole face and setting a hue of the hole face based on the hues of the identified plurality of associated vertices; To carry out the above. The association includes calculating a first weighting factor for directional association for each of a plurality of hue images associated with each face based on a directional vector for the face; calculating, for each of the plurality of hue images associated with each face, a second weighting factor for the resolution of the hue image; Calculating based on the first weighting factor and the second weighting factor; The first weighting value element is set to be higher as the angle between the first direction vector of the first face and the shooting direction is smaller. storage medium.

Images