Image processing apparatus and program

Abstract

To provide an image processing device and a program capable of easily creating high-quality 3DCG information using already-created low-quality 3DCG information.SOLUTION: The image processing device 1 includes a parameter assigning unit 17 that assigns a parameter to an object in a low-quality 3DCG including the object, and a 3DCG data-outputting unit 18 that outputs a high-quality 3DCG including the object to which the parameter is assigned by the parameter assigning unit 17.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and a program. [Background technology]

[0002] Traditionally, catalogs and brochures used to sell automobiles, homes, and other products often contain computer-generated images instead of photographs of the actual items being sold. Nowadays, 3DCG (3-Dimensional Computer Graphics) can be easily created using software. However, the 3DCG images generated using this software tend to be of low quality. On the other hand, a fitting simulation system has been disclosed that generates a three-dimensional shape model of a room, generates a fitting model that is a three-dimensional model of fittings, and changes the color and pattern combination data of at least the fittings, the walls, floors, and baseboards of the room (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-43647 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 generates three-dimensional models of interiors and fixtures, and adjusts the colors and patterns of fixtures placed three-dimensionally within the room. However, creating high-quality 3DCG images of interiors and fixtures from scratch is time-consuming. Therefore, there is a demand for technology that allows for the easy creation of high-quality 3DCG.

[0005] Therefore, an object of the present invention is to provide an image processing device and a program that can easily create high-quality 3DCG data using previously created low-quality 3DCG data. [Means for solving the problem]

[0006] The present invention solves the above problems by the following means. A first invention is an image processing device comprising a parameter assignment means for assigning parameters to an object based on tag information assigned to the object in a three-dimensional spatial image including the object, and an image output means for outputting the three-dimensional spatial image including the object to which the parameters have been assigned by the parameter assignment means. A second invention is an image processing device according to the first invention, comprising a tag estimation means for estimating tag information relating to at least an object and a material based on data relating to the object in the three-dimensional spatial image, and a tag assignment means for assigning the tag information estimated by the tag estimation means to the object. A third invention is an image processing device according to the second invention, wherein the tag estimation means estimates the tag information relating to the object in the three-dimensional spatial image using a learning model that has learned the correspondence between data relating to the object and the tag information. A fourth invention is an image processing device according to any one of the first to third inventions, further comprising a light source identification means for identifying the object that will be a light source based on the tag information assigned to the object, and the parameter assignment means assigns the parameter related to the brightness of light to each object based on orientation information previously stored in the data relating to the three-dimensional spatial image, the light source object identified by the light source identification means, and information regarding the time of day. A fifth invention is an image processing device that is any of the image processing devices of the first to fourth inventions, and includes an object storage unit that stores high-resolution objects, an object extraction means that extracts the high-resolution objects stored in the object storage unit based on the tag information assigned to the objects in the three-dimensional spatial image, and an object modification means that modifies the objects in the three-dimensional spatial image to the high-resolution objects extracted by the object extraction means. A sixth aspect of the present invention is a program for causing a computer to function as any one of the image processing devices according to the first to fifth aspects of the present invention. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device and a program that can easily create high-quality 3DCG data using previously created low-quality 3DCG data. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a schematic diagram of an overall image processing system according to an embodiment of the present invention and a functional block diagram of an image processing apparatus; [Figure 2] FIG. 2 is a diagram illustrating an example of a storage unit of the image processing apparatus according to the present embodiment. [Figure 3] 5 is a flowchart showing image processing of the image processing apparatus according to the present embodiment. [Figure 4] 10A to 10C are diagrams illustrating a specific example of image processing by the image processing device according to the present embodiment. [Figure 5] 10 is a flowchart showing object processing in the image processing device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, this is merely an example, and the technical scope of the present invention is not limited to this example. (Embodiment) <Image processing system 100> FIG. 1 is a schematic diagram of an entire image processing system 100 according to this embodiment and a functional block diagram of an image processing device 1. As shown in FIG. FIG. 2 is a diagram illustrating an example of the storage unit 20 of the image processing device 1 according to this embodiment.

[0010] The image processing system 100 shown in FIG. 1 is a system including an image processing device 1 and a terminal 4. In the image processing system 100, the terminal 4 transmits 3DCG data created using software for creating simple 3DCG (three-dimensional spatial images) (hereinafter, this software will also be referred to as "simple CG generation software") to the image processing device 1. The image processing device 1 then estimates tag data (tag information) of each object included in the three-dimensional spatial image represented by the 3DCG data. The image processing device 1 then converts the objects from low-quality 3DCG to high-quality 3DCG based on the estimated tag data. The image processing device 1 then outputs the 3DCG data of the objects with high quality to the terminal 4.

[0011] The image processing device 1 and the terminal 4 are communicatively connected via a communication network N. The communication network N is, for example, an Internet line or the like, and may be wired or wireless. Although only one terminal 4 is shown in FIG. 1, the image processing device 1 may be communicatively connected to each of a plurality of terminals 4.

[0012] In the following, the image processing device 1 will be described as performing processing using 3DCG data relating to the interior of a model house, for example. However, this is just one example, and 3DCG data relating to the interior of an automobile, for example, may also be used. The image processing device 1 will be described as performing image processing on 3DCG data created using simple CG generation software. Here, the 3DCG data created using simple CG generation software will be referred to as low-quality 3DCG data in the following description.

[0013] <Image processing device 1> The image processing device 1 is a device that performs image processing on low-quality 3DCG data received from the terminal 4 to convert it into high-quality 3DCG data. The image processing device 1 is, for example, a server, or may alternatively be a personal computer (PC) or the like. There is no limit to the number of pieces of hardware that make up the image processing device 1. The image processing device 1 may be configured with one or more pieces of hardware as needed. The image processing device 1 may also be, for example, a cloud.

[0014] The image processing device 1 includes a control unit 10, a storage unit 20, and a communication interface unit 29. The control unit 10 is a central processing unit (CPU) that controls the entire image processing device 1. The control unit 10 appropriately reads and executes an operating system (OS) and application programs stored in the storage unit 20, thereby cooperating with the above-mentioned hardware and executing various functions.

[0015] The control unit 10 includes a processing reception unit 11, a tag estimation unit 12 (tag estimation means), a tag assignment unit 13 (tag assignment means), an object extraction unit 14 (object extraction means), an object modification unit 15 (object modification means), a light source identification unit 16 (light source identification means), a parameter assignment unit 17 (parameter assignment means), and a 3DCG data output unit 18 (image output means).

[0016] The processing reception unit 11 receives low-quality 3DCG data from the terminal 4. In addition to the low-quality 3DCG data, the processing reception unit 11 may also receive from the terminal 4 a designation of a time period to be used when converting the 3DCG data. The time period may be a wide range of time periods such as morning, noon, evening, or night, or may be a time such as 10:00 AM.

[0017] The tag estimation unit 12 estimates tag data related to at least an object and its material based on data related to the object included in the low-quality 3DCG data. Here, the data related to the object refers to information such as color, 3D shape, material, and size. Here, material refers to not only the pattern or design of the object but also texture expressions such as unevenness and reflection. The tag estimation unit 12 can estimate tag data related to the object included in the low-quality 3DCG data, for example, using a learning model stored in the learning model storage unit 22. The learning model stored in the learning model storage unit 22 is, for example, a trained model that has learned the correspondence between data related to the object and tag data.

[0018] The tag assigning unit 13 assigns to the object the tag data estimated by the tag estimation unit 12. The tag assigning unit 13 may store the estimated tag data in the estimated tag storage unit 25, for example, in association with an object ID (IDentification) that identifies the object.

[0019] When the object storage unit 23 contains an object based on the tag data assigned by the tag assignment unit 13, the object extraction unit 14 extracts the object stored in the object storage unit 23. The object stored in the object storage unit 23 is a high-resolution object (part of an image). The object change unit 15 changes the object corresponding to the tag data assigned by the tag assignment unit 13 to the object extracted by the object extraction unit 14 .

[0020] The light source identification unit 16 identifies an object that serves as a light source based on the tag data added by the tag addition unit 13. Examples of objects that serve as a light source include windows made of glass and lighting fixtures. The parameter assigning unit 17 assigns a parameter to the object based on the tag data assigned by the tag assigning unit 13. The parameters assigned here are, for example, for forming a shadow on the object. More specifically, the parameter assigning unit 17 assigns parameters related to the brightness of light to the object based on orientation information previously stored in the low-quality 3DCG data, the object that will become the light source identified by the light source identifying unit 16, information about the time of day, etc. Here, the orientation information includes information about the direction, which can be expressed as north, south, east, and west, as well as height information about the light source. Furthermore, the parameter assigning unit 17 assigns parameters relating to texture expression to the object based on, for example, tag data relating to the material that expresses the characteristics of the material feel.

[0021] The 3DCG data output unit 18 outputs 3DCG data including objects to which parameters have been assigned by the parameter assignment unit 17, objects modified by the object modification unit 15, etc. to the terminal 4. The 3DCG output to the terminal 4 is of high quality, including objects to which parameters have been assigned and modified objects.

[0022] The storage unit 20 is a storage area such as a hard disk or semiconductor memory element for storing programs, data, etc. required for the control unit 10 to execute various processes. The storage unit 20 includes a program storage unit 21, a learning model storage unit 22, an object storage unit 23, and an estimated tag storage unit 25. The program storage unit 21 is a storage area that stores various programs. The program storage unit 21 stores an image processing program 21a. The image processing program 21a is a program for performing various functions executed by the control unit 10 of the image processing device 1. The learning model storage unit 22 is a storage area that stores the learning model and the like used to estimate tag data from data related to an object.

[0023] The object storage unit 23 is a storage area for storing high-resolution objects. Fig. 2(A) shows an example of the object storage unit 23. The object storage unit 23 shown in FIG. 2(A) stores tag data and objects in association with each other. The tag data is various information including information about the object, such as its substance and material. The object is the high-resolution object (part of the image) itself.

[0024] The estimated tag storage unit 25 is a storage area that stores tag data that has been assigned to an object by the tag assigning unit 13. FIG. The estimated tag storage unit 25 shown in FIG. 2(B) stores an object ID, tag data, an object position, etc. in association with each other. The object ID is identification information that identifies an object. The object ID may be unique identification information sequentially assigned by the tag estimation unit 12 based on data related to the object included in the low-quality 3DCG data when estimating tag data. The tag data is information about the object estimated by the tag estimation unit 12. The object position is the position information of an object in 3DCG.

[0025] The communication interface unit 29 in FIG. 1 is an interface for performing communication with the terminal 4 and the like via the communication network N. Here, a computer refers to an information processing device that includes a control unit, a storage device, etc., and the image processing device 1 is an information processing device that includes a control unit, a storage device, etc., and is included in the concept of a computer.

[0026] <Terminal 4> The terminal 4 is, for example, a desktop terminal such as a PC as shown in Fig. 1. Alternatively, the terminal 4 may be a portable terminal that also has the functions of a computer, such as a tablet. The terminal 4 may be, for example, a device in which simple CG generation software is installed in a storage unit and which creates low-quality 3DCG data using the simple CG generation software. Although not shown, the terminal 4 includes at least a control unit, a storage unit, an input unit, an output unit, a communication interface unit, and the like.

[0027] <Processing of image processing device 1> Next, the processing performed by the image processing device 1 will be described with reference to a flowchart. FIG. 3 is a flowchart showing image processing by the image processing device 1 according to this embodiment. FIG. 4 is a diagram showing a specific example of image processing by the image processing device 1 according to this embodiment. FIG. 5 is a flowchart showing object processing by the image processing device 1 according to this embodiment.

[0028] The control unit of terminal 4 transmits the low-quality 3DCG data stored on terminal 4 to the image processing device 1, and in step S (hereinafter, "step S" will be simply referred to as "S") 11 of Figure 3, the control unit 10 (processing reception unit 11) of the image processing device 1 receives the low-quality 3DCG data from terminal 4. 4(A) shows a 3D image 50 represented by low-quality 3DCG data received from terminal 4. 3D image 50 is image data of a home, but because it was created using simple CG generation software, the image itself is of low quality. A low-quality image is, for example, an image that does not express the texture of each object.

[0029] In S12, the control unit 10 extracts each object included in the low-quality 3DCG data. Then, the control unit 10 assigns an object ID to the extracted object and stores it in the estimated tag storage unit 25 together with object position information and the like. In S13, the control unit 10 (tag estimation unit 12) estimates tag data of each object using, for example, a learning model stored in the learning model storage unit 22. In S14, the control unit 10 (tag assignment unit 13) assigns the estimated tag data to the corresponding object. More specifically, the control unit 10 stores the estimated tag data in the estimated tag storage unit 25 in association with the corresponding object ID.

[0030] Here, the estimation of tag data will be described using the 3D image 50 in FIG. 4(A) as an example. 3D image 50 includes a number of objects, including objects 51 through 56 . The object 51 is assigned estimated tag data such as "chair," "backrest," "cloth," etc. The object 52 is assigned estimated tag data such as "floor", "wood", etc. For the object 53, tag data such as "window," "glass," "transparent," etc. are estimated. Tag data can be estimated similarly for objects 54 to 56 and other objects.

[0031] In S15 of FIG. 3, the control unit 10 performs object processing. Here, the object processing will be described with reference to FIG. In S21 of FIG. 5, the control unit 10 uses the tag data stored in the estimated tag storage unit 25 to refer to the object storage unit 23.

[0032] In S22, the control unit 10 determines whether or not there is a changeable object in the object storage unit 23. For example, if tag data corresponding to an object in object storage unit 23 completely matches tag data of an object in estimated tag storage unit 25, control unit 10 may determine that a changeable object is in object storage unit 23. Also, if tag data corresponding to an object in object storage unit 23 partially matches tag data of an object in estimated tag storage unit 25, control unit 10 may determine that a changeable object is in object storage unit 23, provided that multiple pieces of tag data match, including tag data representing a "thing" (object) and tag data representing a color. If the object storage unit 23 contains a modifiable object (S22: YES), the control unit 10 proceeds to S23. On the other hand, if the object storage unit 23 does not contain a modifiable object (S22: NO), the control unit 10 proceeds to S24.

[0033] In S23, the control unit 10 (object extraction unit 14) extracts an object corresponding to a changeable object from the object storage unit 23, and changes the changeable object to the extracted object. In S24, the control unit 10 (parameter assigning unit 17) assigns parameters based on the tag data to each object. By this parameter assigning process, textures and the like are expressed in the objects.

[0034] In S25, the control unit 10 (light source identification unit 16) identifies an object that will become a light source based on the tag data. In S26, the control unit 10 (parameter assigning unit 17) assigns parameters to each object based on the object that will become the light source identified in the process of S25. Here, when assigning the parameters, the control unit 10 uses the position and light intensity of the object that will become the light source, as well as orientation data and data related to the time period that are previously included in the low-quality 3DCG data. For example, by using the orientation data and time period data, the control unit 10 can assign parameters taking into account whether the sun is shining through a window. Thereafter, the control unit 10 moves the process to S16 in FIG.

[0035] 3, the control unit 10 (3DCG data output unit 18) outputs high-quality 3DCG data including objects to which parameters have been assigned and changed objects to the terminal 4. This allows the terminal 4 to display a 3D image 60 as exemplified in FIG. 4(B).

[0036] Fig. 4(B) is a 3D image 60 represented by high-quality 3DCG data output to the terminal 4. The 3D image 60 is a high-quality image of the 3D image 50 shown in Fig. 4(A). The 3D image 60 includes objects 61 to 66 corresponding to the objects 51 to 56 in FIG. 4(A), and an object 67 as a light source. For example, the texture of the chair seat of object 61 has become more realistic because parameters based on tag data have been assigned to it. Also, for example, the floor of object 62 has become more clearly visible as being made of wood because parameters based on tag data have been assigned to it. Furthermore, for example, the window of object 63 has become more clearly visible as being made of glass because parameters based on tag data have been assigned to it. Also, for example, object 65 has been changed from object 55 (see FIG. 4(A)) to a higher resolution object.

[0037] The object 67 is outside the window based on the orientation information and time zone information contained in the low-quality 3DCG data. The object 67 is shown in a visualized state in Fig. 4(B), but in reality, the object itself does not need to be displayed. Objects 68a to 68c are created from object 67. Objects 68a to 68c are assigned parameters based on object 67, and all of them form shadow areas.

[0038] As described above, the image processing device 1 of this embodiment has the following advantages. (1) Parameters based on tag data assigned to each object included in the low-quality 3DCG data are assigned to the object, and high-quality 3DCG data including the object to which the parameters have been assigned is transmitted to terminal 4. Therefore, the terminal 4 can output high-quality 3DCG data including objects to which parameters have been assigned, and since the material of each object is assigned parameters based on the tag data assigned to the object, it is possible to display high-quality 3DCG with a more realistic feel.

[0039] (2) Estimate tag data relating to the object or material based on the data relating to each object, and assign the estimated tag data to the object. Therefore, when low-quality 3DCG data is input, tag data related to the object can be estimated and automatically tagged, thereby reducing the workload of the worker. (3) The tag data is estimated using, for example, a learning model that has learned the correspondence between data related to the object and the tag data. Therefore, using the learning model, tag data can be automatically estimated in accordance with the learning results.

[0040] (4) The object that will be the light source is estimated based on the tag data of the object, and parameters related to the brightness of the light are assigned to each object based on the direction information and time information contained in the 3DCG data. This results in objects that are given brightness and darkness of the light, making for a more realistic image.

[0041] (5) Based on the tag data assigned to the object, the object is extracted from the object storage unit 23 that stores high-resolution objects, and the object is changed to the extracted object. Therefore, for example, if the object is the same, it can be replaced with a high-quality image.

[0042] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments. Note that the above-described embodiments and the modified embodiments described below can be used in appropriate combinations, but detailed description thereof will be omitted.

[0043] (Variations) (1) In this embodiment, tag data is estimated from object data and assigned to each object included in low-quality 3DCG data, but this is not limiting. If tag data has already been assigned to an object, this process may be omitted.

[0044] (2) In this embodiment, an image of a time period when sunlight shines into a room has been described as an example, but the present invention is not limited to this. For example, when the time period is night, an object representing a curtain is changed to a high-resolution object representing the curtain in a closed state, parameters are assigned to an object representing a light so that a light on the ceiling is illuminated, and parameters are assigned to other objects taking into account the illumination of the light on the ceiling. By processing each object in this way using an image processing device, 3DCG images of different time periods can be easily created. Furthermore, for example, a season may be specified from the terminal, and parameters may be assigned to the object taking into further consideration the intensity and angle of sunlight during the day, etc. The season may be specified as one of the four seasons, spring, summer, autumn, or winter, or as a month, such as April or May.

[0045] (3) In the present embodiment, the terminal and the image processing device are described as separate devices, but this is not limiting. For example, the image processing device may be a standalone image processing device equipped with an input unit, an output unit, etc. [Explanation of symbols]

[0046] 1. Image processing device 4. Terminal 10 Control Unit 11 Processing Reception Department 12 Tag estimation part 13 Tagging section 14 Object Extraction 15 Object Modification Section 16 Light source identification section 17 Parameter assignment section 18 3DCG data output section 20 Memory section 21a Image processing program 22 Learning model memory unit 23 Object Storage 25 Estimated tag memory 50,60 3D images 51~56,61~67 Objects 100 Image Processing System

Claims

1. a parameter assigning means for assigning a parameter relating to texture expression to an object based on tag information relating to at least one of a substance and a material assigned to the object in a three-dimensional spatial image including the object created by CG generation software; an image output means for outputting a processed three-dimensional spatial image including the object to which the parameters have been assigned by the parameter assigning means; An image processing device comprising:

2. 2. The image processing device according to claim 1, a tag estimation means for estimating the tag information based on data relating to the object in the three-dimensional space image, the data being at least information relating to the color, 3D shape, material, and size of the object; a tag assigning means for assigning the tag information estimated by the tag estimation means to the object; An image processing device comprising:

3. 3. The image processing device according to claim 2, The tag estimation means estimates the tag information related to the object in the three-dimensional spatial image using a learning model that has learned the correspondence between data related to the object and the tag information.

4. 4. The image processing device according to claim 1, an object storage unit that stores high-resolution objects; an object extraction means for extracting the high-resolution object stored in the object storage unit based on the tag information assigned to the object in the three-dimensional space image; an object modification means for modifying the object in the three-dimensional space image to the high-resolution object extracted by the object extraction means; An image processing device comprising:

5. A program for causing a computer to function as the image processing device according to any one of claims 1 to 4.

Images