Information processing system, information processing method, and program

Abstract

To provide an information processing system, an information processing method, and a program capable of expanding a mode of image conversion.SOLUTION: Original image acquisition means (110) acquires an original image (OI) that indicates an object (O). Virtual image generation means (120) generates a virtual image (VI) that indicates a state of a virtual space (VS) where a 3D model (M) expressing the object (O) is positioned as seen from a virtual view point (VV) on the bassi of the 3D model (M), illumination data (L), and texture data (T). Feature extraction means (130) extracts an original image feature value (OF) from the original image (OI), and extracts a virtual image feature value (VF) from the virtual image (VI). Parameter adjustment means (140) adjusts a parameter of the illumination data (L) and a parameter of the texture data (T) so that similarity between the original image feature value (OF) and the virtual image feature value (VF) increases.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present disclosure relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] There is known a technique for performing image conversion on an original image showing an object to obtain an image showing the object that is different from the original image. For example, a large number of various images can be obtained by performing operations such as rotation, enlargement / reduction, change in brightness, and change in color tone on the original image. The large number of images obtained in this way are used, for example, as learning data for a machine learning model for image recognition (see Patent Document 1 below). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Re-tabled publication 2021 / 038678 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned conventional techniques, the manner of image transformation is limited. For example, it is difficult to change the orientation, pose, shape, or arrangement of an object shown in an original image using the above-mentioned conventional techniques.

[0005] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an information processing system, an information processing method, and a program that expand the manner of image conversion. [Means for solving the problem]

[0006] The information processing system according to the present disclosure includes an original image acquisition means for acquiring an original image showing an object, a virtual image generation means for generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as seen from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data, a feature extraction means for extracting original image features from the original image and extracting virtual image features from the virtual image, and a parameter adjustment means for adjusting parameters of the lighting data and parameters of the texture data so as to increase the similarity between the original image features and the virtual image features. [Brief description of the drawings]

[0007] [Figure 1] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing system according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an overview of processing in the information processing system according to the present embodiment. [Diagram 3] FIG. 2 is a functional block diagram showing an example of functions realized in the information processing system of the present embodiment. [Figure 4] FIG. 13 is a diagram showing how a virtual image is generated. [Diagram 5] FIG. 13 is a diagram showing how virtual images viewed from each of a plurality of virtual viewpoints are generated. [Figure 6] FIG. 2 is a flow chart showing an example of processing executed in the information processing system according to the present embodiment. [Figure 7] FIG. 2 is a flow chart showing an example of processing executed in the information processing system according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [1. Hardware configuration of the information processing system according to this embodiment] An example of an embodiment of an information processing system according to the present disclosure will be described below. Fig. 1 is a diagram showing an example of a hardware configuration of an information processing system 1 according to the present embodiment. As shown in Fig. 1, the information processing system 1 is a computer such as a server computer or a personal computer, and includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, and a display unit 15. Although Fig. 1 shows a case where the information processing system 1 is configured by one computer, the information processing system 1 may be configured by multiple computers.

[0009] The control unit 11 includes at least one processor. The storage unit 12 includes a volatile memory such as a RAM and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a keyboard, a mouse, or a touch panel. The display unit 15 is a display such as a liquid crystal display or an organic EL display.

[0010] The program stored in the storage unit 12 may be supplied via the network N. Also, the program stored in a computer-readable information storage medium may be supplied via a reading unit (e.g., an optical disk drive or a memory card slot) that reads the information storage medium, or an input / output unit (e.g., a USB port) that inputs and outputs data to and from an external device.

[0011] [2. Overview of the Information Processing System of the Present Embodiment] An overview of the information processing system 1 of this embodiment will be described below. Fig. 2 is a diagram showing an overview of the processing of the information processing system 1 of this embodiment.

[0012] As shown in Fig. 2, the information processing system 1 of this embodiment adjusts parameters of illumination data L and texture data T so as to increase the similarity between the appearance of the object O shown in the original image OI and the appearance of the object O shown in the virtual image VI generated based on the 3D model M, illumination data L, and texture data T representing the object O. Specifically, the similarity between the appearance of the object O shown in the original image OI and the appearance of the object O shown in the virtual image VI is evaluated by evaluating the similarity between the original image feature OF extracted from the original image OI and the virtual image feature VF extracted from the virtual image VI. In this specification, the "appearance of the object O" refers to the appearance of the object O (e.g., color, texture, etc.) other than the orientation, pose, shape, and arrangement of the object O.

[0013] According to the information processing system 1 of this embodiment, the lighting data L and texture data T reflecting the appearance of the object O shown in the original image OI can be obtained by adjusting the parameters of the lighting data L and texture data T described above. Then, by using the lighting data L and texture data T obtained in this way to render the 3D model M under various conditions, an image obtained by converting the original image OI in various ways can be obtained. In this specification, the term "conversion" is used in a broad sense. That is, "converting" the original image OI means obtaining an image showing the object O that is different from the original image OI (hereinafter, referred to as a "converted image") based on the original image OI showing the object O, and does not only mean performing some operation on the original image OI and obtaining a converted image directly from the original image OI. Hereinafter, the information processing system 1 of this embodiment will be described in detail.

[0014] [3. Functions realized in the information processing system of this embodiment] Hereinafter, functions realized by the information processing system 1 of the present embodiment shown in FIG. 3 will be described with reference to FIG.

[0015] 3 is a functional block diagram showing an example of functions realized by the information processing system 1 of this embodiment. As shown in FIG. 3, in this embodiment, the original image storage unit 100, the 3D model storage unit 101, the lighting data storage unit 102, the texture data storage unit 103, the feature extractor storage unit 104, the original image acquisition unit 110, the virtual image generation unit 120, the feature extraction unit 130, the parameter adjustment unit 140, and the converted image generation unit 150 are realized by the information processing system 1. The original image storage unit 100, the 3D model storage unit 101, the lighting data storage unit 102, the texture data storage unit 103, and the feature extractor storage unit 104 are realized mainly by the storage unit 12. The original image acquisition unit 110, the virtual image generation unit 120, the feature extraction unit 130, the parameter adjustment unit 140, and the converted image generation unit 150 are realized mainly by the control unit 11.

[0016] [Original image storage section] The original image storage unit 100 stores the original image OI.

[0017] As shown in FIG. 2, an original image OI is an image showing an object O.

[0018] The object O is an object having a certain shape and size. In this embodiment, the object O is a dog, for example. The object O may be a living thing such as a person, an animal, or a plant, or may be an inanimate object such as a stone, a mountain, or a building.

[0019] The original image OI shows a part or the whole of the object O. In this embodiment, the whole of the object O is shown in the original image OI, but the original image OI may show only a part of the object O. Also, in this embodiment, the original image OI shows one object O, but the original image OI may show multiple objects O. When the original image OI shows multiple objects O, the objects O may be of the same type or different types. Explaining based on the example of this embodiment, the original image OI may show birds, trees, mountains, etc. in addition to dogs.

[0020] In this embodiment, the original image OI is a photographed image of an object O arranged in real space. This allows a converted image showing an object O of a form that does not exist in reality to be obtained from a photographed image of the object O arranged in real space. As an example, by using the information processing system 1 of this embodiment to convert a photographed image of a new screw arranged in real space, a converted image showing the screw of a form that does not exist in reality (an image showing a rusty screw, an image showing a bent screw, an image showing a screw with a crushed drive unit, etc.) can be obtained. The converted images showing various forms of screws obtained in this way can be used, for example, as learning data for a machine learning model for detecting defective products. Note that the original image OI is not limited to a photographed image of an object O arranged in real space, and may be, for example, a hand-drawn painting or a rendering image of a 3D model.

[0021] [Original image acquisition section] The original image acquisition unit 110 acquires the original image OI. In this embodiment, the original image OI is stored in the original image storage unit 100, so the original image acquisition unit 110 acquires the original image OI from the original image storage unit 100. Note that the original image OI may be stored in an external information storage medium or a computer. In this case, the original image acquisition unit 110 may acquire the original image OI from the external information storage medium or the computer.

[0022] [3D model storage section] The 3D model storage unit 101 stores data regarding the 3D model M.

[0023] As shown in FIG. 2, the 3D model M is a model representing an object O. The 3D model M is composed of at least one polygon representing the three-dimensional shape of the object O. In this embodiment, the 3D model M corresponds to mesh data representing the three-dimensional shape of the object O, and corresponds to model data not including texture data T described later. In this embodiment, the 3D model M is a solid model, but the 3D model M may be a wireframe model or a surface model. In this embodiment, the 3D model M is created in advance by any 3DCG software such as 3D-CAD. The data about the 3D model M indicates the three-dimensional coordinates of each vertex that defines the polygon of the 3D model M. In this embodiment, the data about the 3D model M is stored in the 3D model storage unit 101, but the data about the 3D model M may be stored in an external information storage medium or computer.

[0024] The number of 3D models M is the same as the number of objects O indicated by the original image OI. In this embodiment, since the original image OI indicates one object O, the number of 3D models M is also one.

[0025] Incidentally, "representing object O" means representing the three-dimensional shape of an object having the same concept as object O shown in the original image OI, and does not necessarily mean representing the three-dimensional shape of object O itself shown in the original image OI. For example, if object O shown in the original image OI is an individual beagle dog named "John", it is sufficient for 3D model M to represent the three-dimensional shape of a beagle dog that is an object having the same concept as object O shown in the original image OI, and it is not necessary for 3D model M to represent the three-dimensional shape of the individual beagle dog named "John" itself.

[0026] It is preferable that the 3D model M represents the three-dimensional shape of the object O itself shown in the original image OI. This makes it possible to obtain a converted image in which the orientation, pose, or position of the object O shown in the original image OI is changed while maintaining the appearance and shape of the object O shown in the original image OI with high accuracy.

[0027] [Lighting data storage unit] The illumination data storage unit 102 stores the illumination data L.

[0028] The lighting data L is data on a light source that irradiates the 3D model M. Specifically, the lighting data L includes, for example, the intensity and color of the light irradiated from the light source, as well as the position and orientation of the light source in the virtual space VS described later, as parameters. Note that the parameters included in the lighting data L are not limited to these, and the lighting data L may include parameters other than these, or may not include these parameters. Any light source, such as a point light source, a line light source, a surface light source, or a volume light source, can be used as the light source. In addition, when image-based lighting is used, an image capturing light information in all directions in the real world may be used as the light source.

[0029] [Texture data storage section] The texture data storage unit 103 stores the texture data T.

[0030] The texture data T is data about a texture set on the surface of the 3D model M. The texture is also called texture, and includes, for example, the color, glossiness, transparency, metallic feel, unevenness, etc. of the object O. Specifically, the texture data T is image data indicating the texture set on the surface of the 3D model M. In this embodiment, the texture data T includes, for example, the diffuse reflectance (albedo), normal vector (normal), specular reflectance, metallic, roughness, gloss, anisotropy, transparency, etc. of the surface of the object O as parameters. Note that the parameters included in the texture data T are not limited to these, and the texture data T may include parameters other than these, or may not include these parameters.

[0031] [Virtual image generation section] Hereinafter, the processing of the virtual image generating unit 120 will be described with reference to Fig. 4. Fig. 4 is a diagram showing how a virtual image is generated.

[0032] As shown in FIG. 2 and FIG. 4, the virtual image generating unit 120 generates a virtual image VI showing the state of the virtual space VS in which the 3D model M is placed, as viewed from the virtual viewpoint VV, based on the 3D model M, the lighting data L, and the texture data T. In other words, the virtual image generating unit 120 can perform a rendering process of the 3D model M with the lighting data L, the texture data T, and the virtual viewpoint VV as rendering conditions. In other words, the virtual image generating unit 120 can perform physically based rendering based on the 3D model M, the lighting data L, and the texture data T to generate a rendering image. In other words, the virtual image generating unit 120 can generate a new viewpoint image related to the object O indicated by the original image OI. The rendering conditions may include, in addition to the above, for example, a background image of the virtual space VS, and a resolution and aspect ratio of the virtual image VI to be generated. Various known rendering methods (rendering pipelines), such as forward rendering and deferred rendering, can be used to generate the virtual image VI.

[0033] The virtual space VS is a virtual three-dimensional space. Three mutually orthogonal coordinate axes are set in the virtual space VS. These three coordinate axes are coordinate axes of a world coordinate system. The origin may be any position, and positions in the virtual space VS are expressed by three-dimensional coordinates. Note that the virtual image generating unit 120 may set a background image in the virtual space VS. The background image is, for example, an image showing scenery such as trees, mountains, and the sky.

[0034] The virtual viewpoint VV is also called a virtual camera, and defines the coordinate axes of a view coordinate system. In the virtual image generating unit 120, parameters (position, angle of view, magnification, etc.) of the virtual viewpoint VV are set.

[0035] Hereinafter, detailed processing of the virtual image generating unit 120 will be described with reference to Fig. 5. Fig. 5 is a diagram showing how virtual images viewed from each of a plurality of virtual viewpoints are generated.

[0036] 5, the virtual image generating unit 120 generates a plurality of virtual images VI1, VI2, and VI3, each of which corresponds to a plurality of different virtual viewpoints VV1, VV2, and VV3 and shows the state of the virtual space VS as seen from the respective virtual viewpoints. Note that, although the number of virtual viewpoints is three in this embodiment, the number of virtual viewpoints may be less than this, or may be more than this.

[0037] As described above, the information processing system 1 of this embodiment generates a virtual image VI showing the state of the virtual space VS as seen from each of the multiple virtual viewpoints VV1, VV2, and VV3. That is, the information processing system 1 of this embodiment can reflect the appearance of the object O shown in the original image OI in the appearance of the object O as seen from each of the multiple virtual viewpoints VV1, VV2, and VV3. This makes it possible to suitably maintain the appearance of the object O shown in the original image OI in a converted image in which the orientation, pose, shape, and arrangement of the object O shown in the original image OI are changed, as will be described later.

[0038] [Feature extractor memory section] The feature extractor storage unit 104 stores a feature extractor used in the processing in the feature extraction unit 130. Specifically, the feature extractor storage unit 104 stores the program and parameters of the feature extractor.

[0039] The feature extractor is a trained machine learning model that extracts image features. The feature extractor is used to extract a first intermediate feature amount and a second intermediate feature amount, which will be described later. In this embodiment, a trained CNN (Convolutional Neural Network) from which a fully connected layer has been removed is used as the feature extractor. Note that, in addition to a CNN from which a fully connected layer has been removed, a trained known machine learning model can also be used as the feature extractor.

[0040] [Feature extraction part] The feature extraction unit 130 extracts original image feature values ​​OF from the original image OI, and also extracts virtual image feature values ​​VF from the virtual image VI. In this embodiment, the feature extraction unit 130 extracts virtual image feature values ​​VF corresponding to each of the multiple virtual images VI1, VI2, and VI3.

[0041] Specifically, the feature extraction unit 130 extracts a first intermediate feature from the original image OI, and extracts an original image feature OF from the first intermediate feature. The feature extraction unit 130 also extracts a second intermediate feature from the virtual image VI, and extracts a virtual image feature VF from the second intermediate feature. The feature extraction unit 130 extracts the first intermediate feature and the second intermediate feature using the feature extractor described above.

[0042] The first intermediate feature includes information about the shape and arrangement of the object O in the original image OI. The second intermediate feature includes information about the shape and arrangement of the object O in the virtual image VI. In this embodiment, a trained CNN excluding the fully connected layer is used as a feature extractor, so specifically, the first intermediate feature and the second intermediate feature are each a feature map output from the convolution layer of the CNN. Here, the number of feature maps is the same as the number of filters (kernels).

[0043] Here, the original image feature OF is a feature indicating the frequency distribution of each color of the original image OI. Moreover, the virtual image feature VF is a feature indicating the frequency distribution of each color of the virtual image VI. In other words, the original image feature OF is a feature that does not include information on the shape and arrangement of the object O in the original image OI. Moreover, the virtual image feature VF is a feature that does not include information on the shape and arrangement of the object O in the virtual image VI. In this embodiment, the original image feature OF and the virtual image feature VF may each be a vector expression equivalent to a feature that can be output from a trained feature extractor such as a CNN. In this embodiment, specifically, the original image feature OF and the virtual image feature VF are each a Gram matrix obtained by calculating the correlation between feature maps output from the convolution layer of the CNN. Specifically, this Gram matrix is ​​obtained by calculating the inner product of feature maps output from the convolution layer of the CNN. The image feature may be a feature that also indicates the frequency distribution of a texture pattern.

[0044] [Parameter adjustment section] The parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature amount OF and the virtual image feature amount VF increases. That is, the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the feature amount indicating the frequency distribution of each color of the original image OI and the feature amount indicating the frequency distribution of each color of the virtual image VI increases. It can also be said that the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the color scheme of the original image OI and the color scheme of the virtual image VI increases. As a result, the information processing system 1 of this embodiment can accurately reflect the appearance of the object O shown in the original image OI in the appearance of the object O shown in the converted image even if the arrangement of the object O in the original image OI is different from the arrangement of the object O in the virtual image VI.

[0045] Specifically, the parameter adjustment unit 140 calculates a loss based on the original image feature amount OF and the virtual image feature amount VF, and adjusts the parameters of the illumination data L and the parameters of the texture data T so as to reduce the loss. The calculation of the loss itself can use a known calculation method such as squared error or cross entropy. Furthermore, the adjustment of the parameters of the illumination data L and the parameters of the texture data T can use a known parameter adjustment method such as gradient descent or backpropagation. Note that in the parameter adjustment unit 140, parameters related to the feature extractor (weighting coefficient, bias, etc.) are not adjusted but remain fixed.

[0046] In this embodiment, the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to each of the multiple virtual images VI1, VI2, and VI3 increases. Specifically, first, the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to the virtual image VI1 increases. Next, the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to the virtual image VI2 increases. Finally, the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to the virtual image VI3 increases.

[0047] Here, it is preferable that the orientation of the object O indicated by the original image OI coincides with the orientation of the object O indicated by the virtual image VI1 corresponding to the virtual image feature VF that is first subjected to processing by the parameter adjustment unit 140. This reduces the processing load related to the adjustment of the parameters of the illumination data L and the parameters of the texture data T.

[0048] When a background image is set in the virtual space VS, the parameter adjustment section 140 may adjust the parameters of the background image in addition to the parameters of the lighting data L and the parameters of the texture data T.

[0049] In addition, the above-mentioned feature extraction unit 130 and parameter adjustment unit 140 may be configured, for example, according to the description in the reference (Gatys, LA, Ecker, AS, Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR. (2016)).

[0050] [Converted image generation part] The converted image generating unit 150 generates a converted image based on the 3D model M and the illumination data L and texture data T whose parameters have been adjusted by the parameter adjusting unit 140. The converted image can also be said to be a variation of the original image OI. The contents of the processing executed by the converted image generating unit 150 are similar to those of the virtual image generating unit 120.

[0051] As described above, the information processing system 1 of this embodiment generates a converted image using the lighting data L and texture data T whose parameters have been adjusted. When generating this converted image, various conditions (position, pose, shape, and orientation of the 3D model M, position, magnification, and angle of view of the virtual viewpoint VV, etc.) can be changed to generate various converted images. Furthermore, when generating the converted image, the parameters of the lighting data L and the parameters of the texture data T can be further adjusted to change the position and brightness of the lighting and the texture set on the surface of the object O, thereby generating even more various converted images.

[0052] Moreover, according to the information processing system 1 of the present embodiment, it becomes easy to generate a large amount of various converted images as described above. The large amount of converted images obtained in this manner can be used, for example, as learning data for a machine learning model for image recognition.

[0053] The use of the information processing system 1 of this embodiment is not limited to generating the converted image described above. For example, the information processing system 1 of this embodiment can also be used to create game objects that appear in a video game. That is, (1) an original image OI showing a game character as an object O is created. (2) A virtual image VI is generated based on a 3D model M, lighting data L, and texture data T that represent the game character. (3) By adjusting the parameters as described above, a game object that reflects the appearance of the original image OI is obtained.

[0054] [4. Processing Executed in the Information Processing System of the Present Embodiment] Finally, the process executed in the information processing system 1 of this embodiment will be described with reference to Figs. 6 and 7. Figs. 6 and 7 are flow charts showing an example of the process executed in the information processing system 1 of this embodiment. The process shown in Figs. 6 and 7 is executed by the control unit 11 operating in accordance with a program stored in the storage unit 12. The process described below is an example of the process executed by the functional blocks shown in Fig. 3.

[0055] 6, first, the control unit 11 acquires an original image OI from the storage unit 12 (S100). The control unit 11 extracts a first intermediate feature from the acquired original image OI (S101). The control unit 11 further extracts an original image feature OF from the first intermediate feature, and stores the extracted original image feature OF in the storage unit 12 (S102).

[0056] 7, the control unit 11 sets a virtual viewpoint VV (S103). Specifically, the control unit 11 sets parameters such as the position, angle of view, and magnification of the virtual viewpoint VV. Next, the control unit 11 acquires a 3D model M, lighting data L, and texture data T from the storage unit 12 (S104), and generates a virtual image VI based on these (S105).

[0057] The control unit 11 extracts a second intermediate feature from the generated virtual image VI (S106). The storage unit 12 further extracts a virtual image feature VF from the second intermediate feature and stores the extracted virtual image feature VF in the storage unit 12 (S107). The control unit 11 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature OF stored in the storage unit 12 and the virtual image feature VF increases (S108).

[0058] When the control unit 11 determines that the similarity between the original image feature amount OF and the virtual image feature amount VF does not satisfy a predetermined condition and that the adjustment of the parameters of the illumination data L and the parameters of the texture data T should be repeated (S109; N), the control unit 11 executes the processes from S105 to S108 again. Specifically, the control unit 11 repeats the adjustment of the parameters of the illumination data L and the parameters of the texture data T until the loss calculated based on the original image feature amount OF and the virtual image feature amount VF becomes less than a predetermined threshold.

[0059] On the other hand, when the control unit 11 determines that the similarity between the original image feature amount OF and the virtual image feature amount VF satisfies a predetermined condition and the parameter adjustment is to be terminated (S109; Y), it then determines whether or not to generate a virtual image VI by changing the virtual viewpoint VV (S110). When the control unit 11 determines that the virtual image VI is to be generated by changing the virtual viewpoint VV (S110; N), it executes the processes from S103 to S109 again. Specifically, the control unit 11 repeats the processes from S103 to S109 for each of a predetermined number of virtual viewpoints VV. On the other hand, when the control unit 11 determines that the virtual image VI is not to be generated by changing the virtual viewpoint VV (S110; Y), it terminates this process.

[0060] [5. Notes] For example, the information processing system according to the present disclosure may be configured as follows.

[0061] (1) An original image acquisition means for acquiring an original image showing an object; a virtual image generating means for generating a virtual image showing a state of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data; a feature extraction means for extracting an original image feature from the original image and a virtual image feature from the virtual image; a parameter adjusting means for adjusting parameters of the illumination data and parameters of the texture data so as to increase a similarity between the original image feature amount and the virtual image feature amount; An information processing system having the above configuration.

[0062] (2) The original image is a photographed image of the object disposed in real space. An information processing system according to (1).

[0063] (3) the original image feature amount indicates a frequency distribution of each color of the original image, the virtual image feature amount indicates a frequency distribution of each color of the virtual image, the feature extraction means extracts, from the original image, a first intermediate feature amount including information on the shape and arrangement of the object in the original image, and extracts the original image feature amount from the first intermediate feature amount, and extracts, from the virtual image, a second intermediate feature amount including information on the shape and arrangement of the object in the virtual image, and extracts the virtual image feature amount from the second intermediate feature amount. An information processing system according to (1) or (2).

[0064] (4) the virtual image generating means generates a plurality of virtual images each corresponding to a plurality of different virtual viewpoints and showing a state of the virtual space as seen from the corresponding virtual viewpoint; the feature extraction means extracts, from each of the plurality of virtual images, a virtual image feature corresponding to the virtual image; the parameter adjustment means adjusts the parameters of the illumination data and the parameters of the texture data so as to increase a similarity between the original image feature amount and a virtual image feature amount corresponding to each of the plurality of virtual images. An information processing system according to any one of (1) to (3).

[0065] (5) The method further comprises a converted image generating means for generating a converted image, which is an image showing the object and is different from the original image, based on the 3D model and the lighting data and the texture data whose parameters have been adjusted by the parameter adjusting means. An information processing system according to any one of (1) to (4).

[0066] According to the information processing system 1 of the present embodiment described above, the manner of image conversion can be expanded. [Explanation of symbols]

[0067] 1 Information processing system, N Network, 11 Control unit, 12 Memory unit, 13 Communication unit, 14 Operation unit, 15 Display unit, 100 Original image memory unit, 101 3D model memory unit, 102 Lighting data memory unit, 103 Texture data memory unit, 104 Feature extractor memory unit, 110 Original image acquisition unit, 120 Virtual image generation unit, 130 Feature extraction unit, 140 Parameter adjustment unit, 150 Transformed image generation unit, OI Original image, O Object, M 3D model, L Lighting data, T Texture data, VI, VI1, VI2, VI3 Virtual image, OF Original image feature, VF Virtual image feature, VV, VV1, VV2, VV3 Virtual viewpoint, VS Virtual space.

Claims

1. An original image acquisition means for acquiring an original image showing an object; a virtual image generating means for generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data; a feature extraction means for extracting an original image feature from the original image and a virtual image feature from the virtual image; a parameter adjustment means for adjusting a parameter of the illumination data and a parameter of the texture data based on a loss indicating a degree of similarity between the original image feature amount and the virtual image feature amount; An information processing system having the above configuration.

2. The original image is a photographed image of the object disposed in real space. The information processing system according to claim 1 .

3. the original image feature amount indicates a frequency distribution of each color of the original image, the virtual image feature amount indicates a frequency distribution of each color of the virtual image, the feature extraction means extracts, from the original image, a first intermediate feature amount including information regarding a shape and an arrangement of the object in the original image, and extracts the original image feature amount from the first intermediate feature amount, and also extracts, from the virtual image, a second intermediate feature amount including information regarding a shape and an arrangement of the object in the virtual image, and extracts the virtual image feature amount from the second intermediate feature amount.

3. The information processing system according to claim 1 or 2.

4. the virtual image generating means generates a plurality of virtual images each corresponding to a plurality of different virtual viewpoints and showing a state of the virtual space as seen from the corresponding virtual viewpoint; the feature extraction means extracts, from each of the plurality of virtual images, a virtual image feature corresponding to the virtual image; the parameter adjustment means adjusts the parameters of the illumination data and the parameters of the texture data so as to increase a similarity between the original image feature amount and a virtual image feature amount corresponding to each of the plurality of virtual images.

3. The information processing system according to claim 1 or 2.

5. a converted image generating means for generating a converted image, which is an image showing the object and is different from the original image, based on the 3D model and the lighting data and the texture data whose parameters have been adjusted by the parameter adjusting means; 3. The information processing system according to claim 1 or 2.

6. An original image acquisition step of acquiring an original image showing an object; a virtual image generating step of generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data; a feature extraction step of extracting an original image feature from the original image and a virtual image feature from the virtual image; a parameter adjustment step of adjusting parameters of the illumination data and parameters of the texture data based on a loss indicating a degree of similarity between the original image feature amount and the virtual image feature amount; An information processing method comprising the steps of:

7. An original image acquisition means for acquiring an original image showing an object; a virtual image generating means for generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data; a feature extraction means for extracting an original image feature from the original image and a virtual image feature from the virtual image; a parameter adjusting means for adjusting a parameter of the illumination data and a parameter of the texture data based on a loss indicating a degree of similarity between the original image feature amount and the virtual image feature amount; A program that makes a computer function as a

Images