Image processing apparatus, image processing method, and program

Abstract

To make it easier for an observer to check the glossiness of a virtual object to be synthesized on a subject, even if the distance between a camera and the subject is far apart.SOLUTION: An image processing apparatus acquires a captured image from a camera 203 for capturing a subject. The image processing apparatus acquires a distance from the subject to the camera 203 (camera-subject distance). The apparatus generates a three-dimensional image of a virtual object corresponding to the subject and synthesizes it with the captured image. The apparatus displays the synthesized image on a display 202. In generating the three-dimensional image of the virtual object, the apparatus performs a correction process for adjusting light reflected from the virtual object observed when the direction of the camera 203 is changed on the basis of the camera-subject distance.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to image processing of virtual objects. [Background technology]

[0002] In recent years, virtual try-on services using AR (Augmented Reality) technology have been used. Specifically, CG (Computer Graphics) images of watches, accessories, backpacks, clothes, shoes, etc. are synthesized and displayed on live-action video captured by a camera, allowing users to experience virtual try-on in real time. Virtual try-on can reduce the time it takes to try on clothes, allowing users to check the impression and size of many options, and is used to promote purchases in stores and EC (Electronic Commerce) sites. Patent Document 1 discloses a technology for synthesizing and displaying three-dimensional CG images on live-action video in real time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-211251 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when performing virtual try-on using a camera on a smartphone or the like, there is an issue that if the distance between the camera and the subject is far, it is difficult to confirm the glossiness of the virtual object being synthesized.

[0005] An object of the present invention is to enable an observer to easily confirm the glossiness of a virtual object to be composited with a subject, even if the distance between the camera and the subject is great. [Means for solving the problem]

[0006] The image processing device of the present invention comprises an image acquisition means for acquiring an image from an imaging device that images a subject, a distance acquisition means for acquiring a distance from the subject to the imaging device, an image generation means for generating a three-dimensional image of a virtual object corresponding to the subject, a synthesis means for synthesizing the three-dimensional image of the virtual object with the captured image, and a display control means for controlling the display of the synthesized image synthesized by the synthesis means, wherein the image generation means performs a correction process for adjusting the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired by the distance acquisition means. Effect of the Invention

[0007] According to the present invention, even if the camera is far from the subject, the viewer can easily confirm the glossiness of a virtual object to be composited with the subject. [Brief description of the drawings]

[0008] [Figure 1] 11A and 11B are diagrams illustrating the relationship between the camera-subject distance and the appearance of gloss of a virtual object. [Diagram 2] FIG. 1 is an external view of an image processing device. [Diagram 3] FIG. 1 is a diagram illustrating a hardware configuration of an image processing device. [Figure 4] FIG. 2 is a diagram illustrating a functional configuration of the image processing device. [Diagram 5] 4 is a flowchart showing a main process of the image processing device. [Figure 6] 5 is a flowchart showing a three-dimensional image generating process according to the first embodiment. [Figure 7] 3 is a schematic diagram showing the reflection characteristics of a surface of a virtual object. FIG. [Figure 8] 11 is a schematic diagram for explaining a method of correcting the position of a virtual light source. FIG. [Figure 9] FIG. 2 is a diagram showing an example of a three-dimensional image of a virtual object. [Figure 10] 10 is a flowchart showing a three-dimensional image generating process according to the second embodiment. [Figure 11]10A and 10B are schematic diagrams for explaining a method of correcting reflection characteristics. [Figure 12] FIG. 11 is a schematic diagram for explaining a method of correcting the camera direction. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0010] First, the relationship between the distance from the camera to the subject and the appearance of gloss of a virtual object will be described with reference to Fig. 1. In this embodiment, an observer takes a picture of the subject with a smartphone camera, and observes a virtual object that is composited with the subject in the captured image. As shown in Fig. 1, when an observer observes while changing the camera position, the angle between the camera direction before and after the change in camera position is called the "observation angle."

[0011] As shown in FIG. 1(a), when the distance from the camera to the subject is short, the observation angle of the virtual object is large relative to the change in the camera (smartphone) position. FIGS. 1(b) and 1(d) are diagrams showing the reflection characteristics of a virtual object. Light incident on a virtual object from a certain direction is reflected with a spread near the regular reflection direction. As shown in FIG. 1(b), when the observation angle of the virtual object is large, the change in the reflected light intensity of the virtual object is large, so that it is easy to confirm the difference in glossiness of virtual objects with different glossiness. On the other hand, as shown in FIG. 1(c), when the distance from the camera to the subject is long, the observation angle of the virtual object is small relative to the change in the camera (smartphone) position. As shown in FIG. 1(d), when the observation angle of the virtual object is small, the change in the reflected light intensity of the virtual object is small, so that it is difficult to confirm the difference in glossiness of virtual objects with different glossiness. In each of the following embodiments, a method of adjusting the reflected light of a virtual object observed when the camera direction is changed based on the distance from the camera to the subject will be described.

[0012] <Embodiment 1> In the first embodiment, a method for emphasizing a change in the reflected light intensity of a virtual object when the camera direction is changed by changing the light source direction of a virtual light source will be described.

[0013] [Hardware configuration of image processing device] FIG. 2 is an external view of an image processing device 201 according to this embodiment. In this embodiment, the image processing device is described as a mobile device such as a smartphone or tablet with a built-in camera, but may be a video see-through type head-mounted display or the like, and is not particularly limited. Any device may be used as long as it can synthesize a three-dimensional image of a virtual object with an image captured in real space and display it. FIG. 2(a) shows a front view of the image processing device 201. FIG. 2(b) shows a rear view of the image processing device 201. In this embodiment, the image processing device 201 is an out-camera type in which a camera is mounted on the side without a display, but may be an in-camera type in which a camera is mounted on the side with a display. As shown in FIG. 2, the image processing device 201 has a display 202 such as a liquid crystal or organic EL, a camera 203, and a distance measurement sensor 204. The display 202 is an example of a display device. The camera 203 is an imaging unit composed of, for example, a CMOS image sensor and a lens. The camera 203 is an example of an imaging device. The distance measurement sensor 204 is, for example, a LiDAR sensor (Light Detection And Ranging). LiDAR sensors measure distance by shining light on an object and capturing the reflected light with an optical sensor.

[0014] 3 is a block diagram showing the hardware configuration of an image processing device 201 according to this embodiment. The image processing device 201 has a CPU (Central Processing Unit) 301, a ROM (non-volatile memory) 302, and a RAM (volatile memory) 303. The image processing device 201 also has an HDD (Hard Disk Drive) 304 and a NIC (Network Interface Card) 305 in addition to the above-mentioned display 202, camera 203, and distance measurement sensor 204. These components are connected to each other via a system bus 306.

[0015] The CPU 301 controls the entire image processing device 201. The CPU 301 executes an OS (operating system) and various programs stored in the ROM 302, HDD 304, etc., using the RAM 303 as a work memory. Note that, in the processing according to the flowchart described below, program codes stored in the ROM 302, HDD 304, etc. are expanded in the RAM 303 and executed by the CPU 301. The NIC 305 connects to the Internet and inputs and outputs information between the external device. The CPU 301 uses the HDD 304 and recording media on the Internet as storage locations for various data. The CPU 301 also displays a UI (user interface) provided by the program on the display 202, and accepts inputs such as user instructions (e.g., changing the type of virtual object) via the touch panel of the display 202.

[0016] [Functional configuration of image processing device] Fig. 4 is a diagram showing the functional configuration of the image processing device 201 according to this embodiment. The CPU 301 executes a program stored in the ROM 302, HDD 304, etc., to realize the functions of the components shown in Fig. 4. The image acquisition unit 401 acquires two-dimensional RGB images captured by the camera 203 at a predetermined frame rate (for example, at 1 / 30 second intervals) and records them in the RAM 303. The images acquired by the image acquisition unit 401 may be a plurality of still images or moving images continuously captured by the camera 203. The image acquisition unit 401 acquires still images one image at a time, or a moving image one frame at a time, and handles them as captured images.

[0017] A camera-to-subject distance acquisition unit 402 acquires the distance from the camera 203 to the subject measured by the distance measurement sensor 204 (hereinafter referred to as the camera-to-subject distance), and records it in the RAM 303 . The three-dimensional image generation unit 403 generates a three-dimensional image of a virtual object in which the change in the reflected light intensity of the virtual object observed when the camera direction is changed is emphasized, according to the camera-to-subject distance acquired by the camera-to-subject distance acquisition unit 402. The synthesis unit 404 superimposes a three-dimensional image of a virtual object on the captured image to generate a synthetic image. The display control unit 405 controls so as to display the composite image on the display 202. The display control unit 405 updates the display of the composite image one image at a time if what has been acquired by the image acquisition unit 401 is a still image, or one frame image at a time if what has been acquired by the image acquisition unit 401 is a moving image.

[0018] [Processing performed by the image processing device] Next, the main processing executed by the image processing device 201 according to this embodiment will be described with reference to the flowchart in Fig. 5. In the following description of the flowchart, each process (step) will be denoted with an S at the beginning, and the process (step) will not be described.

[0019] In S501, the image acquisition unit 401 acquires a captured image from the camera 203 that captures an image of a subject. In S502, the camera-to-subject distance acquisition unit 402 acquires the camera-to-subject distance D from the distance measurement sensor 204. Note that the method by which the camera-to-subject distance acquisition unit 402 acquires the camera-to-subject distance is not limited to the method using the distance measurement sensor 204. For example, the camera-to-subject distance may be acquired from another camera that captures an overhead image of the space in which the subject exists, or may be acquired from the size of the subject in the captured image acquired in S501.

[0020] In S503, the three-dimensional image generating unit 403 generates a three-dimensional image of a virtual object to be synthesized with the image acquired in S501. Here, the three-dimensional image generating unit 403 performs a correction process for adjusting the reflected light of the virtual object observed when the camera direction is changed, based on the camera-subject distance D acquired in S502. Details of the process of this step will be described later with reference to the flowchart in FIG. In S504, the synthesis unit 404 generates a synthetic image by superimposing the three-dimensional image of the virtual object generated in S503 on the captured image acquired in S501. In S505, the display control unit 405 controls so that the composite image generated in S504 is displayed on the display 202. In this manner, the series of steps in this flowchart is completed. The series of steps in this flowchart is repeatedly executed at a predetermined frame rate, whereby the composite image displayed on the display 202 is updated in real time.

[0021] FIG. 6 is a flowchart showing the details of the process of generating a three-dimensional image of a virtual object, which is executed in S503. In S601, the image acquisition unit 401 acquires virtual object information. The virtual object information is pre-stored in a storage unit such as the HDD 304, and the image acquisition unit 401 reads out the virtual object information from the storage unit. Here, the virtual object information in this embodiment will be described. The virtual object information is information that includes at least shape information and reflection property information of a virtual object. The shape information is information such as a plurality of points, faces, and normals that constitute the shape. The reflection property information is information that represents the reflection property of each minute region of each face. Here, the reflection property will be further described.

[0022] FIG. 7 is a schematic diagram for explaining the reflection characteristics at a certain position on the surface of a virtual object. As shown in FIG. 7, when light is irradiated from the direction of a light source 701 toward a point 702 on the surface of a virtual object having a normal direction 703, reflected light 704 has different intensities depending on the direction. This reflected light 704 can be modeled using, for example, a dichromatic reflection model. Specifically, the reflected light 704 is separated into a specular reflected light component 705, which is light reflected at the boundary between the air and the virtual object, and a diffuse reflected light component 706, which is light that passes through the boundary and is scattered and diffused inside the virtual object. The specular reflected light component 705 is characterized by being strongly observed near the regular reflection direction. Therefore, the specular reflected light component 705 is characterized by three elements: a specular reflected light direction 707, which is the direction in which the intensity of the specular reflected light component is maximum, a specular reflected light intensity 708, which is the reflection intensity in the specular reflected light direction, and a specular reflected light width 709, which is the spread width of the specular reflected light component centered on the specular reflected light direction. In addition, the specular reflected light direction 707 changes due to the normal direction 703 of the virtual object. On the other hand, the diffuse reflection component 706 has the characteristic that it is observed with a uniform intensity in all directions. The reflection intensity of the diffuse reflection component is called the diffuse reflection intensity 710.

[0023] The reflectance characteristics at a certain position on an object surface can be described by a four-dimensional function called the Bidirectional Reflectance Distribution Function (BRDF). The reflectance characteristics of an object surface can also be described by a six-dimensional function called the spatially-varying BRDF (SVBRDF), in which the BRDF changes depending on the position. The reflectance characteristics of semi-transparent objects such as skin and marble can be described by an eight-dimensional function called the bidirectional scattering surface reflectance distribution function. Note that the amount of data increases as the dimensions increase. Here, the bidirectional scattering surface reflectance distribution function refers to the Bidirectional Scattering Surface Reflectance Distribution Function (BSSRDF).

[0024] Various methods for measuring reflectance characteristics have been proposed. For example, there is a method in which multiple images are taken while varying the direction of the light source and the direction of the camera, and the reflectance characteristics of the object are measured using a parametric reflectance model. As a model of specular reflected light, for example, there is the Torrance-Sparrow model, which is known as a physically correct model of specular reflected light. As other models of specular reflected light, there are the Cook-Torrance model, which expresses the normal distribution term using a Beckmann distribution that can be directly calculated by absolute intensity, and the Phong model, which is a classical model based on experience. On the other hand, as a model of diffuse reflected light, for example, there is the Lambert model, which is known as an ideal model of diffuse reflected light. As another model of diffuse reflected light, there is the Oren-Nayar model, which takes into account the effects of self-shielding and self-shadowing caused by microfacets. By using these parametric reflectance models, the reflectance characteristics of the object surface can be expressed by parameters.

[0025] Of the parameterized reflection characteristics of the object surface, in this embodiment, information corresponding to the diffuse reflected light component 706 is called diffuse reflected light information, and information corresponding to the specular reflected light component 705 is called specular reflected light information. Also, information corresponding to the normal direction 703 is called normal information, information corresponding to the specular reflected light intensity 708 is called specular reflected light intensity information, and specular reflected light width information corresponding to the specular reflected light width 709. Also, in this embodiment, information including the specular reflected light intensity information and the specular reflected light width information is called specular reflected light information.

[0026] For example, let us consider a case where diffuse reflected light, one of the reflection characteristics of an object surface, is parameterized using the Lambertian model. When the Lambertian model is used, the radiance of diffuse reflected light is expressed by the following equation (1).

number

[0027] Therefore, when the diffuse reflected light is modeled using the Lambertian model, the two-dimensional distribution of this diffuse reflectance K d (x, y) corresponds to the diffuse reflection light information in this embodiment.

[0028] On the other hand, we will explain the case where the specular reflected light, which is one of the reflection characteristics of the object surface, is parameterized by the Torrance Sparrow model. When the Torrance Sparrow model is used, the radiance of the specular reflected light is expressed by the following equation (2).

number

[0029] The normal distribution term D, which expresses the minute normal variation of the object surface, represents the probability density function of the angle α between the normal direction N and the half vector H, which bisects the illumination direction and the observation direction. A specific normal distribution term D is expressed by the following equation (3).

number

[0030] In addition, the geometric attenuation term G, which expresses self-occlusion and self-shadowing caused by microscopic surface irregularities, becomes larger as the lighting direction or observation direction approaches the tangent plane of the object, and is expressed by the following equation (4).

number

[0031] Moreover, the Fresnel term F, which changes the reflectance depending on the refractive index and the angle of incidence of light, is expressed by the following equations (5) to (7).

number

[0032] Therefore, when the specular reflected light is parameterized by the Torrance Sparrow model, the two-dimensional distribution N(x, y) of the normal direction N used in the above formulas (2) to (7) corresponds to the normal information in this embodiment. s The two-dimensional distribution of K s (x, y) corresponds to specular reflected light intensity information in this embodiment, and the two-dimensional distribution n(x, y) of n, which is a parameter representing surface roughness, corresponds to specular reflected light width information in this embodiment.

[0033] In S602, the three-dimensional image generating unit 403 acquires a synthesis position and synthesis direction for synthesizing a virtual object in the captured image acquired in S501. In this embodiment, a method using a marker will be described as a method for detecting the synthesis position and synthesis direction. Specifically, an image including a marker that serves as a landmark for the position where the virtual object is to be synthesized is captured in advance, and the synthesis position of the virtual object is calculated by calculating the center of gravity of the marker from the image. In addition, the synthesis direction of the virtual object is calculated by calculating the degree of deformation of the marker from the image. The marker is assumed to be, for example, a part of the subject.

[0034] In S603, the three-dimensional image generating unit 403 calculates the camera position and the camera direction in the three-dimensional virtual space. The camera position and the camera direction in this embodiment are the position and the direction of the camera in the three-dimensional virtual space with the composition position calculated in S603 as the origin and the composition direction calculated in S603 as the positive direction of the Z axis. The three-dimensional image generating unit 403 calculates the camera position and the camera direction in the three-dimensional virtual space from the camera-subject distance D acquired in S502 and the degree of deformation of the marker also used in S602. The camera direction is an example of the imaging direction. The three-dimensional image generating unit 403 stores the calculated camera position and the camera direction in time series in the RAM 303 or the like. The three-dimensional image generating unit 403 may estimate the camera position and the camera direction from the position and attitude information of the image processing device main body.

[0035] In S604, the three-dimensional image generating unit 403 acquires the position of the virtual light source in the three-dimensional virtual space and performs correction. Here, the principle of the correction method of the position of the virtual light source will be described first. FIG. 8 is a schematic diagram for explaining the principle of the correction method of the position of the virtual light source. It is assumed that the camera is placed at a position 801 at a certain time t0. Then, it is assumed that the camera moves in a direction 808 away from the normal direction 803 of the virtual object, and moves to a position 802 at the next time t1 after the certain time t0. Also shown are reflection characteristics 804 of a virtual object when the virtual light source is placed at a position 805, and reflection characteristics 806 of a virtual object when the virtual light source is placed at a position 807. In this way, when the direction from the virtual object to the virtual light source moves in a direction approaching the normal direction 803 of the virtual object, the specular reflection direction of the reflection characteristics of the virtual object changes in a direction approaching the normal direction of the virtual object.

[0036] Here, assuming that there is a virtual light source at position 805, in the reflection characteristic 804 of the virtual object, the difference between the reflected light intensity observed by the camera at position 801 and the reflected light intensity observed by the camera at position 802 is not large. Therefore, in this embodiment, the position (light source direction) of the virtual light source is changed in conjunction with the direction in which the observation position (camera position) of the virtual object moves, thereby increasing the difference between the reflected light intensity observed by the camera at position 801 and the reflected light intensity observed by the camera at position 802. Specifically, the position 805 of the virtual light source at a certain time t0 is moved to a position 807 at the next time t1. In addition, the change amount 809 of the virtual light source position is increased with an increase in the camera-subject distance D. In other words, as the camera-subject distance increases, the change amount 809 of the virtual light source position relative to the change in the observation position is increased, so that the change in the intensity of the observed reflected light is made larger than the actual one. In addition, as the camera-subject distance increases, the position of the virtual light source is prevented from being changed even if the observation position changes. This is because as the camera and subject come closer to each other, the observation angle becomes larger with respect to the change in observation position, making it unnecessary to change the position of the virtual light source.

[0037] Next, the specific process of S604 will be described. First, the 3D image generator 403 calculates the current time ti Vector Xc representing the camera position calculated using the captured image at i and the vector Pc representing the camera direction i Obtain the camera position vector Xc i and the vector Pc representing the camera direction i is calculated in S603. Next, the 3D image generator 403 calculates the 3D image from the previous time t i-1 Vector Xc representing the camera position calculated from the image taken at i-1 and the vector Pc representing the camera direction i-1 The vector Xc i-1 and Pc represents the camera direction. i-1 is stored in the RAM 303 or the like. Next, the 3D image generator 403 calculates a vector ΔXc i =Xc i -Xc i-1 Calculate.

[0038] Next, the 3D image generator 403 calculates a vector ΔXc i is a direction away from the normal direction 803 of the virtual object, or a direction approaching the normal direction 803 of the virtual object. Next, the 3D image generator 403 calculates the 3D image from the previous time t i-1 The position vector Xs of the virtual light source in i-1 Get the. Next, the three-dimensional image generating unit 403 calculates a correction coefficient k for correcting the position of the virtual light source from the camera-to-subject distance D acquired in S502. Specifically, the correction coefficient k is calculated using a monotonically increasing function of the camera-to-subject distance D.

[0039] Next, the three-dimensional image generator 403 calculates a vector ΔXs for correcting the position vector of the virtual light source. i Specifically, the vector ΔXc i If the direction of the virtual light source is away from the normal direction of the virtual object, a vector ΔXs is used to correct the position vector of the virtual light source. iThe direction of the camera is changed to the normal direction of the virtual object. Also, the vector ΔXc i When the direction of the virtual light source approaches the normal direction of the virtual object, the vector ΔXs is used to correct the position vector of the virtual light source. i The direction of the virtual light source is changed to a direction away from the normal direction of the virtual object. Then, the vector ΔXs is used to correct the position vector of the virtual light source. i is calculated so as to be larger depending on the correction coefficient k. The calculated vector ΔXs i By performing the correction using, the change amount of the light source direction of the virtual light source is increased as the camera-to-subject distance D increases.

[0040] Finally, the 3D image generator 403 calculates the 3D image from the previous time t i-1 The position vector Xs of the virtual light source in i-1 , the calculated vector ΔXs i By adding i The position vector Xs of the virtual light source in i In this way, the previous time t i-1 The position vector Xs of the virtual light source in i-1 By correcting for the current time t i The position vector Xs of the virtual light source in i This changes the light source direction of the virtual light source.

[0041] In S605, the 3D image generating unit 403 calculates a 3D image of the virtual object from the virtual object information acquired in S601, the synthesis position and direction of the virtual object acquired in S602, the camera position and direction calculated in S603, and the position and direction of the virtual light source calculated in S604. This completes the series of processes for generating a 3D image of the virtual object.

[0042] FIG. 9 shows an example of a three-dimensional image of a virtual object when an observer observes the virtual object while moving the camera. FIG. 9(a) shows a state before the camera is moved, with the camera at position 801 and the virtual light source at position 805. FIG. 9(b) and FIG. 9(c) show a state after the camera is moved from position 801 in a direction away from the normal direction of the virtual object to position 802. The position of the virtual light source in FIG. 9(b) is the same position 805 as before the camera is moved. On the other hand, the position of the virtual light source in FIG. 9(c) is changed to position 807 in conjunction with the direction in which the camera moves. Comparing the three-dimensional images in FIG. 9(a) and FIG. 9(b), the difference in glossiness is small. On the other hand, comparing the three-dimensional images in FIG. 9(a) and FIG. 9(c), it can be confirmed that the difference in glossiness is large.

[0043] As described above, according to the first embodiment, by changing the light source direction of the virtual light source according to the distance between the camera and the subject, it is possible to emphasize the change in the reflected light amount of the observed virtual object when the camera direction is changed. This allows the observer to recognize the difference in glossiness between virtual objects with different glossiness even if the distance from the camera to the subject is long.

[0044] <Embodiment 2> In the first embodiment, a method for emphasizing a change in the reflected light intensity of a virtual object when the camera direction changes by changing the light source direction of a virtual light source is described. In the second embodiment, a method for emphasizing a change in the reflected light intensity of a virtual object when the camera direction changes by reducing the specular reflected light width of the reflection characteristics of the virtual object is described. Note that the hardware configuration and functional configuration of the image processing device in this embodiment are the same as those in the first embodiment, and therefore a description thereof will be omitted. The following mainly describes the parts that are different from the first embodiment.

[0045] In the second embodiment, the main processing executed by the image processing device 201 is shown in the flowchart of FIG. 5, similarly to the first embodiment.

[0046] FIG. 10 is a flowchart showing the details of the process of generating a three-dimensional image of a virtual object in this embodiment. The processes in S601 to S603 are the same as those explained in FIG. In S1001, the three-dimensional image generating unit 403 corrects the reflection characteristics of a virtual object. Here, a specific process of the method of correcting the reflection characteristics of a virtual object will be described.

[0047] FIG. 11 is a schematic diagram for explaining the principle of a method for correcting the reflection characteristics of a virtual object. It is assumed that at a certain time t0, the camera is placed at a position 1101. After that, it is assumed that the camera moves in a direction away from the normal direction 1106 of the virtual object, and moves to a position 1102 at a time t1 following the certain time t0. Also, the reflection characteristics 1103 of the virtual object when the virtual light source is placed at a position 1104 are shown. In the reflection characteristics 1103 of the virtual object, the difference between the reflected light intensity observed by the camera at the position 1101 and the reflected light intensity observed by the camera at the position 1102 is not large. Therefore, in this embodiment, the reflection characteristics of the virtual object are corrected to increase the difference between the reflected light intensity observed by the camera at the position 1101 and the reflected light intensity observed by the camera at the position 1102. Specifically, a correction process is performed to reduce the specular reflection light width of the reflection characteristics of the virtual object. Moreover, the degree to which the specular reflection width of the reflection characteristics of the virtual object is narrowed is increased with an increase in the camera-subject distance D.

[0048] Next, the specific process of S1001 will be described. First, the three-dimensional image generating unit 403 acquires a surface roughness parameter n of the reflection characteristics of the virtual object. Next, the three-dimensional image generating unit 403 calculates a correction coefficient l for correcting the surface roughness parameter of the reflection characteristics of the virtual object from the camera-to-subject distance D acquired in S502. Specifically, the correction coefficient l is calculated using a monotonically increasing function of the camera-to-subject distance D. Finally, the three-dimensional image generating unit 403 corrects the surface roughness parameters of the reflection characteristics of the virtual object using the following equation (8). n a =n / l (8) where n ais the corrected surface roughness parameter. By correcting the surface roughness parameters using the above formula (8), the specular reflected light width of the reflection characteristics is reduced. The correction coefficient l is an example of a correction amount for emphasizing the change in the reflected light intensity of the virtual object.

[0049] In S1002, the three-dimensional image generating unit 403 calculates the position and direction of a virtual light source in the three-dimensional virtual space. Specifically, the position and direction of the virtual light source is calculated so that the camera is positioned in the specular reflected light direction of the reflection characteristics.

[0050] In S605, the three-dimensional image generating unit 403 calculates a three-dimensional image of the virtual object from the virtual object information acquired in S601, the composite position and direction of the virtual object acquired in S602, the camera position and direction calculated in S603, and the position and direction of the virtual light source calculated in S1002. i-1 Alternatively, the position vector of the virtual light source in may be used. For the reflection characteristics of the virtual light source, the surface roughness parameters corrected in S1001 are used. This completes the series of processes for generating a 3D image of the virtual object.

[0051] As described above, according to the second embodiment, by reducing the specular reflection width of the reflection characteristics of the virtual object according to the distance between the camera and the subject, it is possible to emphasize the change in the amount of reflected light of the virtual object observed when the camera direction is changed. This allows the observer to recognize the difference in glossiness between virtual objects with different glossiness even if the distance between the camera and the subject is long.

[0052] <Embodiment 3> In the first embodiment, a method for emphasizing a change in the reflected light intensity of a virtual object when the camera direction is changed by changing the light source direction of a virtual light source is described. In the third embodiment, a method for emphasizing a change in the reflected light intensity of a virtual object by changing the camera direction is described. Note that the hardware configuration and functional configuration of the image processing device in this embodiment are the same as those in the first embodiment, and therefore the description will be omitted. The following mainly describes the parts that are different from the first embodiment.

[0053] In the third embodiment, the main processing executed by the image processing device 201 is shown in the flowchart of Fig. 5, similarly to the first embodiment. Hereinafter, the generation processing of a three-dimensional image of a virtual object in this embodiment will be described with reference to Fig. 6.

[0054] In this embodiment, after the camera position and direction are calculated in S603, the three-dimensional image generating unit 403 corrects the calculated camera direction. FIG. 12 is a schematic diagram for explaining the principle of the camera direction correction method. Reflection characteristics 1202 are shown when a virtual light source is placed at position 1201. Assume that the position of the camera is changed from camera direction 1204 to camera direction 1205 (the direction in which the camera approaches the normal direction 1203 of the virtual object) or from camera direction 1205 to camera direction 1204 (the direction in which the camera moves away from the normal direction 1203 of the virtual object). In this embodiment, a correction process is performed to make the observation angle larger than the actual angle. Specifically, the camera direction 1204 is changed to correspond to the camera direction 1206, and the camera direction 1205 is changed to correspond to the camera direction 1207. In addition, the degree of correction of the observation angle is increased as the camera-subject distance D increases.

[0055] In S604, the three-dimensional image generating unit 403 calculates the position and direction of the virtual light source in the three-dimensional virtual space. In S605, the three-dimensional image generation unit 403 calculates a three-dimensional image of the virtual object from the virtual object information acquired in S601, the synthesis position and direction of the virtual object acquired in S602, the changed camera position and direction, and the position and direction of the virtual light source calculated in S604. Note that the three-dimensional image generation unit 403 uses the changed camera position and direction only for calculating reflection characteristics, and uses the pre-change camera position and direction for calculating color and shape.

[0056] As described above, according to the third embodiment, the intensity of the reflected light of the virtual object can be emphasized by making the observation angle of the virtual object larger than the actual angle when the observer observes while changing the camera direction according to the distance between the camera and the subject. This allows the observer to recognize the difference in glossiness of virtual objects with different glossiness even if the distance from the camera to the subject is far.

[0057] (Other embodiments) In the above-described first embodiment, a method for increasing the change amount of the light source direction of the virtual light source with an increase in the camera-subject distance D has been described. Here, the change amount of the light source direction of the virtual light source is an example of a correction amount for emphasizing a change in the reflected light intensity of the virtual object. Instead of the above-described method, the image processing device 201 may perform a correction process for adjusting the reflected light of the virtual object observed when the camera direction is changed when the camera-subject distance D is equal to or greater than a threshold, and may not perform the above correction process when the camera-subject distance D is less than the threshold. Specifically, when the camera-subject distance D is equal to or greater than a predetermined threshold, a process for changing the light source direction of the virtual light source is performed, and when the camera-subject distance D is less than the predetermined threshold, the light source direction of the virtual light source is not changed, and the position of the virtual light source at the previous time, for example, is kept as it is.

[0058] In the above-mentioned second embodiment, the specular reflected light width is narrowed to emphasize the change in the reflected light intensity of the virtual object, but the method is not limited to this as long as it is a method for correcting the reflection characteristics of the virtual object. Specifically, other methods may be used, such as increasing the ratio of the specular reflected light intensity to the diffuse reflected light intensity, increasing the specular reflected light intensity, or decreasing the diffuse reflected light intensity.

[0059] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0060] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) an image acquisition means for acquiring an image from an imaging device that captures an image of a subject; a distance acquisition means for acquiring a distance from the subject to the imaging device; an image generating means for generating a three-dimensional image of a virtual object corresponding to the subject; A synthesis means for synthesizing a three-dimensional image of the virtual object with the captured image; a display control means for controlling the display of a composite image generated by the composition means; having The image processing device characterized in that the image generation means performs correction processing to adjust the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired by the distance acquisition means. (Configuration 2) The image processing device according to configuration 1, characterized in that the image generating means increases an amount of correction applied by the correction process as the distance acquired by the distance acquiring means increases, in order to emphasize a change in the reflected light intensity of the virtual object when the imaging direction of the imaging device changes. (Configuration 3) 3. The image processing device according to claim 1, wherein the image generating means changes a light source direction of a virtual light source in order to emphasize a change in the reflected light intensity of the virtual object when the imaging direction of the imaging device is changed. (Configuration 4) The image processing device according to any one of configurations 1 to 3, wherein the image generation means changes a position of a virtual light source in a direction away from the normal direction of the virtual object when the position of the imaging device changes in a direction approaching the normal direction of the virtual object, and changes a position of a virtual light source in a direction approaching the normal direction of the virtual object when the position of the imaging device changes in a direction away from the normal direction of the virtual object. (Configuration 5) 4. The image processing device according to configuration 3, wherein the image generating means increases an amount of change in a light source direction of a virtual light source as the distance acquired by the distance acquiring means increases. (Configuration 6) The image acquisition means acquires the captured images at a predetermined interval, The image processing device according to any one of configurations 1 to 5, wherein the image generating means changes the light source direction of the virtual light source based on the position of the virtual light source at the previous time and whether the position of the imaging device at the previous time has changed in a direction approaching the normal direction of the virtual object or in a direction moving away from the normal direction of the virtual object. (Configuration 7) The image processing device according to any one of configurations 1 to 6, wherein the image generating means corrects the reflection characteristics of the virtual object in order to emphasize a change in the reflected light intensity of the virtual object when the imaging direction of the imaging device is changed. (Configuration 8) 8. The image processing device according to claim 7, wherein the image generating means reduces a specular reflection light width representing the spread of the specular reflection component of the virtual object as the distance acquired by the distance acquiring means increases. (Configuration 9) 9. The image processing device according to any one of configurations 1 to 8, wherein the image generating means changes the imaging direction so that an amount of change in the imaging direction becomes larger than an actual amount. (Configuration 10) The image processing device according to configuration 9, wherein the image generating means uses the imaging direction after the change to calculate the reflection characteristics of the virtual object, and uses the imaging direction before the change to calculate the color and shape of the virtual object. (Configuration 11) The image processing device according to any one of configurations 1 to 10, wherein the image generation means performs the correction process when the distance acquired by the distance acquisition means is equal to or greater than a threshold value, and does not perform the correction process when the distance acquired by the distance acquisition means is less than the threshold value. (method) an image acquisition step of acquiring an image from an imaging device that images a subject; a distance acquisition step of acquiring a distance from the subject to the imaging device; an image generating step of generating a three-dimensional image of a virtual object corresponding to the subject; a synthesis step of synthesizing a three-dimensional image of the virtual object onto the captured image; a display control step of controlling to display a composite image obtained by the composition step; Including, An image processing method characterized in that, in the image generation process, a correction process is performed to adjust the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired in the distance acquisition process. (program) Computer, an image acquisition means for acquiring an image from an imaging device that captures an image of a subject; a distance acquisition means for acquiring a distance from the subject to the imaging device; an image generating means for generating a three-dimensional image of a virtual object corresponding to the subject; A synthesis means for synthesizing a three-dimensional image of the virtual object with the captured image; a display control means for controlling the display of a composite image synthesized by the synthesis means; Function as a The program is characterized in that the image generation means performs a correction process to adjust the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired by the distance acquisition means. [Explanation of symbols]

[0061] 201: image processing device, 202: display, 203: camera

Claims

1. an image acquisition means for acquiring an image from an imaging device that captures an image of a subject; a distance acquisition means for acquiring a distance from the subject to the imaging device; an image generating means for generating a three-dimensional image of a virtual object corresponding to the subject; a synthesis means for synthesizing a three-dimensional image of the virtual object with the captured image; a display control means for controlling the display of a composite image synthesized by the synthesis means; having The image processing device characterized in that the image generation means performs correction processing to adjust the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired by the distance acquisition means.

2. 2. The image processing device according to claim 1, wherein the image generating means increases an amount of correction applied by the correction process as the distance acquired by the distance acquisition means increases, in order to emphasize a change in the reflected light intensity of the virtual object when the imaging direction of the imaging device changes.

3. 2 . The image processing apparatus according to claim 1 , wherein the image generating means changes a light source direction of a virtual light source in order to emphasize a change in the reflected light intensity of the virtual object when the imaging direction of the imaging device is changed.

4. 2. The image processing device according to claim 1, wherein the image generating means changes a position of a virtual light source in a direction away from the normal direction of the virtual object when the position of the imaging device changes in a direction approaching the normal direction of the virtual object, and changes a position of the virtual light source in a direction approaching the normal direction of the virtual object when the position of the imaging device changes in a direction away from the normal direction of the virtual object.

5. 4. The image processing apparatus according to claim 3, wherein the image generating means increases an amount of change in the light source direction of the virtual light source as the distance acquired by the distance acquiring means increases.

6. The image acquisition means acquires the captured images at a predetermined interval, 2. The image processing device according to claim 1, wherein the image generating means changes a light source direction of a virtual light source based on a position of a virtual light source at a previous time and whether the position of the imaging device at a previous time has changed in a direction approaching a normal direction of the virtual object or in a direction moving away from the normal direction of the virtual object.

7. 2. The image processing apparatus according to claim 1, wherein the image generating means corrects reflection characteristics of the virtual object in order to emphasize a change in the intensity of reflected light from the virtual object when the imaging direction of the imaging device is changed.

8. 8. The image processing apparatus according to claim 7, wherein the image generating means reduces a specular reflection light width that represents the spread of the specular reflection component of the virtual object as the distance acquired by the distance acquiring means increases.

9. 2. The image processing apparatus according to claim 1, wherein the image generating means changes the imaging direction so that an amount of change in the imaging direction becomes larger than an actual amount.

10. 10. The image processing device according to claim 9, wherein the image generating means uses the imaging direction after the change to calculate the reflection characteristics of the virtual object, and uses the imaging direction before the change to calculate the color and shape of the virtual object.

11. 2. The image processing device according to claim 1, wherein the image generating means performs the correction process when the distance acquired by the distance acquisition means is equal to or greater than a threshold value, and does not perform the correction process when the distance acquired by the distance acquisition means is less than the threshold value.

12. an image acquisition step of acquiring an image from an imaging device that images a subject; a distance acquisition step of acquiring a distance from the subject to the imaging device; an image generating step of generating a three-dimensional image of a virtual object corresponding to the subject; a synthesis step of synthesizing a three-dimensional image of the virtual object onto the captured image; a display control step of controlling to display a composite image obtained by the composition step; Including, An image processing method characterized in that, in the image generation process, a correction process is performed to adjust the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired in the distance acquisition process.

13. Computer, an image acquisition means for acquiring an image from an imaging device that captures an image of a subject; a distance acquisition means for acquiring a distance from the subject to the imaging device; an image generating means for generating a three-dimensional image of a virtual object corresponding to the subject; a synthesis means for synthesizing a three-dimensional image of the virtual object with the captured image; a display control means for controlling the display of a composite image synthesized by the synthesis means; Function as a The program is characterized in that the image generation means performs a correction process to adjust the reflected light of the virtual object observed when the imaging direction of the imaging device changes, based on the distance acquired by the distance acquisition means.

Images