Information processing apparatus, method for controlling information processing apparatus, and storage medium

Abstract

The present invention relates to an information processing apparatus, a method for controlling an information processing apparatus, and a storage medium. The information processing apparatus includes an appearance information obtaining unit configured to obtain appearance information indicating an appearance of a target object, an illumination information obtaining unit configured to obtain illumination information for illuminating the target object, a direction calculating unit configured to calculate, based on the appearance information and the illumination information, an observation direction for observing the appearance of the target object, and a depiction unit configured to depict information indicating the observation direction.

Description

Technical Field

[0001] This invention relates to an information processing apparatus, a method for controlling the information processing apparatus, and a storage medium. Background Technology

[0002] Recently, with the advancements in CG and imaging technologies, as well as computer performance, it has become possible to represent the appearance of objects (target objects) on information terminals. To increase product appeal, especially when shopping online, there is a growing demand for users to confirm the appearance of products in various indoor and outdoor environments.

[0003] In the method described in Japanese Patent Publication No. H11-265462, the content creator manually sets the gaze direction in advance and presents it to the user. By presenting the user with a gaze direction in which the appearance of the product (such as glossiness or color) is easily apparent, the user can effectively confirm the appearance of the product without having to work on changing their gaze direction in various ways to observe it.

[0004] However, in the technology described in Japanese Patent Publication No. H11-265462, the content creator needs to manually specify the gaze direction suitable for observing the appearance of the target object. Generally, the gaze direction suitable for observing the appearance changes when the surrounding environment changes. Therefore, there is a problem that it is difficult to present the user with the gaze direction suitable for observing the appearance when the surrounding environment has changed.

[0005] The present invention has been made in view of the above problems and provides a technique for determining a gaze direction suitable for observing the appearance of a target object. Summary of the Invention

[0006] According to one aspect of the present invention, an information processing apparatus is provided, comprising: an appearance information acquisition unit configured to acquire appearance information indicating the appearance of a target object; an illumination information acquisition unit configured to acquire illumination information for illuminating the target object; a direction calculation unit configured to calculate an observation direction for observing the appearance of the target object based on the appearance information and the illumination information; and a rendering unit configured to render information indicating the observation direction.

[0007] According to one aspect of the present invention, a method for controlling an information processing apparatus is provided, comprising: obtaining appearance information indicating the appearance of a target object; obtaining illumination information for illuminating the target object; calculating an observation direction for observing the appearance of the target object based on the appearance information and the illumination information; and depicting information indicating the observation direction.

[0008] Further features of the invention will become clear from the following description of exemplary embodiments (with reference to the accompanying drawings). Attached Figure Description

[0009] Figure 1 It is a diagram used to explain color and gloss.

[0010] Figure 2A , 2B 2C, 2D, 2E, 2F, and 2G are graphs used to interpret unevenness.

[0011] Figure 3 This is an example diagram illustrating the appearance of the hardware of an information processing device.

[0012] Figure 4 This is a diagram illustrating an example of the hardware configuration of an information processing device.

[0013] Figure 5 This is a diagram illustrating an example of the functional configuration of an information processing device.

[0014] Figure 6 It is a flowchart of the processes to be performed in the information processing device.

[0015] Figure 7A and 7B It is a diagram used to interpret an environment diagram.

[0016] Figure 8 This is a flowchart of the process used to calculate the gaze direction.

[0017] Figure 9A , 9B 9C and 9C are diagrams used to explain the calculation of the reflective area.

[0018] Figure 10 It is a flowchart used to depict the processing of an image.

[0019] Figure 11A and 11B An example of a user interface is illustrated.

[0020] Figure 12 This is a flowchart of the process used to calculate the gaze direction.

[0021] Figure 13 This is a diagram used to explain the method used to calculate the gaze direction.

[0022] Figure 14 This is a diagram used to explain the usage of the third embodiment.

[0023] Figure 15 This is a diagram illustrating an example of the system configuration of an information processing device.

[0024] Figure 16 It is a flowchart of the processes to be performed in the information processing device.

[0025] Figure 17 An example of a user interface is illustrated.

[0026] Figure 18 It is a flowchart used to depict the processing of an image. Detailed Implementation

[0027] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Several features are described in the embodiments, but the invention is not limited to requiring all such features, and multiple such features can be suitably combined. Furthermore, in the drawings, the same reference numerals are given the same or similar configuration, and redundant descriptions thereof are omitted.

[0028] (First Embodiment)

[0029] In this embodiment, an example will be described of calculating and presenting a gaze direction suitable for observing the appearance of an object (target object) displayed on a display screen from the surrounding environment (lighting environment). This embodiment makes it possible to confirm the appearance in the current lighting environment as if the target object were in that location. For example, there is a situation where someone is shopping online and viewing a fabric product, with an image of the product displayed on a tablet computer or similar device. In this case, this embodiment makes it possible to determine, while expressing how the product looks in the lighting environment of the user's room, which direction relative to the display screen would be most easily observed.

[0030] Before giving a description of this embodiment, examples of elements constituting the appearance and a description of the gaze direction suitable for observing these elements will be given.

[0031] <Description of appearance elements>

[0032] Elements of an object's (material or target object's) appearance include, for example, color, gloss, and unevenness. Color can be perceived from the diffuse component of a material, and gloss can be perceived from the specular component.

[0033] Figure 1This diagram illustrates diffuse and specular reflection. Light 102 incident on the material surface 101 is reflected according to an intensity distribution 103 that depends on the viewing angle. The reflected light intensity distribution 103 can be represented as the sum of a diffuse reflection component 104 caused by diffuse reflection within the material and a specular reflection component 105 caused by reflection from the material surface. The specular reflection component 105 has a peak in the angular region where the angle of reflection equals the angle of incidence. Therefore, at the viewing direction position 106 in the specular reflection direction of the incident light 102, a larger proportion of the specular reflection component 105 is observed compared to other viewing directions; thus, the viewing direction position 106 is suitable for observing the gloss of the material. Similarly, at the viewing direction position 107 in the specular reflection direction away from the incident light 102, the diffuse reflection component is observed more significantly than the specular reflection component; thus, the viewing direction position 107 is suitable for observing the color of the material.

[0034] Unevenness can be perceived from the changes in diffuse and specular reflection that correspond to subtle unevenness on the material surface. Figure 2A This diagram illustrates a material surface with subtle unevenness. Subtle unevenness exists on material surface 101, and the normal direction 108 varies depending on location. In the case of materials where specular reflection intensity is stronger than diffuse reflection intensity and the change in reflected light intensity near the specular reflection direction is significant, subtle changes in the surface normal can be perceived as changes in the specular reflection component. A description of how the unevenness can be effectively observed when viewed from the specular reflection direction of a small light source in such a case will be given below.

[0035] Figure 2B and 2C This diagram illustrates specular reflections on two planes with different normal directions N on a material surface. Regarding the two rays, ray 202 and ray 203, when specularly reflected light from two planes with different normal directions on material surface 205 is observed at viewing position 204, ray 202 is not observed at viewing position 204 because its origin is in an area without a light source when the surface is illuminated by light source 200. Therefore, the difference between the normals of the two planes is perceived as a difference in luminance due to the presence or absence of specularly reflected light. Meanwhile, when the surface is illuminated by light source 201, which is larger than light source 200, both ray 202 and ray 203 are observed; thus, it is difficult to distinguish the difference between the normals of the two planes by specular reflection. Therefore, in the case of materials with strong specular reflection, unevenness can be effectively observed by observing from the specular reflection direction of a small light source.

[0036] Furthermore, unevenness can be effectively observed by looking from the specular reflection direction of a light source with sharp edges. Light sources with sharp edges are, for example, light sources such as fluorescent lamps and spotlights, while light sources with blurry edges are, for example, light from outside a window covered by curtains. Figure 2D The illustrations show images of a material observed from both the specular reflection direction of a light source with sharp edges and the specular reflection direction of a light source with indistinct edges. In image 206, observed from the specular reflection direction of a light source with sharp edges, the irregularities at the edges of the specular reflection image reflect subtle changes in the normals of the material surface, thus allowing the unevenness of the material to be visually identified. Conversely, in image 207, observed from the specular reflection direction of a light source with indistinct edges, the small difference in luminance at the edges of the regular reflection image makes it difficult to visually identify surface unevenness from the reflected image.

[0037] In materials where specular reflection intensity is weaker than diffuse reflection intensity and the change in reflected light intensity near specular reflection light is small, subtle changes in the surface normal can be perceived as changes in diffuse reflection intensity. A description of how unevenness can be effectively observed in such cases by viewing from a direction perpendicular to the light ray will be given below.

[0038] Figure 2E It is a diagram illustrating the diffuse reflection of light from a light source incident on an uneven material surface. Figure 2E The illustration shows light rays 208 incident from light source 201 onto a plane of material with a normal parallel to the direction of the light source, and light rays 209 incident onto a plane with a normal forming an angle θ with the direction of the light source. Assuming diffuse reflection intensities In and Iθ, the relationship between these two intensities is approximately expressed by Lambert's law of cosines as follows:

[0039] [Equation 1]

[0040] I θ =I n cosθ…(1)

[0041] Differentiating Equation 1 with θ yields the following equation.

[0042] [Equation 2]

[0043]

[0044] The region where the diffuse reflection intensity changes the most is near θ = 90 degrees. Figure 2FThe diagram illustrates the distribution of the normal directions of the material. The normals to the material surface have a distribution 210 centered on the average normal direction Nave. Within distribution 210, the plane of region 211, where the angle formed by the incident ray and the normal vector is approximately 90 degrees, exhibits the greatest change in luminance.

[0045] To observe a plane where the angle between the incident ray and the normal vector is approximately 90 degrees, observe the plane from a direction perpendicular to the light source. Figure 2G These are views of planes formed by the angles of 0 degrees, 5 degrees, 85 degrees, and 90 degrees between the normal and the incident ray, viewed from both a viewing direction parallel to the incident ray and a viewing direction perpendicular to the incident light source. In view 212, viewed from the direction parallel to the incident ray, the plane with the smaller angle between the normal and the incident ray appears larger. Similarly, in view 213, viewed from the direction perpendicular to the incident ray, the plane with the angle close to 90 degrees between the normal and the incident ray appears larger. Furthermore, since the intensity of diffuse reflection varies greatly with the normal forming an angle near 90 degrees between the normal and the incident ray, subtle changes in the normal of the material surface can be observed as changes in luminance. Therefore, even when specular reflection is weak, unevenness can be effectively observed by viewing from a direction perpendicular to the light source.

[0046] In this embodiment, a tablet-type information terminal is used as the information processing device; the terminal calculates and presents a gaze direction suitable for observing the color and gloss among the elements of the aforementioned appearance. Assume an application for observing appearance that displays appearance information mapped to a planar model of the target object on the display of a tablet-type information terminal with a fixed position and posture, and dynamically changes the appearance of the target object according to changes in the user's gaze.

[0047] <Usage>

[0048] An example of a usage scenario of the tablet-type information terminal according to this embodiment will be described. For example, suppose a user wants to confirm the appearance of a product displayed on the Web (such as when shopping online) while viewing and verifying it. When a fabric, which is used as an example product, is being viewed, the tablet-type information terminal obtains the appearance information of the fabric that has been pre-stored on the Web in response to the user's selection of the fabric product.

[0049] Next, the lighting information of the user's room is obtained to determine how the fabric will be observed in the lighting environment (such as the lighting environment of the room the user is currently in). It is conceivable that the lighting information could be obtained, for example, by capturing the entire room while the user holds the tablet-type information terminal in the center of the room and rotates it. Alternatively, a configuration using an omnidirectional camera deployed in the center of the room could be employed to obtain the lighting information.

[0050] When a user displays an image of a fabric on a tablet computer in a room, the appearance of the fabric changes depending on the tablet computer's position and orientation, as well as the user's gaze direction. When the fabric image is displayed on the tablet computer, it's as if the fabric is being held in the same position as the tablet computer, and its appearance is then displayed. At this point, by displaying a viewing direction suitable for observing the appearance, the user can easily determine that their current gaze direction is not aligned with the optimal viewing direction. Therefore, it becomes easier for the user to change their gaze direction to more closely approximate the optimal viewing direction while keeping the tablet computer in a fixed position (i.e., without moving the tablet computer's spatial position and orientation). For example, when initially observing the fabric directly in front of the tablet computer screen, it can be determined that viewing from a slightly right diagonal direction would be the optimal viewing direction. The user changes their gaze direction by shifting their head to the right to view the tablet computer screen from a diagonal angle; this changes the appearance of the fabric image, making it easier to observe.

[0051] <Hardware Configuration>

[0052] Figure 3 This is an external view of the information processing apparatus described in this embodiment. Reference numeral 1 refers to the main body of the information processing apparatus. The information processing apparatus 1 includes a display 304 provided with touch panel functionality configured with a panel such as a liquid crystal or organic EL panel, and a camera 308 located on the same surface as the display 304. The camera 308 acquires, for example, a two-dimensional 8-bit RGB image of 1280×720 pixels using a CMOS sensor.

[0053] Figure 4 This is a block diagram illustrating the hardware configuration of the information processing device 1. In addition to the aforementioned display 304 and camera 308, the information processing device 1 also includes a central processing unit (CPU) 301, a read-only memory (ROM) 302, a random access memory (RAM) 303, and a hard disk drive (HDD) 305. It also includes an accelerometer 306, an orientation sensor 307, and a network interface card (NIC) 309.

[0054] CPU 301 uses RAM 303 as its working memory to execute the operating system (OS) and various programs stored in ROM 302, HDD 305, etc. CPU 301 controls various components via system bus 310. NIC 309 connects to the Internet and inputs and outputs information to external devices. CPU 301 uses HDD 305, ROM 302, and storage media on the Internet as data storage areas. CPU 301 also displays a user interface (UI) provided by the program on display 304 and receives user input via the touch panel of display 304. Accelerometer 306 outputs three axial directions (which are...) via methods such as electrostatic capacitance. Figure 3 The acceleration is shown on the coordinate axes x, y, and z. Regarding this acceleration, when the display 304 is facing directly upwards (towards the sky), assuming the gravitational acceleration is g, the output is (0, 0, g). The orientation sensor 307 is a sensor used to measure the Earth's magnetic field and... Figure 3 The x, y, and z coordinate system shown outputs a three-dimensional vector indicating the north direction.

[0055] <Function Configuration>

[0056] Figure 5 This is a configuration diagram illustrating the functional configuration of the information processing apparatus of this embodiment. OS 3 is an operating system and a set of instructions for controlling input and output, as well as for launching and switching applications. Device driver 408 is a set of instructions included in OS 3 and controls the display 304, camera 308, and various sensors incorporated in the information processing apparatus 1. Various applications can control these devices by sending predetermined instructions to OS 3.

[0057] Appearance display application 2 is an instruction set for reading information from a storage medium on HDD 305 or the Internet and displaying it on display 304. Appearance display application 2 includes appearance information acquisition unit 401, lighting information acquisition unit 402, user gaze direction acquisition unit 403, appearance observation gaze direction calculation unit 404, target object image depiction unit 405, appearance observation gaze direction depiction unit 406, and output image synthesis unit 407.

[0058] The appearance information acquisition unit 401 acquires diffuse reflection intensity, specular reflection intensity, gloss, and normal map as appearance information of the target object. The lighting information acquisition unit 402 acquires an environment map as lighting information. Details of the environment map will be described later. The user gaze direction acquisition unit 403 estimates the position of the user's eye relative to the information processing device 1 from the RGB image acquired by the camera 308 to obtain the gaze direction when viewing the target object displayed on the display 304. The appearance observation gaze direction calculation unit 404 calculates the gaze direction from the environment map that makes it easy to visually identify the appearance of the target object. The target object image depiction unit 405 depicts an image I1 representing the view from the user's gaze direction when the target object is illuminated by the environment map, based on the appearance information, the environment map, and the user gaze direction.

[0059] The appearance observation gaze direction depiction unit 406 depicts an image I2 representing the user's gaze direction and the gaze direction used to observe the appearance. The output image synthesis unit 407 synthesizes the image I1 and the image I2 and generates an image to be displayed on the display 304.

[0060] <Processing>

[0061] Figure 6 This is a flowchart used to explain the processes to be performed in Appearance Display Application 2. References will be made below. Figure 6 Details regarding the processing to be performed in Appearance Display Application 2 are described below. In the following text, each step is indicated by adding an "S" before the reference numerals.

[0062] In S501, based on instructions from the user, the appearance information acquisition unit 401 obtains diffuse reflection intensity ρd(x,y,i), specular reflection intensity ρs(x,y,i), glossiness σ(x,y,i), and normal N(x,y,k) as appearance information from the data storage area. Here, the data storage area is, for example, information pre-stored on a website of a product viewed on an internet shopping website visited by the user. Here, x and y represent the position coordinates on the target object. i indicates which of the R, G, and B color signals it is. k indicates which of the x, y, and z components of the normal direction it is. In this embodiment, the diffuse reflection intensity, specular reflection intensity, and glossiness are arrays that hold 8 bits of information for each set of position coordinates and for each color signal. The normal direction is an array that holds 8 bits of information for each set of position coordinates and for each component, and the range from -1 to 1 for each component is associated with a value from 0 to 255. The type of appearance information in this embodiment is an example, and for example, specular reflection intensity can be held as 32-bit floating-point information.

[0063] In S502, based on instructions from the user, the lighting information acquisition unit 402 obtains the environmental map E from the data storage area. As lighting information. Subsequently, based on the posture of the information processing device 1, the lighting information acquisition unit 402 rotates the environment map E. This ensures that the real world and the coordinate system are consistent.

[0064] Here, we will refer to Figure 7A and Figure 7B Provide a detailed description of the environment map. Figure 7A Indicates the use of environment graph E for creating The method, Figure 7B An example of the created environment map E is shown. Environment map E can be obtained using an omnidirectional camera 601 mounted at any point O (e.g., the center of the floor surface of a room or the location of the planning operation information processing device 1). Specifically, measurements are taken from any point O relative to the gaze direction. The intensity of the light, and by storing it for each gaze direction Furthermore, an environment map E is created based on the light intensity measured for each color signal i. The environment map E is an image representing the light intensity and is an omnidirectional image, etc.

[0065] Here, the position O of the imaging surface of the omnidirectional camera 601 is set as the origin, and θa and These represent the polar angle and azimuth angle relative to the normal direction N of the imaging surface, respectively. In this embodiment, θa is obtained where directly above (sky) is 0 and the north direction is... Environment Map E Furthermore, the environment map E stores 8 bits of intensity information for each angle and for each color signal. The number of pixels is assumed to be 1920×1080.

[0066] Alternatively, an environment map E can be obtained by having the user hold the information processing device 1 at any point O and capture the surrounding environment while rotating it 360° at that position, without using the omnidirectional camera 601.

[0067] The lighting information in this embodiment is in the form of an example, and for example, the signal strength of the environment map can be maintained as 32-bit floating-point information, and the origin of the polar angle and azimuth angle does not need to be determined as described above.

[0068] Next, a three-dimensional vector d representing the north direction is obtained via the orientation sensor 307 of the information processing device 1, and a three-dimensional vector a representing the acceleration of the information processing device 1 is obtained via the acceleration sensor 306. Based on the two vectors, d and a, three rotation angles (r, p, y) in the roll, pitch, and yaw directions are calculated, representing the coordinates from a reference coordinate system pointing east along the x-axis, north along the y-axis, and directly upward (towards the sky) to... Figure 3The transformation of the coordinate system shown is fixed to the information processing device 1.

[0069] The description of the calculation method will be omitted because it is known. Environment Diagram E Rotate based on rotation angle (r, p, y) to make the environment graph E The directions directly above (sky) and north coincide with the directions directly above (sky) and north in real space, thus ensuring consistency between real space and the coordinate system. As a result, optical consistency with real space can be maintained, especially when using an environment map that reflects the surrounding lighting environment in which the observation is performed.

[0070] In S503, the appearance observation gaze direction calculation unit 404 calculates a gaze direction suitable for observing color and gloss from the lighting information. Here, Figure 8 This is a flowchart showing the details of S503. Refer to the following text. Figure 8 Provide a description.

[0071] In S701, the user gaze direction acquisition unit 403 first acquires an RGB image including the user's face via the camera 308. Then, based on the RGB image, the user gaze direction acquisition unit 403 estimates the gaze direction (R, Θ, Φ) of the target object displayed at the center of the display 304 from the position of the user's eyes.

[0072] Here, the gaze direction (R, Θ, Φ) is represented by a three-dimensional polar coordinate system in which the polar angle is Θ, the azimuth angle is Φ, and the origin is the center point of the display 304. The vector (xr, yr, zr) pointing from the camera 308 to the eye position is obtained from the distance from the camera 308 to the user's eye estimated using a known depth estimation technique based on RGB monocular images, and the two-dimensional position of the user's eye in the image. Using the vector (X, Y, 0) pointing from the camera 308 to the center point of the display 304, the gaze direction (R, Θ, Φ) when viewing a target object displayed on the display 304 is obtained through the following equation.

[0073] [Equation 3]

[0074]

[0075] From S702 to S707 onwards, processing is performed while scanning the observation direction, and when there is no observation direction to be scanned, the repetition is terminated according to S707.

[0076] The observation direction to be scanned is determined as follows. First, consider the polar angle θ and azimuth angle along the average normal direction of the target object. Furthermore, the target object is positioned in a three-dimensional polar coordinate system with its center at the origin. In this case, the target object's θ ranges from 0 to 90 degrees. The hemispherical region of the environment map, ranging from 0 to 360 degrees, is the area from which the target object is illuminated. Within this hemispherical region, the viewing directions are scanned sequentially, and their suitability for observing the color and gloss of the target object is determined in a process described later. At this time, the distance from the origin of the viewpoint to be scanned is fixed and set to be equal to the distance R between the eye and the target object obtained in S701. In this embodiment, θ and θ within the hemispherical region... Each segment is divided into 20 segments to determine the observation direction to be scanned.

[0077] here, Figure 9A This diagram illustrates the scanning order. The viewing directions are ordered by the distance 801 between the viewing direction 803 to be scanned and the obtained user gaze direction 802, and scanning is performed sequentially starting with the viewing direction with the shortest distance. Since this allows for priority presentation of the gaze direction closest to the user's gaze direction, the distance the user travels from their original gaze direction to the gaze direction used for observing the appearance can be shortened, thus improving convenience. Although distance is used to order the viewing directions to be scanned in this embodiment, the viewing directions can be ordered based on, for example, the magnitude of their relative angle to the user's gaze direction.

[0078] In S702, the appearance observation gaze direction calculation unit 404 sorts the observation directions in the manner described above and selects the observation direction with the shortest distance to the user's gaze direction. In the second and subsequent repetitions, the observation directions are selected from the sorted observation directions in order of their shortest distance to the user. Therefore, the direction that forms a small angle with the user's gaze direction is determined as the suitable observation gaze direction for observing the appearance of the target object.

[0079] In S703, the appearance observation gaze direction calculation unit 404 calculates a portion of the environment map reflected on the target object 805 (reflection area).

[0080] here, Figure 9B It is a diagram illustrating the calculation of the reflective area, and Figure 9C This is a partial map of the environment map, representing the reflective area. First, the directions 806 from the gaze direction position 804 to be scanned towards the four corners of the rectangular target object 805 are calculated. Here, j is the subscript representing each corner of the target object 805. Assuming the target object 805 is, for example, a rectangular area, when assuming the vertical length of the display 304 of the information processing device 1 is d and the horizontal length is w, the rectangular area is -w / 4≤x≤w / 4, -d / 4≤y≤d / 4, and z=0. Specifically, it is obtained as shown in the following equation.

[0081] [Equation 4]

[0082]

[0083] Here, (xj, yj) represent the x and y components of the coordinates of four points at the four corners of the target object 805, namely (-w / 4, -d / 4), (-w / 4, d / 4), (w / 4, -d / 4), and (w / 4, d / 4). The directions 807 of the reflections from the mirror are calculated with reference to the average normal direction. Specifically, it is obtained through the following equation.

[0084] [Equation 5]

[0085]

[0086] A portion 808 of the environment map, including four directions 807, is designated as a reflection area. As a method of designation, for example, it is based on the range from the minimum to the maximum value of the polar angle θ of the four directions 807 and from the azimuth angle... The region of the environment map corresponding to the range from the minimum to the maximum value is set as the reflection region.

[0087] In S704, the appearance observation gaze direction calculation unit 404 determines a gaze direction suitable for observing the gloss of the target object from the illumination information. As described above, a gaze direction suitable for observing gloss is a gaze direction in which the light source exists in the specular reflection direction of the gaze and the specular reflection intensity and gloss can be visually identified. That is, the observation direction in which the portion of the illumination (light source) in the illumination information is reflected in the target object is determined as the observation gaze direction suitable for observing the gloss of the target object. In this embodiment, the reflective area is grayscaled to determine the gaze direction. The environment map, which is an array holding 8-bit intensity information for each set of coordinates and for each color signal, is converted into an array holding luminance values ​​with 8-bit information for each set of coordinates.

[0088] The conversion method will be omitted as it is a known technique. The format used during grayscale conversion is an example, and for instance, the luminance value can be preserved as 32-bit floating-point information.

[0089] When there are pixels in the reflective region with a luminance greater than or equal to threshold E1 and at least threshold S1, the viewing direction at that point in time is determined as the gaze direction for observing gloss. Here, threshold E1 is assumed to be, for example, a value obtained by subtracting the standard deviation of the luminance value from the maximum luminance value of the entire area of ​​the environment map, and threshold S1 may be, for example, 10% of the total pixels in the reflective region. S704 is repeated when scanning the viewing direction; however, once the gaze direction for observing gloss has been determined, the determination of the gaze direction for observing gloss is skipped in subsequent scans by setting a flag.

[0090] In S705, the appearance observation gaze direction calculation unit 404 determines a gaze direction suitable for observing the color of the target object. As described above, a gaze direction suitable for observing color is a gaze direction in which there is no light source in the specular reflection direction of the gaze and the diffuse reflection component is more dominant than the specular reflection component. In this embodiment, the reflective area is grayscaled, and the observation direction at that point in time is determined as the gaze direction for observing color when the maximum luminance of the reflective area is less than or equal to a threshold E2. Here, the threshold E2 can be, for example, the first quartile of the luminance of the entire area of ​​the environment map. That is, the observation direction in which the portion of the illumination information in which there is no illumination (light source) is reflected in the target object is determined as the observation gaze direction suitable for observing the color of the target object. When scanning the observation direction, S705 is repeated; however, once the gaze direction for observing color has been determined, the determination of the gaze direction for observing color is skipped in subsequent scans by setting a flag.

[0091] In S706, the scanning is terminated once the gaze direction for observing gloss and the gaze direction for observing color have been determined. If neither has been determined, the process proceeds to S707.

[0092] In S707, the appearance observation gaze direction calculation unit 404 determines whether there are other observation directions to be scanned. If there are other observation directions to be scanned, the process returns to S702. If there are no other observation directions to be scanned, the process ends.

[0093] That's all. Figure 8 The flowchart description, and return to Figure 6 A description of the flowchart.

[0094] Repeat the subsequent S504 to S509 processes until a termination command from the user is received.

[0095] In S505, the user gaze direction acquisition unit 403 acquires the user gaze direction (R,Θ,Φ). A description of this process will be omitted, as it is the same as that in S701.

[0096] In S506, the target object image depiction unit 405 depicts image I1 using a known image-based illumination technique. Image I1 represents the target object 805 through the environment map E. The image I1 is a view taken from the user's viewing direction (R, Θ, Φ) during illumination. The size of image I1 is, for example, the same as the size of the target object 805 used in the calculations in S704 and is 50% of the size of the display 304 of the information processing device 1. Furthermore, the drawing is performed such that the normal to the display 304 coincides with the average normal to the target object 805. At this time, the coordinate system of the user's viewing direction (R, Θ, Φ) coincides with the coordinate system used in the calculations in S704.

[0097] In S507, the appearance observation gaze direction depiction unit 406 depicts image I2, which presents the gaze direction used for observing the appearance and the user's gaze direction. The details of the processing will be described later.

[0098] In S508, the output image synthesis unit 407 generates an image to be displayed on the display 304 by synthesizing the two images I1 and I2.

[0099] here, Figure 10 It is a detailed flowchart of S507, and Figure 11A This is an example of an image to be output by the output image synthesis unit 407. Figure 11A The target object image 1001 shown is image I1 depicted in the target object image depiction unit 405. Image I2 is configured with a three-dimensional hemispherical model 1002, a user gaze direction marker 1003, an appearance observation gaze direction marker 1004, and a region of interest marker 1005. In the following text, reference will be made to... Figure 10 Details regarding the handling of S507 are described.

[0100] First, in S901, the appearance observation gaze direction depiction unit 406 determines whether the gaze direction for observing the appearance has been determined in the aforementioned processes S705 and S704. If it has been determined, the process proceeds to S903. Otherwise, if it has not been determined, the process proceeds to S902.

[0101] In S902, the appearance observation gaze direction depiction unit 406 notifies the user that there is no gaze direction for observing the appearance, thereby prompting the user who wants to observe the appearance to change the tablet's posture or the environment map. Here, Figure 11B This is an example illustration of a warning instruction. Warning instruction 1006 notifies the user. In the example shown, the instruction indicates that a suitable gaze direction for observing the color of the target object has not yet been determined.

[0102] In S903, the user's gaze direction (Θ,Φ) calculated by the appearance observation gaze direction depiction unit 406 is compared with the two appearance confirmation gaze directions. and The relative angle between and Relative angles can be calculated, for example, using the following equation.

[0103] [Equation 6]

[0104]

[0105] In S904, the appearance observation gaze direction depiction unit 406 depicts an image indicating the angle calculated in S903 relative to the user's gaze direction. This image is compared with... Figure 11A The three-dimensional hemispherical model 1002, user gaze direction marker 1003, and appearance observation gaze direction marker 1004 correspond to each other. When the polar angle and azimuth angle of the three-dimensional hemispherical model 1002 relative to the direction from the center of the bottom surface towards the zenith portion are assumed to be T and P, the user gaze direction marker 1003 is displayed on the sphere where T=0 and P=0. Then, the appearance observation gaze direction marker 1004 is displayed on the sphere. and At two points. To see the approximate shape, the surface of the 3D hemispherical model is displayed, for example, as a mesh obtained by dividing the polar angle T and azimuth angle P into fixed intervals.

[0106] In S905, the appearance observation gaze direction depiction unit 406 determines the depiction transparency of the region of interest marker 1005 to be depicted in S906 from the magnitude of the relative angle calculated in S903. The magnitude of the relative angle A is defined, for example, by the following equation.

[0107] [Equation 7]

[0108]

[0109] Here, Δθ and These represent the relative polar angle and azimuth angle between the two gaze directions used to confirm the appearance and the user's gaze direction, respectively. When the size of the relative angle A is large, the rendering transparency of the region of interest marker 1005 is considered to be 0%, and as the size of the relative angle A decreases, the rendering transparency of the region of interest marker 1005 approaches 100%. For example, when A is 90 degrees or more, the transparency is 0%, and when A = 10 degrees or less, the transparency is 100%, and the transparency changes linearly with respect to the value of A.

[0110] As a result, when observing the appearance of the region of interest, the user is not prevented from observing by the region of interest marker 1005, and when searching for the gaze direction for observing the appearance, the user can use the region of interest marker 1005 to determine the region of interest.

[0111] In S906, the appearance observation gaze direction depiction unit 406 depicts the region of interest marker 1005 based on the transparency determined in S905. It is assumed that the region of interest marker 1005 is centered, for example, at the center position of the target object image I1, and its size is 10% of the size of the display 304.

[0112] Thus, a three-dimensional hemispherical model is depicted, in which the zenith portion of the sphere (e.g., 1003) is indicated as the user's gaze direction, and the appropriate viewing gaze direction for observing the appearance of the target object is indicated by a point on the sphere (e.g., 1004) as the relative direction to the user's gaze direction. This allows the user to visually determine how to change their gaze direction to suit the observation of the appearance by looking at this display.

[0113] This ended the... Figure 6 A description of the processing flow.

[0114] <Effects of the Example>

[0115] As described above, in this embodiment, appearance information indicating the appearance of the target object and lighting information (environmental map information) for illuminating the target object are obtained. Then, based on that information, a suitable viewing direction for observing the appearance of the target object (a viewing direction that makes it easy to visually identify the color and gloss of the target object) is calculated, and information indicating this viewing direction is depicted and presented to the user.

[0116] As a result, it becomes easier for users to determine which direction to observe from, and thus, it becomes possible to effectively confirm the appearance of the target object, such as its color and gloss.

[0117] <Variants>

[0118] Figure 6 The processing in S503 is performed after obtaining the environment map and appearance information according to the user's instructions; however, the order of processing can be changed as follows. The gaze direction for observing the appearance is pre-calculated from the environment map and appearance information using the same processing as in S503, and stored in the data storage destination. Then, when the environment map and object information are obtained according to the user's instructions, the corresponding gaze direction for observing the appearance is simultaneously obtained from the data storage destination and presented.

[0119] In S502, an environment map is obtained as lighting information; however, it can be obtained by obtaining the three-dimensional position, direction, and intensity information of the light source as illumination information and then converting them into an environment map by performing a bake process (which is a common CG rendering technique).

[0120] In S505 and S701, the user's gaze direction is obtained by incorporating an RGB camera into the information processing device; however, the gaze direction can be estimated in conjunction with a range sensor, or parallax information obtained from multiple cameras can be used. Alternatively, for example, a camera designed for light other than visible light, such as an infrared camera, can be used. Furthermore, incorporating a camera is unnecessary as long as the position relative to the display can be obtained. A gaze tracking device can also be used to obtain the user's gaze direction.

[0121] Furthermore, the method for calculating the reflection region described in S704 is merely an example, and for instance, the configuration can take into account the effect of changes in the reflection region caused by changes in the normal of the target object by expanding a portion of the environment map according to the changes in the normal of the target object. Additionally, by actually performing ray tracing in the opposite direction relative to each point of the target object, the reflection region considering the normal direction at each position of the target object can be obtained.

[0122] (Second Embodiment)

[0123] In the first embodiment, an example of calculating and presenting a gaze direction suitable for the color and gloss of the observed object has been described. In this embodiment, an example of calculating and presenting a gaze direction suitable for the unevenness of the observed object will be described.

[0124] The appearance, hardware configuration, functional configuration, and processing flow (excluding S503) of the information processing device in the second embodiment will be omitted from the description because they are the same as in the first embodiment. Details regarding the processing of S503 in this embodiment are as follows: Figure 12 The instructions are as follows. References will be made below. Figure 12 The flowchart describes the details of the process.

[0125] <Processing>

[0126] In S1101, the user gaze direction acquisition unit 403 acquires the user's gaze direction.

[0127] The description of this process will be omitted because it is the same as that in S701.

[0128] In S1102, the appearance observation gaze direction calculation unit 404 determines the intensity of the gloss of the target object. In this embodiment, an array is obtained by performing grayscale conversion on the diffuse reflection intensity ρs and the specular reflection intensity ρd, maintaining, for example, 8-bit luminance value information for each set of coordinates. It is determined whether the ratio of the average specular reflection luminance to the average diffuse reflection luminance in the entire area of ​​the target object is less than or equal to a threshold R1. Here, the threshold R1 is, for example, 1.

[0129] Processes S1103 to S1108 are steps for calculating the viewing direction suitable for observing unevenness when the gloss of the target object is strong. As mentioned above, the viewing direction suitable for observing unevenness in this case is the specular reflection direction of a small light source or a light source with sharp edges. When these light sources are present in the reflective region, the reflective region contains many high spatial frequency components. Therefore, the presence or absence of these light sources can be determined by calculating the spatial frequency information of the reflective region.

[0130] Descriptions for S1103, S1104, and S1108 will be omitted because the processing is the same as that for S702, S703, and S707, respectively.

[0131] In S1105, the appearance observation gaze direction calculation unit 404 performs two-dimensional FFT processing on the reflection area and converts it into spatial frequency information.

[0132] In S1106, the appearance observation gaze direction calculation unit 404 determines whether the observation direction is suitable for observing unevenness. When the spatial frequency component of the reflective area includes components that are greater than or equal to threshold D1 or more in absolute value, the observation direction at that time point is determined. The gaze direction is determined for observing unevenness. The threshold F1 is, for example, 10 times the reciprocal of the width of the target object 805 (which is 2 / w), and the threshold D1 is, for example, 10% of all frequency components included in the reflective region 808.

[0133] Therefore, when the specular reflection intensity is greater than or equal to the diffuse reflection intensity, the observation direction in which the portion of the illumination information containing a spatial frequency component greater than or equal to the threshold is reflected into the target object is determined as the observation gaze direction suitable for observing the unevenness of the target object.

[0134] In S1107, if a gaze direction for observing unevenness was determined in S1106, the appearance observation gaze direction calculation unit 404 terminates the scanning of the observation direction. Otherwise, if no gaze direction for observing unevenness was determined, the process proceeds to S1108.

[0135] Meanwhile, steps S1109 to S1111 are steps that calculate the viewing direction suitable for observing unevenness when the gloss of the target object is weak. As described above, when the gloss is weak, the viewing direction suitable for observing unevenness is the direction perpendicular to the light source. However, there may be a situation where the environment map includes multiple light sources. Therefore, in this embodiment, the viewing direction orthogonal to both the direction of the strongest light source included in the environment map and the direction of the integral of other light sources is set as the viewing direction suitable for observing unevenness when the gloss of the target object is weak.

[0136] here, Figure 13 This diagram illustrates a method for calculating the optimal viewing direction for observing unevenness when the gloss of a target object is weak.

[0137] In S1109, the visual observation gaze direction calculation unit 404 will calculate the polar angle θ and azimuth angle of the target object's average normal direction. A portion of the environment map corresponding to 0°≤θ≤90° is converted into a grayscale image. Then, θ and Each is divided into smaller regions 1201 by, for example, dividing each into 10 segments. When using parameters θ1, θ2, ... and The small region 1201 is represented as θ1≤θ≤θ2. At that time, for each small region, calculate the integral value of the luminance of the small region in the direction expressed in the following equation. The vector is 1202.

[0138] [Equation 8]

[0139]

[0140] In S1110, the appearance observation gaze direction calculation unit 404 defines the vector with the largest magnitude among the vectors obtained for each small region as the main light source direction L1. The sum of the vectors for each small region excluding the main light source direction L1 is defined as the secondary light source direction L2.

[0141] In S1111, the appearance observation gaze direction calculation unit 404 selects the gaze direction that is closest to the user's gaze direction (Θ,Φ) from among the gaze directions that are orthogonal (perpendicular) to both the main light source direction L1 and the secondary light source direction L2. Set the gaze direction for observing unevenness.

[0142] Therefore, when the specular reflection intensity is less than the diffuse reflection intensity, the viewing direction perpendicular to the light source included in the illumination information is determined as the viewing direction suitable for observing the unevenness of the target object.

[0143] This ended the... Figure 12 A description of the processing flow.

[0144] <Effects of the Example>

[0145] As described above, the information processing apparatus according to this embodiment calculates the gaze direction from the environmental map information that makes it easy to visually identify the unevenness of the target object and presents it to the user. As a result, the user can effectively confirm the unevenness of the target object.

[0146] <Variants>

[0147] In this embodiment, in S1103, it is determined whether the target object has a glossy appearance based on the ratio of specular reflection intensity to diffuse reflection intensity; however, the determination can be performed using, for example, the value of another function of specular reflection intensity and diffuse reflection intensity based on visual characteristics.

[0148] In addition, in S1105, the frequency conversion technique is not limited to two-dimensional FFT and can be, for example, wavelet transform.

[0149] (Third Embodiment)

[0150] In this embodiment, similar to the first and second embodiments, a tablet-type information terminal is used as the information processing device. Figure 14 This illustrates the usage of this embodiment. A target object 1401, as a three-dimensional shape model endowed with appearance information, is displayed as a fixed object in real-space coordinates, and an image 1402 reproducing the view of the target object 1401 under ambient lighting is displayed. Furthermore, an application is assumed to observe the appearance of the target object 1401 by calculating a direction perpendicular to the surface of the display 304 of the information processing device 1 from the position and posture of the information processing device 1 as the user's gaze direction 1403, and dynamically changing the view in response to dynamic changes in the gaze direction. In this embodiment, a gaze direction suitable for observing unevenness is presented in the application.

[0151] The description of the hardware configuration of the information processing apparatus according to this embodiment will be omitted because it is the same as that in the first embodiment.

[0152] <Function Configuration>

[0153] Figure 15 This diagram illustrates the functional configuration of the information processing apparatus according to this embodiment. In addition to the components described in the first embodiment, the appearance display application 2 of the information processing apparatus 1 according to this embodiment also includes a shape information acquisition unit 409, a partial area designation unit 410, and an appearance viewing mode switching unit 411.

[0154] The shape information acquisition unit 409 acquires shape information representing the three-dimensional shape of the target object. Based on instructions from the user, the appearance viewing mode switching unit 411 can switch from a first display mode for displaying only the target object to a second display mode for presenting a viewing direction suitable for observing the appearance, in addition to the view of the target object. The partial area designation unit 410 designates a partial area on the shape of the target object to be used by the appearance viewing viewing direction calculation unit 404. Descriptions of other components will be omitted as they are the same as those represented by the same numbers in the first embodiment.

[0155] <Processing>

[0156] Figure 16 This is a flowchart illustrating the processes to be performed in the appearance display application 2 of the information processing apparatus 1 according to this embodiment. In the following text, reference will be made to... Figure 16 A description of the details of the processing is provided.

[0157] In S1601, the appearance information acquisition unit 401 acquires the appearance information of the target object. A description of this process will be omitted because it is the same as the process in S501.

[0158] In S1602, the shape information acquisition unit 409 acquires shape information representing the three-dimensional shape of the target object. The shape information is, for example, three-dimensional shape data described in Wavefront OBJ format.

[0159] In S1603, the lighting information acquisition unit 402 acquires the environmental map E based on instructions from the user. As lighting information, the data format is the same as that described in S502; however, in this embodiment, in order to dynamically change the orientation of the information processing device 1, it is also necessary to dynamically rotate the environment map. Therefore, rotation is not performed during this process.

[0160] The processes in S1604 and S1605 are the processes for implementing the aforementioned first display mode for displaying only the view of the target object. In S1604, the user gaze direction acquisition unit 403 acquires the posture of the information processing device 1 via the acceleration sensor 306 and the orientation sensor 307. Figure 3 The negative direction on the z-axis of the coordinate system fixed to the information processing device 1 is set as the gaze direction. The gaze direction and position of the information processing device 1 obtained during the first processing S1604 are defined as the initial gaze direction and initial gaze position, respectively. Although the initial gaze position cannot be obtained directly, the relative position with respect to the initial gaze position can be obtained by integrating the acceleration vector a output by the accelerometer 306.

[0161] In S1605, the target object image depiction unit 405 depicts image I1, which represents a view of the user's gaze direction and gaze position of a target object fixed in real space, illuminated by an environment map fixed in real space. The virtual three-dimensional coordinate system in which the target object is located and the environment map rotate according to the posture of the information processing device 1, and the virtual three-dimensional coordinate system in which the target object is located translates according to its relative position with respect to the initial gaze position, such that the environment map and the target object are displayed as fixed in real space. The environment map is displayed such that the overhead (sky) direction and the north direction of the environment map coincide with the overhead (sky) direction and the north direction in real space. A detailed description of the process will be omitted as it is not the main focus of this invention.

[0162] In S1606, the appearance viewing mode switching unit 411 determines whether to switch to a second display mode for presenting a gaze direction suitable for viewing the appearance based on instructions from the user.

[0163] If a user instruction already exists, then perform the switch. If no user instruction exists, then return to S1604.

[0164] here, Figure 17 An example of an image to be displayed on the display area of ​​the display 304 of the information processing device 1 is illustrated. In the first display mode, an image 1702 representing a view of a target object illuminated by an ambient light map from the current user's viewing direction is displayed in the display area 1701 of the display 304. When the user presses the appearance confirmation button 1703, the display mode is switched to the second display mode.

[0165] In S1607, the partial area designation unit 410 designates a region (region of interest) on the surface of the target object for user confirmation of appearance. The region of interest needs to be designated in order to determine the gaze direction for confirming appearance in the next step, S1608. In this embodiment, when the button is pressed, the periphery of a point (xc, yc, zc) on the surface of the target object displayed at the center of the screen is set as the region of interest. A square region with the same normal direction as the point on the target object surface is set around the point. The length l of the side of this square is set to 1 / 100 of the longest dimension among the height, width, and depth of the entire target object. However, this method for setting the region of interest is an example, and the center of the region of interest can be determined, for example, by the user selecting a partial region on the image of the target object displayed on the display 304. Furthermore, the region of interest can be set using, for example, a rectangle or a circle, and the size of the region of interest can be determined, for example, based on the curvature at the point on the target object surface or the sum of the distances between each point on the target object surface and the region of interest.

[0166] In S1608, the appearance observation gaze direction calculation unit 404 determines a gaze direction suitable for observing unevenness. This process is the same as that in S503 of the second embodiment; however, the calculation is performed by replacing the size of the target object in the second embodiment with the length l of the edge of the region of interest.

[0167] Then, the process from S1609 to S1614 is repeated until a termination command from the user is received.

[0168] The processing of S1610 and S1611 will be omitted from the description because it is the same as that of S1604 and S1605, respectively.

[0169] In S1612, the appearance observation gaze direction depiction unit 406 depicts image I2, which presents the gaze direction for observing appearance. Image I2 is configured to include Figure 17 The model includes at least one of a three-dimensional arrow model 1705 indicating the direction of gaze for observing appearance, a two-dimensional plane model 1706 indicating the direction of gaze for observing appearance, and a region of interest marker 1707. Figure 18 This is a flowchart used to explain the detailed processing flow of S1612. It will be referenced below. Figure 18 A description is provided. Descriptions for S1801, S1802, S1803, and S1804 will be omitted because the processing is the same as that for S901, S902, S903, and S905 described in the first embodiment, respectively.

[0170] In S1805, image I2 is depicted with the depiction transparency determined in S1804. Specifically, the depiction transparency of at least one of the three-dimensional arrow model 1705 and the two-dimensional plane model 1706 is processed as the user's gaze direction, which is the gaze direction of the user of the observation information processing device 1, approaches the observation gaze direction suitable for observing the appearance of the target object.

[0171] Regarding the 3D arrow model 1705, the arrow's starting point vi and ending point vf are determined using the center point (xc, yc, zc) of the region of interest, the length l of the edge of the region of interest, and the viewing direction. This indicates the direction of gaze during external observation. The gaze direction used for observing the appearance, calculated in S1608, is obtained by converting it into polar and azimuth angles in a three-dimensional polar coordinate system (where the polar and azimuth angles relative to the direction directly above (sky) are two angular coordinates, with the center point of the region of interest as the origin). Specifically, the starting and ending points of the arrows are determined in the following equation.

[0172] [Equation 9]

[0173]

[0174] Each set of coordinates represents the x, y, and z coordinates of the target object in the virtual 3D coordinate system.

[0175] Furthermore, regarding the two-dimensional plane model 1706 indicating the gaze direction for observing the appearance, the plane represented by the following equation is depicted as passing through the center point (xc, yc, zc) of the region of interest and perpendicular to the gaze direction for observing the appearance. The plane. Its size is, for example, 20 times the length l of the edge of the region of interest.

[0176] [Equation 10]

[0177]

[0178] Regarding the region of interest marker 1707, a square model representing the region of interest is displayed. A three-dimensional arrow model 1705 indicating the gaze direction for observing the appearance; a two-dimensional plane model 1706; and the region of interest marker 1707 are arranged to be superimposed on the virtual three-dimensional coordinates of the target object and depicted taking into account occlusion relationships with the target object. In this embodiment, where the user's eyes are always directly facing the display 304, this method of presenting the direction for observing the appearance allows for a more intuitive determination of the direction for observing the appearance compared to the method of presenting the direction for observing the appearance in the first embodiment.

[0179] The methods used to determine the model's position and pose (which represent the gaze direction used to observe the appearance) are not limited to those described above, and another position and pose can be used to perform the display, as long as they are used to present the gaze direction used to observe the appearance. This concludes the discussion. Figure 18 The description of the processing flow, and return to Figure 16 A description of the processing flow.

[0180] In S1613, the output image synthesis unit 407 generates an image to be displayed on the display 304 of the information processing device 1 by synthesizing the image I1 depicted by the target object image depiction unit 405 and the image I2 depicted by the appearance observation gaze direction depiction unit 406.

[0181] This ended the... Figure 16 A description of the processing flow.

[0182] <Effects of the Example>

[0183] As described above, the information processing apparatus according to this embodiment calculates the gaze direction of the unevenness of a target object with a three-dimensional shape from environmental map information and presents it to the user. As a result, the user can effectively confirm the appearance of the target object.

[0184] <Variants>

[0185] In this embodiment, a gaze direction suitable for observing unevenness has been presented; however, a gaze direction suitable for observing another element of appearance may be displayed, and multiple gaze directions for observing appearance may be displayed simultaneously or switched according to user instructions.

[0186] According to the present invention, a suitable gaze direction for observing the appearance of a target object can be easily determined. Therefore, users can effectively confirm the appearance of an object without having to change their gaze direction in various ways for observation.

[0187] Other embodiments

[0188] Embodiments of the present invention can also be implemented by a computer of a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a 'non-transitory computer-readable storage medium') to perform one or more functions of the above embodiments and / or includes one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above embodiments, and by a method performed by the computer of the system or apparatus by, for example, reading and executing computer-executable instructions from the storage medium to perform one or more functions of the above embodiments and / or controlling one or more circuits to perform one or more functions of the above embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include separate computers or networks of separate processors to read and execute computer-executable instructions. The computer-executable instructions may be provided to the computer, for example, from a network or storage medium. The storage medium may include, for example, a hard disk, random access memory (RAM), read-only memory (ROM), storage devices for distributed computing systems, optical discs (such as CDs, DVDs, or Blu-ray discs). TM One or more of the following: flash memory devices, memory cards, etc.

[0189] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0190] While the invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims is to be given the broadest interpretation in order to cover all such modifications and equivalent structures and functions.

Claims

1. An information processing apparatus, comprising: An appearance information acquisition unit is configured to acquire appearance information indicating the appearance of a target object. A lighting information acquisition unit, configured to acquire lighting information for illuminating the target object; A direction calculation unit is configured to calculate a viewing direction for observing the appearance of the target object based on the appearance information and the lighting information; as well as A drawing unit configured to draw information indicating the viewing direction. The appearance information includes information on diffuse reflection intensity and specular reflection intensity. Wherein, when the specular reflection intensity is greater than or equal to the diffuse reflection intensity, the direction calculation unit determines the observation direction in which the spatial frequency component of the illumination information, which is greater than or equal to a threshold, is reflected in the target object as the observation direction for observing the unevenness of the target object.

2. The information processing apparatus according to claim 1, wherein, When the specular reflection intensity is less than the diffuse reflection intensity, the direction calculation unit determines the observation direction perpendicular to the light source included in the lighting information as the observation direction for observing the unevenness of the target object.

3. The information processing apparatus according to claim 1, wherein, The direction calculation unit determines the observation direction in which the non-illuminated portion of the lighting information is reflected in the target object as the observation direction for observing the color of the target object.

4. The information processing apparatus according to claim 1, wherein, The direction calculation unit determines the observation direction in which the portion of the illumination information is reflected in the target object as the observation direction for observing the gloss of the target object.

5. The information processing apparatus according to claim 1, wherein, The depiction unit depicts at least one of the following: information indicating the observation direction for observing the gloss of the target object, information indicating the observation direction for observing the unevenness of the target object, and information indicating the observation direction for observing the color of the target object.

6. The information processing apparatus according to claim 1, wherein, If there is no viewing direction for observing the appearance of the target object, the drawing unit displays a warning.

7. The information processing apparatus according to claim 1, further comprising: A user gaze direction acquisition unit is configured to obtain the user gaze direction, which is the gaze direction of a user observing the information processing device.

8. The information processing apparatus according to claim 7, wherein, The direction calculation unit determines the direction with a smaller angle to the user's gaze direction as the observation direction for observing the appearance of the target object.

9. The information processing apparatus according to claim 7, wherein, The depiction unit depicts a three-dimensional hemispherical model, in which... The zenith portion of the spherical surface of the three-dimensional hemispherical model is indicated as the user's gaze direction, and The viewing direction for observing the appearance of the target object is indicated by a point on the sphere, as a direction relative to the user's gaze direction.

10. The information processing apparatus according to claim 1, further comprising: A shape information acquisition unit, configured to acquire shape information representing the shape of the target object; as well as A partial region designation unit is configured to designate a partial region of the shape information.

11. The information processing apparatus according to claim 10, wherein, In response to the information processing device receiving a designation from the user, the partial area designation unit designates the partial area.

12. The information processing apparatus according to claim 10, wherein, The direction calculation unit refers to the partial region to calculate the observation direction for observing the appearance of the target object.

13. The information processing apparatus according to claim 10, wherein, The depiction unit also depicts the partial area.

14. The information processing apparatus according to claim 1, wherein, The drawing unit draws at least one of an arrow model and a planar model, wherein the arrow model indicates the same direction as the viewing direction for observing the appearance of the target object, and the planar model is perpendicular to the viewing direction for observing the appearance of the target object.

15. The information processing apparatus according to claim 14, wherein, The closer the user's gaze direction, which is the direction of gaze of the user observing the information processing device, is to the observation direction used to observe the appearance of the target object, the more the drawing unit reduces the drawing transparency of at least one of the arrow model and the planar model.

16. An information processing apparatus, comprising: An appearance information acquisition unit is configured to acquire appearance information indicating the appearance of a target object. A lighting information acquisition unit, configured to acquire lighting information for illuminating the target object; A direction calculation unit is configured to calculate a viewing direction for observing the appearance of the target object based on the appearance information and the lighting information; as well as A drawing unit configured to draw information indicating the viewing direction. The appearance information includes information on diffuse reflection intensity and specular reflection intensity. Wherein, when the specular reflection intensity is less than the diffuse reflection intensity, the direction calculation unit determines the observation direction perpendicular to the light source included in the lighting information as the observation direction for observing the unevenness of the target object.

17. An information processing apparatus, comprising: An appearance information acquisition unit is configured to acquire appearance information indicating the appearance of a target object. A lighting information acquisition unit, configured to acquire lighting information for illuminating the target object; A direction calculation unit is configured to calculate a viewing direction for observing the appearance of the target object based on the appearance information and the lighting information; as well as A drawing unit configured to draw information indicating the viewing direction. The information processing device further includes a user gaze direction acquisition unit configured to acquire the user's gaze direction, wherein the user gaze direction is the gaze direction of a user observing the information processing device. The depiction unit depicts a three-dimensional hemispherical model, in which... The zenith portion of the spherical surface of the three-dimensional hemispherical model is indicated as the user's gaze direction, and The viewing direction for observing the appearance of the target object is indicated by a point on the sphere, as a direction relative to the user's gaze direction.

18. An information processing apparatus, comprising: An appearance information acquisition unit is configured to acquire appearance information indicating the appearance of a target object. A lighting information acquisition unit, configured to acquire lighting information for illuminating the target object; A direction calculation unit is configured to calculate a viewing direction for observing the appearance of the target object based on the appearance information and the lighting information; as well as A drawing unit configured to draw information indicating the viewing direction. The depiction unit depicts at least one of an arrow model and a planar model, wherein the arrow model indicates the same direction as the observation direction for observing the appearance of the target object, and the planar model is perpendicular to the observation direction for observing the appearance of the target object.

19. A method for controlling an information processing device, comprising: Obtain appearance information that indicates the appearance of the target object; Obtain lighting information for illuminating the target object; Calculate the viewing direction for observing the appearance of the target object based on the appearance information and the lighting information; as well as Describe information indicating the direction of observation. The appearance information includes information on diffuse reflection intensity and specular reflection intensity. Wherein, when the specular reflection intensity is greater than or equal to the diffuse reflection intensity, the observation direction in which the spatial frequency component of the illumination information, which is greater than or equal to a threshold, is reflected in the target object is determined as the observation direction for observing the unevenness of the target object.

20. A computer-readable storage medium storing instructions that, when executed by a computer, cause the computer to perform the method for controlling an information processing apparatus according to claim 19.

Images