Information processing systems, information processing methods, and programs

Abstract

To allow users to easily check the surface texture of an object using a general-purpose device such as their smartphone. [Solution] The information processing system comprises a storage unit and a control unit. The storage unit stores multiple posture information of the user's mobile terminal or the user's viewpoint information relative to the mobile terminal, and multiple texture images of an object from different viewpoints, in association with each other. The control unit acquires posture information or viewpoint information detected by the mobile terminal from the mobile terminal, acquires a texture image corresponding to the posture information or viewpoint information, and outputs the acquired texture image to the mobile terminal.

Description

[Technical Field]

[0001] This invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Traditionally, there have been technologies for representing the surface texture of various objects (such as glossiness and roughness) in images.

[0003] For example, Patent Document 1 discloses a multi-view image processing system that displays a ray-space compressed image consisting of multi-view images taken from multiple different viewpoints of a virtual object placed in a virtual three-dimensional space on a stacked display made up of multiple transmissive panels. The system generates ray-space image data from computational information including pattern image data of a pattern drawn on the surface of the object, gloss characteristic data which is gloss information of the surface of the object, normal image data which is normal vector information of the surface of the object, light source characteristic data which indicates the characteristics of the light emitted from a light source that irradiates the surface of the object, and observation condition information. From this ray-space image data, it generates multiple ray-space compressed images to be displayed on each of the transmissive panels of the stacked display, and displays these ray-space compressed images on each of the multiple transmissive panels. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2019-080133 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, the technology described in Patent Document 1 requires a special stacked display made up of multiple transparent panels to simulate texture.

[0006] The object of the present invention is to provide an information processing system, information processing method, and program that enable users to easily confirm the surface texture of an object using a general-purpose device such as a smartphone. [Means for solving the problem]

[0007] An information processing system according to one embodiment of the present invention comprises a storage unit and a control unit. The storage unit stores multiple posture information of a user's mobile terminal or the user's viewpoint information relative to the mobile terminal, and multiple texture images of an object from different viewpoints, in association with each other. The control unit acquires posture information or viewpoint information detected by the mobile terminal from the mobile terminal, acquires a texture image corresponding to the posture information or viewpoint information, and outputs the acquired texture image to the mobile terminal.

[0008] Information processing methods according to other embodiments of the present invention are: Multiple posture information of the user's mobile device or the user's viewpoint information relative to the mobile device is associated with and stored as multiple texture images of the object from different viewpoints. From the aforementioned mobile terminal, acquire posture information or viewpoint information detected by the mobile terminal, and acquire a texture image corresponding to the posture information or viewpoint information. This includes outputting the acquired texture image to the mobile terminal.

[0009] A program according to yet another embodiment of the present invention is provided for an information processing device. The steps include: storing multiple posture information of the user's mobile device or the user's viewpoint information relative to the mobile device in association with multiple texture images of an object from different viewpoints; The steps include obtaining posture information or viewpoint information detected by the mobile device from the aforementioned mobile device, and obtaining a texture image corresponding to the posture information or viewpoint information, The process involves outputting the acquired texture image to the mobile terminal. [Effects of the Invention]

[0010] According to the information processing system according to one aspect of the present invention, it is possible to easily allow a user to confirm the texture of the surface of an object using a general-purpose device such as the user's smartphone. However, this effect does not limit the present invention.

Brief Description of Drawings

[0011] [Figure 1] It is a diagram showing the configuration of a skin texture information providing system according to an embodiment of the present invention. [Figure 2] It is a diagram showing the hardware configuration of a skin texture information providing server according to an embodiment of the present invention. [Figure 3] It is a diagram showing the configuration of a database included in a skin texture information providing server according to an embodiment of the present invention. [Figure 4] It is a flowchart showing the flow of mapping processing of a skin texture image by a skin texture information providing server according to an embodiment of the present invention. [Figure 5] It is a diagram conceptually showing the mapping processing of a skin texture image by a skin texture information providing server according to an embodiment of the present invention. [Figure 6] It is a flowchart showing the flow of skin texture information providing processing by a skin texture information providing server according to an embodiment of the present invention. [Figure 7] It is a diagram showing an example of display of skin texture information provided by a skin texture information providing server according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] [Configuration of System] As shown in FIG. 1, this system includes a skin texture information providing server 100 on the Internet 50 and a plurality of user terminals 200.

[0014] The skin texture emotion information providing server 100 is a server (information processing device) that executes a service for providing the user's skin texture emotion information. The skin texture emotion information providing server 100 is connected to a plurality of user terminals 200 via the Internet 50.

[0015] The skin texture emotion information providing server 100 receives a skin texture emotion information providing request from the user terminal 200 together with the posture information of the user terminal 200, and transmits skin texture emotion information (skin texture image) indicating the texture of the skin according to the posture information to the user terminal 200.

[0016] In the present embodiment, the skin texture image is, for example, an image showing a state in which a cosmetic is applied to the skin or hair as the object, and is an image in which the texture of the skin surface in the state of applying the cosmetic, such as a glossiness or roughness, is expressed.

[0017] The user terminal 200 (200A, 200B, 200C...) is a terminal used by the user, and is, for example, a smartphone, a mobile phone, a tablet PC (Personal Computer), a notebook PC, a desktop PC, or the like. The user terminal 200 accesses the skin texture emotion information providing server 100, receives a web page on which the skin texture emotion information and the like are posted, and displays it on the screen by a browser or the like. In addition, an application corresponding to the skin texture emotion information providing service is installed in the user terminal 200, and the user terminal 200 may access the skin texture emotion information providing server 100 by the application and display the skin texture emotion information.

[0018] [Hardware Configuration of Skin Texture Emotion Information Providing Server] As shown in FIG. 2, the skin texture emotion information providing server 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface 15, and a bus 14 that connects these to each other.

[0019] The CPU 11 accesses RAM 13 and other memory as needed, performing various calculations and comprehensively controlling each block of the skin texture information server 100. Multiple CPUs 11 may be provided depending on the processing. ROM 12 is a non-volatile memory in which the OS, programs, and firmware such as various parameters to be executed by the CPU 11 are permanently stored. RAM 13 is used as a working area for the CPU 11 and temporarily holds the OS, various running applications, and various data being processed.

[0020] The input / output interface 15 is connected to a display unit 16, an operation reception unit 17, a storage unit 18, a communication unit 19, and the like.

[0021] The display unit 16 is a display device that uses, for example, an LCD (Liquid Crystal Display), an OLED (Organic ElectroLuminescence Display), or a CRT (Cathode Ray Tube).

[0022] The operation reception unit 17 is, for example, a pointing device such as a mouse, a keyboard, a touch panel, or other input device. If the operation reception unit 17 is a touch panel, the touch panel may be integrated with the display unit 16.

[0023] The storage unit 18 is a non-volatile memory such as an HDD (Hard Disk Drive), flash memory (SSD; Solid State Drive), or other solid-state memory. The OS, various applications, and various data are stored in this storage unit 18.

[0024] As will be described later, in this embodiment in particular, the storage unit 18 has a skin texture image database, a mapping information database, and a product information database, in addition to programs such as applications necessary for the skin texture information provision processing described later.

[0025] The communication unit 19 consists of various modules for wireless communication, such as a NIC (Network Interface Card) for Ethernet or a wireless LAN, and is responsible for communication processing with the user terminal 200.

[0026] Although not shown in the diagram, the basic hardware configuration of the user terminal 200 is substantially the same as that of the skin texture information server 100 described above.

[0027] [Database configuration of the skin texture information server]

[0028] As shown in Figure 3, the skin texture information providing server 100 has a skin texture image database 31, a mapping information database 32, and a product information database 33 in its storage unit 18. Note that these databases may be stored in a storage device or server externally connected to the skin texture information providing server 100, rather than in the storage unit 18.

[0029] The skin texture image database 31 stores images representing the appearance of cosmetics applied to skin or hair. These skin texture images are taken from multiple different viewpoints, showing the appearance of each cosmetic product applied to a model's skin or hair, for each product of different genres and brands. As will be described later, each skin texture image is associated with information indicating the viewpoint at the time of shooting.

[0030] The skin texture images are, for example, taken from multiple different viewpoints by the operator of the skin texture information server 100 or its contractor. The skin texture information server 100 may also receive images from the user terminal 200, taken from multiple viewpoints of the user's skin or hair after applying cosmetics, as skin texture images, and store them in the skin texture image database 31 for each cosmetic (cosmetic ID that identifies the cosmetic). In this case, viewpoint information at the time of shooting may also be transmitted from the user terminal 200 to the skin texture information server 100 and used in the mapping process described later.

[0031] The skin texture information server 100 may also receive images from multiple user terminals 200, taken from one or more viewpoints, showing the state of each user applying the same cosmetic product to their skin or hair, and store these images as skin texture images for each cosmetic product (cosmetic product ID that identifies the cosmetic product).

[0032] The mapping information database 32, for each of the skin texture images stored in the skin texture image database 31, uses the mapping process described later to obtain viewpoint information (two angles in a spherical coordinate system that indicate reflected light onto the skin: θ) for the skin or hair to which the cosmetic product has been applied. r ,φ r The system stores information that associates the skin texture images with the user's mobile device (smartphone) orientation information (for example, the two angles Roll(x) and Pitch(y) when the mobile device is rotated on two axes, roll and pitch), grouped for each subject being photographed. When receiving skin texture images to be associated from one or more user devices 200, it is sufficient that multiple skin texture images from different viewpoints received from the user device 200 for a single subject are each associated with different orientation information. In other words, even if the skin texture images received from each individual user device 200 do not cover all viewpoints of a single subject, it is possible to associate all viewpoints with the above orientation information by combining skin texture images from different viewpoints received from multiple user devices 200.

[0033] The product information database 33 stores information about cosmetics as products to be applied to the user's skin or hair in texture images, namely the manufacturer name, product name (brand name), price, cosmetic category (e.g., serum, lotion, emulsion, cleansing product, facial wash, makeup base, foundation, sunscreen, mask, cream, blush, lipstick, lip gloss, eyeshadow, shampoo, conditioner, hair oil, hair styling product, etc.), product photograph, expected effects, price, etc., associated with the above cosmetic ID.

[0034] The "texture" represented in the above skin texture image can be categorized as follows: • Smoothness: The surface appears smooth. • Roughness: The surface is uneven. • Glossiness: Has the property of reflecting light. • Matte finish: Absorbs light and reflects less. • Transparency: Clear or nearly transparent. • Moisturizing feel: Appears to contain moisture. • Dryness: It feels dry and lacks moisture.

[0035] The following are examples of cosmetics that can produce these textures. • Smoothness: Serum, lotion, emulsion, foundation • Texture: Facial cleanser, blush (for a rough finish or a matte, slightly rough texture) • Shine finish: Lipstick, lip gloss, foundation, eyeshadow (including those with shimmering or glossy finishes such as pearls), sunscreen • Matte finish: Foundation, eyeshadow, lipstick, blush • Transparency: Toner, serum, foundation (such as one that provides light coverage and a natural finish), facial cleanser • Moisturizing effect: Cream (mainly products that claim to be moisturizing), lotion (mainly products that claim to be moisturizing), serum (mainly products that claim to be moisturizing) • Dryness: Facial cleansers (those that leave the skin feeling refreshed but also make fine lines more noticeable due to dryness), sunscreen

[0036] These databases are referenced and used as needed in the skin texture image mapping process and skin texture image provision process by the skin texture information provision server 100, which will be described later.

[0037] [Operation of the skin texture information server] Next, the operation of the skin texture information server 100 configured as described above will be explained. This operation is performed through the cooperation of the hardware of the skin texture information server 100, such as the CPU 11 and communication unit 19, and the software stored in the storage unit 18. For convenience, in the following explanation, the CPU 11 will be considered the main operator.

[0038] (Mapping process of skin texture images by a skin texture information server) Figure 4 is a flowchart showing the flow of the skin texture image mapping process performed by the skin texture information server 100. Figure 5 is a conceptual diagram illustrating this mapping process.

[0039] As shown in Figure 4, the CPU 11 first obtains the skin texture image to be mapped from the skin texture image database 31 (step 41).

[0040] Next, the CPU 11 maps the skin texture image data with the posture information of the mobile device (step 42).

[0041] As shown in Figure 5(A), the reflected light from an object (the user's viewpoint relative to the object) is in spherical coordinates with an incident direction L (two angles: θ). i ,φ i ) and the observed direction V (two angles: θ) r ,φ r ) is expressed as follows. Also, as shown in Figure (B), the orientation information of the user terminal 200 is indicated by the rotation angles of three rotation axes: roll (x), pitch (y), and yaw (z). As shown in Figure (C), the CPU 11 uses two angles (θ) which are viewpoint information of the skin texture image I. r ,φ r ) is then associated with two angles, Roll(x) and Pitch(y), which are the orientation information of the user terminal 200 (mapping process).

[0042] The CPU 11 then stores the mapping information in the mapping information database 32. The CPU 11 repeats the above process for all skin texture images from different viewpoints of the same object being photographed, and then repeats the process for other objects being photographed.

[0043] (Skin texture information processing by the skin texture information server) Next, we will explain the process of providing skin texture information using the mapping information generated as described above. Figure 6 is a flowchart showing the flow of the process of providing this skin texture information.

[0044] As shown in the figure, the CPU 11 first determines whether or not it has received a request for skin texture information from the user terminal 200 (step 61). This request is sent, for example, via a skin texture information provision application installed on the user terminal 200.

[0045] Specifically, the request is transmitted from the user terminal 200 as a request to view a texture image of a specific cosmetic product applied to the skin or hair. The request includes a cosmetic product ID to identify the cosmetic product applied to the skin or hair, and orientation information of the user terminal 200 (two angles: Roll (x) and Pitch (y)). This orientation information is a value obtained by detection means such as an accelerometer or gyroscope in the user terminal 200.

[0046] If the CPU 11 determines that it has received the above-mentioned request for skin texture information, it obtains the orientation information of the user terminal 200 (two angles: Roll (x) and Pitch (y)) included in the request (step 62).

[0047] Next, the CPU 11 extracts the skin texture image corresponding to the cosmetic ID and posture information from the mapping information database 32 (step 63).

[0048] The CPU 11 then sends the extracted skin texture image to the user terminal 200 (step 64).

[0049] Next, the CPU 11 determines whether or not the user terminal 200 has instructed it to terminate the process of providing skin texture information (step 65).

[0050] If the CPU determines that it has been instructed to terminate the above process (Yes), the CPU 11 terminates the process. On the other hand, if it determines that it has not been instructed to terminate the process (No), the CPU 11 returns to step 62 above, obtains the orientation information of the user terminal 200 from the user terminal 200, and repeats the subsequent processes.

[0051] Figure 7 shows an example of how skin texture information provided by the skin texture information server 100 is displayed. This skin texture information is provided, for example, when a user requests to view product information of cosmetics of interest from a user terminal 200 via the application.

[0052] For example, as shown in Figure (A), on the application's display screen, options 71 for skincare products, makeup products, and fragrance products are displayed as choices for the user to select products of interest, as shown in Figure (A). Suppose the user selects the makeup product option 71 and presses the continue button 72 at the bottom of the screen.

[0053] In practice, users further select specific product categories from the available makeup products. For example, if lip products are selected by the user, as shown in Figure (B), photographs showing multiple types of lip cosmetics with different formulations, such as lipstick, lip gloss, and lip color, applied to the user's skin are displayed for comparison as skin texture images 73 (73a, 73b, 73c). By comparing these skin texture images, the user can understand the differences in texture of each product. The user can also switch the color of the skin texture images 73 using the color icon 74.

[0054] Specifically, in skin texture image 73a, which shows the application of lipstick, specular and diffuse reflections are present, resulting in a matte texture. In skin texture image 73b, which shows the application of lip gloss, specular reflections are present, as well as the luster of pearls in the formulation. In skin texture image 73c, which shows the application of lip color, a glossy and shiny appearance due to specular reflections can be observed.

[0055] Next, when the user selects a specific cosmetic product and presses the details display button 75, as shown in Figure (C), the product name, product photo, price, etc. are retrieved from the product information database 33 and displayed as product recommendation information 76. When the user presses the checkout button 77 at the bottom, payment is executed. In this way, the user can experience the differences in texture of cosmetics firsthand and make an informed purchase without having to try them in a store.

[0056] As described above, according to this embodiment, the skin texture information providing server 100 allows the user to easily check the texture of their skin or hair when cosmetics are applied, using a general-purpose user terminal 200 such as the user's smartphone.

[0057] Furthermore, in photogrammetry using images from multiple viewpoints of an object, similar to this embodiment, the three-dimensional shape of the object can be estimated from these multiple images and converted into computer graphics, thereby adding texture. However, in order to reproduce the texture of an object, it is necessary to define the spectral reflectance characteristics (BRDF (Bidirectional Reflectance Distribution Function), BSSRDF (Bidirectional Scattering Surface Reflectance Distribution Function)) of the object. In particular, if the object is skin or cosmetics applied to the skin, even if spectral reflectance characteristics can be defined, complex textures such as glossiness and transparency, which depend on the viewpoint, cannot be reproduced. However, according to this embodiment, by associating skin texture images from multiple viewpoints with the posture information of the user terminal 200, it is possible to easily confirm the complex texture of the cosmetics applied to the skin, which is important for the user to evaluate cosmetics, without defining the spectral reflectance characteristics as described above.

[0058] [Differentiation] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the present invention.

[0059] In the above embodiment, the skin texture information providing server 100 stored images showing the state of applying cosmetics to the skin or hair of a single model, mapped to the posture information of the user terminal 200, as skin texture images. Alternatively, the skin texture information providing server 100 may map multiple types of texture images showing the state of applying cosmetics to multiple skin or hair of different states (types) to the posture information of the user terminal 200, and output these multiple types of skin texture images to the user terminal 200 so that they can be compared for each different state of skin or hair. Different skin states may be, for example, oily skin (high moisture and sebum levels), dry skin (low moisture and sebum levels), combination skin (low moisture and high sebum levels), normal skin (high moisture and low sebum levels), etc., or they may be differences in skin texture, pore size, color, etc. Furthermore, the skin texture images in this case may be images of real skin of different states taken from different viewpoints, or they may be simulation images that virtually represent different skin states using computer graphics or the like. This allows users to select and display skin texture images that match their own skin condition, enabling them to more accurately understand the texture of the applied cosmetics.

[0060] In the above embodiment, skin texture images were stored by mapping images taken in the same environment with the cosmetic product applied to the user terminal 200 to the user terminal 200's posture information. Alternatively, the skin texture information providing server 100 may map multiple types of texture images taken in multiple different environments with the same cosmetic product applied to the user terminal 200's posture information and output these multiple types of skin texture images to the user terminal 200 so that they can be compared for each different environment. Different environments include, but are not limited to, differences between indoors and outdoors, differences in illumination, time of day (daytime / nighttime), temperature, etc. Furthermore, the skin texture images in this case may also be created by reproducing different environments through simulation. This allows the user to select and display a skin texture image that corresponds to the environment in which they use the cosmetic product, thereby more accurately understanding the texture of the cosmetic product applied under each environment.

[0061] In the above-described embodiment, the skin texture information providing server 100 stored the posture information acquired by detection means such as an acceleration sensor or a gyro sensor of the user terminal 200 in association with a plurality of texture images with different viewpoints. Instead of the posture information, the skin texture information providing server 100 may store the user's viewpoint (line of sight) information (two angles indicating the above observation direction V: θ r , φ r ) in association with a plurality of texture images. Then, the skin texture information providing server 100 may acquire the user's viewpoint information from the user terminal 200, acquire the texture image corresponding to the viewpoint information, and display it on the user terminal 200. The viewpoint information can be acquired, for example, by using an eye-tracking application (a technology for detecting the user's line of sight from information such as the position of the pupil, the line-of-sight vectors of both eyes, and the fixation point) mounted on the user terminal 200. Thereby, the user can confirm the texture of the state where the cosmetic is applied to the skin or hair by changing the viewpoint instead of moving the user terminal 200.

[0062] Also, as the above-described viewpoint information, the skin texture information providing server 100 may store the finger movement values (x, y) (slider information) in a slider operation where the user moves a finger, for example, on the screen of the user terminal 200, in association with a plurality of texture images in a state where an object is rotated about two axes, for example, Roll(x) and Pitch(y), according to the movement values. Then, each time the skin texture information providing server 100 acquires the slider information (x, y) of the user detected on the screen from the user terminal 200, it may acquire the texture image corresponding to the slider information and display it on the user terminal 200. Thereby, the user can confirm the texture of the state where the cosmetic is applied to the skin or hair by sliding a finger on the user terminal 200.

[0063] In the embodiments described above, the skin texture information providing server 100 displayed images showing the state of cosmetics applied to skin or hair as texture images, but the display targets are not limited to these. For example, any object whose texture may be of interest to the user can be adopted as a texture image of the present invention, such as food, fashion products (clothing, accessories, etc.), interior products (furniture, curtains, rugs, etc.), building materials (tiles, flooring, wallpaper, etc.), automobiles (exterior, interior, etc.), natural objects (stone, wood, plants, etc.), and works of art (paintings, sculptures, etc.).

[0064] In the embodiment described above, only one skin texture information providing server 100 is shown, but the processing performed by the skin texture information providing server 100 may be distributed and executed across multiple servers. For example, the texture image mapping process and the process of providing the texture image to the user terminal 200 may be performed on separate servers.

[0065] Of the inventions described in the claims of this application, the invention described as "information processing method" is one in which each step is performed automatically by at least one device such as a computer through information processing by software, and not by a human using a computer or other device. In other words, the "information processing method" is an information processing method using computer software, and not a method in which a human operates a computer as a calculating tool. [Explanation of Symbols]

[0066] 11…CPU 18...Storage section 19… Communications Department 31…Texture Image Information Database 32…Mapping Information Database 33…Product Information Database 73…Texture image 100...Skin texture information server 200... User terminal

Claims

1. A storage unit that stores multiple posture information of the user's mobile device or the user's viewpoint information relative to the mobile device, and multiple texture images of an object from different viewpoints, From the aforementioned mobile terminal, acquire posture information or viewpoint information detected by the mobile terminal, and acquire a texture image corresponding to the posture information or viewpoint information. The acquired texture image is output to the mobile device. control unit An information processing system equipped with the following features.

2. The control unit receives images of the object taken by the user from multiple viewpoints from the mobile terminal as texture images and associates them with the posture information or viewpoint information. The information processing system according to claim 1.

3. The control unit receives images taken by multiple users of the same object from their mobile devices, each of which is used as a texture image. The information processing system according to claim 1 or 2.

4. The control unit acquires an image as the texture image showing the state in which the cosmetic is applied to the target object, which is the skin or hair, and associates it with the posture information or the viewpoint information. The information processing system according to claim 1 or 2.

5. The control unit associates multiple texture images showing the state of applying multiple different cosmetics to the skin or hair with the posture information or viewpoint information, and outputs these multiple texture images to the mobile terminal in a way that allows comparison for each of the different cosmetics. The information processing system according to claim 4.

6. The control unit associates multiple texture images, each showing the state of applying the cosmetic to multiple skin or hair in different conditions, with the posture information or viewpoint information, and outputs these multiple texture images to the mobile terminal in a way that allows comparison for each different state of skin or hair. The information processing system according to claim 4.

7. The control unit associates multiple texture images, taken in different environments with the same cosmetic product applied to the skin or hair, with the posture information or viewpoint information, and outputs these multiple texture images to the mobile terminal in a way that allows comparison for each of the different environments. The information processing system according to claim 4.

8. Multiple posture information of the user's mobile device or the user's viewpoint information relative to the mobile device is associated with and stored as multiple texture images of the object from different viewpoints. From the aforementioned mobile terminal, acquire posture information or viewpoint information detected by the mobile terminal, and acquire a texture image corresponding to the posture information or viewpoint information. The acquired texture image is output to the mobile device. Information processing methods.

9. In an information processing device, The steps include: storing multiple posture information of the user's mobile device or the user's viewpoint information relative to the mobile device in association with multiple texture images of an object from different viewpoints; The steps include obtaining posture information or viewpoint information detected by the mobile device from the aforementioned mobile device, and obtaining a texture image corresponding to the posture information or viewpoint information, The steps include outputting the acquired texture image to the mobile terminal. A program that executes the command.

Images