Information processing device, information processing method, and program

Abstract

This information processing device is provided with: a means for acquiring a user moving image including a beautifying action of a user on the hand of each beautifying target site; a means for comparing the demonstration operation and the beauty operation to determine an operation difference between the demonstration operation and the beauty operation, the operation difference including an operation difference relating to the position of the beauty operation, i.e., a position difference, and an operation difference relating to the speed of the beauty operation, i.e., a speed difference; and a means for generating navigation information corresponding to the difference in motion of each of the beautifying target sites.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method, and a program. Background Art

[0002] Due to digitalization in recent years, value propositions beyond beauty products have become increasingly important for customers of beauty products (skin care products and makeup products).

[0003] As a value other than beauty products, there is customer experience. In particular, it is important for customers who have purchased beauty products to obtain customer experience in which they perform appropriate beauty actions (such as skin care or makeup) by using the beauty products correctly. Therefore, there is a known technology that provides suggestions for skin care and makeup.

[0004] For example, Japanese Patent Application Laid-Open No. 2021-077218 discloses a technology for guiding a user to perform actions according to a demonstration action. Summary of the invention

[0005] Problems to be solved by the invention

[0006] However, Japanese Patent Application Laid-Open No. 2021-077218 does not consider beauty actions for the user's face (facial area), so it is not attractive enough to customers who are interested in beauty.

[0007] An object of the present invention is to provide an incentive to a customer who is interested in beauty to continue his or her beauty activities, thereby motivating the customer to make the beauty activities a daily routine.

[0008] Technical solutions to solve problems

[0009] A technical solution of the present invention is an information processing device, comprising: a unit for obtaining a user dynamic image including the user's hand beauty movements on each beauty object part; a unit for determining the action difference (difference, distinction) between the demonstration action and the beauty action by comparing the demonstration action with the beauty action, the action difference including the action difference related to the position of the beauty action, i.e., the position difference, and the action difference related to the speed of the beauty action, i.e., the speed difference; and a unit for generating navigation (navigation, guidance) information corresponding to the action difference for each of the beauty object parts. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0011] Figure 2 yes Figure 1 Functional block diagram of an information processing system.

[0012] Figure 3 This is an explanatory diagram of the outline of this embodiment.

[0013] Figure 4 It is a diagram showing the data structure of the user database according to the present embodiment.

[0014] Figure 5 It is a diagram showing the data structure of the user log database according to the present embodiment.

[0015] Figure 6 This is a timing chart of the processing in this embodiment.

[0016] Figure 7 It means in Figure 6 Figure 2 is an example of a screen displayed during information processing.

[0017] Figure 8 It means in Figure 6 Figure 2 is an example of a screen displayed during information processing.

[0018] Fig. 9 This is an explanatory diagram of the outline of Modification Example 1.

[0019] Fig.10 This is an explanatory diagram of the outline of Modification Example 2.

[0020] Fig.11 This is an explanatory diagram of the outline of Modification Example 3.

[0021] Fig.12 This is an explanatory diagram of the outline of Modification Example 4.

[0022] Fig.13 This is a timing diagram of information processing in variant example 4.

[0023] Fig.14 It means in Fig.13 Figure 2 is an example of a screen displayed during information processing.

[0024] Fig.15 It means in Fig.13 Figure 2 is an example of a screen displayed during information processing.

[0025] Fig.16 It means in Fig.13 Figure 2 is an example of a screen displayed during information processing.

[0026] Fig.17 This is an explanatory diagram of the outline of Modification Example 5.

[0027] Fig.18 This is a timing diagram of information processing in variant example 5.

[0028] Fig.19 yes Fig.17An illustration of a scenario.

[0029] Fig. 20 It means in Fig.18 Figure 2 is an example of a screen displayed during information processing.

[0030] Fig.21 It means in Fig.18 Figure 2 is an example of a screen displayed during information processing.

[0031] Fig. 22 This is an explanatory diagram of the outline of Modification Example 6.

[0032] Fig.23 This is a timing diagram of information processing in variant example 6.

[0033] Fig.24 It is an explanatory diagram of the expression evaluation model (evaluation of the degree of smile) of Modification Example 6.

[0034] Fig.25 It is an explanatory diagram of the expression evaluation model (evaluation of the degree of seriousness) of Modification Example 6.

[0035] Fig.26 This is an explanatory diagram of the outline of Modification Example 7. DETAILED DESCRIPTION

[0036] Hereinafter, one embodiment of the present invention will be described in detail based on the drawings. In the drawings for describing the embodiments, the same components are denoted by the same reference numerals in principle, and their repeated descriptions are omitted.

[0037] The terms in this embodiment are used as defined below.

[0038] A "beauty action" is a hand action of the user, and is an action performed on the face for the user's face care. The beauty action includes a bare-hand action and an action using a beauty tool (e.g., a flat cotton pad, a triangular sponge, or an applicator). The beauty action is, for example, at least one of the following actions.

[0039] Massage action (for example, acupressure (pressure on acupoints))

[0040] Skin care actions

[0041] Makeup action

[0042] ・Sunscreen action (for example, the action of applying sunscreen)

[0043] The “user dynamic image” is a dynamic image of beauty movements of the hands on various parts of the face.

[0044] The “user position” is the relative position of the hand with respect to each part of the face in each frame of the user's dynamic image.

[0045] The “user speed” is the displacement amount of the user's position between frames of the user's moving image.

[0046] (1) Composition of information processing system

[0047] The configuration of the information processing system will be described. Figure 1 It is a block diagram showing the configuration of the information processing system according to the present embodiment. Figure 2 yes Figure 1 Functional block diagram of an information processing system.

[0048] like Figure 1 As shown, the information processing system 1 includes a client device 10 , a wearable sensor 20 , and a server 30 .

[0049] The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.

[0050] The wearable sensor 20 is communicably connected to the client device 10 .

[0051] The client device 10 is a computer (an example of an “information processing device”) that sends a request to the server 30. The client device 10 is, for example, a smart mirror, a smartphone, a tablet terminal, or a personal computer.

[0052] The wearable sensor 20 is a sensor that can be worn by a user. The wearable sensor measures the value of at least one of the following information, for example, and transmits the measurement result to the client device 10.

[0053] Biological information (for example, body temperature, heart rate, and blood flow)

[0054] Acceleration information (for example, information related to hand acceleration)

[0055] Pressing force information (for example, information related to the pressing force applied by the hand to the face)

[0056] Information related to the three-axis rotation direction of the hand

[0057] Information related to myoelectricity

[0058] The server 30 is a computer (an example of an “information processing device”) that provides the client device 10 with a reply (response) corresponding to a request sent from the client device 10. The server 30 is, for example, a WEB (website) server.

[0059] (1-1) Configuration of client device

[0060] The configuration of the client device 10 will be described.

[0061] like Figure 2 As shown, the client device 10 includes a storage device 11 , a processor 12 , an input / output interface 13 , a communication interface 14 , and a camera 15 .

[0062] The storage device 11 is configured to store programs and data and is, for example, a combination of a ROM (Read Only Memory), a RAM (Random Access Memory), and a storage (for example, a flash memory or a hard disk).

[0063] The program includes, for example, the following programs.

[0064] ·OS (Operating System) Program

[0065] · APPs (applications) that perform information processing (e.g., web browsers)

[0066] The data includes, for example, the following data.

[0067] · Databases referenced in information processing

[0068] Data obtained by executing information processing (that is, the results of executing information processing)

[0069] The processor 12 is configured to realize the functions of the client device 10 by activating a program stored in the storage device 11. The processor 12 is, for example, a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof.

[0070] The input / output interface 13 is configured to obtain a user's instruction from an input device connected to the client apparatus 10 and output information to an output device connected to the client apparatus 10 .

[0071] The input device is, for example, a keyboard, a pointing (pointing, fixed) device, a touch panel, or a combination thereof.

[0072] The output device is, for example, a display, a speaker, or a combination thereof.

[0073] The communication interface 14 is configured to control communication between the client device 10 and the server 30 .

[0074] The camera 15 is configured to obtain a moving image including the user's hand beauty movements on various parts of the face (hereinafter referred to as "user moving image"). The camera 15 includes, for example, at least one of the following.

[0075] Image sensor

[0076] Thermal imaging camera (thermocamera)

[0077] (1-2) Server Configuration

[0078] The configuration of the server 30 will be described.

[0079] like Figure 2 As shown, the server 30 includes a storage device 31 , a processor 32 , an input / output interface 33 , and a communication interface 34 .

[0080] The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a ROM, a RAM, and a storage (for example, a flash memory or a hard disk).

[0081] The program includes, for example, the following programs.

[0082] ·OS program

[0083] ・APP programs that perform information processing

[0084] The data includes, for example, the following data.

[0085] · Databases referenced in information processing

[0086] ·The results of information processing

[0087] The processor 32 is configured to realize the functions of the server 30 by activating a program stored in the storage device 31. The processor 32 is, for example, a CPU, an ASIC, an FPGA, or a combination thereof.

[0088] The input / output interface 33 is configured to obtain a user's instruction from an input device connected to the server 30 and output information to an output device connected to the server 30 .

[0089] The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.

[0090] An output device is, for example, a display.

[0091] The communication interface 34 is configured to control communication between the server 30 and the client device 10 .

[0092] (2) Overview of Implementation Methods

[0093] The outline of this embodiment will be described. Figure 3 This is an explanatory diagram of the outline of this embodiment.

[0094] like Figure 3 As shown, by analyzing the user's dynamic image of the beauty action, the user's position P(t) and the user's speed V(t) in each frame F(t) of the user's dynamic image are determined. t is an example of information for identifying a frame.

[0095] By inputting the user position P(t) and the user speed V(t) into the demonstration model M(Pm(t), Vm(t)), the motion difference between the user position P(t) and the demonstration position Pm(t) (hereinafter referred to as the "position difference") ΔP(t), and the motion difference between the user speed V(t) and the demonstration speed Vm(t) (hereinafter referred to as the "speed difference") ΔV(t) are obtained.

[0096] By inputting the position difference ΔP(t) and the speed difference ΔV(t) into the navigation model NM(ΔP(t), ΔV(t)), navigation information is obtained.

[0097] The navigation information is prompted to the user.

[0098] For example, when the beauty action is a massage action, navigation information of a cheek massage action (for example, an action of pressing the cheek with one's own fingers or an action of pressing the cheek with an acupressure tool) is generated.

[0099] For example, when the beauty action is a skin care action, navigation information for an action of applying lotion, beauty serum, cream, or emulsion is generated.

[0100] For example, in the case where the beauty action is a makeup action, navigation information for an action of using foundation, primer, blush, eyebrow pencil, eye shadow, mascara, or lipstick is generated.

[0101] For example, when the beauty action is a sunscreen action, guidance information for an action of applying UV (Ultra Violet) agent (for example, liquid, powder, or spray) is generated.

[0102] (3) Database

[0103] The database of this embodiment is described. The following database is stored in the storage device 31.

[0104] (3-1) User Database

[0105] The user database of this embodiment is described. Figure 4It is a diagram showing the data structure of the user database according to the present embodiment.

[0106] Figure 4 User information is stored in the user database.

[0107] The user database includes a "user ID" field, a "user name" field, a "user attribute" field, a "user preference (favorite, first choice)" field, and a "skin problem (trouble)" field. Each field is associated with each other.

[0108] The user identification information is stored in the "User ID" field. The user identification information is information for identifying the user.

[0109] The user name information is stored in the "user name" field. The user name information is information related to the name of the user.

[0110] The user attribute information is stored in the "user attribute" field. The user attribute information is information related to the attributes of the user. The "user attribute" field includes a "gender" field and an "age" field.

[0111] The gender information is stored in the "gender" field. The gender information is information related to the gender of the user.

[0112] The age information is stored in the "age" field. The age information is information related to the age of the user.

[0113] The user preference information is stored in the "user preference" field. The user preference information is information related to the user's preference. The "user preference" field includes the "appearance (facial features)" field, the "color tone" field, the "item" field, the "scene" field, and the "feeling of use" field.

[0114] The appearance information is stored in the "appearance" field. The appearance information is information related to the appearance preferred by the user.

[0115] The "hue" field stores hue information, which is information related to the hue preferred by the user.

[0116] The "single product" field stores single product information, which is information related to the single product preferred by the user.

[0117] The scene information is stored in the "scene" field. The scene information is information related to the scene preferred by the user.

[0118] The skin problem information is stored in the "skin problem" field. The skin problem information is information related to the skin problem that the user has. The skin problem includes, for example, at least one of the following.

[0119] Rough skin

[0120] ·dry

[0121] Pigmentation

[0122] ·relaxation

[0123] ·elasticity

[0124] Dark

[0125] The "usage feeling" field stores the usage feeling information. The usage feeling information is information related to the usage feeling of the single item. The usage feeling of the single item is, for example, at least one of the following feelings.

[0126] · Sensation perceived by touch (for example, moisture or freshness)

[0127] · Feelings during or after application (e.g., easy to spread, easy to penetrate, or moisturizing)

[0128] (3-2) User Log Database

[0129] The user log database of this embodiment is described. Figure 5 It is a diagram showing the data structure of the user log database according to the present embodiment.

[0130] Figure 5 The user log database stores user log information.

[0131] The user database includes a "user log ID" field, a "timestamp" field, a "user dynamic image" field, a "motion track" field, and an "action score" field. Each field is associated with each other.

[0132] The user log database is associated with user identification information.

[0133] The user log identification information is stored in the "user log ID" field. The user log identification information is information for identifying the user log.

[0134] The "time stamp" field stores time stamp information, which is information related to the date and time corresponding to the user log.

[0135] The “user dynamic image” field stores the user dynamic image acquired by the camera 15 .

[0136] The "motion trajectory" field stores motion trajectory information. The motion trajectory information is information related to the trajectory of the beauty motion.

[0137] The action score is stored in the "action score" field. The action score is the score of the beauty action performed by the user.

[0138] (4) Information processing

[0139] The information processing in this embodiment is described. Figure 6 This is a timing chart of the processing in this embodiment. Figure 7 It means in Figure 6 Figure 2 is an example of a screen displayed during information processing. Figure 8 It means in Figure 6 Figure 2 is an example of a screen displayed during information processing.

[0140] Figure 6 The start trigger of the process is that the user of the client device 10 gives a user instruction to start the navigation application (applicant) installed in the client device 10. The user identification information of the user is registered in the navigation application.

[0141] like Figure 6 As shown, the client device 10 obtains the user dynamic image (S1110).

[0142] Specifically, the processor 12 converts the picture P0 ( Figure 7 ) is displayed on the monitor.

[0143] The screen P0 includes operation objects B0 to B2.

[0144] The operation object B0 is an object that receives a user instruction to display guide information. The guide information is information related to guidance on how to use the navigation application. The guide information is, for example, at least one of the following.

[0145] Still images

[0146] Dynamic image (video)

[0147] ·voice

[0148] ·text

[0149] When the user operates the operation object B0 , the processor 12 displays the guide information pre-stored in the storage device 11 on the display.

[0150] The operation object B1 is an object that receives a user instruction to start a massage mode. The massage mode is a mode that provides guidance on a beauty operation for a beauty target area using the hand.

[0151] The operation object B2 is an object for accepting a user instruction to start a facial exercise mode. The facial exercise mode is a mode for providing guidance on beauty exercises for the face without using the hands.

[0152] When the user operates the operation object B1, the processor 12 displays the screen P1110 ( Figure 7 ) is displayed on the monitor.

[0153] Screen P1110 includes a display object A1110 and an operation object B1110.

[0154] A guide is displayed in the display object A1110.

[0155] The operation object B1110 is an object that receives a user instruction to start navigation.

[0156] When the user aligns the position of his / her face with the guide of the display object A1110 and operates the operation object B1110, the camera 15 starts capturing a dynamic image of the user.

[0157] The processor 12 acquires the user dynamic image captured by the camera 15. When the user operates the operation object B 1110 and then performs a beauty action, the user dynamic image includes an image of the beauty action.

[0158] After step S1110 , the client device 10 performs image analysis ( S1111 ).

[0159] Specifically, the processor 12 analyzes the user's dynamic image and identifies the feature points of the user's part that is the target of the beauty action (hereinafter referred to as the "beauty target part") and the feature points of the user's hand (for example, fingertips) for each frame constituting the user's dynamic image.

[0160] The beauty target site includes, for example, at least one of the following sites.

[0161] ·head

[0162] · Parts of the face (for example, eyebrows, eyes, nose, mouth, and cheeks)

[0163] Neck

[0164] ·jaw

[0165] ·ear

[0166] ·shoulder

[0167] The processor 12 specifies a region of the user's face (hereinafter referred to as a “target region”) based on the coordinates of each beauty target part of the user for each frame.

[0168] The processor 12 specifies, in the frame F(t), a position of the user's hand that is included in the target area as the user position P(t).

[0169] The processor 12 calculates the user velocity V(t) based on the displacement amount (P(t+1)-P(t)) of the user position between the frames F(t) and F(t+1).

[0170] After step S1111 , the client device 10 performs evaluation of the action ( S1112 ).

[0171] Specifically, the storage device 11 stores a model M. A model action is described in the model M. The model action is defined by a model position Pm(t) and a model speed Vm(t). When the model position Pm(t1) in frame t1 and the model position P(t2) in frame t2 indicate the same position, it means that the position of the beauty action is stationary between frames t1 and t2.

[0172] The processor 12 refers to the model M and calculates the difference between the user position P(t) and the model position Pm(t), ie, the position difference ΔP(t).

[0173] The processor 12 refers to the exemplary model M and calculates the speed difference ΔV(t) which is the difference between the user speed V(t) and the exemplary speed Vm(t).

[0174] The storage device 11 stores a time series scoring model. The time series scoring model describes the correlation between the evaluation results of the action (for example, the position difference ΔP(t) and the speed difference ΔV(t)) and the action score at a time point (hereinafter referred to as "time series action score").

[0175] When the processor 12 inputs the position difference ΔP(t) into the time series scoring model, the scoring model outputs a time series position score corresponding to the position difference ΔP(t).

[0176] When the processor 12 inputs the speed difference ΔV(t) into the time series scoring model, the scoring model outputs a time series speed score corresponding to the speed difference ΔV(t).

[0177] After step S1112 , the client device 10 generates navigation information ( S1113 ).

[0178] The first example of step S1113 is described below. The first example of step S1113 is an example of using an image as navigation information.

[0179] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t) and the speed difference ΔV(t) and the navigation information.

[0180] The processor 12 inputs the position difference ΔP(t) and the speed difference ΔV(t) obtained in step S1112 into the navigation model NM, thereby generating navigation information corresponding to the combination of the position difference ΔP(t) and the speed difference ΔV(t).

[0181] During the execution of the beauty action, the processor 12 displays the screen P1111 ( Figure 8 ) is displayed on the monitor.

[0182] Screen P1111 includes display objects A11110 to A11113 and operation object B1111.

[0183] Display object A11110 is a navigation area. Display object A11110 displays a user dynamic image IMG11110 and images (hereinafter referred to as "navigation images") IMG11111 to IMG11112 indicating navigation information. Navigation images IMG11111 to IMG1112 are displayed overlappingly on the user dynamic image (that is, an image of the user's face) IMG1110. The navigation image includes, for example, at least one of the following images.

[0184] An image showing the location on the beauty target part where the beauty action should be applied

[0185] Animated images of movements (exercises) that guide beauty treatments (for example, animation of arrow images that indicate the direction and speed of hand movements)

[0186] When the navigation image is an animated image, the processor 12 may adjust the speed of the arrow movement according to the speed difference ΔV(t).

[0187] For example, when the speed difference ΔV(t) is a positive value (that is, the beauty motion is faster than the demonstration motion), the processor 12 reproduces the change of the animated image at a speed slower than the standard speed.

[0188] For example, when the speed difference ΔV(t) is a negative value (that is, the beauty movement is slower than the demonstration movement), the processor 12 reproduces the change of the animated image at a speed faster than the standard speed.

[0189] The navigation image IMG11111 includes, for example, at least one of the following forms.

[0190] Message image showing the evaluation of beauty action and suggestion of beauty action (for example, a bubble dialogue image)

[0191] Position guidance image that guides the position of beauty treatment (for example, a dot image that moves in an appropriate direction)

[0192] · Speed ​​guidance image that guides the speed of beauty treatment (for example, a dot image that flashes at an appropriate speed)

[0193] · Computer graphic image of a hand that changes according to the position of a beauty action (for example, a hand that changes according to at least one of an appropriate direction and an appropriate speed)

[0194] The navigation image IMG11112 represents the navigation message. The context of the navigation message includes at least one of the following.

[0195] Evaluation of the beauty action (e.g., "GOOD" or "BAD")

[0196] Beauty action suggestions (e.g., the beauty action that should be performed next, or the beauty action that should be improved (for example, the suggestion of "slow down"))

[0197] Display object A11111 is a tracking area. Image objects IMG11110 and IMG11113 are displayed in display object A11111.

[0198] The image object IMG11113 is a trajectory image. The trajectory image is an image showing the trajectory of a beauty action (for example, the trajectory of the user's hand) during a predetermined period (for example, from 3 seconds before step S1113 is performed to step S1113 is performed).

[0199] Display object A11112 is a score area. Display object A11110 displays graphs G11110 to G11111 showing action scores of beauty actions along a time series (in chronological order).

[0200] Graph G11110 is a graph of time series position scores.

[0201] Graph G11111 is a graph of time series speed scores.

[0202] By displaying the action scores along the time series, the user can easily know the evaluation quality (that is, accuracy) of the action index (speed and position) of each step. As a result, the user can objectively understand his or her own skill level.

[0203] The display object A11113 is an object for displaying a model image. The model image changes along the time series of the beauty action. The model image is, for example, at least one of the following images.

[0204] A still image showing the template at time T1 synchronized with frame t1 of the demonstration action

[0205] A dynamic image showing a template at time T1 synchronized with frame t1 of the demonstration action

[0206] A still image showing a sample at time T2 earlier than frame t1 of the demonstration action

[0207] A dynamic image showing a sample at time T2 earlier than frame t1 of the demonstration action

[0208] Based on the sample image, the user can be prompted to perform beauty actions based on the demonstrated actions.

[0209] The operation object B1111 is an object that receives a user instruction to request a recommendation corresponding to a beauty action.

[0210] The second example of step S1113 is described below. The second example of step S1113 is an example of using voice as navigation information.

[0211] In the storage device 11, similarly to the first example of step S1113, the navigation model NM is stored.

[0212] The processor 12 generates a navigation message in the same manner as in the first example of step S1113.

[0213] The processor 12 outputs voice information corresponding to the navigation message (hereinafter referred to as "navigation voice information") from the speaker. The navigation voice information is an example of "sound information". The context of the navigation voice information is the same as the navigation image in the first example of step S1113. The navigation voice information includes, for example, at least one of the following.

[0214] BGM (Background Music)

[0215] · Voice reading of a predetermined sentence (for example, a commentary text)

[0216] ·Speak the feedback in navigation

[0217] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t) and the speed difference ΔV(t) and the voice conversion parameter.

[0218] The processor 12 inputs the position difference ΔP(t) and the speed difference ΔV(t) obtained in step S1112 to the navigation model NM, thereby generating a sound conversion parameter corresponding to a combination of the position difference ΔP(t) and the speed difference ΔV(t).

[0219] The storage device 11 stores predetermined sound information (for example, information to be reproduced during the execution of a beauty treatment action).

[0220] The processor 12 converts the sound information by using the sound conversion parameter, thereby generating converted sound information.

[0221] The processor 12 outputs the converted sound information from the speaker.

[0222] The third example of step S1113 is described below. The third example of step S1113 is an example of using the display format of the screen as the guide information.

[0223] In the storage device 11, similarly to the first example of step S1113, the navigation model NM is stored.

[0224] The processor 12 generates a navigation message in the same manner as in the first example of step S1113.

[0225] When at least one of the time-series position score and the time-series action score is smaller than a predetermined threshold value, the processor 12 displays the screen P1111 in a warning form (for example, yellow or flashing).

[0226] When both the time-series position score and the time-series action score are equal to or greater than the threshold value, the processor 12 displays the screen P1111 in a display format different from the warning format (for example, green or light).

[0227] The first to third examples of step S1113 can be combined with each other.

[0228] After step S1113 , the client device 10 performs the recommendation ( S1114 ).

[0229] Specifically, the storage device 11 stores an overall motion score model. The overall motion score model describes the correlation between the combination of the overall position difference ΔP(t) and the speed difference ΔV(t) of the user's dynamic image and the overall motion score. The overall motion score includes, for example, at least one of the following scores.

[0230] ·The effect score indicating the overall effect of the beauty action

[0231] ·Proficiency score indicating overall proficiency of beauty actions

[0232] · Comprehensive score indicating the overall evaluation of the beauty action

[0233] · Overall position score indicating the evaluation of the overall position of the beauty action

[0234] · Overall speed score indicating the evaluation of the overall speed of the beauty action

[0235] The storage device 11 stores a recommendation model. The recommendation model describes the correlation between the combination of the overall position difference ΔP(t) and the speed difference ΔV(t) of the user's dynamic image and the recommendation information. The recommendation information includes, for example, at least one of the following suggestions.

[0236] Recommendations related to products that users can use for skin care (for example, skin care tools, skin care products (specifically, massage beauty lotions or creams), makeup tools (specifically, cotton pads, sponges, puffs, or brushes of varying hardness), or cosmetics

[0237] · Suggestions for more effective beauty treatments

[0238] · Suggestions for improving beauty routines

[0239] When the user operates the operation object B11120, the processor 12 determines the overall action score corresponding to the combination of the position difference ΔP(t) and the speed difference ΔV(t) of the overall user dynamic image obtained in step S1112, by inputting the combination into the overall action scoring model.

[0240] The processor 12 inputs the combination of position difference ΔP(t) and speed difference ΔV(t) of the entire user moving image obtained in step S1112 into the recommendation model, thereby generating recommendation information corresponding to the combination of position difference ΔP(t) and speed difference ΔV(t).

[0241] The processor 12 displays the screen P1112 ( Figure 7 ) is displayed on the monitor.

[0242] Screen P1112 includes display objects A11120 to A11121.

[0243] The action scores (eg, effect score, proficiency score, comprehensive score, overall position score, and overall speed score) are displayed in the display object A11120.

[0244] Recommended information (for example, text information and image information) is displayed in display object A11121.

[0245] After step S1114 , the client device 10 executes the update request ( S1115 ).

[0246] Specifically, the processor 12 sends the update request data to the server 30. The update request data includes the following information, for example.

[0247] User identification information

[0248] Information related to the execution date and time of step S1114 (hereinafter referred to as "time stamp information")

[0249] User dynamic image obtained in step S1110

[0250] Information indicating the user position P(t) of the entire user dynamic image obtained in step S1111

[0251] The action score obtained in step S1113

[0252] After step S1115 , the server 30 executes updating of the database ( S1130 ).

[0253] Specifically, the processor 32 updates the user log database ( Figure 5 ) and store the following information in each field of the new record.

[0254] "User log ID" field: New user log identification information

[0255] "Timestamp" field: Update the timestamp information contained in the request data

[0256] "User dynamic image" field: Update the user dynamic image included in the request data

[0257] "Motion trajectory" field: Updates the information indicating the user's position P(t) contained in the request data

[0258] "Action score" field: Updates the action score included in the request data

[0259] (5) Summary of this embodiment

[0260] According to the present embodiment, navigation information corresponding to a combination of the position and speed of each beauty target part of the user is presented to the user.

[0261] This allows the user to take into account the navigation information and perform beauty actions. As a result, it is possible to provide an incentive for users who are customers interested in beauty to continue their beauty actions.

[0262] According to the present embodiment, a navigation image IMG11111 may be generated as navigation information, and the navigation image IMG11111 may be displayed superimposed on the user dynamic image IMG11110 .

[0263] This allows the user to perform beauty treatments while visually recognizing his or her own face and the navigation image IMG11111 at the same time.

[0264] According to this embodiment, a position guide image for guiding the position of the beauty action and a speed guide image for guiding the speed of the beauty action may be generated as navigation information, and the position guide image and the speed guide image may be displayed superimposed on the user dynamic image IMG11110.

[0265] As a result, the user can perform beauty movements according to the guidance of the position and speed of the beauty movements while visually recognizing his or her own face and the navigation image IMG11111 at the same time.

[0266] According to the present embodiment, an image of a hand that changes according to the position of the beauty action may be generated as navigation information.

[0267] This allows the user to perform beauty treatments on his or her face while visually recognizing a navigation image showing that the user is performing beauty treatments on his or her face with his or her hands.

[0268] (6) Modification

[0269] Modifications of this embodiment will be described.

[0270] (6-1) Modification 1

[0271] Modification 1 will be described. Modification 1 is an example in which the user's pressing force is taken into consideration in addition to the user's position and the user's speed in the evaluation of the action ( S1112 ).

[0272] (6-1-1) Overview of Modification 1

[0273] The outline of Modification 1 will be described. Fig. 9 This is an explanatory diagram of the outline of Modification Example 1.

[0274] like Fig. 9 As shown, by analyzing the user dynamic image, the user position P(t) and the user speed V(t) in each frame F(t) of the user dynamic image are determined.

[0275] From the wearable sensor 20 worn by the user, the user's pressing force PR(t) applied to the user's face by the user's hand is determined.

[0276] By inputting the user position P(t), user speed V(t) and user pressure PR(t) into the demonstration model M (Pm(t), Vm(t), PRm(t)), the position difference ΔP(t), speed difference ΔV(t), and the motion difference (hereinafter referred to as "pressure difference") ΔPR(t) between the user pressure PR(t) and the demonstration pressure PRm(t) are obtained.

[0277] By inputting the position difference ΔP(t), the speed difference ΔV(t), and the pressure difference ΔPR(t) into the navigation model NM(ΔP(t), ΔV(t), ΔPR(t)), navigation information is obtained.

[0278] The navigation information is prompted to the user.

[0279] (6-1-2) Information Processing of Modification 1

[0280] The information processing of Modification 1 is described.

[0281] The start trigger of the process of Modification Example 1 is Figure 6 It's the same.

[0282] The client device 10 and Figure 6 The acquisition of user dynamic images is performed in the same manner ( S1110 ).

[0283] After step S1110 , the client device 10 performs image analysis ( S1111 ).

[0284] Specifically, the processor 12 analyzes the user's dynamic image to recognize the beauty target part of the user and the user's hand (for example, fingertips) for each frame F(t) constituting the user's dynamic image.

[0285] The processor 12 determines the target region based on the coordinates of each beauty target part of the user for each frame F(t).

[0286] The processor 12 specifies the coordinates of the user's hand that are included in the target area as the user position P(t) in the frame F(t).

[0287] The processor 12 calculates the user velocity V(t) based on the displacement amount (P(t+1)-P(t)) of the user position between the frames F(t) and F(t+1).

[0288] The processor 12 determines the user pressing force PR(t) applied by the user's hand to the user's face based on changes in the user's hand at the user position P(t) (eg, changes in skin wrinkles) in the frame F(t).

[0289] After step S1111 , the client device 10 performs evaluation of the action ( S1112 ).

[0290] Specifically, the storage device 11 stores a model M. A model action is described in the model M. The model action is defined by a model position Pm(t), a model speed Vm(t), and a model pressing force PRm(t).

[0291] The processor 12 refers to the model M and calculates the difference between the user position P(t) and the model position Pm(t), ie, the position difference ΔP(t).

[0292] The processor 12 refers to the exemplary model M and calculates the speed difference ΔV(t) which is the difference between the user speed V(t) and the exemplary speed Vm(t).

[0293] The processor 12 refers to the exemplary model M and calculates the difference between the user's pressing force PR(t) and the exemplary pressing force PRm(t), that is, the pressing force difference ΔPR(t).

[0294] The storage device 11 stores a time series scoring model. The time series scoring model describes the correlation between the evaluation results of the action (for example, position difference ΔP(t), speed difference ΔV(t), and pressure difference ΔPR(t)) and the time series action score.

[0295] When the processor 12 inputs the position difference ΔP(t) into the time series scoring model, the scoring model outputs a time series position score corresponding to the position difference ΔP(t).

[0296] When the processor 12 inputs the speed difference ΔV(t) into the time series scoring model, the scoring model outputs a time series speed score corresponding to the speed difference ΔV(t).

[0297] When the processor 12 inputs the pressure difference ΔPR(t) into the time series scoring model, the scoring model outputs a time series pressure score corresponding to the pressure difference ΔPR(t).

[0298] After step S1112 , the client device 10 performs navigation ( S1113 ).

[0299] The first example of step S1113 in modification 1 will be described. The first example of step S1113 in modification 1 is an example of using an image as navigation information.

[0300] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the pressure difference ΔPR(t) and the navigation information.

[0301] The processor 12 generates navigation information corresponding to the combination of the position difference ΔP(t), the speed difference ΔV(t) and the pressure difference ΔPR(t) obtained in step S1112 by inputting them into the navigation model NM.

[0302] During the execution of the beauty action, the processor 12 displays the screen P1111 ( Figure 8 ) is displayed on the monitor.

[0303] The second example of step S1113 of variant 1 is the same as Figure 6 The second example of step S1113 is the same.

[0304] The first example and the second example of step S1113 in variation 1 may be combined.

[0305] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the pressure difference ΔPR(t) and the voice conversion parameter.

[0306] The processor 12 generates a sound conversion parameter corresponding to the combination of the position difference ΔP(t), the speed difference ΔV(t) and the pressure difference ΔPR(t) obtained in step S1112 by inputting the position difference ΔP(t), the speed difference ΔV(t) and the pressure difference ΔPR(t) into the navigation model NM.

[0307] The storage device 11 stores predetermined sound information (for example, information to be reproduced during the execution of a beauty treatment action).

[0308] The processor 12 converts the sound information by using the sound conversion parameter, thereby generating converted sound information.

[0309] The processor 12 outputs the converted sound information from the speaker.

[0310] After step S1113 , the client device 10 performs the recommendation ( S1114 ).

[0311] Specifically, the storage device 11 stores a whole motion score model that describes the correlation between the combination of the whole position difference ΔP(t), speed difference ΔV(t), and pressure difference ΔPR(t) of the user's dynamic image and the whole motion score.

[0312] The storage device 11 stores a recommendation model. The recommendation model describes the correlation between the combination of the overall position difference ΔP(t), speed difference ΔV(t), and pressure difference ΔPR(t) of the user's moving image and the recommendation information.

[0313] When the user operates the operation object B11120, the processor 12 determines the overall action score corresponding to the combination of the position difference ΔP(t), speed difference ΔV(t) and pressure difference ΔPR(t) of the overall user dynamic image obtained in step S1112 by inputting the combination into the overall action scoring model.

[0314] The processor 12 generates recommendation information corresponding to the combination of the position difference ΔP(t), speed difference ΔV(t) and pressure difference ΔPR(t) of the entire user dynamic image obtained in step S1112 by inputting it into the recommendation model.

[0315] The processor 12 displays the screen P1112 ( Figure 7 ) is displayed on the monitor.

[0316] After step S1114, the client device 10 and Figure 6 The update request is similarly executed (S1115).

[0317] After step S1115, the server 30 and Figure 6 The database is updated in the same manner (S1130).

[0318] (6-1-3) Summary of Modification 1

[0319] According to the first modification, navigation information corresponding to the combination of the position, speed, and pressing force of each beauty target part of the user is presented to the user.

[0320] This allows the user to take the navigation information into consideration and perform beauty actions. As a result, it is possible to provide greater motivation for users who are customers who pay attention to beauty to continue their beauty actions.

[0321] In particular, variant 1 is particularly suitable for situations where it is preferred to change the pressing force according to the beauty target area, or where it is preferred to slowly change the pressing force locally and sequentially even for the same beauty target area (as an example, when the beauty action is a massage or application action).

[0322] More specifically, when the beauty treatment is acupressure massage, the acupuncture points are pressed with a pressing force corresponding to the beauty target area, thereby maximizing the massage effect.

[0323] When the beauty action is to apply foundation, the user is guided to apply the foundation with a pressing force corresponding to the type of foundation or the desired finishing effect, so that the foundation powder can be appropriately applied to the skin.

[0324] In variation 1, the processor 12 may obtain the user's pressing force PR(t) from a wearable sensor 20 (eg, a strain sensor) worn by the user, instead of determining the user's pressing force PR(t) based on an image.

[0325] (6-2) Modification 2

[0326] Modification 2 is described. Modification 2 is an example in which the user's tempo is considered in addition to the user's position and the user's speed in the evaluation of the action (S1112). The user's tempo refers to the tempo of the beauty action.

[0327] (6-2-1) Overview of Modification 2

[0328] The outline of Modification Example 2 is described. Fig.10 This is an explanatory diagram of the outline of Modification Example 2.

[0329] like Fig. 9 As shown, by analyzing the user dynamic image, the user position P(t), user speed V(t) and user rhythm T(t) in each frame F(t) of the user dynamic image are determined.

[0330] By inputting the user position P(t), user speed V(t) and user rhythm T(t) into the demonstration model M (Pm(t), Vm(t), Tm(t)), the position difference ΔP(t), speed difference ΔV(t), and the motion difference (hereinafter referred to as "rhythm difference") ΔT(t) between the user rhythm T(t) and the demonstration rhythm Tm(t) are obtained.

[0331] By inputting the position difference ΔP(t), the speed difference ΔV(t) and the rhythm difference ΔT(t) into the navigation model NM(ΔP(t), ΔV(t), ΔT(t)), navigation information is obtained.

[0332] The navigation information is prompted to the user.

[0333] (6-2-2) Information Processing of Modification 2

[0334] The information processing of Modification 2 is described.

[0335] The start trigger of the process of Modification Example 2 is Figure 6 It's the same.

[0336] The client device 10 and Figure 6 The acquisition of user dynamic images is performed in the same manner ( S1110 ).

[0337] After step S1110 , the client device 10 performs image analysis ( S1111 ).

[0338] Specifically, the processor 12 analyzes the user's dynamic image to recognize each action target part of the user and the user's hand (for example, fingertips) for each frame F(t) constituting the user's dynamic image.

[0339] The processor 12 determines the target region based on the coordinates of each beauty target part of the user for each frame F(t).

[0340] The processor 12 specifies, in the frame F(t), a position of the user's hand that is included in the target area as the user position P(t).

[0341] The processor 12 calculates the user velocity V(t) based on the displacement amount (P(t+1)-P(t)) of the user position between the frames F(t) and F(t+1).

[0342] A first example of step S1110 in variation 2 is described.

[0343] The processor 12 calculates the user's tempo T(t) based on the position P(t) and the acceleration A(t).

[0344] A second example of step S1110 in variation 2 is described.

[0345] The processor 12 calculates the user rhythm T(t) based on the sequence of the user's hand movements and the number of the movements.

[0346] After step S1111 , the client device 10 performs evaluation of the action ( S1112 ).

[0347] Specifically, the storage device 11 stores a model M. A model action is described in the model M. The model action is defined by a model position Pm(t), a model speed Vm(t), and a model rhythm Tm(t).

[0348] The processor 12 refers to the model M and calculates the difference between the user position P(t) and the model position Pm(t), ie, the position difference ΔP(t).

[0349] The processor 12 refers to the exemplary model M and calculates the speed difference ΔV(t) which is the difference between the user speed V(t) and the exemplary speed Vm(t).

[0350] The processor 12 refers to the exemplary model M and calculates the difference between the user's tempo T(t) and the exemplary tempo Tm(t), ie, the tempo difference ΔT(t).

[0351] The storage device 11 stores a time series scoring model in which the correlation between the evaluation results of the action (eg, position difference ΔP(t), speed difference ΔV(t), and rhythm difference ΔT(t)) and the time series action score is described.

[0352] When the processor 12 inputs the position difference ΔP(t) into the time series scoring model, the scoring model outputs a time series position score corresponding to the position difference ΔP(t).

[0353] When the processor 12 inputs the speed difference ΔV(t) into the time series scoring model, the scoring model outputs a time series speed score corresponding to the speed difference ΔV(t).

[0354] When the processor 12 inputs the tempo difference ΔT(t) into the time series scoring model, the scoring model outputs a time series tempo score corresponding to the tempo difference ΔT(t).

[0355] After step S1112 , the client device 10 performs navigation ( S1113 ).

[0356] A first example of step S1113 in modification 2 will be described. The first example of step S1113 in modification 2 is an example of using an image as navigation information.

[0357] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the rhythm difference ΔT(t) and the navigation information.

[0358] The processor 12 generates navigation information corresponding to the combination of the position difference ΔP(t), the speed difference ΔV(t) and the rhythm difference ΔT(t) obtained in step S1112 by inputting them into the navigation model NM.

[0359] During the execution of the beauty action, the processor 12 displays the screen P1111 ( Figure 8 ) is displayed on the monitor.

[0360] The second example of step S1113 of variant 2 is the same as Figure 6 The second example of step S1113 is the same.

[0361] The first example and the second example of step S1113 in variation 2 may be combined.

[0362] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the tempo difference ΔT(t) and the sound conversion parameter.

[0363] The processor 12 generates a sound conversion parameter corresponding to the combination of the position difference ΔP(t), the speed difference ΔV(t) and the rhythm difference ΔT(t) obtained in step S1112 by inputting the position difference ΔP(t), the speed difference ΔV(t) and the rhythm difference ΔT(t) into the navigation model NM.

[0364] The storage device 11 stores predetermined sound information (for example, sound information to be reproduced during the execution of a beauty treatment).

[0365] The processor 12 converts the sound information (an example of “sound information”) using the sound conversion parameter, thereby generating converted sound information.

[0366] The processor 12 outputs the converted sound information from the speaker.

[0367] After step S1113 , the client device 10 performs the recommendation ( S1114 ).

[0368] Specifically, the storage device 11 stores a whole motion score model that describes the correlation between the combination of the whole position difference ΔP(t), speed difference ΔV(t) and tempo difference ΔT(t) of the user's dynamic image and the whole motion score.

[0369] The storage device 11 stores a recommendation model. The recommendation model describes the correlation between the combination of the overall position difference ΔP(t), speed difference ΔV(t), and tempo difference ΔT(t) of the user's dynamic image and the recommendation information.

[0370] When the user operates the operation object B11120, the processor 12 determines the overall action score corresponding to the combination of the position difference ΔP(t), speed difference ΔV(t) and rhythm difference ΔT(t) of the overall user dynamic image obtained in step S1112 by inputting the combination into the overall action scoring model.

[0371] The processor 12 generates recommendation information corresponding to the combination of position difference ΔP(t), speed difference ΔV(t) and rhythm difference ΔT(t) of the overall user dynamic image obtained in step S1112 by inputting the combination into the recommendation model.

[0372] The processor 12 displays the screen P1112 ( Figure 7 ) is displayed on the monitor.

[0373] After step S1114, the client device 10 and Figure 6 The update request is similarly executed (S1115).

[0374] After step S1115, the server 30 and Figure 6 The database is updated in the same manner (S1130).

[0375] (6-2-3) Summary of Modification 2

[0376] According to the second modification, navigation information corresponding to the combination of the position, speed, and rhythm of each of the user's motion target parts is presented to the user.

[0377] This allows the user to take the navigation information into consideration and perform beauty actions. As a result, it is possible to provide greater motivation for users who are customers who pay attention to beauty to continue their beauty actions.

[0378] In particular, Modification 2 is particularly suitable for cases where it is preferable to change the speed according to the action target part, or where it is preferable to change the acceleration locally and sequentially slowly even for the same beauty target part (for example, when the beauty action is massage).

[0379] More specifically, when the cosmetic action is to lift the cheeks "in piles", in the second half of the treatment, when the hands are on the upper part of the cheeks, the action of slowly lifting the cheeks is guided, and when the hand rotation action is required three times, the third action is performed slowly.

[0380] (6-3) Modification 3

[0381] Modification 3 will be described. Modification 3 is an example in which user acceleration is considered in addition to user position and user speed in the evaluation of the action ( S1112 ).

[0382] (6-3-1) Overview of Modification 3

[0383] The outline of Modification 3 is described. Fig.11 This is an explanatory diagram of the outline of Modification Example 3.

[0384] like Fig.11 As shown, by analyzing the user dynamic image, the user position P(t), user speed V(t) and user hand acceleration (hereinafter referred to as "user acceleration") A(t) in each frame F(t) of the user dynamic image are determined.

[0385] By inputting the user position P(t), user speed V(t) and user acceleration A(t) into the demonstration model M (Pm(t), Vm(t), Am(t)), the position difference ΔP(t), the speed difference ΔV(t), and the motion difference between the user acceleration A(t) and the demonstration acceleration Am(t) (hereinafter referred to as "acceleration difference") ΔA(t) are obtained.

[0386] By inputting the position difference ΔP(t), the velocity difference ΔV(t), and the acceleration difference ΔA(t) into the navigation model NM(ΔP(t), ΔV(t), ΔA(t)), navigation information is obtained.

[0387] The navigation information is prompted to the user.

[0388] (6-3-2) Information Processing of Modification 3

[0389] The information processing of Modification 3 is described.

[0390] The client device 10 and Figure 6 The acquisition of user dynamic images is performed in the same manner ( S1110 ).

[0391] After step S1110 , the client device 10 performs image analysis ( S1111 ).

[0392] Specifically, the processor 12 analyzes the user's dynamic image to recognize each beauty target part of the user and the user's hand (for example, fingertips) for each frame F(t) constituting the user's dynamic image.

[0393] The processor 12 determines the target region based on the coordinates of each beauty target part of the user for each frame F(t).

[0394] The processor 12 specifies, in the frame F(t), a position of the user's hand that is included in the target area as the user position P(t).

[0395] The processor 12 calculates the user velocity V(t) based on the displacement amount (P(t+1)-P(t)) of the user position between the frames F(t) and F(t+1).

[0396] The processor 12 calculates the user acceleration A(t) based on the displacement (V(t+1)-V(t)) of the user velocity between the frames F(t) and F(t+1).

[0397] After step S1111 , the client device 10 performs evaluation of the action ( S1112 ).

[0398] Specifically, the storage device 11 stores a model M. A model action is described in the model M. The model action is defined by a model position Pm(t), a model velocity Vm(t), and a model acceleration Am(t).

[0399] The processor 12 refers to the model M and calculates the difference between the user position P(t) and the model position Pm(t), ie, the position difference ΔP(t).

[0400] The processor 12 refers to the exemplary model M and calculates the speed difference ΔV(t) which is the difference between the user speed V(t) and the exemplary speed Vm(t).

[0401] The processor 12 refers to the exemplary model M and calculates the difference between the user acceleration A(t) and the exemplary acceleration Am(t), ie, the acceleration difference ΔA(t).

[0402] The storage device 11 stores a time series scoring model. The time series scoring model describes the correlation between the evaluation results of the action (for example, position difference ΔP(t), speed difference ΔV(t), and acceleration difference ΔA(t)) and the time series action score.

[0403] When the processor 12 inputs the position difference ΔP(t) into the time series scoring model, the scoring model outputs a time series position score corresponding to the position difference ΔP(t).

[0404] When the processor 12 inputs the speed difference ΔV(t) into the time series scoring model, the scoring model outputs a time series speed score corresponding to the speed difference ΔV(t).

[0405] When the processor 12 inputs the acceleration difference ΔA(t) to the time series scoring model, the scoring model outputs a time series press score corresponding to the acceleration difference ΔA(t).

[0406] After step S1112 , the client device 10 performs navigation ( S1113 ).

[0407] A first example of step S1113 in modification 3 will be described. The first example of step S1113 in modification 3 is an example of using an image as navigation information.

[0408] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the acceleration difference ΔA(t) and the navigation information.

[0409] The processor 12 generates navigation information corresponding to the combination of the position difference ΔP(t), the speed difference ΔV(t) and the acceleration difference ΔA(t) obtained in step S1112 by inputting them into the navigation model NM.

[0410] During the execution of the beauty action, the processor 12 displays the screen P1111 ( Figure 8 ) is displayed on the monitor.

[0411] The second example of step S1113 of variant 3 is the same as Figure 6 The second example of step S1113 is the same.

[0412] The first example and the second example of step S1113 in variation 3 may be combined.

[0413] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the acceleration difference ΔA(t) and the sound conversion parameter.

[0414] The processor 12 generates a sound conversion parameter corresponding to a combination of the position difference ΔP(t), the speed difference ΔV(t) and the acceleration difference ΔA(t) obtained in step S1112 by inputting the position difference ΔP(t), the speed difference ΔV(t) and the acceleration difference ΔA(t) into the navigation model NM.

[0415] The storage device 11 stores predetermined sound information (for example, information to be reproduced during the execution of a beauty treatment action).

[0416] The processor 12 converts the sound information (an example of “sound information”) using the sound conversion parameter, thereby generating converted sound information.

[0417] The processor 12 outputs the converted sound information from the speaker.

[0418] After step S1113 , the client device 10 performs the recommendation ( S1114 ).

[0419] Specifically, the storage device 11 stores a whole motion score model that describes the correlation between the combination of the whole position difference ΔP(t), speed difference ΔV(t), and acceleration difference ΔA(t) of the user's dynamic image and the whole motion score.

[0420] The storage device 11 stores a recommendation model. The recommendation model describes the correlation between the combination of the overall position difference ΔP(t), speed difference ΔV(t), and acceleration difference ΔA(t) of the user's dynamic image and the recommendation information.

[0421] When the user operates the operation object B11120, the processor 12 determines the overall action score corresponding to the combination of the position difference ΔP(t), speed difference ΔV(t) and acceleration difference ΔA(t) of the overall user dynamic image obtained in step S1112, by inputting the combination into the overall action scoring model.

[0422] The processor 12 generates recommendation information corresponding to the combination of position difference ΔP(t), speed difference ΔV(t) and acceleration difference ΔA(t) of the entire user dynamic image obtained in step S1112 by inputting it into the recommendation model.

[0423] The processor 12 displays the screen P1112 ( Figure 7 ) is displayed on the monitor.

[0424] After step S1114, the client device 10 and Figure 6 The update request is similarly executed (S1115).

[0425] After step S1115, the server 30 and Figure 6 The database is updated in the same manner (S1130).

[0426] (6-3-3) Summary of Modification 3

[0427] According to Modification 3, navigation information corresponding to a combination of the position, velocity, and acceleration of each of the user's action target parts is presented to the user.

[0428] This allows the user to take the navigation information into consideration and perform beauty actions. As a result, it is possible to provide greater motivation for users who are customers who pay attention to beauty to continue their beauty actions.

[0429] In particular, Modification 3 is particularly suitable for a case where it is preferred to perform treatment at a constant speed regardless of the method of the beauty action and the action target part (for example, a case where the beauty action is the action of applying lotion or milky lotion).

[0430] (6-4) Modification 4

[0431] Modification 4 is described. Modification 4 is an example of using an avatar image as navigation information.

[0432] (6-4-1) Overview of Modification 4

[0433] The outline of Modification 4 is described. Fig.12 This is an explanatory diagram of the outline of Modification Example 4.

[0434] like Fig.12 As shown, by analyzing the user's dynamic image of the beauty action, the user's position P(t) and the user's speed V(t) in each frame F(t) of the user's dynamic image are determined.

[0435] By inputting the user position P(t) and the user velocity V(t) into the exemplary model M(Pm(t), Vm(t)), the position difference ΔP(t) and the velocity difference ΔV(t) are obtained.

[0436] By inputting the position difference ΔP(t) and the speed difference ΔV(t) into the navigation model NM(ΔP(t), ΔV(t)), navigation information is obtained.

[0437] The navigation information is presented to the user as an avatar image.

[0438] (6-4-2) Information Processing of Modification 4

[0439] The information processing of Modification 4 is described. Fig.13 This is a timing diagram of information processing in variant example 4. Fig.14 It means in Fig.13 Figure 2 is an example of a screen displayed during information processing. Fig.15 It means in Fig.13 Figure 2 is an example of a screen displayed during information processing. Fig.16 It means in Fig.13 Figure 2 is an example of a screen displayed during information processing.

[0440] Fig.13 The start of the processing is triggered with Figure 6 It's the same.

[0441] like Fig.13 As shown, the client device 10 and Figure 6 The acquisition of user dynamic images is performed in the same manner ( S1110 ).

[0442] After step S1110 , the client device 10 displays the avatar image ( S5110 ).

[0443] Specifically, the processor 12 displays the screen P5110 ( Fig.14 ) is displayed on the monitor.

[0444] The screen P5110 includes an operation object B5110 and an image object IMG5110.

[0445] The avatar image IMG5110 is any of the following images.

[0446] Images stored in the storage device 11

[0447] ·Images created according to user instructions

[0448] The operation object B5110 is an object that receives a user instruction to start navigation.

[0449] After step S5110, the client device 10 and Figure 6 Image analysis ( S1111 ) to motion evaluation ( S1112 ) are performed in the same manner.

[0450] After step S1112 , the client device 10 performs navigation ( S5111 ).

[0451] The first example of step S5111 is described below. The first example of step S5111 is an example in which the user's face is exposed by erasing the pixels of the avatar image at the position where the beauty action is performed.

[0452] Specifically, the processor 12 erases the pixels of the avatar image corresponding to the coordinates of the user's hand determined in step S1111 and replaces them with the pixels of the user dynamic image IMG5111.

[0453] Processor 12 displays screen P5111 ( Fig.15 ) is displayed on the monitor.

[0454] Screen P5111 includes display objects A5111 and A11111 to A11113, and operation object B1111. Display objects A11111 to A11113 and operation object B1111 are Figure 8 It's the same.

[0455] The image objects IMG11112, IMG5110, and IMG5111 are displayed in the display object A5111. The image object IMG11112 is Figure 8 It's the same.

[0456] Image object IMG5111 is a portion of the user dynamic image replaced by processor 12.

[0457] In the first example of step S5111, if Fig.14As shown, at the time point before the beauty action begins, the avatar image IMG5110 is displayed, and the user dynamic image is not displayed.

[0458] When the user performs a beauty action, such as Fig.15 As shown, pixels of the user's dynamic image (that is, the user's face) are exposed at the location where the beauty action is performed.

[0459] The second example of step S5111 is described below. The second example of step S5111 is an example of applying makeup to the position where the beauty action is performed on the avatar image by changing the color of the pixel of the avatar image at the position where the beauty action is performed.

[0460] Specifically, processor 12 changes the color of the pixel of the avatar image corresponding to the coordinates of the user's hand specified in step S1111.

[0461] Processor 12 displays screen P5111 ( Fig.16 ) is displayed on the monitor.

[0462] Screen P5111 includes display objects A5111, A11111 to A11113, and operation object B1111. Display objects A11111 to A11113 and operation object B1111 are Figure 8 It's the same.

[0463] The image objects IMG11112, IMG5110, and IMG5111 are displayed in the display object A5111. The image object IMG11112 is Figure 8 It's the same.

[0464] Image object IMG5111 is a pixel whose color is changed by processor 12.

[0465] In the second example of step S5111, if Fig.14 As shown, at the time point before the beauty action begins, the avatar image IMG5110 is displayed, and the user dynamic image is not displayed.

[0466] When the user performs a beauty action, such as Fig.16 As shown, at the location where the beauty action is performed, the color of the pixel IMG5111 of the avatar image IMG5110 changes (that is, makeup is applied to the avatar image).

[0467] After step S5111, the client device 10 and Figure 6 The recommendation ( S1114 ) to the update request ( S1115 ) are performed in the same manner.

[0468] After step S1115, the server 30 and Figure 6The database is updated in the same manner (S1130).

[0469] (6-4-3) Summary of Modification 4

[0470] According to modification example 4, the avatar image is displayed superimposed on the user dynamic image IMG11110, and the pixels of the avatar image at the position where the beauty action is performed are changed.

[0471] Thus, the user can perform beauty actions while enjoying the changes in the avatar image. As a result, the user who becomes a beauty-conscious customer can be given a greater incentive to continue the beauty actions.

[0472] According to Modification 4, by erasing the pixels of the avatar image at the position where the beauty action is performed, the image of the user's face at the position where the beauty action is performed is exposed.

[0473] Thus, the user can perform beauty actions while enjoying the changes in the avatar image. As a result, the user who becomes a beauty-conscious customer can be given a greater incentive to continue the beauty actions.

[0474] According to Modification 4, by changing the color of the pixel of the avatar image at the position where the beauty action is performed, makeup is applied to the position where the beauty action is performed on the avatar image.

[0475] Thus, the user can perform beauty actions while enjoying the changes in the avatar image. As a result, the user who becomes a beauty-conscious customer can be given a greater incentive to continue the beauty actions.

[0476] In Modification 4, an example in which an avatar image is displayed superimposed on the user dynamic image IMG11110 is described, but the scope of Modification 4 is not limited thereto. Modification 4 can also be applied to a case in which both the user dynamic image IMG11110 and the avatar image are displayed.

[0477] In Modification 4, an example of displaying an avatar image is described, but the scope of Modification 4 is not limited thereto. Modification 4 can also be applied to an example of displaying an avatar image and outputting a voice of the avatar image (an example of "navigation information").

[0478] (6-5) Modification 5

[0479] Modification 5 will be described. Modification 5 is an example in which beauty actions are evaluated according to a scenario.

[0480] (6-5-1) Overview of Modification Example 5

[0481] The outline of Modification Example 5 is described. Fig.17 This is an explanatory diagram of the outline of Modification Example 5.

[0482] like Fig.17 As shown, by analyzing the user's dynamic image of the beauty action, the user's position P(t) and the user's speed V(t) in each frame F(t) of the user's dynamic image are determined. t is an example of information for identifying a frame.

[0483] By inputting the user position P(t) and the user speed V(t) into the demonstration model M (Pm(t), Vm(t)), according to a predetermined script, the motion difference between the user position P(t) and the demonstration position Pm(t) (hereinafter referred to as the "position difference") ΔP(t), and the motion difference between the user speed V(t) and the demonstration speed Vm(t) (hereinafter referred to as the "speed difference") ΔV(t) are obtained.

[0484] By inputting the position difference ΔP(t) and the speed difference ΔV(t) into the navigation model NM(ΔP(t), ΔV(t)), navigation information is obtained.

[0485] The navigation information is prompted to the user.

[0486] (6-5-2) Information Processing of Modification 5

[0487] The information processing of Modification 5 is described. Fig.18 This is a timing diagram of information processing in variant example 5. Fig.19 yes Fig.17 An illustration of the script. Fig. 20 It means in Fig.18 Figure 2 is an example of a screen displayed during information processing. Fig.21 It means in Fig.18 Figure 2 is an example of a screen displayed during information processing.

[0488] like Fig.19 As shown, the client device 10 and Figure 6 The acquisition of user dynamic images ( S1110 ) and image analysis ( S1111 ) are similarly performed.

[0489] After step S1111 , the client device 10 performs action evaluation ( S6110 ).

[0490] Specifically, the storage device 11 stores a plurality of demonstration models M. Each demonstration model M corresponds to a script. In the script, demonstration actions along a time series are described for each facial part of the user and each type of beauty action. In other words, each demonstration model M describes a demonstration action corresponding to the script.

[0491] The types of beauty actions include, for example, at least one of the following.

[0492] · How to move your hands (for example, moving in a straight line, moving to lift your cheeks, moving in a circle, etc.)

[0493] · How the hand applies force (for example, pressing at one point)

[0494] The script contains multiple sections. In each section, the beauty action steps that constitute a series of beauty actions are specified. Fig.19 ). The combination of multiple beauty action steps forms a series of beauty actions. The beauty action steps included in each paragraph are common or different. In the case where the beauty action steps included in each paragraph are common, multiple paragraphs mean that the common beauty action steps are repeated.

[0495] In the demonstration model M, elements of the demonstration action are defined for each cosmetic action step. The elements of the demonstration action include at least one of action time, action name, part, track coordinates, description, and display data.

[0496] The processor 12 refers to the model M and calculates the position difference ΔP(t), which is the difference between the user position P(t) and the model position Pm(t), for each cosmetic action step.

[0497] The processor 12 refers to the exemplary model M and calculates the speed difference ΔV(t), which is the difference between the user speed V(t) and the exemplary speed Vm(t), for each cosmetic action step.

[0498] The storage device 11 stores a time series scoring model. The time series scoring model describes the correlation between the evaluation result of the action (for example, the position difference ΔP(t) and the speed difference ΔV(t)) of each beauty action step and the time series action score.

[0499] When the processor 12 inputs the position difference ΔP(t) to the scoring model, the scoring model outputs a time-series position score for each cosmetic action step corresponding to the position difference ΔP(t).

[0500] When the processor 12 inputs the speed difference ΔV(t) to the scoring model, the scoring model outputs a time-series speed score for each beauty action step corresponding to the speed difference ΔV(t).

[0501] After step S6110 , the client device 10 generates navigation information ( S6111 ).

[0502] Specifically, a navigation model NM is stored in the storage device 11. The navigation model NM describes the correlation between the combination of the position difference ΔP(t) and the speed difference ΔV(t) and the navigation information.

[0503] The processor 12 generates navigation information for each beauty action step corresponding to the combination of the position difference ΔP(t) and the speed difference ΔV(t) obtained in step S6110 by inputting the position difference ΔP(t) and the speed difference ΔV(t) of each beauty action step into the navigation model NM.

[0504] During the execution of the beauty action, the processor 12 displays the screen P6110 ( Fig. 20 ) is displayed on the monitor.

[0505] The screen P6110 includes display objects A5111, A11111, A11113, A61100 to A61102, and an operation object B6110. Figure 8 is the same. Display object A5111 and Fig.15 It's the same.

[0506] The display object A61100 is an object indicating the current beauty action step relative to the entire beauty action steps.

[0507] The display object A61101 is an object representing a time-series position score.

[0508] The display object A61102 is an object representing the time-series speed score.

[0509] The operation object B6110 is an object that receives a user instruction to display an outline of the current beauty action step.

[0510] After step S6111 , the client device 10 performs the recommendation ( S6112 ).

[0511] Specifically, the storage device 11 stores a whole motion score model. In the whole motion score model, the correlation between the combination of the position difference ΔP(t) and the speed difference ΔV(t) of the user's dynamic image and the whole motion score is described for each beauty motion step.

[0512] The storage device 11 stores a recommendation model. In the recommendation model, the correlation between a combination of a position difference ΔP(t) and a speed difference ΔV(t) of a user's dynamic image and recommendation information is described for each beauty action step.

[0513] When the user operates the operation object B11120, the processor 12 determines the overall action score of each beauty action step corresponding to the combination of the position difference ΔP(t), speed difference ΔV(t) and acceleration difference ΔA(t) of the user's dynamic image of each beauty action step obtained in step S1112 (hereinafter referred to as the "step-by-step overall action score") and the overall action score of the beauty action as a whole including all beauty action steps by inputting the combination of the position difference ΔP(t), speed difference ΔV(t) and acceleration difference ΔA(t) into the overall action scoring model.

[0514] The processor 12 generates recommendation information corresponding to the combination of position difference ΔP(t), speed difference ΔV(t) and acceleration difference ΔA(t) of the user's dynamic image of each beauty action step obtained in step S1112 by inputting the combination into the recommendation model.

[0515] Processor 12 displays screen P6111 ( Fig.21 ) is displayed on the monitor.

[0516] Screen P6111 includes display objects A11120 to A11121 and A6111. Display objects A11120 to A11121 and Figure 7 It's the same.

[0517] The display object A6111 is an object that displays the overall action score by step (for example, the overall position score by step and the overall speed score by step).

[0518] After step S6112, the client device 10 and Figure 6 The update request is similarly executed (S1115).

[0519] After step S1115, the server 30 and Figure 6 The database is updated in the same manner (S1130).

[0520] (6-5-3) Summary of Modification 5

[0521] According to the fifth modification, navigation information is generated using a plurality of model examples M. Each model example M corresponds to one scenario. This makes it easy to add and change the pattern of the beauty action.

[0522] (6-6) Modification 6

[0523] Modification 6 is described. Modification 6 is an example in which navigation information is changed according to a combination of a beauty gesture and an expression.

[0524] (6-6-1) Overview of Modification Example 6

[0525] The outline of Modification Example 6 is described. Fig. 22 This is an explanatory diagram of the outline of Modification Example 6.

[0526] like Fig. 22 As shown, by analyzing the user dynamic image, the present embodiment ( Figure 3 )Similarly, determine the user position P(t) and user velocity V(t).

[0527] By inputting the user position P(t) and the user speed V(t) into the demonstration model M(Pm(t), Vm(t)), the present embodiment ( Figure 3 )Similarly, the position difference ΔP(t) and speed difference ΔV(t) are obtained.

[0528] By inputting the user's dynamic image into the expression evaluation model M(F(t)), the user's expression F(t) along the time series is estimated.

[0529] By inputting the position difference ΔP(t), the speed difference ΔV(t), and F(t) into the navigation model NM(ΔP(t), ΔV(t), F(t)), navigation information is obtained.

[0530] The navigation information is prompted to the user.

[0531] (6-6-2) Information Processing of Modification Example 6

[0532] The information processing of Modification Example 6 is described. Fig.23 This is a timing diagram of information processing in variant example 6. Fig.24 It is an explanatory diagram of the expression evaluation model (evaluation of the degree of smile) of Modification Example 6. Fig.25 It is an explanatory diagram of the expression evaluation model (evaluation of the degree of seriousness) of Modification Example 6.

[0533] like Fig.23 As shown, the client device 10 and Figure 6 The acquisition of the user's dynamic image ( S1110 ) to the evaluation of the action ( S1112 ) are performed in the same manner.

[0534] After step S1112 , the client device 10 performs evaluation of the expression ( S7110 ).

[0535] Specifically, the storage device 11 stores an expression evaluation model M(F(t)). The expression evaluation model M(F(t)) describes the correlation between the relative positional relationship of each part of the user's face (for example, eyebrows, eyes, and mouth) and the evaluation of the expression. The evaluation of the expression refers to the degree of emotion (for example, joy, anger, sadness, and happiness) shown on the user's face. For example, the evaluation of the expression is at least one of the degree of a smiling face, the degree of a serious face, and the degree of an unhappy face. The evaluation of the expression is an indicator of the user's subjective reaction to the beauty action.

[0536] like Fig.24 As shown, when evaluating the degree of smile, the evaluation target parts are the eyes and the mouth.

[0537] In the evaluation of eyes and mouth, the following values ​​were used as evaluation target indices.

[0538] ·Standstill time

[0539] Tilt

[0540] ·size

[0541] The difference between the position of a serious face and the position of a smiling face (as an example, the amount of change)

[0542] Number of repetitions of the action

[0543] For example, the degree of smile is evaluated based on at least one of the change in the position of the corner of the mouth and the descent of the corner of the eye. As an example, in at least one of the following cases, the degree of smile is evaluated to be high (that is, the user feels happy).

[0544] · Eye corners downward

[0545] · Mouth corners up

[0546] like Fig.25 As shown, when evaluating the degree of seriousness of a face, the evaluation target parts are the face, eyes, mouth, neck, chin, ears, and hands.

[0547] In the evaluation of the face, the following values ​​are used as evaluation object indicators.

[0548] ·size

[0549] In the evaluation of eyes and mouth, the following values ​​were used as evaluation target indices.

[0550] ·Standstill time

[0551] Tilt

[0552] ·size

[0553] The difference between the position of a serious face and the position of a smiling face (as an example, the amount of change)

[0554] Number of repetitions of the action

[0555] In the evaluation of the neck, the following values ​​were used as evaluation indexes.

[0556] ·Standstill time

[0557] Tilt

[0558] The difference between the position of a serious face and the position of a smiling face (as an example, the amount of change)

[0559] Number of repetitions of the action

[0560] In the evaluation of the chin, the following values ​​were used as evaluation target indices.

[0561] ·speed

[0562] ·Standstill time

[0563] Tilt

[0564] The difference between the position of a serious face and the position of a smiling face (as an example, the amount of change)

[0565] Number of repetitions of the action

[0566] In the evaluation of the ear, the following values ​​were used as evaluation target indices.

[0567] ·speed

[0568] Tilt

[0569] In the evaluation of the hand, the following values ​​are used as evaluation target indicators.

[0570] ·speed

[0571] ·Standstill time

[0572] Tilt

[0573] For example, the degree of seriousness is evaluated based on at least one of the opening of the eyelids, the change in the position of the eyebrows, and the shape of the mouth. As an example, in at least one of the following cases, the degree of seriousness is evaluated to be high (that is, the user feels uncomfortable).

[0574] Narrowing of the brows

[0575] Squint

[0576] Pursed lips

[0577] The processor 12 inputs the user dynamic image into the expression evaluation model M(F(t)).

[0578] The expression evaluation model M(F(t)) is constructed by Fig.24 or Fig.25 The value of the evaluation target index is calculated for each evaluation target part, and the evaluation of the expression is output.

[0579] After step S7110 , the client device 10 performs navigation ( S7111 ).

[0580] Specifically, a navigation model NM is stored in the storage device 11. The navigation model NM describes the correlation between the combination of the position difference ΔP(t), the speed difference ΔV(t), and the evaluation of the expression, and the navigation information.

[0581] The processor 12 inputs the position difference ΔP(t) and the speed difference ΔV(t) obtained in step S1112 into the navigation model NM, thereby generating navigation information corresponding to a combination of the position difference ΔP(t), the speed difference ΔV(t), and the evaluation of the expression.

[0582] After step S7111, the client device 10 and Figure 6 The recommendation ( S1114 ) to the update request ( S1115 ) are performed in the same manner.

[0583] After step S1115, the server 30 and Figure 6 The database is updated in the same manner (S1130).

[0584] (6-6-3) Summary of Modification 6

[0585] According to the sixth modification, the navigation information presented to the user is changed according to the combination of the motion and expression of each beauty target part of the user.

[0586] This makes it possible to present navigation information that satisfies the user in the form of an expression, thereby providing the user with an incentive to continue the beauty treatment.

[0587] (6-7) Modification 7

[0588] Modification 7 will be described. Modification 7 is an example of presenting navigation information according to the movement of the head, neck, or face.

[0589] (6-7-1) Overview of Modification Example 7

[0590] The outline of Modification Example 7 is described. Fig.26 This is an explanatory diagram of the outline of Modification Example 7.

[0591] like Fig.26 As shown, by analyzing the user's motion image of beauty movements (movements of the head, neck, or face), the user's position P(t) in each frame F(t) of the user's motion image is determined. t is an example of information for identifying a frame.

[0592] By inputting the user position P(t) into the demonstration model M(Pm(t)), the position difference ΔP(t) is obtained.

[0593] By inputting the position difference ΔP(t) into the navigation model NM(ΔP(t), ΔV(t)), navigation information is obtained.

[0594] The navigation information is prompted to the user.

[0595] (6-7-2) Information Processing of Modification Example 7

[0596] The information processing of Modification Example 7 is described.

[0597] like Figure 6 As shown, the client device 10 obtains the user dynamic image (S1110).

[0598] Specifically, the processor 12 converts the picture P0 ( Figure 7 ) is displayed on the monitor.

[0599] When the user operates the operation object B2, the processor 12 displays a screen image P1110 on the display.

[0600] When the user aligns the position of his / her face with the guide of the display object A1110 and operates the operation object B1110, the camera 15 starts capturing a dynamic image of the user.

[0601] The processor 12 acquires the user dynamic image captured by the camera 15. When the user operates the operation object B 1110 and then performs a beauty action, the user dynamic image includes an image of the beauty action.

[0602] After step S1110 , the client device 10 performs image analysis ( S1111 ).

[0603] Specifically, the processor 12 analyzes the user's dynamic image to recognize feature points of the beauty target part for each frame constituting the user's dynamic image.

[0604] The beauty target site includes, for example, at least one of the following sites.

[0605] ·head

[0606] ·eyebrow

[0607] ·Eye

[0608] ·nose

[0609] ·Mouth

[0610] ·cheek

[0611] ·neck

[0612] For example, the beauty action includes at least one of the following actions.

[0613] · Movement of the head (for example, up or down)

[0614] Eyebrow movements (for example, raising or lowering eyebrows)

[0615] Nose movement (for example, moving the nose or keeping the nose still)

[0616] Eye movements (for example, opening or closing the eyes)

[0617] Mouth movements (for example, opening or closing the mouth)

[0618] Cheek movements (for example, puffing up the cheeks or hollowing the cheeks)

[0619] Neck movement (for example, tilting the neck or straightening the neck)

[0620] After step S1111 , the client device 10 performs evaluation of the action ( S1112 ).

[0621] Specifically, the storage device 11 stores a model M. A model action is described in the model M. The model action is defined by a model position Pm of each part of the head or face. When the model position Pm(t1) in frame t1 and the model position P(t2) in frame t2 indicate the same position, it means that the position of the beauty action is stationary in frames t1 to t2.

[0622] The processor 12 refers to the model M and calculates the difference between the user position P(t) and the model position Pm(t), ie, the position difference ΔP(t).

[0623] The time series scoring model is stored in the storage device 11. The time series scoring model describes the correlation between the evaluation result of the action (for example, the position difference ΔP(t)) and the time series action score.

[0624] When the processor 12 inputs the position difference ΔP(t) into the time series scoring model, the scoring model outputs a time series position score corresponding to the position difference ΔP(t).

[0625] After step S1112 , the client device 10 generates navigation information ( S1113 ).

[0626] The storage device 11 stores a navigation model NM. The navigation model NM describes the correlation between the position difference ΔP(t) and the navigation information.

[0627] The processor 12 generates navigation information corresponding to the position difference ΔP(t) obtained in step S1112 by inputting the position difference ΔP(t) into the navigation model NM.

[0628] A specific example of the navigation information is at least one of the first to third examples in step S1113.

[0629] After step S1113, the client device 10 and Figure 6 The recommendation ( S1114 ) to the update request ( S1115 ) are performed in the same manner.

[0630] After step S1115, the server 30 and Figure 6 The database is updated in the same manner (S1130).

[0631] (6-7-3) Summary of Modification 7

[0632] According to Modification 7, navigation information (that is, navigation information when massaging the user's face without hands) is presented to the user according to the beauty action for each part of the user's face.

[0633] This allows the user to perform hands-free beauty actions in consideration of the navigation information, thereby providing an incentive for the user to continue the beauty actions.

[0634] In Modification 7, an example is shown in which the position difference ΔP(t) of the face parts is input to the navigation model NM (that is, the navigation information is generated based on the position difference ΔP(t)), but the scope of Modification 7 is not limited to this. Modification 7 can also be applied to an example in which the navigation information is generated by inputting a combination of the position difference ΔP(t) and the velocity difference ΔV(t) of the face parts to the navigation model NM (that is, the navigation information is generated based on the combination of the position difference ΔP(t) and the velocity difference ΔV(t)).

[0635] (7) Other Modifications

[0636] Other modified examples will be described.

[0637] The storage device 11 may be connected to the client device 10 via the network NW. The storage device 31 may be connected to the server 30 via the network NW.

[0638] Each step of the above-mentioned information processing may be executed in either the client device 10 or the server 30 .

[0639] For example, when the client device 10 can execute all the steps of the above-described information processing, the client device 10 functions as an information processing device that operates independently without sending a request to the server 30 .

[0640] In this embodiment, at least one of the following hand images may be used as a navigation image in the screen P1111. In this case, the processor 12 changes the hand image according to the user's position (for example, when the cheek should be cared for, the hand image is generated in a manner that indicates the movement of the hand suitable for cheek care).

[0641] · Pre-captured image of the user’s hand

[0642] · Pre-registered computer graphics of hands

[0643] In the present embodiment, the navigation model NM may be set for each problem (trouble) of the user.

[0644] For example, in navigation (S1113), the processor 12 refers to the "skin problem" field of the user database to determine the user's skin problem information.

[0645] The processor 12 selects a navigation model NM corresponding to the determined skin problem information from among the navigation models NM stored in the storage device 11 .

[0646] The processor 12 generates navigation information using the selected navigation model NM.

[0647] In this embodiment, an example is shown in which the navigation model NM uses at least one of a navigation image and a navigation voice to prompt the user of navigation information, but the present invention is not limited thereto. This embodiment can also be applied to an example in which the navigation model NM prompts the user of navigation information through vibration.

[0648] In this embodiment, if Figure 6 As shown, an example is shown in which recommendation ( S1114 ) is performed after navigation ( S1113 ), but the scope of the present embodiment is not limited to this.

[0649] This embodiment can also be applied to an example in which the recommendation ( S1114 ) is performed when a predetermined condition is satisfied.

[0650] The predetermined condition is, for example, at least one of the following.

[0651] The action score reaches above the predetermined threshold

[0652] The change in the action score (e.g., the difference from the immediately previous action score) exceeds a predetermined threshold.

[0653] In this embodiment, an example of determining the user position P(t), user speed V(t), user pressing force PR(t), user rhythm T(t), and user acceleration A(t) according to the independent variable t of the frame is shown, but the scope of this embodiment is not limited to this.

[0654] This embodiment can also be applied to an example in which the user position, user speed, user pressing force, user rhythm, and user acceleration are determined based on a combination of a plurality of frames in a predetermined period (hereinafter referred to as a “frame group”).

[0655] For example, in the image analysis ( S1111 ), the processor 12 calculates the average value of the user position, the average value of the user speed, the average value of the user pressing force, the average value of the user rhythm, and the average value of the user acceleration for each frame group.

[0656] Thus, even if the user's action at a certain moment deviates from the demonstration action, but the user's action within a predetermined period does not deviate greatly from the demonstration action, it is possible to indicate navigation information as if the user's action does not deviate from the demonstration action. For example, when the user's action rotates the hand, even if the user deviates to the left or right within a certain distance from the demonstration action, it is possible to indicate navigation information as if the user's action does not deviate from the demonstration action. Therefore, even if the user improves the user's action after seeing the navigation information, it is possible to guide the user to appropriately improve the user's action.

[0657] In this embodiment, the screen P1111 ( Figure 8 ) shows an example of displaying the action scores along the time series in the form of a curve graph, but the scope of this embodiment is not limited to this. This embodiment can also be applied to an example of displaying the action scores along the time series in the form of a heat map of the trajectory.

[0658] Thus, in the evaluation of the position, the user can be prompted with a more understandable visual effect whether the action of each part is good or bad. For example, when the user uniformly applies foundation on the face, he can easily know whether he has not applied it or applied too much.

[0659] In this embodiment, an example in which navigation information is presented during the execution of a beauty action is shown, but the scope of this embodiment is not limited thereto. This embodiment can also be applied to an example in which navigation information is presented after the execution of a beauty action.

[0660] In this case, for example, when the user gives a user instruction to the client device 10 to perform a beauty treatment action, the client device 10 transmits the user instruction to the server 30 .

[0661] The server 30 transmits navigation information corresponding to the beauty action to the client device 10 according to the user instruction.

[0662] The client device 10 displays the navigation information on a display.

[0663] This allows the user to confirm the navigation information after completing the beauty treatment.

[0664] In the present embodiment, an example in which a common model M is used in the evaluation of the action ( S1112 ) is shown, but the scope of the present embodiment is not limited thereto. The present embodiment can also be applied to an example in which the model M is changed for each user.

[0665] In the first example, the storage device 11 stores a model M of each user attribute. In the evaluation of the action (S1112), the processor 12 refers to the user database ( Figure 4 ), determine the user attribute (for example, gender) associated with the user identification information, and select a demonstration model M corresponding to the user attribute.

[0666] In the second example, the storage device 11 stores the exemplary model M preferred by each user. In the evaluation of the action (S1112), the processor 12 refers to the user database ( Figure 4 ), determine the user preference (for example, appearance) associated with the user identification information, and select a demonstration model M corresponding to the user preference.

[0667] In the third example, the storage device 11 stores a model M for each user attribute, a model M for each user preference, and a model M for each skin problem. In the evaluation of the action (S1112), the processor 12 refers to the user database ( Figure 4 ), determine the skin problem of the user associated with the user identification information, and select a demonstration model M corresponding to the skin problem.

[0668] The embodiments of the present invention are described in detail above, but the scope of the present invention is not limited to the above embodiments. In addition, the above embodiments can be variously improved and changed within the scope of the gist of the present invention. In addition, the above embodiments and modifications can be combined.

[0669] Description of Reference Numerals

[0670] 1: Information Processing System

[0671] 10: Client Device

[0672] 11: Storage device

[0673] 12: Processor

[0674] 13: Input and output interface

[0675] 14: Communication interface

[0676] 15: Camera

[0677] 20: Wearable Sensors

[0678] 30: Server

[0679] 31: Storage device

[0680] 32: Processor

[0681] 33: Input and output interface

[0682] 34: Communication interface

Claims

1. An information processing device comprising: a unit for acquiring a user dynamic image including a user's hand beauty action on each beauty target part; a unit for determining a motion difference between the demonstration motion and the beauty motion by comparing the demonstration motion and the beauty motion, the motion difference including a motion difference related to a position of the beauty motion, i.e., a position difference, and a motion difference related to a speed of the beauty motion, i.e., a speed difference; and A generating unit generates navigation information corresponding to the motion difference of each of the beauty target parts.

2. The information processing device according to claim 1, A presenting unit is provided for presenting the navigation information to the user.

3. The information processing device according to claim 2, The generating unit generates a navigation image as the navigation information, The prompting unit displays a navigation image superimposed on the user dynamic image.

4. The information processing device according to claim 3, The navigation image includes a position guidance image and a speed guidance image, wherein the position guidance image is an image for guiding the position of the beauty action, and the speed guidance image is an image for guiding the speed of the beauty action. The prompt unit, causing the position guide image to be displayed superimposed on the user dynamic image, The speed guide image is displayed superimposed on the user dynamic image and changes according to the speed of the beauty action.

5. The information processing device according to claim 3 or 4, The generation unit generates, as the navigation image, an image of a hand that changes according to a position of the cosmetic action.

6. The information processing device according to any one of claims 1 to 4, The generating unit generates a navigation sound as the navigation information by converting predetermined sound information according to the motion difference.

7. The information processing device according to any one of claims 1 to 4, The motion difference includes an acceleration difference associated with an acceleration of a hand of the user.

8. The information processing device according to any one of claims 1 to 4, The motion difference includes a pressing pressure difference associated with the pressing force applied to the face by the user, that is, the user pressing force.

9. The information processing device according to any one of claims 1 to 4, The motion difference includes a rhythm difference associated with a rhythm of movement of the user's hand.

10. The information processing device according to any one of claims 1 to 4, comprising: A unit for displaying the user's avatar image superimposed on the user's facial image; and A changing unit changes pixels of the avatar image at a position where the beauty action is performed.

11. The information processing device according to claim 10, The changing unit exposes the facial image of the user at the position where the beauty action is performed by erasing pixels of the avatar image at the position where the beauty action is performed.

12. The information processing device according to claim 10, The changing unit applies makeup to the avatar image by changing the color of pixels of the avatar image at a position where the beauty action is performed.

13. The information processing device according to any one of claims 1 to 4, comprising: a calculation unit for calculating a score of the beauty action; and A unit for prompting the user with the navigation information and the score during the performance of the cosmetic action.

14. The information processing device according to claim 13, The calculation unit calculates the score based on a scenario describing a time-series model action for each of the beauty target parts and each of the beauty action types.

15. The information processing device according to any one of claims 1 to 4, A unit for analyzing the user's expression is provided, The unit for generating the navigation information generates navigation information according to a combination of the motion difference and the analysis result of the expression for each of the beauty target parts.

16. An information processing method is an information processing method using a computer, comprising: A step of acquiring a user dynamic image including a user's hand beauty action on each beauty target part; The step of determining a motion difference between the demonstration motion and the beauty motion by comparing the demonstration motion and the beauty motion, wherein the motion difference includes a motion difference related to a position of the beauty motion, i.e., a position difference, and a motion difference related to a speed of the beauty motion, i.e., a speed difference; and A step of generating navigation information corresponding to the motion difference of each of the beauty target parts.

17. A program that causes a computer to operate as the following unit: a unit for acquiring a user dynamic image including a user's hand beauty action on each beauty target part; a unit for determining a motion difference between the demonstration motion and the beauty motion by comparing the demonstration motion and the beauty motion, the motion difference including a motion difference related to a position of the beauty motion, i.e., a position difference, and a motion difference related to a speed of the beauty motion, i.e., a speed difference; and A unit for generating navigation information corresponding to the movement difference of each of the beauty target parts.

Images