Treatment system and treatment program

Abstract

To provide a treatment system which can estimate a factor of emotion arisen in a customer, by detecting a visual line direction of the customer when calculating the customer's emotion.SOLUTION: A treatment system includes: a camera 10 for imaging the upper half area of a body including at least an expression of a treated person C when a practitioner H applies a prescribed treatment to a treatment object of the treated person C; a display 20 which is disposed opposite a front face of the treated person C, in order to display imaging information photographed by the camera 10 and / or processed imaging information that has been processed on the basis of the imaging information; a visual line detection part 46 for detecting a visual line direction of the treated person C on the basis of the imaging information; an emotion calculation part 45 for calculating emotion information of the treated person C, on the basis of the expression of the treated person C contained in the imaging information; a visual line emotion storage part 47 for storing a target object of a destination indicated by the visual line direction of the treated person C, in association with the emotion information of the treated person C when the visual line is directed toward the target object; and a practitioner terminal 30 for notifying the practitioner of the emotion information of the treated person for each target object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a treatment system that calculates the emotions of a person receiving a treatment when a prescribed treatment is given. [Background technology]

[0002] For example, in a service such as a beauty salon where a therapist performs a specific treatment on a patient, it is important to increase customer satisfaction and attract more customers. For example, Patent Document 1 discloses a method of cutting the back of the head of a patient while presenting a back image of the back of the head, which is not visible to the patient, to the patient on a display device. It also discloses displaying a past image, a current image, or a simulated image of the treatment on the display device.

[0003] Patent Document 2 discloses a system for displaying a customer's satisfaction level as a numerical value by performing image recognition (image analysis) on the facial expression of the customer from information captured by a camera in a beauty salon or the like (see, for example,

[0055] ). Patent Document 3 discloses a system for detecting a user's reaction to a makeup simulation being performed (the angle and deformation of the user's facial expression parameters (eye direction and face), gestures, voice and dialogue content, and biological signals such as blood pressure and heart rate) in a makeup simulation, and detecting the user's satisfaction level, or more specifically, the user's interest level plus the user's evaluation (high evaluation, low evaluation) according to the detection results (see, for example,

[0029] and

[0030] ). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-13914 A [Patent Document 2] JP 2021-105916 A [Patent Document 3] Patent No. 6748981 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 cuts a customer's hair while showing the back of the customer's head, but it is not possible to grasp the customer's feelings regarding the haircut.

[0006] In addition, the technologies shown in Patent Documents 2 and 3 detect a customer's satisfaction and emotions during treatment, but even if it is possible to detect an individual customer's satisfaction and emotions at that time, it is not possible to detect the causes of that satisfaction or emotion (for example, what emotions a customer had in response to a particular area or type of treatment), which makes it difficult to fully satisfy customers.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to provide a treatment system and a program therefor that can estimate the cause of the emotion a customer has by detecting the direction of the customer's gaze when calculating the customer's emotion. [Means for solving the problem]

[0008] The treatment system of the present invention comprises an imaging means for imaging at least the upper body region of the recipient, including facial expressions, when a therapist performs a specified treatment on the recipient's treatment target, a display means installed facing the front of the recipient and displaying imaging information captured by the imaging means and / or processed imaging information processed based on the imaging information, a gaze detection means for detecting the gaze direction of the recipient based on the imaging information, an emotion calculation means for calculating emotional information of the recipient based on the facial expression of the recipient contained in the imaging information, an emotion area storage means for storing an attention target indicated by the gaze direction of the recipient and emotional information of the recipient when his / her gaze is directed at the attention target in association with each other, and a notification means for notifying the emotional information of the recipient for each attention target.

[0009] In this way, in the treatment system of the present invention, an imaging means captures an image of at least the upper body area of ​​the recipient, including at least the facial expression, when a therapist performs a specified treatment on the treatment target of the recipient, the imaging means is installed facing the front of the recipient, and the display means displays the imaging information captured by the imaging means and / or processed imaging information processed based on the imaging information, detects the gaze direction of the recipient based on the imaging information, calculates emotional information of the recipient based on the facial expression of the recipient contained in the imaging information, associates and stores the target of attention indicated by the gaze direction of the recipient with the emotional information of the recipient when the gaze is directed at the target of attention, and notifies the therapist of the emotional information of the recipient for each target of attention, thereby achieving the effect of presuming that the cause of the recipient's emotions is in the area indicated by the gaze direction at that time, and notifying the therapist of this, so that the therapist can take appropriate action corresponding to the emotions of the recipient. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a system configuration of a treatment system according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram showing the configuration of a calculation device in the treatment system according to the first embodiment. [Diagram 3] 3 is a diagram showing an example of gaze emotion information stored in a gaze emotion storage unit in the treatment system according to the first embodiment; FIG. [Figure 4] FIG. 4 is a diagram showing an example of an image displayed on a therapist terminal in the treatment system according to the first embodiment. [Diagram 5] 4 is a flowchart showing a processing operation of the treatment system according to the first embodiment. [Figure 6] FIG. 11 is a functional block diagram showing the configuration of a calculation device in the treatment system according to the second embodiment. [Figure 7] 13 is a graph showing changes in emotion information of a subject over time in a moving image captured by a camera of the treatment system according to the second embodiment. [Figure 8]FIG. 13 is a diagram showing a system configuration of a treatment system according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (First embodiment of the present invention) The treatment system according to the present embodiment will be described with reference to Figures 1 to 5. The treatment system according to the present embodiment is a treatment system used, for example, in a beauty salon when a hairdresser who is a practitioner performs a predetermined treatment such as cutting or dyeing hair on the hair of a customer who is a patient.

[0012] Fig. 1 is a diagram showing the system configuration of the treatment system according to the present embodiment. In Fig. 1, the treatment system 1 includes a camera 10 that captures at least the upper body region including the hair and facial expression of the person C from the front of the person C, a display 20 that is installed facing the front of the person C and displays the image information captured by the camera 10, a therapist terminal 30 that is carried by the therapist H and has a display that displays at least the entire hair region of the person C and the emotional information of the therapist C, and a calculation device 40 that is connected to each device in a communicable manner and is a computer that performs calculation processing while managing the entire treatment system.

[0013] One or more cameras 10 are installed around the position where the patient C receives treatment. In Fig. 1, in addition to the camera 10a facing the patient C, cameras 10b and 10c that capture images of the patient C diagonally from the front and rear, and cameras 10d and 10e that capture images of the patient C diagonally from the rear and rear, totaling five cameras, are provided.

[0014] The display 20 displays the image information captured by the camera 10 via the computing device 40. This display 20 has the function of a mirror that is usually installed in a beauty salon, and therefore the computing device 40 edits the image information captured by the camera 10 into image information that looks like a mirror when viewed from the patient C, and displays it on the display 20. At this time, the front image information captured by the camera 50a is mainly displayed on the display 20, but the image information captured by the cameras 50b to 50e is also displayed in the free space in the display 20, so that the patient C can see his or her own treatment state from multiple angles.

[0015] The therapist terminal 30 receives and displays the emotional information of the person C calculated by the arithmetic device 40. At this time, the arithmetic device 40 calculates the emotional information of the person C and displays the emotional information on the therapist terminal 30 in real time, which enables the therapist H to take appropriate action corresponding to the emotion of the person C.

[0016] The arithmetic device 40 performs processing such as analyzing image information captured by the camera 10 to calculate emotion information of the person C to be treated, and detecting the line of sight of the person C to be treated. The configuration and processing of the arithmetic device 40 will be described in detail below.

[0017] 2 is a functional block diagram showing the configuration of the arithmetic device 40 in the treatment system 1 according to this embodiment. The arithmetic device 40 includes a receiving unit 41 that receives imaging information transmitted from the camera 10, an editing processing unit 42 that performs image conversion (e.g., conversion of coordinates, direction, size, etc.), image editing (e.g., adjustment of color tone, contrast, noise, etc.) and / or processing (e.g., processing into a virtual hairstyle, hair color, makeup, etc. for simulation) on the received imaging information for displaying it on the display 20, and an output control unit 43 that outputs the edited imaging information to the display 20 or other devices as necessary.

[0018] The calculation device 40 also includes an image analysis unit 44 that performs image analysis of the imaging information from the camera 50 received by the receiving unit 41, an emotion calculation unit 45 that calculates emotional information of the recipient C from the facial expression of the recipient C in the imaging information, a gaze detection unit 46 that extracts the eye area from the facial area of ​​the recipient C in the imaging information and calculates where on the display 20 the recipient C is looking at, a gaze emotion memory unit 47 that stores gaze emotion information that corresponds the gaze direction detected by the gaze detection unit 46 with the emotion information calculated by the emotion calculation unit 45, and an image information generation unit 48 that generates image information for expressing the gaze emotion information as an image from the gaze emotion information stored in the gaze emotion information memory unit 47 and sends the image information to the editing processing unit 42.

[0019] As described above, the camera 10 has at least the camera 10a that captures the image of the person C from the front, and the editing processing unit 42 edits the image information captured by the camera 10a, so that the image of the person C is displayed on the display 20 as if it were reflected in a mirror as seen by the person C. Specifically, the editing processing unit 42 changes the angle and size of the image information and displays it on the display 20 in a left-right inverted state, taking into consideration, for example, the information on the position and angle of the camera 10a, the information on the position and angle of the display 20, and the information on the position and angle of the seat on which the person C is seated, thereby editing the image information so that the person C is reflected in a mirror. In addition to the above, the editing processing unit 42 also edits the emotional information of the person C to be displayed on the therapist terminal 30, etc., as will be described in detail later. The image information edited by the editing processing unit 42 is transmitted to the display 20 or the therapist terminal 30 by the output control unit 43 and displayed.

[0020] The image analysis unit 44 is a processing unit that performs image analysis of the imaging information captured by the camera 50. Specifically, the image analysis unit 44 extracts the imaging target included in the imaging information by image recognition and calculates which imaging target is displayed at which position on the display 20. The image recognition of the imaging target included in the imaging information is performed by using machine learning (e.g., neural network or deep learning), and for example, identifies in which area of ​​the display 20 the hair, face, neck, chest, shoulders, arms, abdomen, etc. of the patient C are displayed based on the coordinate range of the display 20. In addition to this, a process is performed to identify in which area of ​​the display 20 the area in which the therapist H is displayed and the imaging target in the background (e.g., treatment tools, other patients C, other therapists H, wall clocks, decorations, posters, walls, etc.) are displayed based on the coordinate range of the display 20. The results identified here are stored in a storage unit (not shown) as image analysis information that associates the imaging target with its coordinate area, for example.

[0021] For objects in the background that do not change much, such as walls and wall clocks, the coordinates of the range displayed on the display 20 may be registered in advance as fixed values.

[0022] The emotion calculation unit 45 calculates the emotion of the treated person C from the facial expression of the facial region of the treated person C in the imaging information specified by the image analysis unit 44. A known technique can be used for the process of calculating emotion from a person's facial expression by image analysis. Specifically, for example, in the extracted facial region of the treated person C, a plurality of feature points in the face are specified, and the emotion of the treated person C is estimated from the positional relationship of these feature points. Examples of the estimated emotion information include anger, disgust, fear, happiness, loneliness, surprise, and no emotion. In addition, for example, the pulse rate is obtained from the color change caused by the capillaries on the skin surface, which changes extremely minutely according to the pulse rate, and an emotion according to the change in the pulse rate (for example, tension when the pulse rate is fast, and relaxation when the pulse rate is slow, etc.) may be estimated. In addition, the emotion calculation unit 45 may calculate the customer satisfaction level from statistics of the calculated emotion information. For example, a score may be set according to each emotion information and the depth of the emotion (the calculation of the depth of the emotion will be described in detail later), and the total score of the emotion information generated during the treatment may be calculated as the customer satisfaction level.

[0023] The gaze detection unit 46 calculates the gaze direction of the subject C from the relationship between the area of ​​the pupil and the area of ​​the white of the eye, for example, for the eye area of ​​the facial area of ​​the subject C in the imaging information identified by the image analysis unit 44. For gaze detection, known techniques may be used. For example, see Kohei Arai, “Method for leaning efficiency improvements based on gaze location notifications on e-learning content screen display”, International Journal of Advanced Research in Artificial Intelligence, 1, 3, 1-6, 2012. (Reference 1); Kohei Arai, “Ronny Mardiaynto, Method for psychological status estimation by gaze location monitoring using eye-based Human-Computer Interaction”, International Journal of Advanced Computer Science and Applications, 4, 3, 199-206, 2013. (Reference 2); and Kohei Arai, Akihiro Yamashita, Hiroshi Okumura, Pedestrian safety with eye contact between autonomous car and pedestrian, International Journal of Advanced Computer Science and Applications IJACSA, 10, 5, 161-165, By using technologies such as "(Reference 3), it is possible to detect with high accuracy which position (coordinates) on the display 20 the gaze direction of the subject C is pointing to.

[0024] The gaze emotion storage unit 47 stores gaze emotion information in which the coordinates of the display 20 indicated by the gaze direction detected by the gaze detection unit 46 correspond to emotion information calculated by the emotion calculation unit 45 when the gaze direction is detected. FIG. 3 is a diagram showing an example of gaze emotion information stored by the gaze emotion storage unit 47 in the treatment system 1 according to this embodiment. As shown in FIG. 3, the coordinates of the display 20 where the gaze direction is detected in chronological order and the emotion information calculated at that time are stored. In addition, as shown in FIG. 3, if necessary, the time when the gaze is stationary or the time when it remains within a predetermined coordinate range, the target of attention indicated by the gaze direction, etc. may also be registered. This target of attention can be extracted from the image analysis information registered by the image analysis unit 44 described above.

[0025] In addition, the column of the target of attention shown in FIG. 3 stores the names of blocks into which the hair region of the recipient C displayed on the display 20 is divided; details of this division into blocks will be described later.

[0026] 3 shows the gaze emotion information when the gaze detection unit 46 detects the gaze only when the gaze direction of the patient C remains still for a predetermined time (set to 3 seconds here) or remains within a predetermined coordinate range. That is, for example, when the gaze direction remains still for a short time less than the predetermined time or remains within a predetermined coordinate range, it is presumed that the patient C does not pay high attention to the gaze emotion information, and it is not included in the gaze emotion information. Conversely, when the gaze direction remains still or remains within a predetermined coordinate range for a predetermined time or more, it is presumed that the patient C is aware of that area and pays high attention to it, and it is included in the gaze emotion information.

[0027] Based on the gaze emotion information as shown in FIG. 3 stored in the gaze emotion storage unit 47, the image information generating unit 48 generates image information to be displayed on the display of the therapist terminal 30, specifically, image information serving as a parameter for editing the captured image information into an image as described below. Here, the image to be displayed on the display of the therapist terminal 30 will be described. FIG. 4 is a diagram showing an example of an image to be displayed on the therapist terminal 30 in the treatment system 1 according to this embodiment. In FIG. 4, the part above the shoulders of the treated person C in the captured image information is displayed, and the hair region is displayed in a color-coded manner for each block (A1 to A3, B1 to B3, C1 to C2). In FIG. 4, the different colors are expressed by hatching for convenience of the drawing, and the division of the blocks can be set arbitrarily. Emotion information is associated with the difference in the displayed color, and for example, the closer to the first color (any color) the more negative the emotion information (disgust, sadness, anger, etc.) is indicated, and the closer to the second color (any color other than the first color) the more positive the emotion information (happiness, satisfaction, joy, etc.) is indicated. That is, the image information generating unit 48 extracts hair blocks and emotion information from the coordinates indicated by the gaze direction based on the gaze emotion information shown in Fig. 3, and sets the color indicated by the emotion information corresponding to the block to the color of the block. Then, each time the coordinates indicated by the gaze direction change, color information indicated by the emotion information is set for the hair block corresponding to each coordinate. The color information set for each block is sent to the editing processing unit 42 as image information, and is displayed on the practitioner terminal 30 as an image as shown in Fig. 4 via the output control unit 43.

[0028] The practitioner H can refer to the image displayed on the practitioner terminal 30 without the patient C knowing, and instantly understand from the difference in color what emotion the patient C has for each block of hair. The practitioner H can then take action to shift the patient C's emotion to a positive emotion by focusing on the block that has a negative emotion, or by adding additional treatment or modification.

[0029] A series of processes from the imaging process by the camera 10 to the display of the image on the display 20 and the therapist terminal 30 is performed in real time. That is, the therapist H can recognize the emotion information of the patient C in real time.

[0030] In addition, even if the gaze direction of the patient C moves from one block to another block, it is preferable that the color of the one block is maintained as it is and displayed on the therapist terminal 30. Then, when the gaze direction points to the one block again, the color of the one block may be updated according to the emotion information at that time.

[0031] Furthermore, regarding the shade of the color indicating the emotion information, an emotion intensity calculation unit (not shown) may calculate the shade according to the time during which the gaze direction remains still or remains within a predetermined coordinate range (specifically, within a block). For example, if the gaze direction remains within a block for a predetermined time (for example, about 3 seconds) or less, it is presumed that the emotion at that time is shallow, and the block may be displayed in a light color corresponding to this emotion information, and if the gaze direction remains for a time significantly longer than the predetermined time, it is presumed that the emotion at that time is deep, and the block may be displayed in a dark color corresponding to this emotion information. At this time, the shade of the displayed color may be changed according to the continuous time during which the gaze direction remains within the block, or the proportion of the time during which the gaze direction remains within a predetermined time range (for example, within a range of 3 minutes) even if not continuous. In this case, the color of the block may be displayed so as to gradually change as the time during which the gaze direction remains still or remains within a predetermined coordinate range passes. In this way, by displaying the depth of emotion with the shade of color, the therapist H can recognize the depth of emotion of the treated person C.

[0032] 4 shows an example in which the hair region is colored according to emotion information for each block, but other regions constituting the recipient C, including the facial region of the recipient C, and other imaged objects displayed on the display 20 as background may also be colored according to emotion information. In this way, it is possible to collect emotion information of the recipient C about items that are not directly related to the treatment but should be considered in terms of customer satisfaction, such as the interior decoration, decorations, and atmosphere of the store, and this information can be used to increase customer satisfaction throughout the store.

[0033] Furthermore, in the above description, the emotional information of the recipient C is displayed as an image on the practitioner terminal 30, but the practitioner terminal 30 may be a mobile terminal such as a smartphone or tablet, or may be a monocular display or a head-mounted display that is worn by the practitioner H. Also, instead of the practitioner terminal 30, the recipient C may wear earphones, and the emotional information for each hair block of the recipient C may be notified by voice through the earphones. For example, if the hair region is divided into blocks A1, A2, A3, . . . , C3, C3, the emotional information at that time may be read out by voice in order from A1, and this may be constantly repeated from A1 to C3.

[0034] Furthermore, the above series of processes may be performed on imaging information obtained by directly imaging the patient C, or on a simulation image that virtually simulates the hairstyle, hair color, and / or makeup of the patient C. By recognizing the emotions of the patient C with respect to the simulation image, it becomes possible to select a treatment method that is closer to the intention of the patient C. This virtual simulation image may be displayed not only for a simulation before treatment, but also periodically or irregularly during treatment, and the direction of gaze with respect to the simulation image may be associated with emotion information.

[0035] Furthermore, after the treatment, the state before the treatment and the state after the treatment may be displayed on the display 20 in comparison, and emotional information may be associated with the gaze direction. This makes it possible to collect information indicating the satisfaction level of the treated person C after the treatment.

[0036] Next, the processing operation of the treatment system 1 will be described. FIG. 5 is a flowchart showing the processing operation of the treatment system 1 according to this embodiment. First, as a premise, the treatment system 1 is started before the treatment is started, and the processing operation is started from the point when the treated person C enters the imaging range of the camera 10. The camera 10 captures the image when the treated person C enters the imaging range (S1). The imaging information captured by the camera 10 is transmitted to the calculation device 40, and edited by the editing processing unit 42 so that it becomes an image reflected in a mirror as seen by the treated person C (S2), and is transmitted to the display 20 and displayed (S3). At this time, if the gaze detection and the calculation of the emotion information of the treated person C described later have already been performed, the output control unit 43 displays an image in which image information based on the gaze emotion information is added to the imaging information on the therapist terminal 30 (S4). Depending on whether or not the treatment is to be ended (S5), if the treatment is to be ended, a series of processes is ended.

[0037] If the treatment is continued in S5, or if the gaze detection or the calculation of the emotion information of the treated person C is not performed, the image analysis unit 44 performs image analysis processing on the imaging information displayed on the display 20 (S6). From the result of the image analysis, the emotion calculation unit 45 calculates emotion information based on the facial expression of the treated person C (S7), and the gaze detection unit 46 performs gaze detection of the treated person C (S8). The image information generation unit 48 generates image information required for editing an image to display the emotion information of the treated person C on the therapist terminal 30 from the gaze emotion information stored in the gaze emotion storage unit 47 in which the emotion information is associated with the area indicated by the gaze direction (S9). Returning to S2, the editing processing unit 42 edits the image to be displayed on the display 20, and creates an image by adding the image information generated in S9 to the imaging information (S2), and the display on the display 20 and the therapist terminal 30 are performed (S3, S4).

[0038] In this manner, in the treatment system according to the present embodiment, the camera 10 captures at least the upper body region of the recipient C, including the facial expression, when the therapist H performs a specified treatment on the treatment target of the recipient C, and the camera 10 is installed facing the front of the recipient C. The imaging information captured by the camera 10 and / or processed imaging information processed based on the imaging information are displayed on the display 20. The direction of the recipient C's gaze is detected based on the imaging information. Emotional information of the recipient C is calculated based on the facial expression of the recipient C contained in the imaging information. The target of attention (e.g., a block of hair) indicated by the direction of the recipient C's gaze is associated with and stored as emotional information of the recipient C when the recipient C's gaze is directed at the target of attention. The emotional information of the recipient C for each target of attention is notified to the therapist terminal 30, earphones, etc., so that the cause of the recipient C's emotion is estimated to be in the area indicated by the direction of the gaze at that time, and this is notified to the therapist H, which can lead to appropriate behavior corresponding to the emotion of the recipient C.

[0039] In addition, since the object of attention is any part of the entire hair area of ​​the recipient C that is the target of treatment, which is displayed on the display 20, and the practitioner terminal 30 notifies the practitioner H of the emotional information of the recipient C for each arbitrary part of the hair area, the practitioner H can know what emotion arose in which part of the hair area that was treated by the recipient C, and can provide the recipient with appropriate treatment and advice corresponding to that emotion, thereby increasing customer satisfaction.

[0040] Furthermore, the therapist terminal 30 is carried by the therapist H and is equipped with a display that displays at least the entire hair area and emotional information of the person being treated C, and the entire hair area of ​​the person being treated C displayed on the display is displayed in a predetermined display mode according to the emotional information corresponding to any part of the hair area stored in the gaze emotion memory unit 47. Therefore, with just one glance at the display, the therapist H can recognize what emotion is being felt in which area of ​​the person being treated C, and can provide the person being treated with appropriate treatment and advice in real time accordingly, thereby increasing customer satisfaction.

[0041] Furthermore, since the practitioner wears earphones and emotional information corresponding to any part of the hair region stored in the gaze emotion memory unit 47 is notified by voice through the earphones, the practitioner H can recognize the emotional information of the person C without the person C noticing.

[0042] Furthermore, the gaze detection unit 46 detects the gaze direction of the patient C when the gaze direction remains stationary for a predetermined period of time or more, so that emotional information is calculated for an area that the patient C is paying attention to to a certain extent, thereby reducing the processing load and improving the calculation accuracy of the emotional information.

[0043] Furthermore, since the strength of emotional information is calculated according to the length of continuous time that the gaze direction of the therapist H is stationary and / or the cumulative length of time that the gaze direction of the recipient C is stationary, it is estimated that the longer the gaze of the recipient C focuses attention, the deeper the emotion at that time is, and the emotion of the recipient C can be calculated with high accuracy.

[0044] Furthermore, the display 20 displays processed image information obtained by processing the hairstyle, hair color, and / or makeup of the person C in the image information, and the gaze detection unit 46 detects the gaze direction of the person C with respect to the processed image information, so that, for example, in pre-treatment counseling, the therapist H can recognize the emotions of the person C regarding the virtually simulated hairstyle, hair color, and / or makeup, thereby enabling high-quality counseling to be performed. In addition, when an image comparing the state before and after the treatment is displayed on the display 20 after the treatment, information can be collected by finding the degree of satisfaction with the treatment from the gaze direction and emotion information.

[0045] (Second embodiment of the present invention) The treatment system according to this embodiment will be described with reference to Figures 6 and 7. The treatment system according to this embodiment is a treatment system according to the first embodiment with additional functions, in which the camera 10 captures an image of an area including the treatment motion of the therapist H and stores the captured image information as a moving image, so that the therapist H can refer to his or her own treatment motion after the treatment and use it as material for improving his or her skills. Note that in this embodiment, explanations that overlap with those in the first embodiment will be omitted.

[0046] Fig. 6 is a functional block diagram showing the configuration of the arithmetic device 40 in the treatment system 1 according to this embodiment. The difference from the configuration in Fig. 2 in the first embodiment is that the imaging information captured by the camera 50 includes a moving image including the treatment action of the therapist H, and the system is provided with an imaging information storage unit 61 that stores the moving image received by the receiving unit 41, and an extraction and editing processing unit 62 that extracts and edits a moving image of a time period desired by the therapist H from the moving images stored in the imaging information storage unit 61 and passes the extracted and edited moving image to the output control unit 43, and the output control unit 43 outputs the extracted and edited moving image to a desired device 60 (for example, the therapist terminal 30 held by the therapist H during treatment, or another device 60 that can be used elsewhere).

[0047] The imaging information storage unit 61 stores moving images including the treatment motions of the practitioner H (for example, if the practitioner H is a hairdresser, the motions of the tips of both hands while cutting hair). These moving images can be referenced by accessing the computing device 40 from the external device 60, and the practitioner H can refer to his or her own treatment motions to help improve his or her treatment skills.

[0048] The video stored in the imaging information storage unit 61 can be referred to as it is without editing, but it takes a lot of time to refer to all the video of the treatment, which may result in poor efficiency in the confirmation work of the video. Therefore, by extracting and referring to only the time period that the therapist H judges necessary, it is possible to significantly improve the efficiency of the confirmation work of the video. FIG. 7 is a graph showing the change in emotional information of the patient C over time in the video captured by the camera 10 of the treatment system 1 according to this embodiment. FIG. 7(A) is a graph showing the change in emotional information over time regardless of the gaze direction, and FIG. 7(B) is a graph showing the case in FIG. 7(A) where the gaze direction is on the bangs. In FIG. 7, the vertical axis shows normalized values ​​with the deepest negative emotional information being a score of 0 and the highest positive emotional information being a score of 1.

[0049] In FIG. 7(A), the emotion of the treated person C changes over time, and particularly in the A section (A1-A5), the emotion changes suddenly, while in the B section (B1-B3), the emotion state is maintained stably. That is, if the therapist H wants to refer to only the time period when the emotion of the treated person C is changing, he / she inputs a condition to that effect into the device 60, and the extraction / editing processing unit 62 extracts only the A section, connects the A section, and passes it to the output control unit 43. If the therapist H wants to refer to only the time period when the emotion of the treated person C is maintained stably, he / she inputs a condition to that effect into the device 60, and the extraction / editing processing unit 62 extracts only the B section, connects the B section, and passes it to the output control unit 43. In addition, it is also possible to extract and edit only the time period when positive emotions changed to negative emotions (A2 section in FIG. 7A), or only the time period when negative emotions changed to positive emotions (A1, A3, A4, and A5 sections in FIG. 7A). It is also possible to extract and edit a time period when emotions are stable and maintained for a predetermined period of time or more.

[0050] In FIG. 7(B), the solid line indicates the change in emotional information when the bangs are cut, and the dashed line indicates the change in emotional information when the other parts are cut. In this graph, in the A2 and B1 sections, the treatment is performed on the patient C in a way that causes him to have negative feelings about the bangs, causing his emotions to change from positive to negative, and the negative emotions are maintained for a while. Then, while the patient C's gaze is directed to another part, the practitioner H checks the image on the practitioner terminal 30, and the practitioner H confirms that the patient C has negative feelings about the bangs. After checking the practitioner terminal 30, the practitioner H starts a follow-up treatment on the bangs from around the A4 section, which causes the patient C's emotions to change upward. Furthermore, in the final finishing stage, A5, the practitioner H performs further follow-up treatment on the patient C's bangs, which shows that the patient C is highly satisfied with the bangs. In response to such changes in emotional information, practitioner H inputs conditions into device 60 to extract the time periods when the gaze direction of patient C is directed toward the bangs, and extraction / editing processor 62 extracts only the relevant time periods (time periods shown by solid lines in FIG. 7(B)), connects them together to create a video image, and passes it to output controller 43. Practitioner H can confirm what emotional changes occurred when different treatments were performed on patient C's bangs, and can utilize this information in future treatments as well as refer to how he or she deals with customers.

[0051] When conditions are input from the device 60, the extraction / editing processing unit 62 may generate a graph from the gaze emotion information in the gaze emotion storage unit 47 so that the graph as shown in Fig. 7 is displayed on the search condition screen. This allows the practitioner H to input conditions so as to extract a video of a required portion while referring to the emotional changes of the patient C.

[0052] In addition, in Fig. 7, the change in emotional information during treatment is shown in chronological order, but it is also possible to show the change in emotional information for each location in the gaze direction in chronological order. In other words, it is also possible to display a graph showing the change in emotional information for only one arbitrary block in Fig. 4, for example.

[0053] Furthermore, when the moving images extracted and edited by extraction and edit processing unit 62 are displayed on device 60, it is desirable to extract and edit the moving images so that a graph showing changes in emotional information as shown in FIG. 7 is displayed in conjunction with playback of the edited moving images.

[0054] In this way, in the treatment system 1 of this embodiment, the camera 10 captures treatment moving images including the treatment movements of the therapist H and is equipped with an imaging information storage unit 61 that stores the treatment moving images, so that the therapist H can check his or her own treatment movements after treatment and use the results to improve his or her skills.

[0055] In addition, the extraction and editing processing unit 62 extracts and edits video from a time period corresponding to a specified emotional information of the subject C from the treatment video stored in the imaging information storage unit 61. This makes it possible to selectively refer to video before and after a change in the subject C's emotion (for example, before and after a change from joy to anger, or before and after a change from emotionless to happy), allowing the therapist H to look back on his or her own treatment movements from an objective perspective and to efficiently check the video.

[0056] (Third embodiment of the present invention) The treatment system according to this embodiment will be described with reference to Fig. 8. In the treatment system according to this embodiment, the calculation of emotion information is performed based on voice information of the treatment recipient C in the treatment system according to each of the above-mentioned embodiments. Note that in this embodiment, the description overlapping with each of the above-mentioned embodiments will be omitted.

[0057] Fig. 8 is a diagram showing the system configuration of the treatment system according to this embodiment. In Fig. 8, what is different from the configuration of Fig. 1 in the first embodiment is that a microphone 70 is installed somewhere in the treatment system 1 as a voice collecting means. In the case of Fig. 8, it is installed in front of the person C to be treated, but it may be installed anywhere as long as it is located in a position where it can collect the voice of the person C to be treated, and it may be integrated with the camera 10. The voice information collected by the microphone 70 is transmitted to the calculation device 40 in real time.

[0058] In the arithmetic device 40, the receiving unit 41 receives the voice information transmitted from the microphone 70, and the voice information is directly passed to the emotion calculation unit 45 without passing through the image analysis unit 44. The emotion calculation unit 45 calculates emotion information corresponding to the acquired voice information from the voice information. In addition, as a technique for calculating emotion information from voice information, for example, a known technique such as "Taiga Haruta, Mariko Oda, Kohei Arai, Emotion Estimation Method with Mel-frequency Spectrum, Voice Power Level and Pitch Frequency of Human Voices through CNN Learning Processes, International Journal of Advanced Computer Science and Applications, 13, 11, 215-220, 2022" (Reference 4) can be used.

[0059] At this time, the emotion calculation unit 45 may calculate the emotion information by image analysis as described in each embodiment, may calculate the emotion information by analyzing the voice information, or may calculate the emotion information from the results of both. When calculating the emotion information from the results of both, the emotion information may be calculated by prioritizing the results of the voice analysis for emotions that are difficult to distinguish by image analysis. Specifically, for example, a smile and a wry smile are both smiles and may be difficult to distinguish by image analysis, but by taking the results of the voice analysis into consideration, it is possible to distinguish between a voice when happy and a voice when the voice is raised due to a wry smile. Conversely, for emotions that are difficult to distinguish by voice analysis, the emotion information may be calculated by prioritizing the results of the image analysis. The ratio (weighting) of which analysis result is to be prioritized may be registered in advance for each type of emotion. By using the results of the image analysis and the results of the voice analysis, it is possible to calculate the emotion information in a complementary manner.

[0060] In this way, the microphone 70 is provided as a voice collecting means for collecting voice information of the person being treated C while he or she is receiving treatment, and the emotion calculation unit 45 calculates the emotion information of the person being treated C based on the voice information, so that the emotion information can be accurately calculated by adding not only the facial expression but also the tone of voice of the person being treated C to the parameters.

[0061] Other embodiments of the present invention Another embodiment of the treatment system according to the present invention will now be described. The treatment system 1 according to this embodiment presents hair care products, cosmetics, and the like suitable for the person C to be treated and displays them on the display 20.

[0062] As described in each embodiment, by using the treatment system 1 according to the present invention, it is possible to recognize which part the treated person C feels what kind of emotion. That is, for parts where the treated person C feels negative emotion such as dissatisfaction or dislike, the treatment side (including the practitioner H himself or the treatment shop) can respond to follow up on that part. Specifically, for example, if negative emotion is associated with the eye area, cosmetics to be used in the eye area can be presented, and if negative emotion is associated with the hair tip area, hair care products for hair tip care can be presented. As a method of presentation, the practitioner H may present the product directly, or a video introducing the product may be displayed on the display 20.

[0063] In addition, the condition of the skin (e.g., texture, wrinkles, blemishes, skin color, pores, moisture state, etc.) and the condition of the hair (e.g., the presence or absence of hair medulla, the presence or absence of melanin pigment, hair hardness (vellus hair, hard hair, medium hair), hair length, hair shape (straight hair, curly hair), etc.) may be judged simply based on the image information captured by the camera 10, and if the condition is judged to be poor by comparing it with the data of many other recipients C, pre-registered products suitable for that area and condition may be presented in the same manner as above.

[0064] Furthermore, when presenting products suitable for an area determined to be in poor condition, it is possible to present products (pre-registered) used by other recipients C who have a good condition in that area, or to present products (pre-registered) used by other recipients C who have a similar poor condition but have improved. [Explanation of symbols]

[0065] C Subject H Practitioner 1 Treatment system 10(10a~10e) Camera 20 Display 30 Therapist terminal 40 Arithmetic unit 41 Receiving section 42 Editing Processing Section 43 Output control section 44 Image Analysis Unit 45 Emotion calculation part 46 Gaze detection unit 47 Gaze emotion memory unit 48 Image information generation unit 60 Devices 61 Imaging information storage unit 62 Extraction and editing processing section 70 Mike

Claims

1. An imaging means for imaging an upper body region including at least a facial expression of a patient when a therapist performs a predetermined treatment on a treatment target of the patient; A display means is installed facing the front of the patient and displays image information captured by the imaging means and / or processed image information processed based on the image information; A gaze detection means for detecting a gaze direction of the patient based on the imaging information; An emotion calculation means for calculating emotion information of the recipient based on a facial expression of the recipient included in the imaging information; an emotion area storage means for storing an attention target indicated by the gaze direction of the subject and emotion information of the subject when the gaze is directed to the attention target in association with each other; A treatment system comprising a notification means for notifying emotional information of the subject for each of the attention targets.

2. The treatment system according to claim 1, The treatment system is characterized in that the object of attention is any part of the entire hair area of ​​the recipient that is the treatment target displayed on the display means, and the notification means notifies the practitioner of emotional information of the recipient for each of the any part of the hair areas.

3. In the treatment system according to claim 2, The treatment system is characterized in that the notification means is carried by the practitioner and is equipped with a display that displays at least the entire hair area of ​​the person to be treated and the emotional information, and the entire hair area of ​​the person to be treated displayed on the display is displayed in a predetermined display mode according to the emotional information corresponding to any part of the hair area stored in the emotional area storage means.

4. In the treatment system according to claim 2, The treatment system is characterized in that the notification means is an earphone worn by the therapist, and the emotion information corresponding to any part of the hair region stored in the emotion region storage means is notified by voice through the earphone.

5. In the treatment system according to any one of claims 1 to 4, The treatment system is characterized in that the gaze detection means detects the gaze direction of the person to be treated when the gaze direction of the person to be treated remains stationary for a predetermined period of time or more.

6. The treatment system according to claim 5, an emotion intensity calculation means for calculating an intensity of the emotion information according to a continuous time during which the gaze direction of the subject is stationary and / or a cumulative time during which the gaze direction of the subject is stationary; The treatment system, wherein the notification means notifies information on the intensity of the emotion information in addition to the emotion information.

7. The treatment system according to claim 1, A treatment system characterized in that the display means displays processed image information in which the hairstyle, hair color and / or makeup of the person to be treated in the image information has been processed, and the gaze detection means detects the gaze direction of the person to be treated relative to the processed image information.

8. In the treatment system according to any one of claims 1 to 4, The imaging means captures a treatment video including the treatment motion of the therapist, A treatment system comprising an imaging information storage means for storing the treatment moving images.

9. The treatment system according to claim 8, A treatment system comprising an extraction and editing processing means for extracting and editing a moving image of a time period corresponding to predetermined emotional information of the subject from the treatment moving image stored in the imaging information storage means.

10. In the treatment system according to any one of claims 1 to 4, A voice collecting means is provided for collecting voice information while the person is receiving treatment, The treatment system, wherein the emotion calculation means calculates emotion information of the person to be treated based on the voice information.

11. a receiving means for receiving image information captured by an imaging means for capturing an image of an upper body region including at least a facial expression of a person to be treated when a therapist performs a predetermined treatment on the hair of the person to be treated; an imaging information transmitting means for transmitting the imaging information and / or the processed imaging information to a display means for displaying the imaging information captured by the imaging means and / or the processed imaging information processed based on the imaging information; A gaze detection means for detecting a gaze direction of the patient based on the imaging information; an emotion calculation means for calculating emotion information of the recipient based on a facial expression of the recipient included in the imaging information; An emotion area storage means for storing an attention target indicated by the gaze direction of the patient and emotion information of the patient when the gaze is directed to the attention target in association with each other. A treatment program that causes a computer to function as emotion information transmitting means that transmits information about the target of attention and the corresponding emotion information to a notifying means that notifies the target of attention of the treatment recipient.

Images