Wearable device and action evaluation system

Abstract

To provide a wearable device and an action evaluation system which measure and evaluate a direct relation between a wearer and a target person, and which are capable of giving awareness to the wearer regarding their action by clearly indicating the evaluation result, and to contribute to one of the Sustainable Development Goals (SDGs): "health and well-being for all", by making it possible to maintain human mental and physical health.SOLUTION: A wearable device is provided with: an imaging device which captures a subject image by imaging a target person present in front of or around a wearer; a subject image processing device which detects and performs face recognition of the target person included in the subject image, and processes the subject image so as to give an expression evaluation point to the wearer based on the facial expression of the target person obtained by the face recognition; and an output processing device which executes an evaluation process for obtaining an action evaluation point to evaluate an action of the wearer based on the expression evaluation point and an action evaluation time period being a time period during which the wearer, who is the evaluation target, performs the action, and outputs the action evaluation point.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a wearable device and a behavior evaluation system. [Background technology]

[0002] Patent Document 1 proposes a system that detects when people wearing face-to-face sensors meet each other in order to measure the relationships between people belonging to an organization, and visualizes the level of activity in the organization through the creation and analysis of face-to-face encounter histories between people.

[0003] In addition, wearable devices known as "action cams" are available on the market that use a camera to capture images of the scene in front of the user's line of sight and to create content based on the user's sports or other activities.

[0004] Furthermore, as described in Patent Document 2, for example, in the field of autonomous driving, a distance measurement sensor (LiDAR: Light Detection And Ranging) is known that obtains the distance from the background by irradiating a laser light while scanning and measuring the arrival time and phase difference of the reflected light. There are also known cases where distance measurement sensors are used in portable information devices.

[0005] However, the technology described in Patent Document 1 focuses on estimating the activity level of tissues, and does not mention anything about utilizing direct relationships between people. In addition, the technology described in Patent Document 1 has a problem in that face-to-face measurement between people can only be performed between people wearing face-to-face sensors. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2008-210363 A [Patent Document 2] International Publication No. 2019-082926 Summary of the Invention [Problem to be solved by the invention]

[0007] As described above, there is no description of utilizing direct relationships between people in Patent Document 1. In addition, the technology described in Patent Document 1 has a problem in that all people belonging to an organization must wear face-to-face sensors.

[0008] An object of the present invention is to provide a wearable device and behavior evaluation system that can measure and evaluate the direct relationship between the wearer and a target person, and provide awareness of the wearer's behavior by clearly displaying the evaluation results. [Means for solving the problem]

[0009] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0010] A wearable device according to a representative embodiment of the present invention comprises an imaging device that captures an image of a subject in front of or around the wearer to generate a subject image, a subject image processing device that detects and recognizes the face of the subject contained in the subject image and processes the subject image so as to assign to the wearer an expression evaluation score based on the facial expression of the subject obtained by the face recognition, and an output processing device that executes an evaluation process to determine a behavioral evaluation score for evaluating the behavior of the wearer from the expression evaluation score and a behavioral evaluation time that represents the behavioral time of the wearer that is the subject of evaluation, and outputs the behavioral evaluation score.

[0011] A behavioral evaluation system according to a representative embodiment of the present invention includes the above-mentioned wearable device, and a server device that acquires the behavioral evaluation points output from the output processing device and provides the acquired behavioral evaluation points to an external device owned by a person related to the subject. Effect of the Invention

[0012] The effects obtained by the representative inventions among those disclosed in this application will be briefly described as follows.

[0013] In other words, according to a representative embodiment of the present invention, it is possible to provide a wearable device and a behavioral evaluation system having a behavioral evaluation function that can measure and evaluate the direct relationship between the wearer and a target person and provide awareness of the wearer's behavior by clearly displaying the evaluation results. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of a wearable device according to a first embodiment. [Diagram 2] 1 is a diagram for explaining the relationship between a wearer of a wearable device and a target person. [Diagram 3] FIG. 1 is a perspective view illustrating the configuration of a head mounted display (HMD), which is one form of a wearable device. [Figure 4] 4 is a flowchart for explaining the operation of the head mounted display in the first embodiment. [Diagram 5] 13 is a graph for explaining the change over time in evaluation points for the behavior of a wearer of a wearable device. [Figure 6] 13 is a diagram illustrating an example of a table in which types of facial expressions of a target person (a set person) are associated with facial expression evaluation scores. FIG. [Figure 7] 13 is a diagram illustrating an example of a table in which the distance between a wearer and a target person (a set person) is associated with a distance evaluation score. FIG. [Figure 8] 13A and 13B are diagrams illustrating examples of display of behavior evaluation results for a wearer of a wearable device. [Figure 9] 13 is a flowchart showing a process for acquiring a cumulative behavior evaluation result. [Figure 10A] FIG. 13 is a diagram showing an example of a display of a cumulative behavioral evaluation result. [Figure 10B] FIG. 13 is a diagram showing an example of a display of a cumulative behavioral evaluation result. [Figure 11]13 is a diagram illustrating the relationship between the angle of view of a camera and the measurement angle of a distance measuring device in the second embodiment. FIG. [Figure 12] 11A and 11B are diagrams illustrating a process of tracking a target person from a distance image. [Figure 13] 11 is a flowchart illustrating a process executed by a controller in the second embodiment. [Figure 14] FIG. 13 is a diagram for explaining the configuration of a third embodiment, showing the system configuration of a behavior evaluation system including a wearable device. [Figure 15] FIG. 13 is a diagram illustrating a data configuration of a behavior evaluation result. [Figure 16] FIG. 11 is a sequence diagram for monitoring behavior evaluation results. [Figure 17] FIG. 1 is a diagram illustrating a scene in which multiple subjects are present. [Figure 18] 13 is a flowchart showing an example of processing executed by a wearable device in the fourth embodiment. [Figure 19] FIG. 13 is a diagram illustrating the configuration of a database of behavioral evaluation results corresponding to a plurality of subjects. [Figure 20] FIG. 13 is a diagram showing an example of a display of behavior evaluation results. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In the embodiments and examples disclosed below, a device and a system that contribute to maintaining the mental health of a user and the health of those they interact with by analyzing and evaluating the degree of human contact and encouraging the user to act will be disclosed. In addition, the technology disclosed herein contributes to the achievement of the Sustainable Development Goals (SDGs) of "Good Health and Well-Being" advocated by the United Nations (UN) by enabling people to maintain their mental and physical health.

[0016] In the following embodiment, a wearable device is used to analyze the degree of contact between people. Specifically, the degree of contact between a user wearing the wearable device (hereinafter referred to as the "wearer") and a person photographed or recognized by the wearable device (hereinafter referred to as the "target person") is obtained through image analysis such as person detection and face recognition.

[0017] Here, the degree of contact between the wearer and the target person (i.e., between people) is calculated as an "action evaluation point" based on the facial expression of the target person obtained by face recognition, the distance between the wearer and the target person, the time during which the wearer acts with respect to the target person, and other factors. This "action evaluation point" can be calculated by obtaining an "expression evaluation point" based on the facial expression of the target person, a "distance evaluation point" according to the distance between the wearer and the target person, and an "action evaluation time" as the time during which the wearer acts with respect to the target person, and substituting these points and times into, for example, a predetermined calculation formula and performing a calculation. The calculated "action evaluation point" is a point that comprehensively evaluates the wearer's action (degree of contact) with respect to the target person, and by showing such a point to the wearer, it is possible to encourage the wearer to act and contribute to maintaining the mental and physical health of both the wearer and the target person.

[0018] More specifically, in order to determine the above-mentioned "facial expression evaluation score," the wearable device is equipped with an imaging device that captures an image of a subject in front of or around the wearer to generate a subject image, and a subject image processing device that detects and recognizes the faces of the subjects contained in the subject image, and processes the subject image so as to give the wearer an facial expression evaluation score based on the facial expression of the subject obtained by the facial recognition.

[0019] In addition, in order to determine the above-mentioned "distance evaluation point," the wearable device is equipped with a distance measuring device that measures the distance between the wearer and the target person to obtain distance data, and a distance image processing device that determines the distance between the wearer and the target person by analyzing a distance image obtained by mapping the distance data, and obtains a distance evaluation point corresponding to that distance.

[0020] Furthermore, in order to determine and output the above-mentioned "behavioral evaluation score," the wearable device is equipped with an output processing device that executes an evaluation process to determine a behavioral evaluation score for evaluating the wearer's behavior from the facial expression evaluation score, distance evaluation score, and behavioral evaluation time which is the behavior time of the wearer that is the subject of evaluation, and outputs the behavioral evaluation score.

[0021] Of the above, the "subject image processing device," "distance image processing device," and "output processing device" can be realized by a processor (such as a CPU) implemented in the wearable device or an external server device. Each of these devices can be realized by either the same processor or separate processors, but the following assumes an example in which they are realized by a single processor implemented in the wearable device.

[0022] With the above-described configuration, it is possible to measure and evaluate the direct relationship between the wearer and the subject, and output the evaluation result, without the subject having to wear a device such as a sensor.

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Each embodiment shown in the drawings is an example for realizing the present invention, and does not limit the technical scope of the present invention. In the examples, members having the same function are given the same reference numerals, and repeated explanations thereof will be omitted unless particularly necessary.

[0024] [Embodiment 1] Hereinafter, a first embodiment will be described with reference to Fig. 1 to Fig. 10. The first embodiment discloses a wearable device worn by a user (hereinafter, referred to as "wearer"). This wearable device includes an imaging device such as a camera and a distance measuring device such as a distance sensor, and has a behavior evaluation function that measures the degree of interaction between the wearer and a person (target person) captured by these devices.

[0025] (Block diagram and usage scenarios) FIG. 1 is a block diagram of a wearable device according to a first embodiment.

[0026] As shown in FIG. 1, the wearable device 1 of embodiment 1 includes a camera 10 (imaging device), a distance measuring device 11, a sensor unit 12, an image display unit 13, an audio input / output unit 14, an operation input unit 15, a communication unit 16, a main processor 17, a RAM 18, and a flash ROM (FROM) 19.

[0027] The main processor 17 is a main control unit that controls the entire wearable device 1 according to a predetermined program. The main processor 17 is realized by a CPU (Central Processor Unit) or a microprocessor unit (MPU). The main processor 17 performs operation control processing of the entire wearable device 1 by executing programs such as an operating system (OS) and various operation control applications stored in the storage device 110. The main processor 17 also controls the startup operations of various applications.

[0028] Among these, the sensor unit 12, the communication unit 16, the main processor 17, the RAM 18, and the flash ROM (FROM) 19 function as a control device or a controller 21 in the wearable device 1 (see the dotted line frame in FIG. 1).

[0029] In the wearable device 1, the camera 10 is an imaging device that captures an image of a target person in front of or around the wearer to generate a subject image.

[0030] In one specific example, the camera 10 includes an image sensor, such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and an optical lens. In this case, the camera 10 captures an image of the background in front of the wearer's line of sight within the range of its angle of view (see also Figs. 2 and 3 as appropriate). For this reason, when capturing an image of a surrounding subject not in front of the wearer's line of sight, the wearer faces or faces the face or body of the subject to be captured.

[0031] Meanwhile, in the wearable device 1, the distance measuring device 11 is a distance measuring device generally called a distance sensor or a distance measuring sensor, and has a function of measuring the distance between the wearer and the target person and acquiring distance data.

[0032] In one specific example, the distance measuring device 11 is a device equipped with a distance measuring sensor such as LiDAR (Light Detection And Ranging) (also called optical radar).

[0033] The distance measuring device 11 has a configuration for, for example, irradiating a laser light from a light source (such as a laser diode) inside the sensor in a scanning manner and measuring the time it takes for the reflected light reflected by the object to be received by the light receiving element of the sensor. The distance measuring device 11 configured as above measures the distance to the background (various objects, people, etc.) in front of the wearer's line of sight and acquires distance data. The distance measuring device 11 also generates a distance image by two-dimensionally mapping the measured and acquired distance data to the object to be measured.

[0034] The sensor unit 12 includes various sensors other than the distance measuring sensor (distance measuring device 11) described above, such as a GPS sensor, a gyro sensor, a geomagnetic sensor, an acceleration sensor, etc., as shown in Fig. 1. Note that the configurations of the above sensors in the sensor unit 12 are publicly known and have little relevance to the first embodiment, so detailed description will be omitted.

[0035] The image display unit 13 plays a role of displaying an image generated by the controller 21. There are various types of images that are generated and displayed by the controller 21, but in this embodiment, for example, an image of the behavior evaluation result of the wearer is included, and the details of this image will be described later.

[0036] The voice input / output unit 14 includes a microphone (hereinafter, abbreviated as "mic") for inputting (collecting) voice, and a speaker for outputting (pronouncing) voice. In this embodiment, the behavior evaluation result can be output as voice from the speaker of the voice input / output unit 14 to notify the wearer of the result.

[0037] Various devices for inputting operations by the wearer may be used as the operation input unit 15. For example, the operation input unit 15 may be any device such as a touch sensor for inputting operations of the wearer's fingers, a microphone for voice input of the wearer, or a sensor for detecting the movement of the wearer's hand captured by the camera 10, or may be a combination of a plurality of these.

[0038] The communication unit 16 includes, for example, a wireless LAN that performs 4G (Generation) and 5G mobile communication. When necessary, the communication unit 16 selects an appropriate process from among communication protocols and connects the wearable device 1 to a network or the like.

[0039] FROM 19 is a non-volatile memory medium and includes a basic operation program 91 and a behavioral evaluation program 92 (hereinafter sometimes simply referred to as processing programs 91, 92) as processing programs executed by the main processor 17, and a data section 93 in which various data are stored.

[0040] The processing programs 91, 92 are executed by being read by the main processor 17 and expanded in the RAM 18. The data section 93 stores data required for executing the processing programs 91, 92. The FROM 19 may be a single memory medium as shown in the figure, or may be composed of multiple memory media. Furthermore, it may be a non-volatile memory medium other than a flash ROM.

[0041] Fig. 2 is a diagram for explaining the relationship between a wearer of a wearable device and a target person. Referring to Fig. 2, a wearer 3 wearing a wearable device 1 and a target person 4 who is the subject of the behavior of the wearer 3 and behavior evaluation are shown.

[0042] In the illustrated example, the wearer 3 is a parent, the target person 4 is an infant, and the behavior of the wearer 3 (and thus the type of behavior evaluation) is childcare. The following description is based on the assumption that the wearable device 1 is a head mounted display (HMD).

[0043] The wearer 3 can appropriately use the attachment 2 such as a fastener to hold the wearable device 1 so that it does not slip off or fall off the face. The attachment 2 may be in a form in which the wearable device 1 is attached to the head of the wearer 3, like a hair band, or the wearable device 1 (HMD in this example) and the attachment 2 may be integrated.

[0044] Thus, the wearable device 1 captures (photographs) the subject 4 with the camera 10 (see Figure 1) and executes processing programs 91, 92 to evaluate the behavior of the wearer 3 toward the subject 4 and record the results of such behavior evaluation (hereinafter referred to as the "behavior evaluation result", or sometimes abbreviated to the "evaluation result" as appropriate).

[0045] When the wearable device 1 is an HMD as in this embodiment, the behavioral evaluation results can be output to the display of the HMD, that is, to a screen displayed in front of the eyes of the wearer 3. This allows the wearer 3 to use the wearable device 1 in a hands-free manner.

[0046] FIG. 3 is an example of an external view of an HMD, which is one type of wearable device.

[0047] 1 and 3, the wearable device 1 (HMD) includes a camera 10, a distance measuring device 11, an image display unit 13, an audio input / output unit 14, an operation input unit 15, and a controller 21 having various blocks (see the dotted frame in FIG. 1). Of these, the image display unit 13 is implemented as projectors 13a and 13b and a main screen display unit 13c (a semi-transparent screen in this example) as shown in FIG. 3. The camera 10 and the distance measuring device 11 are arranged side by side in the approximate center of the upper part of the housing (front frame 22c) of the HMD. The controller 21 is provided on one side of the horizontal frame 22b, which corresponds to the temples (of glasses) of the HMD, in this example, on the right side.

[0048] 1, a speaker 14a and a microphone 14b are provided on the other (left side in this example) horizontal frame 22a corresponding to the temple. As shown in FIG. 3, the front frame 22c is provided with an L projector 13a on the left side, an R projector 13b on the right side, and a main screen display section 13c having a larger area than the projectors 13a and 13b as the image display section 13.

[0049] Furthermore, a nose pad 23 is provided in the center of the main screen display unit 13c as a component of the wearing equipment 2 described above in Fig. 2. The user wears the HMD 1 on their face (head) by hanging the tip sides of the horizontal frames 22a and 22b on both ears and placing the nose pad 23 on their nose.

[0050] Of the above, the main screen display unit 13c is a semi-transmissive screen in this example, but may be a non-transmissive display as another example. In either case, the user can check the background in front of him / her through the main screen display unit 13c.

[0051] In the above, an example in which the display unit is configured with a semi-transmissive screen and a projector has been described, but a retinal projection display that projects onto the user's retina without using a screen may also be used. Furthermore, non-transmissive displays include display devices that use, for example, lasers, liquid crystal panels, organic EL (EL: Emitting Diode), etc., and a system in which the user directly views the display device through a lens, etc. may also be used.

[0052] Here, when the main screen display unit 13c is a semi-transmissive screen, the user visually recognizes the forward background transmitted through the semi-transmissive screen. On the other hand, when the main screen display unit 13c is a non-transmissive display, the user recognizes the forward background by displaying a camera image of the forward background on the non-transmissive display.

[0053] The camera 10 is attached to the front frame 22c (housing) of the HMD so as to capture an image of the background in front of the user's line of sight. A distance measuring device 11 arranged next to the camera 10 measures the distance to an object in the background in front of the user's line of sight.

[0054] Controller 21 inputs an image captured by camera 10 (hereinafter referred to as "camera image") and a distance image generated by distance measuring device 11, and supplies them to an internal memory (RAM 18 or data unit 93) and main processor 17. Controller 21 also generates images to be projected or displayed on the above-mentioned image display unit 13 (L projector 13a, R projector 13b, main screen display unit 13c) and sounds to be output from speaker 14a.

[0055] In relation to the above-mentioned characteristic parts, the controller 21, particularly the main processor 17, performs functions such as "object image processing device", "distance image processing device", and "output processing device".

[0056] As described above, the controller 21, camera 10, distance measuring device 11, speaker 14a, and microphone 14b are arranged on the corresponding frames 22a to 22c, but the locations of these components do not necessarily have to be as shown in FIG.

[0057] (flowchart) Fig. 4 is a flowchart for explaining the operation of the head mounted display in embodiment 1. The flowchart in Fig. 4 shows a flow of processing based on the behavior evaluation program 92, and the main processor 17 that has read the behavior evaluation program 92 is the main subject of the processing. Then, according to a control signal output from the main processor 17, the corresponding block of the wearable device 1 operates to perform each process in the flowchart.

[0058] In step S11 after starting execution of the behavior evaluation program 92, the main processor 17 outputs control information to the camera 10 to acquire a camera image. The camera image may be acquired in synchronization with the timing of image capture by the camera 10, or images may be captured continuously at, for example, 30 fpS (frames per second) to acquire images at any timing.

[0059] In the next step S12, the main processor 17 performs a process of registering new face data for the face captured in the acquired camera image, or a process of facial recognition based on a comparison with already registered face data. Here, when registering new face data, the main processor 17 can assign and register an identification number to the target person based on the operation of the operation input unit 15 by the user (wearer).

[0060] In step S13, main processor 17 determines whether or not a target person has been registered or recognized.

[0061] Here, if the main processor 17 determines that the target person has not been registered or recognized (step S13: NO), it determines that the face captured in the camera image is not the face of the target person or that the face is not captured in the camera image. In this case, the main processor 17 returns to step S11 to reacquire the camera image, and repeats the processes of steps S11 to S13 described above.

[0062] On the other hand, when the main processor 17 determines that the target person has been registered or recognized (step S13: YES), it specifies (sets) the target person who is the target of the user's (wearer's) behavior, and proceeds to step S14.

[0063] In step S14, the main processor 17 starts integrating the program execution time.

[0064] In the next step S15, the main processor 17 determines whether or not a predetermined periodic timing has arrived.

[0065] If main processor 17 determines that the timing of the predetermined period has not arrived (step S15: NO), main processor 17 repeats the determination of step S15 until it determines that the timing of the predetermined period has arrived (step S15: YES).

[0066] If main processor 17 determines that the timing of the predetermined period has arrived (step S15: YES), it executes the camera image processing process (steps S16 to S19) and the distance image processing process (steps S20 to S22) described below.

[0067] 4 shows an example of multi-processing in which main processor 17 executes in parallel (concurrently) the camera image processing process (steps S16 to S19) and the distance image processing process (steps S20 to S22). As another example, main processor 17 may execute serial processing, for example, the camera image processing process (steps S16 to S19) followed by the distance image processing process (steps S20 to S22).

[0068] (Camera image processing) In step S16 at the start of the camera image processing process, the main processor 17 acquires a camera image captured by the camera 10. In the following step S17, the main processor 17 performs face recognition processing and person detection processing of faces captured (shown) in the acquired camera image.

[0069] Then, in step S18, main processor 17 determines whether the target person (hereinafter also referred to as "target person") set in step S13 is captured in the acquired camera image. More specifically, in step S18, main processor 17 checks the detection or recognition result in step S17. In one specific example, main processor 17 determines whether a person is detected in the camera image, and if so, whether the person is the target person. Main processor 17 also determines whether a face is recognized in the camera image, and if so, whether the face is the face of the target person.

[0070] Furthermore, when the main processor 17 can recognize the face of a person detected in a camera image, the main processor 17 judges whether or not the person is the set target based on the result of the face recognition. On the other hand, when the face of a person detected in a camera image is unclear (when face recognition cannot be performed), for example, when the detected person is facing away from the camera, the main processor 17 estimates whether or not the detected person is the set target.

[0071] To make such an estimation, main processor 17 judges, for example, the similarity of the color and texture of the clothes of the person recognized immediately before (when the target person (set person) before step S14 was set), the physical balance between the face and the body, the validity of the moving distance considering the motion vector, etc. The detected target person shares the position and size in the image with the distance image processing process (see step S25).

[0072] Thus, when the main processor 17 determines that the set person (set target person) is not captured in the camera image (step S18: NO), the process returns to step S15 and executes the processes from step S23 onward. The contents of the processes from step S23 onward will be described later.

[0073] On the other hand, when the main processor 17 determines that the person to be set (the set target person) is captured in the camera image (step S18: YES), the process proceeds to step S19. In step S19, the main processor 17 acquires an expression evaluation score based on the recognized facial expression of the person to be set. In one specific example, the main processor 17 acquires the expression evaluation score using a table that associates types of facial expressions (laughing, anger, crying, etc.) with expression evaluation scores. An example of this table will be described later with reference to FIG. 6.

[0074] Alternatively, the main processor 17 may modify or calculate the facial expression evaluation score based on the facial expression of the person to be set, while taking into consideration other gestures of the person to be set. Here, the "other gestures" of the person to be set include the degree of emotion in the voice uttered by the person to be set (so-called tone of voice) and the emotional expression by the gestures of the hands (arms) and body of the person to be set that are captured in the camera image. In this case, the main processor 17 obtains the final facial expression evaluation score by adding the score based on the above-mentioned other gestures to the facial expression evaluation score obtained corresponding to the facial expression of the person to be set.

[0075] (Distance image processing) At the start of the distance image processing process, in step S20, the main processor 17 acquires a distance image measured by the distance measuring device 11.

[0076] In the next step S21, the main processor 17 receives or appropriately refers to the presence information of the person to be set and the position of the person to be set in the camera image obtained in the above-mentioned camera image processing process (step S18, etc.). Through this processing, the main processor 17 obtains or calculates a more accurate three-dimensional coordinate position of the person to be set who is in front of the user (wearer) and obtains the distance between the wearer and the person to be set. Then, in step S22, the main processor 17 obtains a distance evaluation point based on the distance between the wearer and the person to be set.

[0077] (Processing of evaluation points, etc.) In step S25 after the facial expression evaluation points are acquired in step S19 and the distance evaluation points are acquired in step S22, the main processor 17 records and accumulates these evaluation points. Such recording and accumulation processing may be performed by recording (accumulating) in the data section 93 in the wearable device 1, or by recording (accumulating) in a recording medium of a server via a network.

[0078] If the main processor 17 determines in the above-mentioned camera image processing process that the person to be set is not captured in the camera image (the person to be set cannot be confirmed) (step S18: NO), the process returns to step S15 to reacquire the camera image and distance image, and executes the processes from step S23 onwards.

[0079] In step S23, the main processor 17 starts counting the time during which the settee cannot be confirmed (the time during which the determination of NO in step S18 is repeated). In the following step S24, the main processor 17 determines whether the counted time exceeds the threshold value TH.

[0080] Here, if the main processor 17 determines that the counted time has not yet exceeded the threshold TH (step S24: NO), the process returns to step S15 and repeats the above-mentioned process. On the other hand, if the main processor 17 determines that the counted time has exceeded the threshold TH (step S24: YES), the process determines that the behavior of the user (wearer) toward the set subject has reached a turning point, and proceeds to step S26.

[0081] In step S26, main processor 17 calculates a statistical value of the evaluation points recorded (accumulated) in the above-mentioned step S25. In one specific example, main processor 17 calculates a value obtained by normalizing the evaluation points (facial expression evaluation points and distance evaluation points) recorded (accumulated) at that time point by the behavior evaluation time (for example, the execution time of the program).

[0082] The main processor 17 outputs the calculated value as a behavior evaluation result (step S27), and proceeds to step S28.

[0083] In step S28, the main processor 17 determines whether a program termination event has occurred. This "termination event" may be, for example, a program shutdown or power off command input by the user (wearer).

[0084] Here, if the main processor 17 determines that the reason for terminating the program has not yet occurred (step S28: NO), it determines that the user (wearer) has not yet finished the task, etc., and returns to step S11 to repeat the above-mentioned processing. In this case, the main processor 17 initializes the program execution time (behavior evaluation time for the user (wearer)) in step S14, which is executed again, and starts accumulating the time.

[0085] On the other hand, if the main processor 17 determines that a reason to terminate the program has occurred (step S28: YES), it determines that the user (wearer) has completed his / her work, and terminates the series of processes shown in FIG.

[0086] (Examples of evaluation and display) Fig. 5 is a graph for explaining the change over time in the evaluation score for the behavior of the wearer of the wearable device. In the graph of Fig. 5, the vertical axis indicates the evaluation score, and the horizontal axis indicates the change over time or the execution time of the program (the behavior evaluation time of the wearer). In addition, in order to make it easier to understand, the step numbers of the process explained in Fig. 4 are appropriately added in Fig. 5.

[0087] As shown in FIG. 5, the main processor 17 of the wearable device 1 begins evaluating the wearer's behavior (assigning, recording, and accumulating evaluation points) from the point in time when a person (target person) in the camera image is recognized and set as the set person (when the result of the first execution of step S18 is determined to be YES).

[0088] The above-mentioned camera image processing process, distance image processing process, etc. are performed at regular intervals T shown in Fig. 5, and main processor 17 obtains an evaluation point at each interval T. Here, interval T corresponds to the timing at which step S15 is determined as YES.

[0089] The "evaluation score" shown on the vertical axis of the graph in Fig. 5 is the sum of the facial expression evaluation score and the distance evaluation score. Generally, when face recognition is possible, the facial expression evaluation score is added, so the evaluation score is high (see the evaluation score for "(S18: YES) Face recognition available" in Fig. 5). In contrast, when the set-person cannot be identified by face recognition, the evaluation score is only the distance evaluation point, so the evaluation score is relatively low (see the evaluation point for "(S18: NO) Face recognition not available" in Fig. 5).

[0090] Also, in the example shown in Fig. 5, if the set person is not captured in the camera image (see step S18: NO branch in Fig. 4 as appropriate) and the target person is not captured in the distance image either, the evaluation score will be zero (see the evaluation score for "set person not measured" in Fig. 5). Then, if this "set person not measured" time (here, the time when the evaluation score is zero) exceeds a threshold TH (see step S24 in Fig. 4 as appropriate), a value based on an accumulation of the evaluation results in the first period (see period "NT" in Fig. 5) is output as the behavior evaluation result (see steps S26 and S27 in Fig. 4 as appropriate).

[0091] Of the above, the period "NT" shown in FIG. 5 corresponds to the "behavior evaluation time" which is the behavior time of the wearer who is the subject of evaluation.

[0092] Furthermore, if the set person is subsequently detected in the camera image or distance image (see step 18: YES in Figure 4 and "Re-recognition of set person" in Figure 5), the facial expression evaluation or distance evaluation for the wearer's next behavior period (new behavior evaluation period) is recorded and accumulated (see step S25 in Figure 4, etc.).

[0093] By repeating this process, points (behavioral evaluation points) that comprehensively evaluate the wearer's behavior (degree of contact) with respect to the target person (designated person) are recorded and accumulated in chronological order, and the behavior evaluation results are output for each behavior evaluation period.

[0094] The method of assigning behavior evaluation points and the timing of program execution are not limited to the above example, and may be, for example, as follows. That is, if the target person (designated person) is not captured in the camera image (step S18 in FIG. 4: NO), the evaluation point is set to zero even if the designated person can be detected in the distance image. In this case, if the period during which the target person is not captured in the camera image exceeds a certain period (see step S24 in FIG. 4 as appropriate), execution of the program is paused and the behavior evaluation result is output. Then, when the target person is recognized again in the camera image, behavior evaluation is resumed.

[0095] FIG. 6 is a diagram illustrating a table in which the types of facial expressions of a target person (a set person) are associated with facial expression evaluation points. Such a table can be used in the facial expression evaluation process in step S19 described above. In the table shown in FIG. 6, the types of facial expressions (Category) are set in the upper row, and the corresponding facial expression evaluation points (Evaluation Points) are set in the lower row. In this example, when the type of facial expression is "calm," the facial expression evaluation point is an average value (50 points in this example), and the facial expression evaluation points are high in the order of "smile" and "laugh" (80 points and 100 points, respectively, in this example), and the facial expression evaluation points are low in the order of "anger" and "crying" (20 points and 0 points, respectively).

[0096] FIG. 7 is a diagram illustrating a table in which the distance between the wearer and the target person (the person being set) is associated with the distance evaluation points. This table can be used in the distance evaluation process in step S22 described above. In the table shown in FIG. 7, the distance from the person being set is set in the upper row, and the corresponding distance evaluation points are set in the lower row. In this setting example, the highest evaluation point (100 points) is given when the distance from the person being set is close to "less than 0.5 m," and the evaluation points are reduced in the order of "less than 1 m," "less than 3 m," "less than 10 m," and "10 m or more," to "80 points," "50 points," "20 points," and "0 points."

[0097] Fig. 8 is a diagram showing an example of a display of a behavior evaluation result for a wearer of a wearable device. Display screen 30 shown in Fig. 8 is displayed on image display unit 13 of wearable device 1 during the processing of step S27 described above in Fig. 4, for example. For ease of understanding, an example of a calculation formula for calculating the behavior evaluation result is shown above display screen 30 in Fig. 8.

[0098] In the illustrated example, the behavior evaluation score is obtained by accumulating the facial expression evaluation score (i) and the distance evaluation score (i) recorded at each timing, calculating the statistical value of each of the accumulated values, and normalizing each of the calculated statistical values ​​by the above-mentioned behavior evaluation time (see the period NT shown in FIG. 5, etc.). This calculation (computation) is executed by the main processor 17 in step 26 of FIG. 4 described above.

[0099] Here, the "statistical value" calculated by main processor 17 may be an average value such as an arithmetic average value or a weighted average value. In the case of an arithmetic average value, main processor 17 calculates the arithmetic average value of each of the accumulated facial expression evaluation points and distance evaluation points (see the formula shown in FIG. 8). Alternatively, in the case of a weighted average value, main processor 17 calculates the weighted average value by applying a predetermined coefficient (weight value) to the elements constituting the accumulated facial expression evaluation points (or distance evaluation points), for example, each evaluation point (Evaluation Points) described above in FIG. 6 (for example, evaluating "laughing" higher, evaluating a distance "less than 0.5 m" higher, etc.).

[0100] Then, the main processor 17 calculates a behavior evaluation score by dividing the calculated average value (or weighted average value) by the program execution time N (which is substantially equal to the above behavior evaluation time) and normalizing it. The calculated behavior evaluation score is thus displayed at the "XXX" position on the display screen 30.

[0101] 8 shows the display screen 30 in a simplified display format. The "time" on the display screen 30, in this example, the time period from 9:12 AM to 10:45 AM on January 26, 2021, is the time period during which the program described above in FIG. 4 was executed, and is equal to the time period during which the wearer's behavior was evaluated.

[0102] More specifically, in addition to the above-mentioned "time", the display screen 30 shown in Fig. 8 displays the normalized and calculated behavior evaluation score, a general comment on the behavior evaluation result ("Good" in this example), and an additional comment saying "You can continue to evaluate." The additional comment shown in the figure is a message that you can continue to evaluate the behavior if the set person is recognized again.

[0103] As another display format for the display screen 30 displayed on the image display unit 13 of the wearable device 1, for example, a graphic display format as shown in FIG. 5 may be selected, which graphically displays the time progression of the evaluation.

[0104] 9 is a flowchart showing a process for acquiring a cumulative behavior evaluation result. Here, the cumulative behavior evaluation result can be defined as a cumulative value of behavior evaluation points in a specific period, and can be acquired by executing the behavior evaluation program 92.

[0105] The behavior evaluation score described above in Fig. 8 is an evaluation score normalized by the behavior time, and is an evaluation result related to the so-called "quality of behavior." In contrast, the cumulative evaluation score calculated according to the flow in Fig. 9 is an evaluation score obtained by accumulating the evaluation points for each evaluation time for a specific period without normalization, and corresponds to the so-called "amount of behavior." Here, the "specific period" can be set arbitrarily, for example, in units of "one day," "one week," "one month," or "one year."

[0106] In step S101 after starting the process of acquiring the accumulated evaluation points, the main processor 17 sets the above-mentioned "specific period" in response to, for example, an operation (setting instruction) by the user (wearer).

[0107] In the next step S102, the main processor 17 sequentially reads the evaluation data within the set period. Then, in step S103, the main processor 17 accumulates (sequentially adds) the evaluation points within the set period, and proceeds to step S104 after accumulating all evaluation points within the period. In step S104, the main processor 17 outputs the finally obtained accumulated value as the accumulated evaluation result.

[0108] In the next step S105, the main processor 17 determines whether or not to end the process of obtaining the accumulated evaluation points.

[0109] Here, if the main processor 17 determines that the process of acquiring the accumulated evaluation points is not yet completed (step S105: NO), the process returns to the above-mentioned step S101 and repeats the above-mentioned process. At this time, the user can recalculate and output a different accumulated evaluation point by, for example, setting a different specific period.

[0110] On the other hand, when it is determined that the process of obtaining the accumulated evaluation points is to be ended (step S105: YES), the main processor 17 ends the series of processes described above.

[0111] 10A and 10B are diagrams showing examples of displaying cumulative behavioral evaluation results. Of these, FIG. 10A is an example of displaying cumulative behavioral evaluation results when the evaluation period has been completed. Meanwhile, FIG. 10B is an example of displaying cumulative behavioral evaluation results when the evaluation period has not yet been completed (is in progress). Specifically, in a case where the evaluation period is set to be from Sunday, January 31, 2021 to Saturday, February 6, 2021, FIG. 10B shows the case where the display time is Friday, February 5 within the period.

[0112] As shown in Fig. 10A and Fig. 10B, a preset evaluation period (one week in this example) and accumulated evaluation points within the evaluation period are displayed. In this example, the evaluation period is set to one week, and accumulated evaluation points on a daily basis and on a weekly basis are displayed. By looking at such evaluation points, the wearer can self-manage the goal of a weekly behavior pattern, for example, if there is little interaction on weekdays, then to interact more on weekends.

[0113] Also, when the period is completed, the cumulative evaluation points for this week calculated this time are displayed as a ratio to the cumulative evaluation calculated last time (last week), as shown in the lower part of Fig. 10A. On the other hand, in the example of Fig. 10B, where the period is not yet completed, the remaining points to reach the weekly target points (in this example, the evaluation points (1888 points) to be obtained on the final day, Saturday, February 6th) are displayed, as shown in the lower part of the same figure.

[0114] By displaying the above-mentioned information, it is possible to increase the motivation of the wearer, or to help the wearer set new goals or action guidelines for the next week.

[0115] As described above, according to the wearable device 1 of the first embodiment, it is possible to provide a wearable device having a behavior evaluation function that evaluates direct behavior between the wearer and a target person and provides awareness of the behavior, etc. In addition, there is no need for the target person to wear anything, and therefore, there is a feature that the target person can be freely selected (high degree of freedom in selecting the target person).

[0116] Note that, although an integrated HMD (head mounted display) is exemplified here as a specific configuration of the wearable device 1, the present invention is not limited to this. As another configuration example of the wearable device 1, for example, the controller 21 (subject image processing device, distance image processing device, output processing device) may be configured separately from the HMD (camera 10, distance measuring device 11, image display unit 13, etc.). Alternatively, as another configuration example, the camera 10, distance measuring device 11, etc. may be disposed in a neck-type or watch-type wearable device.

[0117] [Embodiment 2] Next, a second embodiment will be described with reference to FIGS.

[0118] Fig. 11 is a diagram for explaining the relationship between the angle of view of the camera and the measurement angle of the distance measuring device in embodiment 2. As shown by comparison in Fig. 11, in embodiment 2, the measurement angle 40 of the distance measuring device 11 is set to be wider than the angle of view of the camera 10 (camera angle of view 41). Methods for widening the measurement angle 40 of the distance measuring device 11 include a method of widening the scanning range of the laser light output from the above-mentioned light source, or a method of using a combination of multiple distance measuring sensors (optical radar, etc.), and any of these may be used.

[0119] Fig. 12 is a diagram illustrating the process of tracking a target person from a distance image. In Fig. 12, a distance image 51 generated by distance measuring device 11 is shown in an outer frame, and an area 50 of a camera image generated by camera 10 is shown in an inner frame of distance image 51.

[0120] 12, it can be seen that a target person who was initially captured in approximately the center of camera image 50 (position indicated by reference symbol 52a) moves sequentially to positions indicated by reference symbols 52b ⇒ 52c ⇒ 52d in chronological order. Here, the position indicated by reference symbol 52b is the right end side of camera image 50, and the positions indicated by reference symbols 52c and 52d are positions outside camera image 50 and on the right side in distance image 51, and a non-target person 53 is captured in a position outside camera image 50 and on the left side in distance image 51. Note that camera image region 50 is also a region within distance image 51, and is therefore a region where the camera image and the distance image overlap, and is hereinafter also referred to as a "composite region 50."

[0121] Here, the controller 21 (mainly the main processor 17, the same below) of the HMD of this embodiment distinguishes the target person (and thus the person to be set) in the composite area 50 by analyzing the camera image. For example, even if the person to be set moves and changes direction after being recognized and face authentication is no longer possible, the person to be set can be recognized by human detection. More specifically, the person at the positions of symbols 52a and 52b in the composite area 50 can be easily identified as the person to be set in the distance image corresponding to the person to be set recognized in the camera image. Therefore, as a function of a distance image processing device, the controller 21 extracts the features of the person (not limited to the target person or the person to be set) in the distance image, the size of the person, the body balance, the motion vector, etc. as feature quantities.

[0122] In the example shown in FIG. 12, the set person (the same target person) moves from the position of reference symbol 52a to the positions indicated by reference symbols 52b, 52c, and 52d in order. Here, the people indicated by reference symbols 52c and 52d are located outside the range of the camera image. Meanwhile, the controller 21 extracts the features of the person in the distance image, the size of the person, the body balance, the motion vector, and the like as feature quantities, and recognizes that the set person (the same target person) has moved by comparing the feature quantities of the person at the position of reference symbol 52c with the feature quantities of the people at the positions of reference symbols 52a and 52b. Similarly, the controller 21 recognizes that the person at the position of reference symbol 52d is the set person (the same target person) who has moved from the position of reference symbol 52c. Meanwhile, the controller 21 can determine that the person indicated by reference symbol 53 is not the same target person (is a non-target person) because the amount of movement is too large compared to the amount of movement estimated from the motion vector of the person at the position of reference symbol 52a, etc., and because of the difference in body balance, and the like.

[0123] FIG. 13 is a flowchart for explaining the processing executed by the controller of the HMD in the second embodiment. The flowchart in FIG. 13 corresponds to the flowchart described above in FIG. 4, and similarly, the processing is performed by the main processor 17 that reads and executes the behavior evaluation program 92. In FIG. 13, the same numbers are used for steps that perform the same processing as in the first embodiment described above in FIG. 4. In the second embodiment, the processing flow differs from that in FIG. 4 in the processing of steps S30 and S31 that are added to the distance image processing process. Also, the branch destination of the determination in S18 in the camera image processing process when it is NO and the processing of step S21 in the distance image processing process are slightly different from those in the first embodiment (determination processing is inserted), and therefore are marked with the symbol S21A.

[0124] In embodiment 2, if the judgment result of step S18 in the camera image processing process is NO (i.e., if it is judged that the set person (the set target person) is not captured in the camera image), main processor 17 determines that facial expression evaluation (step S19) cannot be performed and proceeds to step S21A.

[0125] In step S21A after obtaining a result indicating that the set person (set target person) is not captured in the camera image, the main processor 17 determines whether the set person is within the above-mentioned composite area 50.

[0126] Here, when main processor 17 determines that the set subject is within composite area 50 (step S21A: YES), it performs the processes of step S25 and thereafter through distance evaluation in step S22, similarly to the flow of FIG.

[0127] On the other hand, when the main processor 17 determines that the person to be set is not within the composite area 50 (step S21A: NO), it detects a person within the distance image and performs processing to track the person to be set as described above with reference to Fig. 12 (step S30). By performing the processing of step S30, it is possible to confirm the presence or absence of the person to be set in a wider area outside the camera image, i.e., an area that cannot be captured by the camera image.

[0128] In the next step S31, the main processor 17 judges whether or not the person to be set is present in the distance image. Here, when the main processor 17 judges that the person to be set is present in the distance image (step S31: YES), the main processor 17 performs the process of step S25 and thereafter through the distance evaluation of step S22, as in the flow of FIG.

[0129] On the other hand, when the main processor 17 determines that the set person does not exist in the distance image (step S31: NO), it executes the processes from step S23 onwards, which are similar to the flow of FIG.

[0130] Specifically, in step S23, the main processor 17 starts counting the time during which the settee cannot be confirmed (here, the time during which the determination of NO in step S21A is repeated). In the following step S24, the main processor 17 determines whether the counted time exceeds a threshold value TH.

[0131] Here, if the main processor 17 determines that the counted time has not yet exceeded the threshold TH (step S24: NO), the process returns to step S15 and repeats the above-mentioned process. On the other hand, if the main processor 17 determines that the counted time has exceeded the threshold TH (step S24: YES), the process determines that the behavior of the user (wearer) toward the set subject has reached a turning point, and proceeds to step S26.

[0132] The processes in steps S27 and S28 are the same as those in the flow shown in FIG.

[0133] Thus, according to the second embodiment, the same effects as those of the wearable device 1 in the first embodiment can be obtained, and the following unique effects can also be obtained: That is, according to the wearable device 1 in the second embodiment, it is possible to perform behavior evaluation with high tracking ability to the movement of the target person (set person) by utilizing the distance measuring device 11 having a characteristic wide measurement angle.

[0134] [Embodiment 3] Next, a third embodiment of the present disclosure will be described with reference to FIGS.

[0135] Fig. 14 is a diagram for explaining the configuration of the embodiment 3, and shows the system configuration of a behavior evaluation system including a wearable device. In Fig. 14, in addition to the wearer 3 and the subject 4 in the embodiment 1 described above in Fig. 2, an administrator 64 is further added as a relation between the wearer 3 or the subject 4.

[0136] In one example, the wearer 3 is a babysitter, and the manager 64 is the babysitter's employer (i.e., a person related to the wearer 3). As another example, the manager 64 may be a person related to the target person 4 (e.g., a guardian such as a parent). In either case, in the behavior evaluation system in the third embodiment, the manager 64 possesses a portable information device 65, and is configured to be able to use the portable information device 65 to check online the evaluation results of the behavior of the wearer 3 (babysitter) toward the target person 4.

[0137] In addition, when checking the behavior evaluation results online, the administrator 64 may use not only the portable information device 65 as exemplified in Fig. 14 but also other information terminals such as a stationary PC. On the other hand, when using the portable information device 65, there is an advantage that the behavior evaluation results can be checked anywhere, such as when on the go.

[0138] In Fig. 14, blocks having the same functions as those in Fig. 2 are given the same numbers. Regarding further differences between Fig. 14 and Fig. 2, the behavior evaluation system in embodiment 3 uses a server device (behavior evaluation service server 63) capable of communicating with wearable device 1 and portable information device 65.

[0139] The behavior evaluation service server 63 is placed in a network 62 such as a LAN or the Internet, and is capable of wireless communication with the wearable device 1 and the portable information device 65 through an access point 61. Figure 14 shows how the communication unit 16 of the wearable device 1 (see Figure 1) communicates with the behavior evaluation service server 63 via the access point 61, using wireless communication signals 60a, 60b.

[0140] In one specific example of such a behavior evaluation system (hereinafter referred to as "this system"), the wearable device 1 transmits data such as the facial expression evaluation score generated in step S19 and the distance evaluation score generated in step S22 to the behavior evaluation service server 63. The behavior evaluation service server 63 receives this data and performs the processes from step S25 (recording and storing behavior evaluation results) onward, or in parallel with the processes. The integrated value of the behavior evaluation results is stored in a memory medium (such as a HDD) of the behavior evaluation service server 63, and the stored behavior evaluation results can be monitored by the administrator 64 using the portable information device 65.

[0141] The behavior evaluation results stored in the behavior evaluation service server 63 (hereinafter, appropriately abbreviated to "server") and monitored by the portable information device 65, and the flow of use of this system, etc. will be described below with reference to Fig. 15 and Fig. 16. Here, Fig. 15 is a diagram explaining the data configuration of the behavior evaluation results. Also, Fig. 16 is a sequence diagram for monitoring the behavior evaluation results.

[0142] In this system, a service user is assigned a service ID when he or she logs into a server in network 62 by inputting a pre-registered USername. The service ID shown in Fig. 15 is a long one, "550e8400-e29b...", and in this case, the service user (administrator 64 in this example) inputs the assigned service ID ("550e8400-e29b...") by dragging and dropping or the like, and a data display screen 70 as shown in Fig. 15 is displayed on the display unit of portable information device 65.

[0143] 15 shows a plurality of data sets (two data sets, data set 71 and data set 72) that accumulate behavior evaluations of the wearer of the wearable device 1. The data configurations of data sets 71 and 72 will be described below in that order.

[0144] At the top of the data structure, a behavioral evaluation score (Score) is displayed. Here, for the data set 71, a score of "71" is displayed as the value of the behavioral evaluation result for which the calculation process has already been completed (in other words, the normalization in step S26 described above has been performed).

[0145] In addition, the start time and end time of the behavioral evaluation are displayed in the field below the behavioral evaluation score (Score). In the dataset 71, 9:15:37 AM on January 26, 2021 is displayed as the Start Time, and 12:00:12 PM on the same day of the same year is displayed as the Stop Time. Therefore, the service user (administrator 64) can know what time period the wearer's behavior was evaluated.

[0146] Furthermore, in the field below Stop Time, pairs of facial expression evaluation points and distance evaluation points are recorded in time series at regular intervals (see T in FIG. 5). The above data are collected together to form a data set 71. In one example, this data set 71 corresponds to the period NT shown in FIG. 5.

[0147] 15, the above-mentioned data set 71 is followed by a data set 72. Note that the data structure of the data set 72 is similar to that of the data set 71, so the following will describe the differences.

[0148] Dataset 72 does not display a numerical score because the behavioral evaluation results have not yet been finalized. Specifically, dataset 72 is data on behavioral evaluation results evaluated after the time period of dataset 71, and the Start Time is displayed as 13:22:46 on January 26, 2021, but the Stop Time is displayed as "Running," meaning that the evaluation is in progress and has not yet been completed.

[0149] Therefore, the service user (administrator 64) can view the behavioral evaluation and results of the wearer in the morning by looking at the data set 71 in the data display screen 70 displayed on the display unit of the portable information device 65. In addition, the administrator 64 can monitor the behavioral evaluation of the wearer from the afternoon in real time by looking at the data set 72 currently being displayed (updated) on the data display screen 70.

[0150] In one specific example, after connecting to the behavior evaluation service server 63, the mobile information device 65, in response to a predetermined operation (e.g., an operation to specify a date) by the service user (administrator 64), obtains from the behavior evaluation service server 63 a list of start times of behavior times on a specified date and displays it on the display unit. In this case, the mobile information device 65 can display a data display screen 70 as shown in Fig. 15 by specifying any start time in the list. Note that, as described above, when any start time in the list is specified, only the data set corresponding to the specified start time (e.g., only one of data set 71 or data set 72 in Fig. 15) may be obtained and displayed.

[0151] From another perspective, the behavior evaluation service server 63 transmits to the portable information device 65 data (behavior evaluation score, and the facial expression evaluation score and the distance evaluation score which are elements of the behavior evaluation score) of the data display screen 70 corresponding to the start time of the behavior time specified by the portable information device 65 (external device).

[0152] With the above-described configuration, the service user (administrator 64) can display and view the facial expression evaluation score, distance evaluation score, and behavior evaluation score on the portable information device 65 using the start time of the behavior time as a search index.

[0153] As another example, the start time of the activity time may be used as a search index, and a data display screen 70 as shown in FIG. 15 (both or only one of the data sets 71, 72 as shown in FIG. 15) may be displayed on the image display unit 13 of the wearable device 1.

[0154] Next, a process flow will be described with reference to Fig. 16 when the administrator 64 monitors the behavior evaluation of the wearer in real time using the portable information device 65. In Fig. 16, symbols T10 to T19 indicate the time from when an application (hereinafter abbreviated as "app") pre-stored (installed) in the portable information device 65 is started to when the service usage ends (logs out).

[0155] First, the administrator 64 operates the portable information device 65 to launch the above-mentioned application (time T10). When this application is launched, the portable information device 65 (the processing entity is the processor of the portable information device 65, the same applies below) sends a preset Username and Password to the behavior evaluation service server 63 at time T11, and then, at time T12, acquires a Service ID (see FIG. 15 as appropriate) from the behavior evaluation service server 63 and receives a setting input screen (not shown).

[0156] This setting input screen is displayed on the display unit of the portable information device 65 at time T13. At the following time T14, the administrator inputs setting parameters ("Setting P" in the figure). The input setting parameters are transmitted to the behavior evaluation service server 63 at time T15. In one specific example, the setting parameter is the Start Time of the data set to be monitored, and in this case, it is input directly or selected from a pull-down menu. The setting parameter may also include a parameter for selecting the display format of the monitoring. As an example of the selection of the display format of the monitoring, it is possible to select a tabular display such as the data set 72 in FIG. 15, or a graphical display as shown in FIG. 5.

[0157] Thus, the behavior evaluation service server 63 that has received the setting parameters as described above sends out data of a behavior evaluation result screen corresponding to the setting parameters (time T16). The portable information device 65 that has received the data of the behavior evaluation result screen displays the behavior evaluation result screen on its display unit at time T17 (see FIG. 15 as appropriate). When the administrator 64 finishes monitoring the behavior evaluation result screen, he operates the portable information device 65 to perform a logout process to end the application (time T18). When the behavior evaluation service server 63 receives this logout signal (time T19), it ends the connection with the service and the portable information device 65.

[0158] As described above, according to the third embodiment, in addition to the effects obtained in the first and second embodiments described above, there is a feature that the administrator 64 can check the behavioral evaluation results of the wearer online and even monitor them in real time.

[0159] [Embodiment 4] A fourth embodiment of the present disclosure will be described with reference to FIGS.

[0160] Fig. 17 is a diagram for explaining a scene in which multiple targets are present. Fig. 17 shows the relationship between the wearer 3 and targets 4, and assumes that there are multiple targets 4 (in this example, four targets 4a, 4b, 4c, and 4d).

[0161] As a specific example, for example, in a nursery school, a nursery teacher is taking care of multiple children and the behavior of the nursery teacher toward the children is evaluated. In this case, the nursery teacher is the wearer 3 of the wearable device 1 (HMD), and each child 4 (4a to 4d) is the target person, and the target person does not need to wear anything.

[0162] Other specific examples include a caregiver (wearer 3) and a care recipient in a care facility, a teacher (wearer 3) and a student in an educational setting, and a store clerk (wearer 3) and a customer (person being served) in a store.

[0163] Fig. 18 is a flowchart showing an example of processing executed by wearable device 1 in embodiment 4. In Fig. 18, the same numbers are given to steps that perform similar processing to the processing flow of embodiment 1 shown in Fig. 4.

[0164] Fig. 18 differs from Fig. 4 in that a face image database DB1 is used. That is, in the fourth embodiment, face data of all target persons is stored in advance in the face image database DB1.

[0165] Then, in a face recognition step (step S12A) after acquiring the camera image in step S11, the main processor 17 performs face recognition processing on the face captured in the acquired camera image based on a comparison with face data registered in advance in the face image database DB1. The processing in the following step S13 is the same as that in FIG.

[0166] Similarly, in a face recognition and person detection step (step S17) after acquiring the camera image in step S16, the main processor 17 performs face recognition processing on the face captured in the acquired camera image based on a comparison with face data registered in advance in the face image database DB1. The processing from the subsequent step S18 onwards is the same as that in FIG.

[0167] More specifically, in the face recognition process of step S12A and the face recognition and person detection process of step S17, main processor 17 refers to face image data for the number of subjects registered in face image database DB1. Then, main processor 17 executes the process for each subject in the facial expression evaluation process of step S19, the distance evaluation process of step S22, the evaluation recording and accumulation process of step S25, and the normalization of the evaluation result of step S26.

[0168] Fig. 19 is a diagram for explaining the configuration of a database of behavioral evaluation results corresponding to multiple subjects. As with Fig. 15 described above, the example shown in Fig. 19 displays multiple data sets (two data sets, data set 73 and data set 74) that accumulate behavioral evaluations of the wearer of the wearable device 1. For ease of understanding, data set 73 shows the same time period as data set 71 shown in Fig. 15, i.e., the start time and end time of the behavioral evaluation.

[0169] 19, the behavior evaluation result is made up of sub-data sets, designated by reference characters 73a to 73d, for each of subjects A, B, C, and D. In other words, according to the configuration of the fourth embodiment using the face image database DB1, it becomes easy to simultaneously perform processing such as evaluation of the behavior of the wearer for a plurality of subjects for each subject (subjects A, B, C, and D in this example) in parallel.

[0170] 20 is a diagram showing an example of the display of a behavior evaluation result in the fourth embodiment. The behavior evaluation result is output as a display screen 30 to the image display unit 13 of the wearable device 1 based on a control signal from the controller 21 (output processing device). The controller 21 of the wearable device 1 may transmit data of this display screen 30 to the behavior evaluation service server 63 via the above-mentioned network 62. In this case, the display screen 30 can be displayed on a display unit of a portable information device 65 connected to the behavior evaluation service server 63.

[0171] In the example of the display screen 30 shown in Fig. 20, the evaluation points are displayed for each subject (A, B, C, D) (66, 48, 35, 75 points), and the comparison result of the evaluation points for each subject is displayed in a graph. Also, in this example, the difference of the evaluation points for each subject (A, B, C, D) from the average score (Ave.) of 65 points is displayed as a percentage. Furthermore, in this example, the graph for subject C, who received the lowest evaluation, is displayed in a different color from the graphs for the other subjects A, B, D.

[0172] By displaying the display screen 30 as described above, for example, it is possible to visually and easily present that the behavior evaluation of the subject C is relatively inferior compared to the other subjects (A, B, D). Furthermore, in this example, advice (message output) of "try your best to take care of the subject C" is presented as a general comment. Note that this advice (message output) may alternatively or additionally be output as a voice from the voice input / output unit 14.

[0173] As another example of the display screen 30, the evaluation points for a specific subject (e.g., subject C) may be displayed in chronological order over a plurality of behavior evaluation times, for example, by controlling the controller 21 in response to a user's operation of the operation input unit 15. Alternatively, the display screen 30 for the behavior evaluation corresponding to the start time of a behavior time may be displayed using the start time as a search index, for example, by controlling the controller 21 in response to a user's operation of the operation input unit 15.

[0174] In this manner, the configuration of the fourth embodiment in which the evaluation score (behavior evaluation score of the wearer 3) is displayed for each target person (A, B, C, D) has the characteristic that the behavior evaluation for each target person can be performed objectively and feedback on the behavior can be easily obtained. Also, as shown in FIG. 20, according to the configuration in which the behavior evaluation score for each target person (A, B, C, D) is graphed and displayed in a list for comparison, the evaluation result for each target person can be easily understood even if the area for displaying the display screen 30 is small. In addition, by configuring to output a general comment on the behavior evaluation, the wearer 3 can quickly understand what he or she needs to reflect on this time.

[0175] It should be noted that the present invention is not limited to the specific examples of the embodiments described in Fig. 1 to Fig. 20. For example, it is possible to replace part of the configuration of one embodiment with another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these belong to the scope of the present invention, and furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0176] For example, some of the functions performed by the controller 21 of the wearable device 1 (for example, functions as a subject image processing device, a distance image processing device, and an output processing device) may be performed by the behavior evaluation service server 63 described above.

[0177] In addition, the functions of the devices described in this specification and the drawings may be implemented in part or in whole in hardware, for example by designing them as integrated circuits. They may also be implemented in software by a microprocessor unit, CPU, etc. interpreting and executing an operating program. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. [Explanation of symbols]

[0178] 1: wearable device, 2: wearing equipment, 3: wearer, 4, 4a to 4d: target person (setting subject), 10: camera (imaging device), 11: distance measuring device, 13: image display unit, 13c: main screen display unit, 14: voice input / output unit, 15: operation input unit, 16: communication unit, 17: main processor, 19: flash ROM, 21: controller (subject image processing device, distance image processing device, output processing device), 92: behavior evaluation program, 93: data unit (data storage unit), 30: display screen, 51: distance image, 63: behavior evaluation service server, 64: administrator, 65: portable information device, DB1: face image database

Claims

1. an imaging device that captures an image of a target person in front of or around the wearer and generates a subject image; a subject image processing device that detects and recognizes the face of the subject included in the subject image, and processes the subject image so as to assign an expression evaluation score to the wearer based on the facial expression of the subject obtained by the face recognition; an output processing device that executes an evaluation process to obtain a behavior evaluation score for evaluating the behavior of the wearer from the facial expression evaluation score and a behavior evaluation time as a behavior time of the wearer to be evaluated, and outputs the behavior evaluation score; A wearable device comprising:

2. The wearable device according to claim 1 , The output processing device starts execution of the evaluation process from a first time point when the target person is set as a set person, sets a period from the first time point to a second time point when the set person is not detected from the generated subject image for a threshold time as the behavior evaluation time, and outputs the behavior evaluation score based on an integrated value of the facial expression evaluation score during the behavior evaluation time. Wearable devices.

3. The wearable device according to claim 1 , The processing of the object image by the object image processing device is executed at a constant cycle. Wearable devices.

4. The wearable device according to claim 3, the output processing device executes the evaluation process by accumulating the facial expression evaluation score at each timing of the fixed period, calculating a statistical value of the accumulated facial expression evaluation score, and normalizing each calculated statistical value by the behavior evaluation time to obtain the behavior evaluation score. Wearable devices.

5. The wearable device according to claim 4, The statistical value is an average value calculated by averaging or weighting the accumulated facial expression evaluation scores. Wearable devices.

6. The wearable device according to claim 2, The output processing device includes: the facial expression evaluation score is integrated at a constant cycle, and the evaluation process is executed so that a cumulative value obtained by accumulating the integrated value during the behavior evaluation time is set as a behavior evaluation score. Wearable devices.

7. The wearable device according to claim 1 , The output processing device outputs a screen showing the behavior evaluation result including the behavior evaluation score to a display unit. Wearable devices.

8. The wearable device according to claim 2, Equipped with a data storage unit, the output processing device records the facial expression evaluation score in the data storage unit together with the first time point, which is a start time of the behavior evaluation time, and the second time point, which is an end time of the behavior evaluation time; Wearable devices.

9. The wearable device according to claim 2, A communication unit for connecting to a network is provided. the output processing device outputs the facial expression evaluation score together with the first time point, which is a start time of the behavior evaluation time, and the second time point, which is an end time of the behavior evaluation time, to a server device via the communication unit. Wearable devices.

10. The wearable device according to claim 2, the subject image processing device detects and recognizes faces of the plurality of subjects included in the subject image, and processes the subject image so as to assign the facial expression evaluation score to each of the subjects; The output processing device executes an evaluation process for obtaining a behavior evaluation score for evaluating the behavior of the wearer for each of the subjects. Wearable devices.

11. The wearable device according to claim 10, The output processing device outputs the behavior evaluation score for each of the subjects. Wearable devices.

12. A wearable device according to claim 1; and a server device that acquires the behavioral evaluation points output from the output processing device and provides the acquired behavioral evaluation points to an external device owned by a related person of the subject. Behavioral assessment system.

13. The behavior evaluation system according to claim 12, the server device transmits to the external device the behavior evaluation score corresponding to a start time of the behavior period designated by the external device, and the facial expression evaluation score, which is an element of the behavior evaluation score; Behavioral assessment system.

Images