Wearable devices and behavioral assessment systems

Abstract

To provide a wearable device and an action evaluation system which measure and evaluate a direct relation between a wearer and a target person, and which are capable of giving awareness to the wearer regarding their action by clearly indicating the evaluation result, and to contribute to one of the Sustainable Development Goals (SDGs): "health and well-being for all", by making it possible to maintain human mental and physical health.SOLUTION: A wearable device is provided with: an imaging device which captures a subject image by imaging a target person present in front of or around a wearer; a subject image processing device which detects and performs face recognition of the target person included in the subject image, and processes the subject image so as to give an expression evaluation point to the wearer based on the facial expression of the target person obtained by the face recognition; and an output processing device which executes an evaluation process for obtaining an action evaluation point to evaluate an action of the wearer based on the expression evaluation point and an action evaluation time period being a time period during which the wearer, who is the evaluation target, performs the action, and outputs the action evaluation point.SELECTED DRAWING: Figure 4

Description

Technical Field

[0006] , , , ,

[0001] The present invention relates to wearable devices and behavior evaluation systems.

Background Art

[0002] In Patent Document 1, in order to measure the relationship between people belonging to an organization, a state where people wearing face sensors face each other is detected, and through the creation and analysis of the face-to-face history of people, a system for visualizing the degree of activity of the organization has been proposed.

[0003] In addition, there is a commercially available wearable device, so-called "action cam", which is equipped with a camera to photograph the background in front of a person's line of sight and produce actions such as sports as content. ​​​​​​​​​​​​​​​​​​​​​​​​​​​ [Problems that the invention aims to solve]

[0007] As described above, Patent Document 1 does not describe the use of direct relationships between people. Furthermore, the technology described in Patent Document 1 has the problem that all members of the organization must be fitted with face-to-face sensors.

[0008] The object of the present invention is to provide a wearable device and behavioral evaluation system that can measure and evaluate the direct relationship between the wearer and the target person, and provide awareness of the wearer's behavior by clearly indicating the evaluation results. [Means for solving the problem]

[0009] A brief overview of some of the representative inventions disclosed in this application is as follows:

[0010] A wearable device according to a typical embodiment of the present invention comprises: an imaging device that captures a subject person in front of or around the wearer to generate a subject image; a subject image processing device that detects and recognizes the subject person contained in the subject image and processes the subject image to assign the wearer an expression evaluation score based on the facial expression of the subject person obtained by the face recognition; and an output processing device that performs an evaluation process to obtain an action evaluation score for evaluating the wearer's actions from the expression evaluation score and the action evaluation time, which is the time the wearer is acting as the subject of evaluation, and outputs the action evaluation score. The output processing device starts executing the evaluation process from the first time point in time when the subject is set as the target, and defines the period from the first time point to the second time point in time, from the generated subject image until the target is no longer detected for the threshold time, as the behavior evaluation time, and outputs the behavior evaluation score based on the cumulative value of the facial expression evaluation score during the behavior evaluation time. .

[0011] A typical embodiment of the present invention provides a behavioral evaluation system that includes the above-mentioned wearable device and a server device that acquires the behavioral evaluation points output from the output processing device and provides the acquired behavioral evaluation points to an external device owned by a person related to the subject. [Effects of the Invention]

[0012] The effects obtained by some of the representative inventions disclosed in this application can be briefly explained as follows.

[0013] In other words, according to a typical embodiment of the present invention, it is possible to provide a wearable device and a behavioral evaluation system that have a behavioral evaluation function capable of measuring and evaluating the direct relationship between the wearer and the target person, and providing awareness of the wearer's behavior by clearly indicating the evaluation results. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram of the wearable device according to Embodiment 1. [Figure 2] This diagram illustrates the relationship between the wearer of a wearable device and the target person. [Figure 3] This is a perspective view illustrating the configuration of a head-mounted display (HMD), a type of wearable device. [Figure 4] This is a flowchart illustrating the operation of the head-mounted display in Embodiment 1. [Figure 5] This graph illustrates how the evaluation score for the behavior of wearers of wearable devices changes over time. [Figure 6] This diagram illustrates a table that associates the types of facial expressions of the target person (the person being evaluated) with their corresponding facial expression evaluation scores. [Figure 7] This diagram illustrates a table that maps the distance between the wearer and the target person (the person being targeted) to distance evaluation points. [Figure 8] This figure shows an example of how behavioral evaluation results for wearers of wearable devices are displayed. [Figure 9] This flowchart shows the process for obtaining cumulative behavioral evaluation results. [Figure 10A] This figure shows an example of how cumulative behavioral assessment results are displayed. [Figure 10B] This figure shows an example of how cumulative behavioral assessment results are displayed. [Figure 11]It is a diagram for explaining the relationship between the angle of view of the camera and the measurement angle of the distance measurement device in Embodiment 2. [Figure 12] It is a diagram for explaining the process of tracking a target person from a distance image. [Figure 13] It is a flowchart for explaining the process executed by the controller in Embodiment 2. [Figure 14] It is a diagram for explaining the configuration of Embodiment 3, and shows the system configuration of an action evaluation system including a wearable device. [Figure 15] It is a diagram for explaining the data structure of the action evaluation result. [Figure 16] It is a sequence diagram when monitoring the action evaluation result. [Figure 17] It is a diagram for explaining a scene where there are a plurality of target persons. [Figure 18] It is a flowchart showing an example of the process executed by the wearable device in Embodiment 4. [Figure 19] It is a diagram for explaining the configuration of a database of action evaluation results corresponding to a plurality of target persons. [Figure 20] It is a diagram showing an example of display of the action evaluation result.

Embodiments for Carrying Out the Invention

[0015] In the embodiments and examples disclosed below, devices and systems that contribute to the mental health of users and the maintenance of the health of touch partners by analyzing and evaluating the degree of human touch and prompting the behavior of users are disclosed. Also, the technology of the present disclosure contributes to "health and well-being for all people" of the Sustainable Development Goals (SDGs) advocated by the United Nations (International United Nations) by enabling the maintenance of human mental and physical health.

[0016] In the following embodiment, a wearable device is used to analyze the degree of human-to-human interaction. Specifically, the degree of interaction between a user wearing the wearable device (hereinafter referred to as the "wearer") and a person photographed or recognized by the wearable device (hereinafter referred to as the "target person") is determined through image analysis such as person detection and facial recognition.

[0017] Here, the degree of interaction between the wearer and the target person (i.e., person to person) is calculated as an "action evaluation score" that takes into account factors such as the target person's facial expression obtained through facial recognition, the distance between the wearer and the target person, and the time the wearer spends interacting with the target person. This "action evaluation score" can be calculated by determining an "expression evaluation score" based on the target person's facial expression, a "distance evaluation score" corresponding to the distance between the wearer and the target person, and an "action evaluation time" representing the time the wearer spends interacting with the target person, and then substituting these scores and times into a predetermined calculation formula. The calculated "action evaluation score" is a point that comprehensively evaluates the wearer's actions (degree of interaction) with the target person, and by showing this point to the wearer, it is possible to encourage the wearer's actions and contribute to maintaining the mental and physical health of both the wearer and the target person.

[0018] More specifically, in order to determine the "facial expression evaluation score" mentioned above, the wearable device comprises an imaging device that captures subjects in front of or around the wearer to generate subject images, and a subject image processing device that detects subjects included in the subject images and performs face recognition, and processes the subject images to assign the wearer facial expression evaluation scores based on the facial expressions of the subjects obtained through face recognition.

[0019] Furthermore, in order to determine the "distance evaluation points" mentioned above, the wearable device includes a distance measuring device that measures the distance between the wearer and the target person and acquires distance data, and a distance image processing device that analyzes a distance image obtained by mapping the distance data to determine the distance between the wearer and the target person and acquires distance evaluation points corresponding to that distance.

[0020] Furthermore, in order to determine and output the above-mentioned "behavioral evaluation score," the wearable device includes an output processing device that performs an evaluation process to determine the behavioral evaluation score for evaluating the wearer's behavior from the facial expression evaluation score, the distance evaluation score, and the behavioral evaluation time, which is the time the wearer is acting as the subject of evaluation, and outputs the behavioral evaluation score.

[0021] Of the above, the "subject image processing device," "distance image processing device," and "output processing device" can be implemented by a processor (such as a CPU) installed in the wearable device or by an external server device. Each of these devices can be implemented by the same processor or by separate processors, but the following example assumes implementation by a single processor installed in the wearable device.

[0022] By using the configuration described above, it is possible to measure and evaluate the direct relationship between the wearer and the subject without attaching sensors or other devices to the subject, and to output the evaluation results.

[0023] Embodiments of the present invention will be described below with reference to the drawings. The embodiments shown in the drawings below are examples for realizing the present invention and do not limit the technical scope of the present invention. In the embodiments, the same reference numerals are used for members having the same function, and repeated descriptions are omitted unless particularly necessary.

[0024] [Embodiment 1] Embodiment 1 will be described below with reference to Figures 1 to 10. Embodiment 1 discloses a wearable device worn by a user (hereinafter exclusively referred to as "wearer"). This wearable device is equipped with an imaging device such as a camera and a distance measuring device such as a distance measuring sensor, and has an action evaluation function that measures the degree of interaction between the wearer and a person (target person) captured by these devices.

[0025] (Block diagram and usage scenarios) Figure 1 is a block diagram of the wearable device according to Embodiment 1.

[0026] As shown in Figure 1, the wearable device 1 of Embodiment 1 includes a camera 10 (imaging device), a distance measuring device 11, a sensor unit 12, an image display unit 13, an audio input / output unit 14, an operation input unit 15, a communication unit 16, a main processor 17, RAM 18, and flash ROM (FROM) 19.

[0027] The main processor 17 is the main control unit that controls the entire wearable device 1 according to a predetermined program. The main processor 17 is implemented as a CPU (Central Processor Unit) or a microprocessor unit (MPU). The main processor 17 performs operation control processing for the entire wearable device 1 by executing programs such as the operating system (OS) and various operation control applications stored in the storage device 110. The main processor 17 also controls the startup operations of various applications.

[0028] Of these, the sensor unit 12, communication unit 16, main processor 17, RAM 18, and flash ROM (FROM) 19 play the role of a control device or controller 21 (see the dotted line frame in Figure 1) in the wearable device 1.

[0029] In the wearable device 1, the camera 10 is an imaging device that captures images of people in front of or around the wearer and generates subject images.

[0030] In one specific example, camera 10 includes an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and an optical lens. In this case, camera 10 captures the background in front of the wearer's line of sight within its field of view (see Figures 2 and 3 as appropriate). Therefore, when photographing surrounding objects not located in front of the wearer's line of sight, the wearer should turn their face or body towards the object to be photographed.

[0031] On the other hand, in the wearable device 1, the distance measuring device 11 is a distance measuring device generally called a distance sensor or distance measuring sensor, and has the function of measuring the distance between the wearer and the target person and acquiring distance data.

[0032] In one specific example, the rangefinder 11 is a device equipped with a rangefinder sensor (also called optical radar) such as LiDAR (Light Detection and Ranging).

[0033] Such a distance measuring device 11 includes a configuration that, for example, irradiates a laser beam from a light source (such as a laser diode) inside the sensor in a scanning manner, and measures the time it takes for the reflected light from the object to be measured to be received by the light-receiving element of the sensor. With a distance measuring device 11 configured in this way, the distance to the background (various objects or people, etc.) in front of the wearer's line of sight is measured and distance data is acquired. The distance measuring device 11 also generates a distance image by mapping the measured and acquired distance data with the object to be measured in two dimensions.

[0034] The sensor unit 12 includes various sensors other than the distance measuring sensor (distance measuring device 11) described above, such as a GPS sensor, gyro sensor, geomagnetic sensor, acceleration sensor, etc., as shown in Figure 1. The configuration of each of the above sensors in the sensor unit 12 is publicly known and has little relevance to Embodiment 1, so a detailed description is omitted.

[0035] The image display unit 13 is responsible for displaying images generated by the controller 21. While various images can be generated and displayed by the controller 21, in this embodiment, for example, an image of the wearer's behavior evaluation results is an example, and details of this image will be described later.

[0036] The audio input / output unit 14 includes a microphone (hereinafter abbreviated as "microphone") for inputting (collecting) sound and a speaker for outputting (speaking) sound. In this embodiment, the above behavior evaluation results can also be output as sound from the speaker of the audio input / output unit 14 to notify the wearer of the results.

[0037] The operation input unit 15 can use various devices for inputting operations by the wearer. For example, the operation input unit 15 may be any of the following devices: a touch sensor for inputting the wearer's finger movements, a microphone for inputting the wearer's voice, or a sensor for detecting the wearer's hand movements captured by the camera 10. Furthermore, it may be configured in combination with multiple such devices.

[0038] The communication unit 16 includes, for example, a wireless LAN that performs 4G (Generation) and 5G mobile communication. When necessary, the communication unit 16 selects an appropriate process from among the communication protocols and connects the wearable device 1 to the network.

[0039] FROM19 is a non-volatile memory medium and includes a basic operation program 91 and an action evaluation program 92 (hereinafter sometimes simply referred to as processing programs 91 and 92) which are processing programs executed by the main processor 17, and a data section 93 in which various data are stored.

[0040] The processing programs 91 and 92 are read by the main processor 17 and executed by being loaded into RAM 18. The data section 93 stores the data necessary to execute the processing programs 91 and 92. FROM 19 may be a single memory medium as shown in the figure, or it may consist of multiple memory mediums. Furthermore, it may be a non-volatile memory medium other than flash ROM.

[0041] Figure 2 is a diagram illustrating the relationship between the wearer of a wearable device and the subject. Referring to Figure 2, we see wearer 3, who is wearing wearable device 1, and subject 4, whose behavior and behavioral evaluation are being conducted.

[0042] In the illustrated example, wearer 3 is a parent, subject 4 is an infant, and wearer 3's behavior (and therefore the type of behavioral evaluation) is childcare. Furthermore, the following explanation assumes that wearable device 1 is a head-mounted display (HMD).

[0043] The wearer 3 can use fasteners or other attachments 2 as appropriate to hold the wearable device 1 in place so that it does not slip or fall from their face. The attachments 2 may also be in the form of a headband, which attaches the wearable device 1 to the wearer 3's head, or the wearable device 1 (HMD in this example) and the attachments 2 may be integrated into a single unit.

[0044] Thus, the wearable device 1 captures (photographs) the subject person 4 with the camera 10 (see Figure 1) and executes processing programs 91 and 92 to evaluate the wearer 3's actions toward the subject person 4, and records the results of this behavioral evaluation (hereinafter referred to as "behavioral evaluation results," and sometimes abbreviated as "evaluation results").

[0045] As in this embodiment, when the wearable device 1 is an HMD (Head-Mounted Display), the behavioral evaluation results can be output to the HMD's display, i.e., a screen displayed in front of the wearer's eyes (3). Therefore, the wearer (3) can use the wearable device 1 hands-free.

[0046] Figure 3 shows an example of the external view of an HMD, which is a type of wearable device.

[0047] Referring to Figures 1 and 3, the wearable device 1 (HMD) comprises a camera 10, a rangefinder 11, an image display unit 13, an audio input / output unit 14, an operation input unit 15, and a controller 21 having various blocks (see the dotted line frame in Figure 1). Of these, the image display unit 13 is implemented as a projector 13a, 13b, and a main screen display unit 13c (a semi-transparent screen in this example), as shown in Figure 3. The camera 10 and the rangefinder 11 are arranged side by side approximately in the center of the top of the HMD housing (front frame 22c). The controller 21 is located on one of the HMD's (glasses) temples, in this example, the right-hand horizontal frame 22b.

[0048] Furthermore, as components constituting the audio input / output unit 14 described above in Figure 1, a speaker 14a and a microphone 14b are provided on the other horizontal frame 22a (the left side in this example), which corresponds to the stalk. Also, as shown in Figure 3, the front frame 22c is provided with an image display unit 13, which includes a left L projector 13a, a right R projector 13b, and a main screen display unit 13c having a larger area than these projectors 13a and 13b.

[0049] Furthermore, a nose pad 23, which is a component of the mounting device 2 described above in Figure 2, is provided in the center of the main screen display unit 13c. The user attaches the HMD 1 to their face (head) by hooking the ends of the horizontal frames 22a and 22b over both ears and placing the nose pad 23 on their nose.

[0050] Of the above, the main screen display unit 13c is a semi-transparent screen in this example, but in other examples, it may be an opaque display, and in either case, the user can see the background in front of them through the main screen display unit 13c.

[0051] The above example describes a display unit consisting of a semi-transparent screen and a projector, but a retinal projection display that projects onto the user's retina without using a screen is also acceptable. Furthermore, non-transparent displays include display devices using lasers, liquid crystal panels, organic EL (EL: Emitting Diode), etc., and the user may view the display device directly through lenses or the like.

[0052] Here, if the main screen display unit 13c is a semi-transparent screen, the user views the front background through the semi-transparent screen. On the other hand, if the main screen display unit 13c is an opaque display, the user confirms the front background by displaying the camera image of the front background on the opaque display.

[0053] Camera 10 is mounted on the front frame 22c (housing) of the HMD to capture the background in front of the user's line of sight. A rangefinder 11, positioned next to camera 10, measures the distance to objects in the background in front of the user's line of sight.

[0054] The controller 21 receives images captured by the camera 10 (hereinafter referred to as "camera images") and distance images generated by the distance measuring device 11, and supplies them to its internal memory (RAM 18 or data unit 93) and the main processor 17. The controller 21 also generates images to be projected or displayed on the image display unit 13 (L projector 13a, R projector 13b, main screen display unit 13c) and sounds to be output from the speaker 14a.

[0055] In relation to the feature components described above, the controller 21, and especially the main processor 17, performs functions such as "subject image processing device," "distance image processing device," and "output processing device."

[0056] The controller 21, camera 10, distance measuring device 11, speaker 14a, and microphone 14b are arranged in the corresponding frames 22a to 22c as described above, but the placement of these components does not necessarily have to be as shown in Figure 3.

[0057] (flowchart) Figure 4 is a flowchart illustrating the operation of the head-mounted display in Embodiment 1. The flowchart in Figure 4 shows the processing flow based on the behavioral evaluation program 92, with the main processor 17, which reads the behavioral evaluation program 92, being the primary processor. Then, according to the control signals output from the main processor 17, the corresponding blocks of the wearable device 1 operate to perform each process in the flowchart.

[0058] In step S11, after the start of execution of the behavior evaluation program 92, the main processor 17 outputs control information to the camera 10 to acquire a camera image. This acquisition of the camera image may be synchronized with the timing of the camera 10's shooting, or it may be done by continuously taking images at, for example, 30 fpS (frames per second) and acquiring an image at any arbitrary timing.

[0059] In the subsequent step S12, the main processor 17 performs either the process of registering new face data or the process of face recognition based on comparison with already registered face data for the face captured in the acquired camera image. Here, when registering new face data, the main processor 17 can assign and register an identification number for the target person based on the operation content of the operation input unit 15 by the user (wearer).

[0060] In step S13, the main processor 17 determines whether or not it has registered or recognized the subject.

[0061] Here, if the main processor 17 determines that the subject has not been registered or recognized (step S13: NO), it determines that the face captured in the camera image is not the face of the subject or that there is no face captured in the camera image. In this case, the main processor 17 returns to step S11 to reacquire the camera image and repeats the processing of steps S11 to S13 described above.

[0062] On the other hand, if the main processor 17 determines that it has registered or recognized a target person (step S13: YES), it identifies (sets) the target person who will be the target of the user's (wearer's) actions and proceeds to step S14.

[0063] In step S14, the main processor 17 starts accumulating the program execution time.

[0064] In the subsequent step S15, the main processor 17 determines whether or not a predetermined cycle timing has arrived.

[0065] Here, if the main processor 17 determines that the predetermined timing cycle has not yet arrived (step S15: NO), it repeats the determination in step S15 until it determines that the predetermined timing cycle has arrived (step S15: YES).

[0066] Then, when the main processor 17 determines that a predetermined period timing has arrived (step S15: YES), it executes the camera image processing process (steps S16 to S19) and the distance image processing process (steps S20 to S22) described below.

[0067] In Figure 4, a multi-process configuration is illustrated in which the main processor 17 executes the camera image processing process (steps S16 to S19) and the distance image processing process (steps S20 to S22) in parallel (concurrently). As another example, the main processor 17 may be configured to perform serial processing, for example, executing the camera image processing process (steps S16 to S19) followed by the distance image processing process (steps S20 to S22).

[0068] (Camera image processing process) In step S16, which marks the start of the camera image processing process, the main processor 17 acquires the camera image captured by the camera 10. In the following step S17, the main processor 17 performs face recognition processing and person detection processing for the face captured (imaged) in the acquired camera image.

[0069] Then, in step S18, the main processor 17 determines whether or not the person set in step S13 (hereinafter also referred to as the "target person") is visible in the acquired camera image. More specifically, in step S18, the main processor 17 checks the detection or recognition result in step S17. In one specific example, the main processor 17 determines whether or not a person has been detected in the camera image, and if so, whether or not that person is the target person. The main processor 17 also determines whether or not a face has been recognized in the camera image, and if so, whether or not that face is the face of the target person.

[0070] Furthermore, if the main processor 17 can perform face recognition on a person detected in the camera image, it determines whether or not the person is a designated person based on the face recognition result. On the other hand, if the face of a person detected in the camera image is unclear (i.e., face recognition cannot be performed), for example, if the detected person is facing away, the main processor 17 makes an estimate of whether or not the detected person is a designated person.

[0071] To perform such estimations, the main processor 17 determines, for example, the similarity of the color and texture of the clothing of the person recognized immediately before (when setting the subject (target) before step S14), the balance of the face and body, and the validity of the distance traveled considering the motion vector. The detected subject shares its position and size in the image with the distance image processing process (see step S25).

[0072] Thus, if the main processor 17 determines that the designated person (the person who has been designated) is not visible in the camera image (step S18: NO), it returns to step S15 and executes the processes from step S23 onwards. The details of the processes from step S23 onwards will be described later.

[0073] On the other hand, if the main processor 17 determines that the designated person (the person who has been designated) is visible in the camera image (step S18: YES), it proceeds to step S19. In step S19, the main processor 17 obtains an expression evaluation score based on the recognized facial expression of the designated person. In one specific example, the main processor 17 obtains the expression evaluation score using a table that associates the type of facial expression (laughing, anger, crying, etc.) with the expression evaluation score. An example of this table will be described later in Figure 6.

[0074] Alternatively, the main processor 17 may modify or calculate the facial expression evaluation score based on the subject's facial expression, while also considering the subject's other gestures. Here, the subject's "other gestures" include the degree of emotion in the voice the subject makes (so-called tone of voice), and emotional expressions made by the subject's hand (arm) or body gestures as seen in the camera image. In this case, the main processor 17 obtains the final facial expression evaluation score by adding the score based on the aforementioned other gestures to the facial expression evaluation score obtained in accordance with the subject's facial expression.

[0075] (Distance image processing process) In step S20, which marks the start of the distance image processing process, the main processor 17 acquires the distance image measured by the distance measuring device 11.

[0076] In the subsequent step S21, the main processor 17 receives or appropriately references the presence information of the person to be assigned and the position of the person to be assigned in the camera image obtained in the camera image processing process described above (step S18, etc.). Through this process, the main processor 17 obtains or calculates the more accurate three-dimensional coordinate position of the person to be assigned who is in front of the user (wearer) and obtains the distance between the wearer and the person to be assigned. Then, in step S22, the main processor 17 obtains distance evaluation points based on the distance between the wearer and the person to be assigned.

[0077] (Processing such as accumulating evaluation points) In step S25, after the facial expression evaluation points are acquired in step S19 and the distance evaluation points are acquired in step S22, the main processor 17 records and accumulates these evaluation points. This recording and accumulation process may be performed by recording (accumulating) the data in the data unit 93 within the wearable device 1, or by recording (accumulating) the data on a server's recording medium via a network.

[0078] If the main processor 17 determines in the camera image processing process described above that the person to be set is not visible in the camera image (the person to be set cannot be identified) (step S18: NO), it returns to step S15 to acquire the camera image and depth image again, and executes the processes from step S23 onwards.

[0079] In step S23, the main processor 17 starts counting the time during which the configured user cannot be confirmed (the time during which the NO determination in step S18 is repeated). In the following step S24, the main processor 17 determines whether the counted time exceeds the threshold TH.

[0080] Here, if the main processor 17 determines that the counted time has not yet exceeded the threshold TH (step S24: NO), it returns to step S15 and repeats the process described above. On the other hand, if the main processor 17 determines that the counted time has exceeded the threshold TH (step S24: YES), it determines that the user's (wearer's) actions toward the configured person have come to an end and proceeds to step S26.

[0081] In step S26, the main processor 17 calculates statistical values ​​of the evaluation scores recorded (accumulated) in step S25 described above. In one specific example, the main processor 17 calculates a value normalized by the action evaluation time (for example, the program execution time) of the evaluation scores (facial expression evaluation score and distance evaluation score) recorded (accumulated) at that time.

[0082] The main processor 17 outputs the calculated value as the behavioral evaluation result (step S27) and proceeds to step S28.

[0083] In step S28, the main processor 17 determines whether or not a program termination event has occurred. This "termination event" may include, for example, receiving a command from the user (wearer) to shut down the program or turn off the power.

[0084] Here, if the main processor 17 determines that a program termination event has not yet occurred (step S28: NO), it determines that the user's (wearer's) work has not yet finished and returns to step S11, repeating the process described above. In this case, in step S14, which is executed again, the main processor 17 initializes the program execution time (the time for evaluating the user's (wearer's) actions) and begins accumulating that time.

[0085] On the other hand, if the main processor 17 determines that a program termination event has occurred (step S28: YES), it determines that the user's (wearer's) work has ended and terminates the series of processes shown in Figure 4.

[0086] (Example of rating and display) Figure 5 is a graph illustrating how the evaluation score for the wearer's behavior with a wearable device changes over time. In the graph in Figure 5, the vertical axis represents the evaluation score, and the horizontal axis represents the passage of time or the execution time of the program (the time spent evaluating the wearer's behavior). For ease of understanding, the step numbers of the process explained in Figure 4 are added to Figure 5 as appropriate.

[0087] As shown in Figure 5, the main processor 17 of the wearable device 1 starts evaluating the wearer's actions (assigning and recording evaluation points, and accumulating them) from the moment it recognizes and sets a person (target) in the camera image as the target (when it is determined to be YES in the first executed step S18).

[0088] Furthermore, the camera image processing process, distance image processing process, etc., described above are performed at regular intervals T as shown in Figure 5, and the main processor 17 obtains evaluation points at each interval T. Here, the interval T corresponds to the timing at which YES is determined in step S15.

[0089] The "evaluation score" shown on the vertical axis of the graph in Figure 5 is the sum of the facial expression evaluation score and the distance evaluation score. Generally, when face recognition is possible, the facial expression evaluation score is added, resulting in a higher evaluation score (see the evaluation score for "(S18:YES) Face Recognition Available" in Figure 5). Conversely, if the target person cannot be identified by face recognition, the evaluation score is based solely on the distance evaluation score, resulting in a relatively lower evaluation score (see the evaluation score for "(S18:NO) Face Recognition Not Possible" in Figure 5).

[0090] Furthermore, in the example shown in Figure 5, if the person being targeted is not visible in the camera image (see step S18:NO branch in Figure 4 as appropriate), and the person is not captured in the depth image, the evaluation score will be zero (see the evaluation score for "Person not measured" in Figure 5). If this "Person not measured" time (in this case, the time when the evaluation score is zero) exceeds the threshold TH (see step S24 in Figure 4 as appropriate), a value based on the cumulative evaluation results for the initial period (see period "NT" in Figure 5) will be output as the behavioral evaluation result (see steps S26 and S27 in Figure 4 as appropriate).

[0091] Of the above, the "NT" period shown in Figure 5 corresponds to the "behavioral evaluation time," which is the time spent by the wearer during their activities.

[0092] Furthermore, if the person being monitored is subsequently detected within the camera image or depth image (see step 18: YES in Figure 4 and "Person being monitored again" in Figure 5), the facial expression evaluation or distance evaluation during the wearer's next action period (new action evaluation period) will be recorded and accumulated (see step S25 in Figure 4, etc.).

[0093] By repeating this process, points (behavioral evaluation scores) that comprehensively evaluate the wearer's behavior (degree of interaction) with the target person (the person being targeted) are recorded and accumulated over time, and the behavioral evaluation results are output for each behavioral evaluation period.

[0094] The method of assigning behavioral evaluation points and the timing of program execution are not limited to the above example; for example, the following may also be used. That is, if the target person (targeted person) is not captured in the camera image (step S18: NO in Figure 4), the evaluation point is set to zero even if the target person can be detected in the depth image. In this case, if the period during which the target person is not captured in the camera image exceeds a certain period (see step S24 in Figure 4 as appropriate), the program execution is paused and the behavioral evaluation results are output. Then, when the target person is recognized in the camera image again, the behavioral evaluation is resumed.

[0095] Figure 6 illustrates a table that associates the type of facial expression of a subject (target person) with their facial expression evaluation score. This table can be used in the facial expression evaluation process described in step S19 above. In the table shown in Figure 6, the type of facial expression (Category) is set in the upper row, and the corresponding facial expression evaluation score (Evaluation Points) is set in the lower row. In this example, the facial expression evaluation score is an average value (50 points in this example) when the type of facial expression is "calm," and the facial expression evaluation score is high for "smile" and "laugh" in that order (80 points and 100 points in this example), and low for "anger" and "crying" in that order (20 points and 0 points in this example).

[0096] Figure 7 illustrates a table that associates the distance between the wearer and the target person (target person) with distance evaluation points. This table can be used in the distance evaluation process in step S22 described above. In the table shown in Figure 7, the distance to the target person is set in the upper row, and the corresponding distance evaluation points are set in the lower row. In this example, the highest evaluation score (100 points) is given when the distance to the target person is close to "less than 0.5m", and the evaluation points decrease to "80 points", "50 points", "20 points", and "0 points" in the order of "less than 1m", "less than 3m", "less than 10m", and "10m or more".

[0097] Figure 8 shows an example of how the behavioral evaluation results for a wearable device are displayed. The display screen 30 shown in Figure 8 is displayed on the image display unit 13 of the wearable device 1, for example, during the processing of step S27 described in Figure 4. For ease of understanding, an example of a calculation formula for calculating the behavioral evaluation results is shown above the display screen 30 in Figure 8.

[0098] In the illustrated example, the behavioral evaluation score is obtained by accumulating the facial expression evaluation score (i) and distance evaluation score (i) recorded at each timing, calculating the statistical value of each of these accumulated values, and normalizing each calculated statistical value by the behavioral evaluation time (see the period NT shown in Figure 5). This calculation (operation) is performed by the main processor 17 in step 26 of Figure 4 above.

[0099] Here, the "statistical value" calculated by the main processor 17 may be an average value such as an additive average or a weighted average. In the case of an additive average, the main processor 17 calculates the respective additive average values ​​by adding up the accumulated facial expression evaluation points and distance evaluation points (see the formula shown in Figure 8). Alternatively, in the case of a weighted average, the main processor 17 calculates the weighted average value by applying predetermined coefficients (weight values) to the elements that make up the accumulated facial expression evaluation points (or distance evaluation points), for example, each evaluation point (Evaluation Points) mentioned above in Figure 6 (for example, giving a higher value to "laughing", giving a higher value to a distance of "less than 0.5m", etc.).

[0100] The main processor 17 then calculates the behavior evaluation score by normalizing the average value (or weighted average value) calculated as described above by dividing it by the program execution time N (which is approximately equal in value to the behavior evaluation time). Thus, the calculated behavior evaluation score is displayed at the position of "XXX" on the display screen 30.

[0101] Furthermore, the display screen 30 shown in Figure 8 represents the simplified display format. The "Time" in the display screen 30, in this example, the time period from 9:12 AM to 10:45 AM on January 26, 2021, is the time period during which the program described above in Figure 4 was executed, and is equivalent to the time period during which the wearer's behavior was evaluated.

[0102] More specifically, the display screen 30 shown in Figure 8 displays, in addition to the "time" mentioned above, a normalized behavioral evaluation score, an overall comment on the behavioral evaluation result (in this example, "Good"), and an additional comment, "Evaluation can be continued." The additional comment shown in the figure is a message indicating that behavioral evaluation can be continued if the assigned person is recognized again.

[0103] Furthermore, as an alternative display format for the display screen 30 shown in the image display unit 13 of the wearable device 1, it may be possible to select a graphic display format that graphically displays the time progression of the evaluation, such as the one shown in Figure 5.

[0104] Figure 9 is a flowchart showing the process for obtaining cumulative behavioral evaluation results. Here, cumulative behavioral evaluation results can be defined as the cumulative value of behavioral evaluation scores over a specific period, and can be obtained by executing the behavioral evaluation program 92.

[0105] The behavioral evaluation score shown in Figure 8 is a score normalized by the duration of the behavior, and represents the evaluation result related to the so-called "quality of behavior." In contrast, the cumulative evaluation score obtained according to the flow in Figure 9 is an evaluation score obtained by accumulating the evaluation score for each evaluation time over a specific period without normalization, and corresponds to the so-called "quantity of behavior." Here, the "specific period" can be arbitrarily set in units such as "one day," "one week," "one month," or "one year."

[0106] In step S101, after the process of acquiring cumulative evaluation points has started, the main processor 17 sets the aforementioned "specific period" in response to, for example, the user's (wearer's) operation (setting instruction).

[0107] In the following step S102, the main processor 17 sequentially reads the evaluation data for the set period. Then, in step S103, the main processor 17 accumulates (sequentially adds up) the evaluation scores for the set period, and after accumulating all the evaluation scores for the period, proceeds to step S104. In step S104, the main processor 17 outputs the finally obtained accumulated value as the accumulated evaluation result.

[0108] In the following step S105, the main processor 17 determines whether or not to terminate the process of acquiring the cumulative evaluation score.

[0109] If the main processor 17 determines that it has not yet finished the process of acquiring the cumulative evaluation score (step S105: NO), it returns to step S101 described above and repeats the process described above. At this time, the user can recalculate and output different cumulative evaluation scores by, for example, setting a different specific period.

[0110] On the other hand, if the main processor 17 determines that it has finished the process of acquiring the cumulative evaluation score (step S105: YES), it terminates the series of processes described above.

[0111] Figures 10A and 10B show examples of how cumulative behavioral evaluation results are displayed. Figure 10A shows an example of how cumulative behavioral evaluation results are displayed when the evaluation period is completed. On the other hand, Figure 10B shows an example of how cumulative behavioral evaluation results are displayed when the evaluation period is not yet completed (is in progress). Specifically, in a case where the evaluation period is set from January 31st (Sun) to February 6th (Sat), 2021, Figure 10B shows the display date as February 5th (Fri) within that period.

[0112] As shown in Figures 10A and 10B, a pre-set evaluation period (one week in this example) and the cumulative evaluation score within that period are shown. In this example, the evaluation period is set to one week, and the cumulative evaluation score on a daily basis, as well as the cumulative evaluation score on a weekly basis, are shown. By looking at these evaluation scores, the wearer can self-manage their weekly behavioral patterns, for example, by making sure to have more contact on weekends if they had less contact on weekdays.

[0113] Furthermore, once the period is complete, as shown in the lower part of Figure 10A, the cumulative evaluation score calculated for this week is displayed along with its ratio to the cumulative evaluation score calculated last week. On the other hand, in the case of Figure 10B, where the period is not yet complete, as shown in the lower part of the same figure, the remaining points to reach the weekly target score (in this example, the evaluation score to be obtained on the final day, February 6th (Saturday) (1888 points) is displayed.

[0114] By displaying information as described above, it is possible to increase the wearer's motivation or help them set new goals and action plans for the following week.

[0115] As described above, the wearable device 1 of Embodiment 1 provides a wearable device that has an action evaluation function that evaluates the direct actions between the wearer and the target person and provides awareness of those actions. Furthermore, it does not require the target person to wear anything, and therefore has the advantage of allowing for free selection of the target person (a high degree of freedom in selecting the target person).

[0116] While an integrated HMD (head-mounted display) is used as an example of the specific configuration of wearable device 1, it is not limited to this. Other configurations of wearable device 1 include, for example, a configuration in which the controller 21 (subject image processing device, distance image processing device, output processing device) is separated from the HMD (camera 10, rangefinder 11, image display unit 13, etc.). Alternatively, as another configuration example, the camera 10, rangefinder 11, etc., may be placed in a neck-type or watch-type wearable device.

[0117] [Embodiment 2] Next, Embodiment 2 will be described with reference to Figures 11 to 13.

[0118] Figure 11 is a diagram illustrating the relationship between the camera's field of view and the rangefinder's measurement angle in Embodiment 2. As shown in the comparison in Figure 11, in Embodiment 2, the measurement angle 40 of the rangefinder 11 is set wider than the camera's field of view (camera field of view 41). Methods for widening the measurement angle 40 of the rangefinder 11 include widening the scanning range of the laser beam output from the light source mentioned above, or using a combination of multiple rangefinder sensors (such as optical radar), and any of these may be used.

[0119] Figure 12 illustrates the process of tracking a subject from a distance image. In Figure 12, the distance image 51 generated by the distance measuring device 11 is shown in the outer frame, and the area 50 of the camera image generated by the camera 10 is shown in the inner frame of the distance image 51.

[0120] Referring to Figure 12, it can be seen that the subject, initially positioned near the center of the camera image 50 (at the position indicated by reference numeral 52a), sequentially moves to the positions indicated by reference numerals 52b, 52c, and 52d in chronological order. Here, the position indicated by reference numeral 52b is at the right edge of the camera image 50, the positions indicated by reference numerals 52c and 52d are outside the camera image 50 but on the right side of the distance image 51, and a non-subject person 53 is captured at the position outside the camera image 50 but on the left side of the distance image 51. Since the region of the camera image 50 is also a region within the distance image 51, it is the region where the camera image and the distance image overlap, and will hereafter be referred to as the "composite region 50".

[0121] In this embodiment, the controller 21 (primarily the main processor 17, hereinafter the same) of the HMD identifies the target person (and thus the designated person) in the composite region 50 by analyzing the camera image. For example, even if the designated person moves and changes direction after being recognized, making face recognition impossible, the designated person can still be recognized by person detection. More specifically, a person at the positions of symbols 52a and 52b within the composite region 50 can be easily identified as a designated person in the distance image corresponding to the designated person recognized in the camera image. Therefore, as a function of the distance image processing device, the controller 21 extracts features of a person (not limited to the target person or designated person), such as the person's size, body balance, and motion vector, as feature quantities within the distance image.

[0122] The example shown in Figure 12 illustrates the case where the designated person (the same subject) moves sequentially from position 52a to positions 52b, 52c, and 52d. Here, the people indicated by 52c and 52d are located outside the range of the camera image. On the other hand, the controller 21 extracts the person's features, size, body balance, motion vector, etc., within the depth image as features, and by comparing the features of the person at position 52c with the features of the people at positions 52a and 52b, it recognizes that the designated person (the same subject) has moved. Similarly, the controller 21 recognizes that the person at position 52d is the designated person (the same subject) who moved from position 52c. On the other hand, the controller 21 can determine that the person indicated by 53 is not the same subject (not a subject) because the amount of movement is too large compared to the amount of movement estimated from the motion vector of the person at positions 52a, etc., and because of differences in body balance, etc.

[0123] Figure 13 is a flowchart illustrating the processing performed by the HMD controller in Embodiment 2. The flowchart in Figure 13 corresponds to the flowchart described above in Figure 4, and similarly, the main processor 17, which reads and executes the behavior evaluation program 92, is the main processor. In Figure 13, steps that perform the same processing as Embodiment 1 described above in Figure 4 are numbered the same. The difference in Embodiment 2 from the processing flow in Figure 4 is the processing of steps S30 and S31, which are added to the distance image processing process. Also, the branch destination for the determination of NO in S18 in the camera image processing process and the processing of step S21 in the distance image processing process are slightly different from Embodiment 1 (a determination process is added), so they are denoted as S21A.

[0124] In Embodiment 2, if the determination result of step S18 in the camera image processing process is NO (i.e., if it is determined that the designated person is not visible in the camera image), the main processor 17 determines that facial expression evaluation (step S19) cannot be performed and proceeds to step S21A.

[0125] In step S21A, after obtaining a result indicating that the person to be set (the designated target person) is not visible in the camera image, the main processor 17 determines whether or not the person to be set is within the composite region 50 described above.

[0126] Here, if the main processor 17 determines that the person to be configured is within the composite area 50 (step S21A: YES), it performs the same processing as in the flow of Figure 4, after the distance evaluation in step S22, and then the processing from step S25 onwards.

[0127] On the other hand, if the main processor 17 determines that the person to be targeted is not within the composite region 50 (step S21A: NO), it detects a person in the depth image and performs a process to track the person to be targeted, as described above in Figure 12 (step S30). By performing this process in step S30, it is possible to confirm the presence or absence of the person to be targeted in a wider area outside the camera image, that is, an area that cannot be captured by the camera image.

[0128] In the following step S31, the main processor 17 determines whether or not the person to be set exists within the depth image. If the main processor 17 determines that the person to be set exists within the depth image (step S31: YES), it performs the same processing as in the flow shown in Figure 4, going through the distance evaluation in step S22 and then processing from step S25 onwards.

[0129] On the other hand, if the main processor 17 determines that no person to be assigned exists within the distance image (step S31: NO), it executes the same process as in the flow shown in Figure 4, from step S23 onwards.

[0130] Specifically, in step S23, the main processor 17 starts counting the time during which the configured person cannot be confirmed (in this case, the time during which the determination of step S21A: NO is repeated). In the following step S24, the main processor 17 determines whether the counted time exceeds the threshold TH.

[0131] Here, if the main processor 17 determines that the counted time has not yet exceeded the threshold TH (step S24: NO), it returns to step S15 and repeats the process described above. On the other hand, if the main processor 17 determines that the counted time has exceeded the threshold TH (step S24: YES), it determines that the user's (wearer's) actions toward the configured person have come to an end and proceeds to step S26.

[0132] Note that the processing in steps S27 and S28 is the same as the flow shown in Figure 4.

[0133] Thus, according to Embodiment 2, the same effects as the wearable device 1 in Embodiment 1 can be obtained, as well as the following unique effects. Specifically, the wearable device 1 of Embodiment 2 can utilize the distance measuring device 11, which has a characteristic wide measurement angle, to perform behavioral evaluation with high tracking ability to the movements of the target person (the person being monitored).

[0134] [Embodiment 3] Next, Embodiment 3 of this disclosure will be described with reference to Figures 14 to 16.

[0135] Figure 14 is a diagram illustrating the configuration of Embodiment 3, showing the system configuration of a behavioral evaluation system including a wearable device. In Figure 14, in addition to the wearer 3 and subject 4 described in Embodiment 1 above in Figure 2, a manager 64 is further added as a relationship between the wearer 3 and the subject 4.

[0136] In one example, wearer 3 is a babysitter, and administrator 64 is the babysitter's employer (i.e., a person related to wearer 3). In another example, administrator 64 may be a person related to subject 4 (for example, a parent or guardian). In either case, in the behavior evaluation system of Embodiment 3, administrator 64 possesses a portable information device 65 and is configured to use the portable information device 65 to check the evaluation results of wearer 3's (babysitter's) behavior towards subject 4 online.

[0137] Furthermore, when administrator 64 checks the results of such behavioral evaluations online, they may use not only the portable information device 65 exemplified in Figure 14, but also other information terminals such as a stationary PC. On the other hand, using the portable information device 65 has the advantage of allowing the results of behavioral evaluations to be checked regardless of location, such as when out and about.

[0138] In Figure 14, blocks having the same function as in Figure 2 are assigned the same number. Further explaining the differences between Figure 14 and Figure 2, the behavioral evaluation system in Embodiment 3 uses a server device (behavioral evaluation service server 63) that can communicate with the wearable device 1 and the portable information device 65.

[0139] The behavioral evaluation service server 63 is located within a network 62 such as a LAN or the Internet, and is capable of wireless communication with the wearable device 1 and the personal information device 65 via an access point 61. Figure 14 shows how the communication unit 16 of the wearable device 1 (see Figure 1) communicates with the behavioral evaluation service server 63 via the access point 61 using wireless communication signals 60a and 60b.

[0140] In one specific example of such an behavioral evaluation system (hereinafter referred to as "this system"), the wearable device 1 sends data such as facial expression evaluation points generated in step S19 and distance evaluation points generated in step S22 to the behavioral evaluation service server 63. Upon receiving this data, the behavioral evaluation service server 63 performs the processing described above from step S25 (recording and saving behavioral evaluation results) onwards, either on behalf of the user or in parallel. Furthermore, the cumulative value of the behavioral evaluation results is saved in the memory medium (such as an HDD) of the behavioral evaluation service server 63, and the saved behavioral evaluation results can be monitored by the administrator 64 using a portable information device 65.

[0141] The behavioral evaluation results stored in the behavioral evaluation service server 63 (hereinafter abbreviated as "server" as appropriate) and monitored by the portable information device 65, as well as the flow of using this system, will be explained below with reference to Figures 15 and 16. Here, Figure 15 is a diagram illustrating the data structure of the behavioral evaluation results. Figure 16 is a sequence diagram for monitoring the behavioral evaluation results.

[0142] In this system, users of the service are assigned a Service ID when they log in to the server on network 62 by entering a pre-registered Username. The Service ID shown in Figure 15 is long, "550e8400-e29b...", and in this case, the service user (administrator 64 in this example) inputs the assigned Service ID ("550e8400-e29b...") using operations such as drag and drop, and a data display screen 70 as shown in Figure 15 is displayed on the display unit of the portable information device 65.

[0143] In the example shown in Figure 15, multiple datasets (two datasets, dataset 71 and dataset 72) containing behavioral evaluations of wearers of wearable device 1 are displayed. The data structure of datasets 71 and 72 will be described below.

[0144] At the beginning of the data structure, the behavioral evaluation score (Score) is displayed. Here, for dataset 71, the Score "71" is displayed as the value of the behavioral evaluation result for which the calculation process has already been completed (in other words, the normalization in step S26 described above has been performed).

[0145] Furthermore, the start time and end time of the behavioral evaluation are displayed in the column below the behavioral evaluation score. In dataset 71, the Start Time is displayed as 9:15:37 AM on January 26, 2021, and the Stop Time is displayed as 12:00:12 PM on the same day of the same year. Therefore, the service user (administrator 64) can find out what time period the wearer's behavior was evaluated.

[0146] Furthermore, in the column below Stop Time, pairs of facial expression evaluation points and distance evaluation points are recorded in time series at regular intervals (see T in Figure 5). All of this data together constitutes dataset 71. In one example, this dataset 71 corresponds to the period NT shown in Figure 5.

[0147] In the example shown in Figure 15, dataset 72 follows dataset 71 as described above. Since the data structure of dataset 72 is the same as that of dataset 71, the differences will be explained below.

[0148] Dataset 72 does not display a score because the behavioral evaluation results have not yet been finalized. Specifically, Dataset 72 contains data on the behavioral evaluation results that were evaluated after the time period of Dataset 71. The Start Time is displayed as January 26, 2021, 13:22:46, but the Stop Time is displayed as "Running," meaning it is in progress and the evaluation is not yet complete.

[0149] Therefore, the service user (administrator 64) can view the wearer's morning behavior evaluation and results by looking at the dataset 71 on the data display screen 70 displayed on the display unit of the mobile information device 65. In addition, the administrator 64 can monitor the wearer's afternoon behavior evaluation in real time by looking at the dataset 72 that is currently being displayed (updated) on the data display screen 70.

[0150] In one specific example, after connecting to the behavior evaluation service server 63, the portable information device 65 obtains a list of start times for the specified day from the behavior evaluation service server 63 and displays it on the display unit, based on a predetermined operation by the service user (administrator 64) (for example, an operation to specify a date). In this case, the portable information device 65 can display a data display screen 70 as shown in Figure 15 by specifying one of the start times in the list. Note that, as described above, when specifying one of the start times in the list, the configuration may be such that only the dataset corresponding to that start time is obtained and displayed (for example, only one of datasets 71 or 72 in Figure 15).

[0151] From another perspective, the behavior evaluation service server 63 transmits data from the data display screen 70 (behavior evaluation points, and the facial expression evaluation points and distance evaluation points, which are elements of the behavior evaluation points) corresponding to the start time of the behavior specified by the portable information device 65 (external device) to the portable information device 65.

[0152] With the configuration described above, service users (administrator 64) can view facial expression evaluation points, distance evaluation points, and behavior evaluation points on their mobile information device 65, using the start time of the activity as a search index.

[0153] As another example, the start time of the activity may be used as a search index, and a data display screen 70 (either or both of the datasets 71 and 72 shown in Figure 15) may be displayed on the image display unit 13 of the wearable device 1.

[0154] Next, referring to Figure 16, the processing flow when the administrator 64 monitors the wearer's behavior evaluation in real time using the personal information device 65 will be explained. In Figure 16, symbols T10 to T19 indicate the time from when the application (hereinafter abbreviated as "app") pre-installed on the personal information device 65 is launched until the service usage is terminated (logged out).

[0155] First, administrator 64 operates the personal information device 65 to launch the above application (time T10). Once the application is launched, personal information device 65 (the processing unit is the personal information device 65's processor, the same applies hereafter) sends the pre-configured Username and Password to the behavioral evaluation service server 63 at time T11. Then, at the following time T12, it receives the ServiceID (see Figure 15 as appropriate) from the behavioral evaluation service server 63 and also receives a configuration input screen (not shown).

[0156] This settings input screen is displayed on the display unit of the mobile information device 65 at time T13. At the following time T14, the administrator enters the setting parameters ("Setting P" in the figure). The entered setting parameters are sent to the behavioral evaluation service server 63 at time T15. In one specific example, the setting parameter is the Start Time of the dataset to be monitored, which can be entered directly or selected from a pull-down menu. The setting parameters may also include a parameter for selecting the display format of the monitoring. Examples of display format selections include, for example, a tabular display like the dataset 72 in Figure 15, or a graphical display like the one shown in Figure 5.

[0157] Thus, the behavior evaluation service server 63, having received the above-mentioned configuration parameters, sends out data for the behavior evaluation result screen corresponding to those configuration parameters (time T16). The personal information device 65, having received this data for the behavior evaluation result screen, displays the behavior evaluation result screen on its own display at time T17 (see Figure 15 as appropriate). When the administrator 64 finishes monitoring this behavior evaluation result screen, they operate the personal information device 65 to log out in order to terminate the application (time T18). When the behavior evaluation service server 63 receives this logout signal (time T19), it terminates its connection to the service and the personal information device 65.

[0158] As described above, Embodiment 3 has the advantage of not only providing the effects obtained in Embodiments 1 and 2 described above, but also allowing the administrator 64 to check the wearer's behavior evaluation results online and even monitor them in real time.

[0159] [Embodiment 4] Embodiment 4 of this disclosure will be described with reference to Figures 17 to 20.

[0160] Figure 17 illustrates a scenario involving multiple subjects. Figure 17 shows the relationship between wearer 3 and subject 4, and assumes a case where there are multiple subjects 4 (in this example, four people indicated by symbols 4a, 4b, 4c, and 4d).

[0161] One concrete example is a scenario in a daycare center where a caregiver is responsible for the care of multiple children, and the caregiver's behavior towards the children is being evaluated. In this case, the caregiver becomes the wearer of the wearable device 1 (HMD) 3, and each child 4 (4a-4d) becomes the subject, and the subjects do not need to wear anything.

[0162] Other specific examples include applications between caregivers (wearer 3) and those receiving care in nursing facilities, teachers (wearer 3) and students in educational settings, and shop staff (wearer 3) and customers (those being served) in stores.

[0163] Figure 18 is a flowchart showing an example of the processing performed by the wearable device 1 in Embodiment 4. In Figure 18, the same numbers are assigned to steps that perform the same processing as in the processing flow of Embodiment 1 shown in Figure 4.

[0164] The difference between Figure 18 and Figure 4 is that Figure 18 uses the face image database DB1. In other words, in Embodiment 4, the face data of all subjects is stored in the face image database DB1 beforehand.

[0165] Then, in the face recognition step (step S12A) after acquiring the camera image in step S11, the main processor 17 performs face recognition processing on the faces captured in the acquired camera image based on a comparison with face data previously registered in the face image database DB1. The processing in the subsequent step S13 is the same as in Figure 4.

[0166] Similarly, in the face recognition / person detection step (step S17) after acquiring the camera image in step S16, the main processor 17 performs face recognition processing on the faces captured in the acquired camera image based on comparison with face data previously registered in the face image database DB1. The processing from step S18 onwards is the same as in Figure 4.

[0167] More specifically, in the face recognition process in step S12A and the face recognition / person detection process in step S17, the main processor 17 refers to face image data for each person registered in the face image database DB1. Then, in the expression evaluation process in step S19, the distance evaluation process in step S22, the evaluation recording and accumulation process in step S25, and the normalization of the evaluation results in step S26, the main processor 17 performs the process for each person.

[0168] Figure 19 illustrates the structure of a database of behavioral assessment results for multiple subjects. Similar to Figure 15 described earlier, the example shown in Figure 19 displays multiple datasets (two datasets, dataset 73 and dataset 74) that store behavioral assessments of wearers of wearable device 1. For ease of understanding, dataset 73 shows the same time period as dataset 71 shown in Figure 15, i.e., the start time and end time of the behavioral assessment.

[0169] Furthermore, as can be seen by comparing with Figure 15, in the dataset 73 shown in Figure 19, the behavioral evaluation results consist of sub-datasets for each subject A, B, C, and D, indicated by symbols 73a to 73d. In other words, according to the configuration of Embodiment 4 using the face image database DB1, it becomes easy to perform processes such as evaluating the wearer's behavior for multiple subjects simultaneously and in parallel for each subject (in this example, subjects A, B, C, and D).

[0170] Furthermore, Figure 20 shows an example of displaying the behavior evaluation results in Embodiment 4. These behavior evaluation results are output as a display screen 30 to the image display unit 13 of the wearable device 1 based on the control signals of the controller 21 (output processing unit). The controller 21 of the wearable device 1 may also transmit the data of this display screen 30 to the behavior evaluation service server 63 via the network 62 described above. In this case, the display screen 30 can be displayed on the display unit of a portable information device 65 connected to the behavior evaluation service server 63.

[0171] In the example of display screen 30 shown in Figure 20, the evaluation scores are displayed for each subject (A, B, C, D) (66, 48, 35, 75 points), and the comparison results of the evaluation scores for each subject are displayed in a graph. In this example, the difference for each subject (A, B, C, D) from the average score (Ave.) of 65 points is also displayed as a percentage. Furthermore, in this example, the graph color for subject C, who had the lowest evaluation, is displayed in a different color from the graphs of the other subjects A, B, and D.

[0172] By displaying the screen 30 as described above, it is possible to visually and clearly show, for example, that the behavioral evaluation of subject C is relatively inferior compared to other subjects (A, B, D). Furthermore, in this example, the overall comment includes the advice (message output) "Let's do our best to care for subject C." This advice (message output) may also be output as voice from the voice input / output unit 14 as an alternative or additional measure.

[0173] Another example of the display screen 30 is that, for example, the controller 21 may control the display of evaluation scores for a specific person (e.g., person C) over multiple behavioral evaluation times, based on the user's operation of the operation input unit 15. Alternatively, for example, the controller 21 may control the display of behavioral evaluation screen 30 corresponding to the start time of an behavioral time, using the start time of the behavioral time as a search index, based on the user's operation of the operation input unit 15.

[0174] Thus, the configuration of Embodiment 4, which displays evaluation scores (behavioral evaluation scores of wearer 3) for each subject (A, B, C, D), has the advantage of allowing for objective behavioral evaluation of each subject and easy provision of feedback on behavior. Furthermore, as shown in Figure 20, the configuration that displays the behavioral evaluation scores for each subject (A, B, C, D) in a graph for easy comparison allows for easy understanding of the evaluation results for each subject, even if the display area of ​​the display screen 30 is small. In addition, by outputting a comprehensive comment on the behavioral evaluation, wearer 3 can quickly grasp areas for improvement and other points to reflect on.

[0175] It should be noted that the present invention is not limited to the specific embodiments described in Figures 1 to 20. For example, it is possible to replace a part of the configuration of one embodiment with that of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention, and the numbers, messages, etc. that appear in the text and figures are merely examples, and using different ones will not impair the effects of the present invention.

[0176] For example, some of the functions performed by the controller 21 of the wearable device 1 (e.g., functions as a subject image processing device, a distance image processing device, and an output processing device) may be assigned to the behavior evaluation service server 63 described above.

[0177] Furthermore, some or all of the functions of the apparatus described in this specification and the drawings may be implemented in hardware, for example, by designing them as integrated circuits. Alternatively, they may be implemented in software by having a microprocessor unit, CPU, etc., interpret and execute an operating program. Moreover, the scope of software implementation is not limited, and hardware and software may be used in combination. [Explanation of symbols]

[0178] 1: Wearable device, 2: Attachment, 3: Wearer, 4, 4a~4d: Target person (person being set), 10: Camera (imaging device), 11: Distance measuring device, 13: Image display unit, 13c: Main screen display unit, 14: Audio input / output unit, 15: Operation input unit, 16: Communication unit, 17: Main processor, 19: Flash ROM, 21: Controller (subject image processing unit, distance image processing unit, output processing unit), 92: Behavioral evaluation program, 93: Data unit (data storage unit), 30: Display screen, 51: Distance image, 63: Behavioral evaluation service server, 64: Administrator, 65: Portable information device, DB1: Face image database

Claims

1. An imaging device that captures images of subjects in front of or around the wearer and generates subject images, A subject image processing device that processes the subject image to detect and recognize the subject person contained within the subject image, and to assign an expression evaluation score to the wearer based on the facial expression of the subject person obtained by the facial recognition, An output processing device that performs an evaluation process to obtain an action evaluation score for evaluating the wearer's actions from the aforementioned facial expression evaluation score and the action evaluation time, which is the time the wearer was acting, and outputs the action evaluation score. Equipped with, The output processing device starts executing the evaluation process from a first time point in time when the subject is set as the target, and defines the period from the first time point to the second time point in time, from the first time point to the second time point in time when the target is no longer detected for a threshold time from the generated subject image, as the behavior evaluation time, and outputs the behavior evaluation score based on the cumulative value of the facial expression evaluation score during the behavior evaluation time, as a wearable device.

2. In the wearable device described in claim 1, The output processing device is The evaluation process is performed such that the facial expression evaluation points are accumulated at regular intervals, and the cumulative value obtained by accumulating these accumulated values ​​during the behavior evaluation time is used as the behavior evaluation score. Wearable devices.

3. In the wearable device described in claim 1, Equipped with a data storage unit, The output processing device records the facial expression evaluation points in the data storage unit, along with the first time point which is the start time of the behavior evaluation time and the second time point which is the end time of the behavior evaluation time. Wearable devices.

4. In the wearable device described in claim 1, Equipped with a communication unit that connects to a network, The output processing device outputs the facial expression evaluation points, along with the first time point which is the start time of the behavior evaluation time and the second time point which is the end time, to the server device via the communication unit. Wearable devices.

5. In the wearable device described in claim 1, The subject image processing device processes the subject image in such a way that it detects and recognizes the faces of multiple subjects contained within the subject image and assigns the facial expression evaluation points to each subject. The output processing device performs an evaluation process for each of the target persons to obtain an evaluation score for evaluating the wearer's behavior. Wearable devices.

6. In the wearable device described in claim 5, The output processing device outputs the behavioral evaluation score for each subject. Wearable devices.

7. A wearable device according to claim 1, A server device which acquires the behavioral evaluation score output from the output processing device and provides the acquired behavioral evaluation score to an external device owned by a person related to the subject, Behavioral evaluation system.

8. In the behavioral evaluation system described in claim 7, The server device transmits to the external device the action evaluation point corresponding to the start time of the action time specified by the external device, and the facial expression evaluation point which is an element of the action evaluation point. Behavioral evaluation system.

Images