Device, Program, and Display Method for Controlling User's Field of Vision According to Biological Activity Amount

Abstract

To provide a device for controlling reflection of objects in a field of view of a user to reduce psychological stress of the user, a program and a display method.SOLUTION: A device having a camera that captures video visible from a field of view of a user, and a lens display that projects an augmented reality image into the field of view of the user, has: object recognition means of detecting an image region of a predetermined object by object recognition from the video taken by the camera; biological activity amount acquisition means of acquiring a biological activity amount of the user; determination means of determining whether a biological activity amount satisfies a predetermined condition; and augmented reality image generation means of generating an augmented reality image obtained by processing the image region of the predetermined object in the video seen in the field of view of the user when the determination means determines that it is true. The biological activity amount acquisition means acquires the biological activity amount from a wearable device attached to the user's body or a sensor adhering to the user's body.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technology of smart glasses based on Augmented Reality.

Background Art

[0002] Smart glasses are worn like glasses in front of the user's field of view and are equipped with a camera and a retinal projection type lens display. In the user's field of view, an unreal space of Virtual Reality is superimposed on the real space.

[0003] Conventionally, for a presentation in front of an audience, there is a technology that allows a user to practice in which direction to turn next (see, for example, Patent Document 1). According to this technology, the orientation of the user's head is detected from a device worn on the user's head, and a visual field image in the corresponding direction is displayed. Then, a position on the visual field image different from the position on the visual field image indicating the orientation of the head is displayed on the lens display. The user can recognize that they should turn in that direction.

[0004] In addition, there is also a technology of smart glasses that make the expressions of people in front of the user's field of view look like smiles (see, for example, Non-Patent Document 1). According to this technology, the expressions of people around through the smart glasses are image-corrected to look like smiles. The correction to a smile is to raise the corners of the mouth of the person's face. A smile has the effect of evoking positive emotions effective for stress relief, and it is said that the expressions of neighbors and friends are contagious. By creating a situation where the expressions of people around look like smiles, smiles and positive emotions can be induced in the user wearing the device.

[0005] Recently, the development of digital devices clinically proven to prevent, manage, and treat specific medical conditions, known as "Digital Therapeutics," has attracted attention (see, for example, Non-Patent Document 7). For example, for users with psychological disorders, they may feel fear or anxiety when faced with people who visually catch their eye. Improving mental health based on such interpersonal relationships has become an important issue. [Prior Art Documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-180503 [Non-Patent Documents]

[0007] [Non-Patent Document 1] Momotake Murata, Yusuke Yamamoto, "Smile Glasses: AR Glasses That Promote Smile Formation," Faculty of Information Science, Shizuoka University, Forum on Data Engineering and Information Management, DEIM2020 A1-4 (day1 p2), [online], [searched on August 28, 2022], Internet <URL:https: / / proceedings-of-deim.github.io / DEIM2020 / papers / A1-4.pdf> [Non-Patent Document 2] KDDI R & D Laboratories Inc., "Developing Facial Expression Recognition AI Technology When Wearing a Mask," [online], [searched on August 28, 2022], Internet <URL:https: / / www.kddi-research.jp / newsrelease / 2021 / 022401.html> [Non-Patent Document 3] KDDI Corporation, KDDI R & D Laboratories Inc., Acordia, Inc., "Cutting-edge Technology Anticipating the 5G Era: Sports Behavior Recognition AI × IoT", [online], [searched on August 28, 2022], Internet <URL:https: / / news.kddi.com / kddi / corporate / newsrelease / 2019 / 10 / 07 / 4060.html>

Non-Patent Document 4

Non-Patent Document 5

Non-Patent Document 6

Non-Patent Document 7

Summary of the Invention

Problems to be Solved by the Invention

[0008] When a user gives a presentation in front of an audience, they are often strongly focused by the audience and forced to feel nervous. At this time, the user may not be able to give a smooth presentation. For some users, the more the audience, the greater the psychological stress. Also, by overpaying attention to the individual facial expressions of the audience, they may easily fall into a negative psychological state. In these cases, the user tries to calm down and proceed with the presentation by deliberately looking away from the audience.

[0009] According to Patent Document 1, it does not affect the psychological situation of the user during a presentation. On the other hand, according to Non-Patent Document 1, by forcibly showing the smiling facial expressions of the audience to the user wearing the device, a positive psychological state may be created.

[0010] However, depending on the situation reflected in the field of view, the degree of stress and the resulting negative psychological state vary greatly among users. In contrast, the inventors of the present application considered whether it is possible to reduce the psychological stress of the user by controlling the reflection of objects entering the field of view. For example, as digital therapeutics, they considered whether it would be useful to improve the mental health based on the interpersonal relationship of the user by controlling the user's field of view.

[0011] Therefore, an object of the present invention is to provide a device, a program, and a display method that can reduce the psychological stress of the user by controlling the reflection of objects entering the field of view.

Means for Solving the Problem

[0012] According to the present invention, in a device having a camera that captures an image visible from the user's field of view and a lens display that projects an augmented reality image onto the user's field of view, From the video captured by the camera, by object recognition Person object recognition means for detecting the image area of biological activity amount acquisition means for acquiring the biological activity amount of the user, biological activity amount Set the reduction rate of the number of people in the video reflected in the user's field of view according to the stress level estimated from the person determination means for in the video reflected in the user's field of view Hide the image area of the person so that the number of people decreases at the rate augmented reality image generation means for generating a processed augmented reality image, and It is characterized by having.

[0013] According to another embodiment of the device of the present invention, the device is in the form of glasses worn in front of the eyes of the user's head, The augmented reality image generation means generates an augmented reality video that conceals the image area of a predetermined object from the video reflected in the user's field of view This is also preferable.

[0014] According to another embodiment of the device of the present invention, the biological activity amount acquisition means acquires the biological activity amount from a wearable device worn on the user's body or a sensor attached to the user's body This is also preferable.

[0015] According to another embodiment of the device of the present invention, the biological activity amount is any one or a combination of heart rate, blood oxygen concentration, skin temperature, blood pressure, pulse, sweating amount, atmospheric pressure, environmental sound be This This is also preferable.

[0023] According to the present invention, in a program that causes a computer mounted on a device having a camera that captures a video visible from the user's field of view and a lens display that projects an augmented reality image onto the user's field of view, From the video captured by the camera, by object recognition Person object recognition means for detecting the image area of biological activity amount acquisition means for acquiring the biological activity amount of the user, Biological activity amount Set the reduction rate of the number of people in the video reflected in the user's field of view according to the stress level estimated from the person Determination means for performing, In the video reflected in the user's field of view Hide the image area of the person so that the number of people decreases at the rate Augmented reality image generation means for generating a processed augmented reality image, and Causing a computer to function.

[0024] According to the present invention, in a display method of a device having a camera for photographing a video visible from the user's field of view and a lens display for projecting an augmented reality image into the user's field of view, The device is, From the video captured by the camera, by object recognition Person A first step of detecting an image area of, A second step of acquiring the biological activity amount of the user, Biological activity amount Set the reduction rate of the number of people in the video reflected in the user's field of view according to the stress level estimated from the person A third step of performing, In the video reflected in the user's field of view Hide the image area of the person so that the number of people decreases at the rate A fourth step of generating a processed augmented reality image, and A display method of a device, characterized by executing.

Effects of the Invention

[0025] According to the device, program, and display method of the present invention, by controlling the reflection of an object entering the field of view according to the biological activity amount of the user, the psychological stress of the user can be reduced.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Best Mode for Carrying Out the Invention

[0027] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0028] FIG. 1 is a system configuration diagram in which a smart glass and a smart watch are connected.

[0029] <Smart Glass 1> Smart Glass 1 is a see-through type of glasses-like device worn in front of the user's eyes on the head. According to FIG. 1, Smart Glass 1 includes, as hardware, a lens display 101, a camera 102, and a wireless communication unit 103. The lens display 101 projects an augmented reality image into the user's field of view. The camera 102 captures an image of the video visible from the user's field of view. For example, in the case of a presentation, the audience in front of the user will be reflected in the video. The wireless communication unit 103 is, for example, Bluetooth (registered trademark), and wirelessly communicates with other devices such as a smart watch or a wearable device.

[0030] Here, as an embodiment of the present invention, the device is described as a transmissive type (see-through) such as smart glasses, but a non-transmissive type such as a head-mounted lens display may also be used. In the case of a transmissive device, an augmented reality image obtained by performing interpolation processing on the pixels around the image captured by the camera 102 is projected into the user's field of view. In the case of a non-transmissive device, an image obtained by performing image processing on the image captured by the camera 102 is projected into the user's field of view. In any device, image interpolation or image processing is performed on the video according to the orientation of the user's head.

[0031] <Smart Watch 2> Smart Watch 2 is a wearable device worn on the user's wrist. The smartwatch 2 is equipped with one or more biosensors and can measure the "biological activity level" of the user. Examples of biosensors include a heart rate sensor, a blood oxygen concentration sensor, a skin temperature sensor, a blood pressure sensor, a barometric pressure sensor, an environmental microphone, and the like.

[0032] The "heart rate sensor" contacts the skin of the arm from the back of the smartwatch 2 and measures the "heart rate" of the user. For example, in the case of an optical type, light is emitted from an LED (Light Emitting Diode), the amount of scattered light reflected by the blood flow is sensed, and the heart rate is measured based on the dynamic changes in hemoglobin in the blood. The "blood oxygen concentration sensor" measures the "blood oxygen concentration (SpO2)" from the attenuation of light of arterial blood components irradiated on the skin. The "skin temperature sensor" contacts the skin on the back of the smartwatch 2 and detects the "skin temperature". The "blood pressure sensor" measures the "blood pressure" based on a combination of the heart rate and the blood flow volume. The "barometric pressure sensor" measures the ambient "barometric pressure". The "environmental microphone" detects the ambient "environmental sound". The biological activity level may be any one of these measured values (heart rate, blood oxygen concentration, skin temperature, blood pressure, pulse, sweating amount, barometric pressure, environmental sound) or a combination thereof.

[0033] According to FIG. 1, a smartphone 3 may be interposed between the smart glasses 1 and the smartwatch 2. The smartphone 3 is paired with the smartwatch 2 via Bluetooth (registered trademark). Then, the smartphone 3 constantly receives the biological activity level from the smartwatch 2 and processes it with a predetermined application. In addition, the smartphone 3 is also paired with the smart glasses 1 via Bluetooth and with the smartphone 1. Then, the smartphone 3 can control the smart glasses 1 from a predetermined application.

[0034] FIG. 2 is a functional configuration diagram of the smart glasses in the present invention.

[0035] According to FIG. 2, the smart glasses 1 include an object recognition unit 11, a biological activity amount acquisition unit 12, a determination unit 13, and an augmented reality image generation unit 14. These functional configuration units are realized by executing a program that causes a computer mounted on the smart glasses to function. Also, the processing flow of these functional configuration units can be understood as a display method of the smart glasses. Note that the software processing of the smart glasses 1 described in FIG. 2 may be executed by a predetermined application on the smartphone 1.

[0036] [Object Recognition Unit 11] The object recognition unit 11 detects an image area of a "predetermined object" by object recognition from the video captured by the camera 102. As processing functions of the object recognition unit 11, it has a region cutting function and a feature detection function.

[0037] [Region Cutting Function] The object recognition unit 11 cuts out an image area (for example, a bounding box) of a predetermined object from the image of the video frame captured by the camera 102. Specifically, R-CNN (Regions with Convolutional Neural Networks) or SSD (Single Shot Multibox Detector) is used. R-CNN combines a rectangular image area with the features of a convolutional neural network to detect a subset of a predetermined object (region proposal). Next, CNN feature amounts are extracted from the region proposal. Then, the bounding box of the region proposal is adjusted by a support vector machine pre-trained using the CNN feature amounts. SSD is an algorithm for general object detection using machine learning, and determines rectangular frames (bounding boxes) called default boxes. A large number of default boxes with different sizes can be superimposed on one image.

[0038] <Feature detection function> The object recognition unit 11 identifies a predetermined object from the image area of the bounding box. The "predetermined object" to be detected may be a person or an object not limited thereto. For example, when assuming a presentation, features of an object that causes the user to be psychologically stressed are set in advance. Generally, when the object that causes the user to be psychologically stressed is the audience, by setting the features of the person in advance, only the person can be detected from the video reflected in the user's field of view.

[0039] The object recognition unit 11 has a lightweight learning model so that it can be implemented on a small IoT device. By selecting intermediate layer features (deleting layers) of the learned model and replacing the lightweight architecture, the amount of calculation is significantly reduced. Specifically, even with a small device such as smart glasses using the C++ Native library, 10,000 people can be identified in 1 second.

[0040] [Biological activity amount acquisition unit 12] The biological activity amount acquisition unit 12 acquires the "biological activity amount" of the user. The biological activity amount acquisition unit 12 acquires the biological activity amount from a wearable device worn on the user's body or a sensor attached to the user's body. According to FIG. 1, the biological activity amount is received from the smartwatch 2. As described above, the biological activity amount may be, for example, any one of or a combination of a heart rate, a blood oxygen concentration, a skin temperature, a blood pressure, a pulse, a sweating amount, an air pressure, and ambient sound.

[0041] [Determination unit 13] The determination unit 13 determines whether the biological activity amount satisfies a predetermined condition. Specifically, the determination unit 13 estimates the "stress level" of the user from the biological activity amount, and determines true when the stress level exceeds a predetermined threshold.

[0042] Generally, the autonomic nervous system in the human body is a system that controls functions such as blood circulation, respiration, and body temperature regulation without the intervention of consciousness, and consists of the "sympathetic nervous system" and the "parasympathetic nervous system" (see, for example, Non-Patent Document 5). The "sympathetic nervous system" acts in a direction to increase the body's activity level and motor ability, while the "parasympathetic nervous system" acts in a direction to calm the mind and body and suppress energy consumption and storage. Heart rate variability (HRV) affects both the sympathetic and parasympathetic nerves. Here, the stress level is determined using the following algorithm. (S1) Measure the time between heartbeats (RRI) from the heart rate per unit time. (S2) Perform resampling by linear interpolation. (S3) Convert to a power spectrum by frequency analysis (Fourier transform). (S4) Divide into an LF (Low Frequency) component of 0.05 Hz to 0.15 Hz and an HF (High Frequency) component of 0.15 Hz to 0.40 Hz. The LF component increases when either the sympathetic or parasympathetic nerve is activated. The HF component increases only when the parasympathetic nerve is activated. (S5) If the measured ratio of the LF component to the HF component (LF / HF) is high, it can be said that there is stress, and if it is low, the user is relaxed. That is, a threshold ratio is set in advance, and if the measured ratio is equal to or higher than the threshold, it can be determined that the stress of the user is high.

[0043] [Augmented Reality Image Generation Unit 14] When the determination unit 13 determines it to be true, the augmented reality image generation unit 14 generates an augmented reality image obtained by processing the "image area of a predetermined object" in the video reflected in the user's field of view.

[0044] Here, as a process in the augmented reality image, there may be a case where the image area of a predetermined object is made to disappear. That is, the visual color in the image area of the predetermined object is made the same color as the surroundings to conceal it from the user's field of view (see, for example, Non-Patent Document 4). Most simply, the image area may be overlaid and filled with the color around the image area of the predetermined object.

[0045] Then, the augmented reality image generation unit 14 outputs the image reality image to the lens display 101. For example, in the case of see-through type smart glasses, the augmented reality image generated by the augmented reality image generation unit 14 is only the filled color range at the same visual position as the image area of the predetermined object. As a result, it appears to the user that the predetermined object has disappeared from the field of view.

[0046] According to the present invention, assuming that the predetermined object that disappears from the user's field of view is a "person", for example, there are the following embodiments. For the user, the stress on the people reflected in the field of view is alleviated, and it functions as digital therapeutics. <First Embodiment> Reduce the number of people reflected in the user's field of view. <Second Embodiment> Among the people reflected in the user's field of view, make the people with a preset facial expression disappear. <Third Embodiment> Among the people reflected in the user's field of view, make the people with a preset attribute disappear. <Fourth Embodiment> Among the people reflected in the user's field of view, make the people with a preset posture disappear.

[0047] <First Embodiment> Reduce the number of people reflected in the user's field of view.

[0048] FIG. 3 is a flowchart of the first embodiment in the smart glasses of the present invention. According to FIG. 3, it is assumed that the user is giving a presentation. Suppose a number of audiences (people) are visible in the user's field of vision. At this time, assume that the user is nervous due to the large number of audiences and has a personality that is prone to anxiety.

[0049] The object recognition unit 11 recognizes a number of people from the video reflected in the user's field of vision. Next, the biological activity amount acquisition unit 12 acquires the biological activity amount of the user who is giving a presentation. Next, the determination unit 13 determines whether the stress level is equal to or higher than a predetermined threshold value from the biological activity amount of the user. Then, when it is determined to be true by the determination unit 13, the augmented reality image generation unit 14 reduces the number of people reflected in the user's field of vision by a predetermined ratio (%). When it is determined to be false, the augmented reality image generation unit 14 is made not to function.

[0050] Here, the determination unit 13 and the augmented reality image generation unit 14 may determine the deletion ratio of the number of people reflected in the user's field of vision according to the stress level (LF / HF×α). That is, the higher the stress level, the higher the deletion ratio is determined, and the larger the number of people to be deleted becomes. On the other hand, the lower the stress level, the lower the deletion ratio is determined, and the smaller the number of people to be deleted becomes. Of course, the stress level and the deletion ratio according to the stress level may be settable in advance by the user himself / herself.

[0051] <Second Embodiment> Among the people reflected in the user's field of vision, the people with a preset facial expression are made to disappear.

[0052] FIG. 4 is a flowchart of the second embodiment in the smart glasses of the present invention. Similar to FIG. 3, FIG. 4 also assumes that the user is giving a presentation. Suppose a number of audiences (people) are visible in the user's field of vision. At this time, assume that the user is prone to anxiety when seeing people with negative facial expressions.

[0053] The object recognition unit 11 recognizes a large number of people from the video reflected in the user's field of view. Here, the object recognition unit 11 may detect the facial expressions of people and recognize the facial expressions from the facial features of those people (see, for example, Non-Patent Document 2). The cut-out person region is converted into a feature amount (Euclidean distance) of 128 / 256 / 512 dimensions using a face recognition model. The feature amount obtained from the face region is not limited to the feature amount of the peripheral region of the eyes (for example, the eyes are closed, and there are wrinkles between the eyebrows), but also detects, for example, the regions of the nose, mouth, cheeks, and facial muscles. Thereby, three expressions of positive / negative / neutral can be determined. Of course, it may recognize the facial expressions themselves such as "the eyes are closed" and "there are wrinkles between the eyebrows". Specifically, the Google (registered trademark) Facenet (registered trademark) algorithm can also be used as the face recognition model. Thereby, the feature amount of a multi-dimensional vector can be converted from the face region.

[0054] Next, the biological activity amount acquisition unit 12 acquires the biological activity amount of the user who is giving a presentation.

[0055] Next, the determination unit 13 determines whether or not the stress level is equal to or higher than a predetermined threshold value from the biological activity amount of the user.

[0056] Then, when it is determined to be true by the determination unit 13, the augmented reality image generation unit 14 generates an augmented reality image in which a person with a preset facial expression among the people reflected in the user's field of view is made to disappear. For example, by presetting "negative" as the facial expression, the person recognized as negative is made to disappear. Of course, it may be preset whether or not to make the person disappear for each facial expression such as "the eyes are closed" and "there are wrinkles between the eyebrows".

[0057] As another embodiment, for example, for a person recognized as "negative", an augmented reality image may be generated in which the facial expression of the person is converted into a "smiling face" using the technique of Non-Patent Document 1. Whether to convert a person recognized as negative into a smiling face can also be preset by the user himself / herself.

[0058] In this way, when the user is making a presentation, only the audience with a friendly facial expression can be seen, so the stress can be relieved.

[0059] <The Third Embodiment> Among the people reflected in the user's field of vision, the people with preset attributes are made to disappear. The object recognition unit 11 detects only the people with a predetermined attribute as a predetermined object. The predetermined attribute may be, for example, gender or age. For example, only men over 50 years old or only women under 30 years old may be recognized. In this way, the user is not stressed because the people with attributes that the user is not good at do not appear in the field of vision, not only in the presentation scene. Even when a person with an appearance that is likely to cause fear or anxiety to the user appears in the field of vision, the mental health can be improved.

[0060] <The Fourth Embodiment> Among the people reflected in the user's field of vision, the people with a preset posture are made to disappear. The object recognition unit 11 recognizes the posture from the image area of the person, and detects only the person whose posture satisfies a predetermined condition as a predetermined object. For example, an AI engine for action recognition such as "looking down", "lying down", "crossing arms", "shaking the body", "talking to the person next to" can also be used (see, for example, Non-Patent Document 6). In this way, the user is not stressed because the people with unfriendly actions do not appear in the field of vision, not only in the presentation scene. Even when a person with a posture that is likely to cause fear or anxiety to the user appears in the field of vision, the mental health can be maintained.

[0061] As described in detail above, according to the device, program, and display method of the present invention, by controlling the reflection of an object entering the field of view according to the user's physical activity amount, the psychological stress of the user can be reduced.

[0062] In addition, thereby, for example, from "reducing the psychological stress of the user", it becomes possible to contribute to Goal 8 of the Sustainable Development Goals (SDGs) led by the United Nations, "Promote inclusive and sustainable economic growth, employment, and decent work for all".

[0063] Regarding the various embodiments of the present invention described above, various changes, modifications, and omissions within the scope of the technical idea and perspective of the present invention can be easily made by those skilled in the art. The above description is merely an example and is not intended to impose any restrictions. The present invention is limited only by the claims and their equivalents.

Explanation of Reference Numerals

[0064] 1 Smart Glass 101 Lens Display 102 Camera 103 Wireless Communication Unit 11 Object Recognition Unit 12 Physical Activity Amount Acquisition Unit 13 Determination Unit 14 Augmented Reality Image Generation Unit 2 Smart Watch, Wearable Device 3 Smartphone

Claims

1. In a device having a camera that captures an image visible from the user's field of view and a lens display that projects an augmented reality image onto the user's field of view, object recognition means for detecting a person's image area by object recognition from the image captured by the camera; biological activity amount acquisition means for acquiring the biological activity amount of the user; determination means for setting a reduction rate for reducing the number of people in the image visible to the user according to the stress level estimated from the biological activity amount; augmented reality image generation means for generating an augmented reality image processed to conceal the person's image area so that the number of people in the image visible to the user is reduced at the rate A device characterized by comprising.

2. The device is in the form of glasses worn in front of the user's eyes on the head, The augmented reality image generation means generates an augmented reality video that conceals the image area of a predetermined object from the video visible to the user The device according to claim 1, characterized in that.

3. The biological activity amount acquisition means acquires the biological activity amount from a wearable device worn on the user's body or a sensor attached to the user's body The device according to claim 1, characterized in that.

4. The biological activity amount is any one or a combination of heart rate, blood oxygen concentration, skin temperature, blood pressure, pulse, sweating amount, atmospheric pressure, and ambient sound The device according to claim 3, characterized in that.

5. In a program that causes a computer mounted on a device having a camera that captures an image visible from the user's field of view and a lens display that projects an augmented reality image onto the user's field of view to function, object recognition means for detecting a person's image area by object recognition from the image captured by the camera; biological activity amount acquisition means for acquiring the biological activity amount of the user; determination means for setting a reduction rate for reducing the number of people in the image visible to the user according to the stress level estimated from the biological activity amount; augmented reality image generation means for generating an augmented reality image processed to conceal the person's image area so that the number of people in the image visible to the user is reduced at the rate A program characterized by causing a computer to function.

6. In a display method of a device having a camera that captures an image visible from the user's field of view and a lens display that projects an augmented reality image onto the user's field of view, The device is A first step of detecting a person's image area by object recognition from an image captured by a camera; A second step of acquiring the user's biological activity amount; A third step of setting a reduction ratio for reducing the number of people in the video reflected in the user's field of view according to the stress level estimated from the biological activity amount; A fourth step of generating an augmented reality image processed to conceal the person's image area so that the number of people in the video reflected in the user's field of view is reduced at the ratio; A method for displaying a device, characterized by executing the above steps.

Images