Display system, display method, and program

Abstract

To provide a display system, a display method, and a program that are capable of providing information related to an object of interest.SOLUTION: A display system includes an object of interest determination unit that determines an object of interest, and an image generation unit that generates an eye image by superimposing and displaying an image of the object of interest associated with the determined object of interest in or on an eye, and displays the generated eye image on a display device. The image generation unit displays the eye image, which is a moving image obtained by changing the image of the object of interest displayed by superimposing it in or on the eye according to a determination result of the object of interest, on the display device.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a display system, a display method, and a program. [Background technology]

[0002] In recent years, development of robots that can communicate with people has been progressing. For example, a method is known in which images simulating human eyes are displayed on the robot's facial area to stimulate communication with people. Also, a method has been proposed in which the robot's eye images are made variable depending on the situation, thereby enhancing the expressiveness of the robot's eyes (see, for example, Patent Document 1).

[0003] Another technology that has been proposed is a method of displaying information about the position in a virtual space where a user is gazing at on the front of a head-mounted display, so that the content that a user wearing a head-mounted display is focusing on in the virtual space can be confirmed from the outside (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7068709 [Patent Document 2] U.S. Patent No. 1,107,999 Summary of the Invention [Problem to be solved by the invention]

[0005] Information about the direction of human eye gaze plays an important role in communication. Specifically, by observing the direction of eye gaze, it is possible to understand where a person is looking. Furthermore, by tracking the direction of eye gaze, it is possible to understand what the person is focusing on. Conventional robots have been configured to allow users to grasp rough information about the direction the robot is gazing by, for example, moving the image of the robot's eyes displayed on a display unit. However, such information based on the movement of the robot's eyes on a two-dimensional screen is ambiguous for understanding the target the robot is focusing on in three-dimensional real space. In particular, when there are many potential targets in three-dimensional real space, users may misinterpret the target the robot is focusing on.

[0006] Furthermore, in conventional head-mounted display techniques, a third party must first recognize the configuration of the virtual space displayed on the front of the head-mounted display, then confirm the position at which the user wearing the head-mounted display is focusing, and then confirm the object corresponding to that position. This makes it difficult for a third party to intuitively and easily grasp the object at which the user is focusing. Furthermore, when a virtual space is displayed on the front of a head-mounted display, the user's appearance appears unnatural to a third party, which hinders communication.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide a display system, a display method, and a program that are capable of providing information related to an object of interest. [Means for solving the problem]

[0008] (1) In order to achieve the above object, a display system according to one aspect of the present invention comprises a focus object determination unit that determines a focus object, and an image generation unit that generates an eye image by superimposing an image of the focus object associated with the determined focus object in or on the eye, and causes the generated eye image to be displayed on a display device, wherein the image generation unit causes the display device to display the eye image, which is a moving image obtained by changing the image of the focus object that is displayed by superimposing it in or on the eye depending on the determination result of the focus object.

[0009] (2) In the display system of (1) above, the image generating unit may be capable of adjusting the transparency of the image of interest.

[0010] (3) In the display system of (1) above, the image generation unit may be capable of adjusting at least one of the position and size of the image of interest displayed in the eye image.

[0011] (4) In the display system of (1) above, the image generating unit may generate the image of the target object by modifying a captured image of the target object.

[0012] (5) In the display systems (1) to (4) above, the object of interest may be a person present in the surrounding environment of the display system, or a person present in another environment communicatively connected to the display system.

[0013] (6) The display system according to (1) to (4) above may include an image of the background of the target object that changes in accordance with the change in the target object.

[0014] (7) In the display system of (6) above, the image generation unit may generate the eye image as a moving image that combines movement of the eye display position within the eye image with movement of the background.

[0015] (8) In the display systems (1) to (4) above, the target object determination unit may determine the target object based on a captured image of the surrounding environment of the display system or another environment communicatively connected to the display system.

[0016] (9) In the display system of (8) above, the image generating unit may generate the image of the target object by extracting a portion corresponding to the target object from the captured image.

[0017] (10) In the display system according to any one of (1) to (4) above, the target object determination unit may determine the target object based on an operating state of the display system.

[0018] (11) In the display system of (10) above, the target image may be an image associated with an operating state of the display system.

[0019] (12) In the display systems (1) to (4) above, the display system may be a communication system for communicating with people, and the target object may be an object that the communication system is focusing on.

[0020] (13) In the display systems described in (1) to (4) above, the display system may be a wearable device that can be worn by a person, and the target object may be an object that the person is focusing on.

[0021] (14) In the display system of (13) above, the image generation unit may display the generated eye image on a display device provided on an outer surface of the wearable device.

[0022] (15) In the display system of (13) above, the image generation unit may display the generated eye image on an external display device communicatively connected to the wearable device.

[0023] (16) In the display system of (13) above, the image generation unit may display the generated eye image on a display device of another wearable device communicatively connected to the wearable device.

[0024] (17) Another aspect of the present invention relates to a display method in which a computer of a display system determines an object of interest, generates an eye image by superimposing an image of the object of interest associated with the determined object of interest in or on the eye, displays the generated eye image on a display device, and displays the eye image, which is a moving image obtained by changing the image of the object of interest that is displayed by superimposing it in or on the eye depending on the determination result of the object of interest, on the display device.

[0025] (18) A program according to another aspect of the present invention causes a computer of a display system to determine an object of interest, generate an eye image in which an image of the object of interest associated with the determined object of interest is superimposed in or on the eye, display the generated eye image on a display device, and display on the display device the eye image, which is a moving image in which the image of the object of interest displayed in or on the eye is changed according to the determination result of the object of interest. [Effects of the Invention]

[0026] According to (1) to (18), information about the target object can be provided. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a diagram showing an example of the external shape of a communication robot according to a first embodiment. [Figure 2] 1 is a functional block diagram showing an example of the configuration of a communication robot according to a first embodiment. [Figure 3] FIG. 4 is a diagram showing a first display mode (pupil display) according to the first embodiment. [Figure 4] 5A to 5C are diagrams showing various display patterns in the first display mode (pupil display) according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing a second display mode (lens display) according to the first embodiment. [Figure 6] 5A to 5C are diagrams showing various display patterns in the second display mode (lens display) according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing a third display mode (full display) according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing an example in which an image of an icon is displayed in the first display mode (pupil display) according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing another example in which icon images are displayed in the first display mode (pupil display) according to the first embodiment. [Figure 10] 4 is a flowchart showing an example of the flow of an operation of the communication robot according to the first embodiment. [Figure 11] FIG. 4 is a diagram illustrating an example of a change in an eye image according to the first embodiment. [Figure 12] 1 is a diagram illustrating an example of use of a communication robot according to a first embodiment. FIG. [Figure 13] 1 is a diagram illustrating an example of use of a communication robot according to a first embodiment. FIG. [Figure 14] FIG. 10 is a diagram showing an example of a usage environment of a display device 3 according to a second embodiment. [Figure 15] FIG. 10 is a diagram showing another example of the usage environment of the display device 3 according to the second embodiment. [Figure 16] FIG. 10 is a diagram showing yet another example of the usage environment of the display device 3 according to the second embodiment. [Figure 17] FIG. 10 is a functional block diagram showing an example of the configuration of a display device 3 according to a second embodiment. [Figure 18] 10 is a flowchart showing an example of the flow of operations of the display device 3 according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0028] [First embodiment] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component has been appropriately changed so that each component can be recognized. In all drawings used to explain the embodiments, components having the same function are designated by the same reference numerals, and repeated explanations will be omitted. In addition, "based on XX" in this application means "based on at least XX" and includes cases where the component is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where the component is based on XX after calculation or processing. "XX" is any element (for example, any information).

[0029] <Communication Robot Overview> The communication robot according to the embodiment is a robot that communicates with one or more people (hereinafter referred to as "users"). Communication is mainly through dialogue and gestures (movements). Movements are expressed not only by actual movements but also by images displayed on a display unit. The communication robot communicates with multiple people, for example, in a place where multiple people gather (such as a classroom or conference room).

[0030] The human eye has a smooth outer surface that reflects a portion of light, and can act as a convex mirror for the scene it is facing. Inspired by this characteristic of the human eye, a communication robot according to an embodiment displays an object that the communication robot is focusing on (hereinafter referred to as "object of focus") within an image that simulates the appearance of a human eye (hereinafter referred to as "eye image"), allowing the communication robot to provide information about the object of focus to the user.

[0031] The image of the communication robot's eyes is displayed on the screen at a much larger size than human eyes, allowing the user to simultaneously and clearly understand the direction (eye focus) and information about the object the communication robot is gazing at, and easily attracting the user's attention. In normal human-to-human communication, two steps are required for another person to understand the object they are looking at: understanding the direction the other person is gazing at and then mapping this direction in space to identify the object. On the other hand, the communication robot according to the embodiment allows the user to perform these two steps in a single step, thereby achieving effective communication.

[0032] The user responds to the actions presented by the communication robot with facial expressions, justification, voice, etc. For example, a communication robot achieves communication by using machine learning technology to learn the content of conversations and actions that constitute communication with humans. A communication robot is an example of a "display system." Note that a "display system" is not limited to a robot, but can be any device capable of communication.

[0033] <Example of communication robot appearance> Next, an example of the external shape of the communication robot 1 will be described. FIG. 1 is a diagram illustrating an example of the external shape of the communication robot 1 according to the first embodiment. In FIG. 1, a front view g101 and a side view g102 illustrate an example of the external shape of the communication robot 1 according to the first embodiment. The communication robot 1 includes, for example, three display units 111 (eye display unit 111a, eye display unit 111b, and mouth display unit 111c). The image capturing unit 102a is attached to the top of the eye display unit 111a, and the image capturing unit 102b is attached to the top of the eye display unit 111b. The eye display units 111a and 111b present images and image information that are visible to the user. The eye display units 111a and 111b are, for example, liquid crystal image display devices or organic EL (Electro Luminescence) image display devices. The screen size of the eye display units 111a and 111b is, for example, 3 inches. Speaker 112 is attached to housing 120 near mouth display unit 111c, which displays an image corresponding to a person's mouth. Mouth display unit 111c is composed of, for example, a plurality of LEDs (light-emitting diodes), and each LED is addressable and can be individually turned on and off. Sound collection unit 103 is attached to housing 120.

[0034] The communication robot 1 also includes a boom 121. The boom 121 is movably attached to the housing 120 via a movable part 131. A horizontal bar 122 is rotatably attached to the boom 121 via the movable part 132. An eye display unit 111a is rotatably attached to the horizontal bar 122 via a movable part 133, and an eye display unit 111b is rotatably attached to the horizontal bar 122 via a movable part 134. Note that the external shape of the communication robot 1 shown in FIG. 1 is an example and is not limited to this.

[0035] <Example of communication robot configuration> Next, we will explain an example of the configuration of the communication robot 1. Figure 2 is a functional block diagram showing an example of the configuration of the communication robot 1 according to the first embodiment. The communication robot 1 includes, for example, a photographing unit 102, a sound collecting unit 103, an actuator 104, a display unit 111, a speaker 112, a control unit 200, and a storage unit 300.

[0036] The image capturing unit 102 captures an image of the environment in which the communication robot 1 is installed. As shown in FIG. 1, the image capturing unit 102 includes, for example, image capturing unit 102a and image capturing unit 102b. The image capturing unit 102 may be installed in the environment in which the communication robot 1 and the user are present. The image capturing unit 102 is, for example, a CMOS (Complementary Metal Oxide Semiconductor) image capturing element or a CCD (Charge Coupled Device) image capturing element. The image information captured by the image capturing unit 102 is, for example, an image including the user's face. The image may be any one of a still image, a series of still images, or a video.

[0037] The sound collection unit 103 collects (collects) sounds (acoustic signals) in the environment in which the communication robot 1 is installed. The sound collection unit 103 is, for example, a microphone array having a plurality of microphones.

[0038] The actuator 104 drives each part of the communication robot 1 in accordance with the control of the control unit 200.

[0039] 1, the display unit 111 includes, for example, an eye display unit 111a, an eye display unit 111b, and a mouth display unit 111c. The display unit 111 displays an image or an animation under the control of the control unit 200.

[0040] The speaker 112 outputs an acoustic signal under the control of the control unit 200 .

[0041] The control unit 200 includes, for example, an acquisition unit 211, an image information processing unit 213, a sound information processing unit 215, a focus object determination unit 217, and an action processing unit 219. Each of these functional units is realized by a hardware processor (computer) such as a central processing unit (CPU) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as a large-scale integration (LSI), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU), or a system-on-chip (SOC), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a hard disk drive (HDD) or flash memory of the communication robot 1, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the hard disk drive or flash memory of the communication robot 1 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0042] The acquisition unit 211 acquires image information captured by the imaging unit 102 and audio information collected by the audio collection unit 103. The acquisition unit 211 may also acquire desired image information via a network such as the Internet.

[0043] The image information processing unit 213 performs well-known image processing on the image captured by the imaging unit 102. Examples of well-known image processing include feature detection, binarization, edge detection, contour detection, clustering processing, etc. Furthermore, the image information processing unit 213 uses the image-processed information to detect, for example, a human face.

[0044] The sound information processing unit 215 performs well-known sound processing on the sound signal collected by the sound collection unit 103. Examples of well-known sound processing include voice recognition and speaker identification.

[0045] The attention target determination unit 217 determines the attention target of the communication robot 1 at each timing. For example, the attention target determination unit 217 determines the attention target based on the image captured by the image capture unit 102 (processing result of the image information processing unit 213) or the sound signal collected by the sound collection unit 103 (processing result of the sound information processing unit 215). Alternatively, the attention target determination unit 217 may determine the attention target based on the content of the current action of the communication robot 1.

[0046] Furthermore, the attention target determination unit 217 may determine the degree of attention for the attention target in addition to determining the attention target. For example, the attention target determination unit 217 determines the degree of attention based on the image captured by the image capture unit 102 and the audio signal collected by the audio collection unit 103 so that the face of a person closer to the communication robot 1 receives a higher degree of attention than a face farther away, the face of a person in motion receives a higher degree of attention than a still face, and the face of a person who is talking receives a higher degree of attention than a silent face. Furthermore, for example, the attention target determination unit 217 determines the degree of attention so that an attention target that is highly relevant to the content of the current action of the communication robot 1 receives a higher degree of attention.

[0047] The focus target determination unit 217 is an example of an "focus target determination unit." The focus target determination unit 217 determines the focus target on which the communication system is focusing. The focus target is a person present in the surrounding environment of the communication system, or a person present in another environment communicatively connected to the communication system. The display position of the focus target image in the eye image moves as the person moves. The eye image includes an image of the background of the focus target that changes as the focus target changes (e.g., the person moves). The image generation unit 221 generates an eye image that is a moving image combining the movement of the eye display position in the eye image with the movement of the background. The focus target determination unit 217 determines the focus target based on a captured image of the surrounding environment of the communication system, or a captured image of another environment communicatively connected to the communication system. The focus target determination unit 217 determines the focus target based on the operating state of the communication system.

[0048] The movement processing unit 219 controls the movement of the communication robot 1. The movement processing unit 219 generates, for example, images to be displayed on the eye display unit 111a, the eye display unit 111b, and the mouth display unit 111c, generates a drive signal to drive the actuator 104, and generates sound to be output from the speaker 112. The movement processing unit 219 includes, for example, an image generation unit 221, a sound generation unit 223, and a drive unit 225.

[0049] Image generation unit 221 generates eye images (moving images) to be displayed on eye display unit 111a and eye display unit 111b based on the determination result of the target of interest by target of interest determination unit 217, and causes the generated eye images to be displayed on eye display unit 111a and eye display unit 111b. The displayed eye images correspond to the movement of a person's eyes. Furthermore, image generation unit 221 generates an image (still image, continuous still images, or moving image) to be displayed on mouth display unit 111c, and causes the generated image to be displayed on mouth display unit 111c. The displayed image corresponds to the movement of a person's mouth.

[0050] An image corresponding to the target of interest is included within or above the eyes of the eye image generated by the image generation unit 221. For example, if the target of interest is the face of a person captured in an image captured by the image capture unit 102, an image of the person's face is included within or above the eyes of the eye image. Alternatively, if the target of interest is an image corresponding to the content of the current action of the communication robot 1 (for example, an image of an icon), an image of the icon is included within or above the eyes of the eye image. The image generation unit 221 generates an image of the target of interest by correcting the captured image of the person by performing left-right flipping processing to obtain a mirror image, cropping, applying a smoothing filter, changing brightness and contrast, etc. That is, the image generation unit 221 generates an image of the target of interest by correcting the captured image of the target of interest. Details of the processing by the image generation unit 221 will be described later.

[0051] The image generation unit 221 is an example of an "image generation unit." The image generation unit 221 generates an eye image in which an image of the target of interest associated with the determined target of interest is superimposed in or on the eye, and causes the generated eye image to be displayed on a display device. During communication, the image generation unit 221 causes the display device to display an eye image, which is a moving image in which the image of the target of interest is changed and superimposed in or on the eye depending on the determination result of the target of interest. The image generation unit 221 generates the image of the target of interest by extracting a portion corresponding to the target of interest from the captured image. The image of the target of interest is an image associated with the operating state of the communication system.

[0052] The voice generation unit 223 generates an output voice signal to be output to the speaker 112, and outputs the generated output voice signal to the speaker 112. The voice generation unit 223 generates the output voice signal using, for example, a learning model that has learned the content of conversations and actions, which are communication with people, using machine learning technology.

[0053] The driving unit 225 generates, for example, a driving signal for driving the neck actuator 104, and drives the actuator 104 using the generated driving signal. The driving unit 225 generates the driving signal using, for example, the above-described learning model.

[0054] The storage unit 300 stores programs, algorithms, learning models, predetermined values, thresholds, and the like required for various controls and processes of the communication robot 1. The storage unit 300 stores, for example, a language model database, an acoustic model database, a dialogue corpus database, and acoustic features used in speech recognition, a comparison image database and image features used in image recognition, and the like. The storage unit 300 may be realized by the various storage devices described above, or by an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), RAM (Random Access Memory), or the like. The storage unit 300 may be located on the cloud or connected via a network.

[0055] 2 is merely an example, and the configuration of the communication robot 1 is not limited to this. For example, the communication robot 1 may include a single display unit, a single image capture unit, and a processing device that interacts with multiple people.

[0056] <Example of communication robot control> Next, a description will be given of an example of controlling the communication robot 1. In the first embodiment, the movement of the eyes and neck is controlled in order to move the line of sight of the communication robot 1 in the direction of an object of attention of the communication robot 1.

[0057] First, the movement of the eyes will be described. The image generation unit 221 moves the center position of the eye to match the target of attention. In the eye image, a range within which the eyeball can move is determined in advance, for example. The image generation unit 221 moves the focus of the eye within this range so that it matches the target of attention.

[0058] Next, neck movement will be described. If the position of the target object is outside the range, image generation unit 221 first moves the planned display position of the eye image to the focal position. Then, drive unit 225 starts moving the head in the same direction to return the eye positions on eye display unit 111a and eye display unit 111b to within the central range.

[0059] If there is an object to be focused on around the communication robot 1, the eyes of the communication robot 1 first move to that object, and then the head of the communication robot 1 begins to move in the same direction at the same angle. In this way, the line of sight is controlled to be automatically guided to the object of attention.

[0060] <Example of eye display> (1) First display mode (pupil display) Next, examples of eye display modes displayed on the eye display units 111a and 111b of the communication robot 1 will be described. FIG. 3 is a diagram showing a first display mode (pupil display) according to the first embodiment. FIG. 3 illustrates an example of a situation in which the communication robot 1 is used in a place P1 where multiple users are present. The place P1 may be, for example, a classroom or a conference room. In this example, the communication robot 1 determines the face of person H1, who is sitting on the far left of the three people present and speaking with his or her hand raised, as the target of attention TA. Under these circumstances, an eye image E is displayed on the eye display units 111a and 111b of the communication robot 1. The eye image E includes an iris area Ir and a pupil area Pu. In the first display mode (pupil display), an image of the target of attention TA (the face of person H1) is displayed in the pupil area Pu. The three people present can see that the communication robot 1 is focusing on the target object TA (the face of person H1) by checking the image of the target object TA (the face of person H1) displayed on the eye display unit 111a and the eye display unit 111b.

[0061] The image pattern displayed in the pupil area Pu can be changed depending on the degree of attention paid to the target of interest TA in the communication robot 1, the situation in which the communication robot 1 is used, the user's preferences, and the like. FIG. 4 illustrates various display patterns in the first display mode (pupil display) according to the first embodiment. As shown in FIG. 4(A), the transparency of the image displayed in the pupil area Pu may be changed. For example, the higher the degree of attention paid to the target of interest TA, the lower the transparency of the image displayed in the pupil area Pu (so that the image can be clearly recognized). Furthermore, as shown in FIG. 4(B), the size of the pupil area Pu (i.e., the size of the image) may be changed. For example, the higher the degree of attention paid to the target of interest TA, the larger the pupil area Pu may be. Furthermore, as shown in FIG. 4(C), the position (distance, depth, angle) of the target of interest TA included in the image displayed in the pupil area Pu may be changed. For example, the higher the degree of attention paid to the target of interest TA, the closer the target of interest TA may be (the shorter the distance or depth). 4(D), the size of the iris Ir (and the size of the pupil Pu) may be changed. For example, the iris Ir (and the size of the pupil Pu) may be made larger as the degree of attention to the target of attention TA increases.

[0062] In addition, the image pattern displayed in the pupil area Pu may be changed in terms of the saturation, brightness, resolution, and distortion of the image, the shape and color of the pupil area, the shape and color of the iris area, the relative positions of the pupil area and the iris area, the color of the sclera, the timing, direction, continuity, and position of the image display, etc. Also, the left eye and the right eye may be displayed in different patterns.

[0063] (2) Second display mode (lens display) 5 is a diagram showing a second display mode (lens display) according to the first embodiment. Under the same circumstances as in FIG. 3, an eye image E is displayed on the eye display units 111a and 111b of the communication robot 1. The eye image E includes a lens unit Le (crystalline lens) that entirely encompasses an iris unit Ir and a pupil unit Pu. In the second display mode (lens display), an image of the target object TA (the face of person H1) is displayed over the entire lens unit Le.

[0064] The image pattern displayed on the lens unit Le can be changed depending on the degree of attention paid to the target of interest TA in the communication robot 1, the situation in which the communication robot 1 is used, the user's preferences, and the like. FIG. 6 illustrates various display patterns in the second display mode (lens display) according to the first embodiment. As shown in FIG. 6(A), the transparency of the image displayed on the lens unit Le may be changed. For example, the higher the degree of attention paid to the target of interest TA, the lower the transparency of the image displayed on the lens unit Le (so that the image can be clearly recognized). Also, as shown in FIG. 6(B), the size of the pupil may be changed. For example, the higher the degree of attention paid to the target of interest TA, the larger the pupil may be. Also, as shown in FIG. 6(C), the position (distance, depth, angle) of the target of interest TA included in the image displayed on the lens unit Le may be changed. For example, the higher the degree of attention paid to the target of interest TA, the closer the target of interest TA may be (the shorter the distance and depth). Also, as shown in FIG. 6(D), the size of the iris Ir (and the size of the pupil Pu) may be changed. For example, the iris portion Ir may be made larger (and the size of the pupil portion Pu may be made larger) as the degree of attention to the target of attention TA increases.

[0065] In addition, the image pattern displayed on the lens unit Le may be changed in terms of the saturation, brightness, resolution, and distortion of the image, the shape and color of the pupil unit, the shape and color of the iris unit, the relative positions of the pupil unit and the iris unit, the color of the sclera, the timing, direction, continuity, and position of the image display, etc. Also, the left eye and the right eye may be displayed in different patterns.

[0066] (3) Third display mode (full display) 7 is a diagram showing a third display mode (full display) according to the first embodiment. Under the same circumstances as in FIG. 3, an eye image E is displayed on the eye display units 111a and 111b of the communication robot 1. In the third display mode (full display), an image of the target of attention TA (the face of person H1) and a background image of the target of attention TA are displayed across the entire eye display units 111a and 111b. The part of the target of attention TA (the face of person H1) that the communication robot 1 pays the most attention to (near the mouth of person H1) is positioned at the position corresponding to the eye image E.

[0067] The images displayed in or superimposed on the eyes of eye display unit 111a and eye display unit 111b may be icon images instead of or in addition to the images captured by image capture unit 102. Fig. 8 is a diagram showing an example in which icon images are displayed in the first display mode (pupil display) according to the first embodiment. Fig. 8 shows an example in which icon images representing an hourglass or gear are displayed in pupil area Pu of eye image E.

[0068] Fig. 9 is a diagram showing another example in which an icon image is displayed in the first display mode (pupil display) according to the first embodiment. Fig. 9 shows an example in which an icon indicating a part of the face of a person that is the target of attention is displayed in the pupil part Pu of the eye image E. This icon indicates, for example, that attention is being paid to the "eyes" of the face of the person that is the target of attention.

[0069] The icons include any icons associated with an object that the communication robot 1 is focusing on, or icons associated with the operating state of the communication robot 1. The icon images may be pre-stored in the storage unit 300, or may be acquired via a network such as the Internet.

[0070] That is, the image generating unit 221 can adjust the transparency of the image of interest. The image generating unit 221 can adjust at least one of the display position and size of the image of interest in the eye image.

[0071] <Processing flow> Next, a description will be given of the process of generating and displaying the eye images of the communication robot 1. Fig. 10 is a flowchart showing an example of the flow of the operation of the communication robot according to the first embodiment.

[0072] First, the acquisition unit 211 acquires image information captured by the image capturing unit 102 and audio information collected by the audio collecting unit 103 (step S101).

[0073] Next, the attention target determination unit 217 determines the attention target based on the image captured by the imaging unit 102 (the processing result of the image information processing unit 213) and the sound signal collected by the sound collection unit 103 (the processing result of the sound information processing unit 215) (step S103). Alternatively, the attention target determination unit 217 may determine the attention target based on the content of the current behavior of the communication robot 1.

[0074] Next, the image generation unit 221 generates eye images to be displayed on the eye display units 111a and 111b based on the determination result of the target of attention by the target of attention determination unit 217 (step S105), and displays the generated eye images on the eye display units 111a and 111b (step S107). The eye images generated by the image generation unit 221 include an image corresponding to the target of attention within or above the eyes. For example, if the target of attention is a person's face captured in an image captured by the image capture unit 102, the eye image includes an image of the person's face within or above the eyes. Alternatively, if the target of attention is an image corresponding to the current action of the communication robot 1 (for example, an image of an icon), the eye image includes an image of the icon within or above the eyes. The user can understand what the communication robot 1 is focusing on by checking the images of the target of attention displayed on the eye display units 111a and 111b.

[0075] When the position (relative position) of a target object (e.g., a person) relative to the communication robot 1 changes, the image generation unit 221 generates an eye image (video) in which both the position of the pupil (and iris) (= the position where the target object image is displayed) and the background of the target object move. Here, the image generation unit 221 does not change the size of the target object image (reflection image), but instead covers a wide field of view by using an eye model that can be made significantly smaller than the captured image. This is achieved by mapping a large movement of the captured image captured by the image capture unit 102, resulting in a small movement of the pupil (and iris). In other words, a slight movement of the pupil can generate an eye image that represents a large movement of the target object. This allows for the effect of the eye's perspective on the target object (the effect of a three-dimensional spatial reflection), similar to that of looking in a mirror from different viewpoints. As the target object (the person) moves, different parts of the scene are displayed. In other words, the above mapping allows the illusion of a shift in viewpoint as the focus changes (moves) in space, with only simple horizontal and vertical translations being used. The use of different spatial offsets for each eye allows for the creation of slightly different virtual reflections (images of the focus) in each eye, giving the user the impression of a three-dimensional visual perception through the eyes.

[0076] The distance-dependent calculation of the spatial offset of each eye will now be described. FIG. 11 is a diagram illustrating an example of a change in the eye image according to the first embodiment. A change in the eye image occurs, for example, when the target of interest moves or when the target of interest is changed from one target to another. FIG. 11 shows a captured image IMG1 captured by the image capturing unit 102 before the target of interest moves, and a captured image IMG2 captured by the image capturing unit 102 after the target of interest moves. The size (pixels) of the captured images IMG1 and IMG2 is 1 / 32 in width and 1 / 32 in height. w × vertical c h The position corresponding to the eye image in the captured images IMG1 and IMG2 is the target position TL_im(c x ,c y) (pixels). In the photographed images IMG1 and IMG2, the eye image E generated (cut out) by the image generating unit 221 is approximately circular, and its size (pixels) is e w ×Vertical e h An image of the target object is superimposed on the pupil area Pu included in the eye image E. For example, a background image of the target object is superimposed on the area other than the pupil area Pu in the eye image E (on the iris area Ir and outside thereof). The center position of the pupil area Pu is defined as TL_pc(e x ,e y ) (pixels). The center position of the eye image E is TL_ec(m x ,m y ) and the horizontal offset is o x and the vertical offset is o y Let's say.

[0077] In the photographed image IMG1, the center position EC of the photographed image IMG1 and the center position TL_pc(e x ,e y ) and the center position of the eye image E, TL_ec(m x ,m y ) coincide with each other. On the other hand, in the photographed image IMG2, the photographed image is shifted to the left (-X direction) with respect to the eye image E, and the position of the eye image E is moved to match the target of attention. As a result, the background image superimposed on the eye image E is moved in the X direction by the movement amount MV1. Furthermore, the center position TL_pc(e x ,e y ) is shifted in the X direction from the center position of the eye image E by a movement amount MV2 (the position of the pupil area Pu relative to the sclera is shifted). The movement amount MV2 is smaller than the movement amount MV1. This continuous or parallel movement of the background image and the pupil area Pu can simulate the movement and stopping of a human eye.

[0078] The center position TL_pc(e x ,e y ) can be calculated using the following equations (1) and (2). e y =c y-m y +e h / 2···(1) e x =c x -n x (c w -e w )···(2)

[0079] Also, the center position TL_ec(m x ,m y ) can be calculated using the following equations (3) and (4). m y =n y (c h -e h )+e h / 2···(3) m x =(1-n x )(c w -e w )+e w / 2···(4)

[0080] In the above equations (1) to (4), n x and n y is a coefficient related to the offset and is defined by the following equations (5) and (6). n x =c x / c w +o x ···(5) n y =c y / c h +o y ···(6)

[0081] The horizontal and vertical offsets can be functions of the distance between the target object and the position of the eye model used in three-dimensional space (e.g., the distance between them, or the distance between the camera and the target object). In this way, simulating the parallax between the two eyes allows the eyes of the communication robot 1 to behave more human-like. Note that the above equations (1) to (6) are merely examples, and modifications or other equations may be used.

[0082] Next, the control unit 200 determines whether or not to end the operation of the communication robot 1 (step S109). For example, if the communication robot 1 receives an end instruction from the user, the control unit 200 determines to end the operation of the communication robot 1 (step S109; YES). On the other hand, if the control unit 200 determines not to end the operation of the communication robot 1 (step S109; NO), the process returns to step S101 and repeats the subsequent processes. By repeating these operations, a moving image of the target of attention is displayed on the eye display units 111a and 111b. In this moving image, the eyes of the eye display units 111a and 111b follow the target of attention, and the eyes of the communication robot 1 move so that the target of attention is included in the image of the eyes. This completes the processing of this flowchart.

[0083] <Examples of communication robot use> Next, an example of how a communication robot is used will be described. FIGS. 12 and 13 are diagrams illustrating an example of how a communication robot according to the first embodiment is used. FIG. 12 illustrates an example of how a communication robot is used at an event held in an environment where two different locations (locations P1 and P2) are connected via a network NW. A communication robot 1 is placed at location P1, and a communication robot 1A is placed at location P2. At a certain timing in the event, a person H2 at location P2 is speaking. If the communication robot 1A determines that this person H2 is a target of attention TA, an image of the person H2 determined to be the target of attention TA is displayed on the eye display unit 111a and the eye display unit 111b of the communication robot 1A. Here, the eye display unit 111a and the eye display unit 111b of the communication robot 1 placed at location P1 may also display the person H2 at location P2 determined by the communication robot 1A to be the target of attention TA. This configuration allows the target of attention of communication robots at other locations to be identified, creating a sense of unity at the event.

[0084] 13, while the communication robot 1 is communicating with the person U, the eye display unit 111a and the eye display unit 111b display the target of interest TA located behind the communication robot 1. With this configuration, the person U can also check the target of interest TA located in a blind spot of the communication robot 1, which is not visible to the person U, through the eye display unit 111a and the eye display unit 111b.

[0085] Additionally, the eye display units 111a and 111b may be displayed to imitate the movement of the target of attention TA. For example, when the person who is the target of attention TA closes his or her eyes (winks), the eyes displayed on the eye display units 111a and 111b may also be closed. Also, a 3D face mesh of the person who is the target of attention TA may be superimposed on the face and displayed on the eye display units 111a and 111b. This allows the user to better understand what the communication robot is looking at and how it is looking at it.

[0086] As described above, according to the first embodiment, it is possible to provide the user with information about the object of the communication robot's attention. The image of the communication robot's eyes is displayed on the screen much larger than a human's eyes, so it is possible to provide the user with information about the object of the communication robot's attention in an easy-to-understand manner and easily attract the user's attention. The communication robot according to the first embodiment makes it possible to perform, in a single step, the two steps normally taken to understand the object of attention of another person in normal human-to-human communication (understanding the direction the other person is gazing and then mapping this direction in space to identify the object), thereby achieving effective communication.

[0087] [Second embodiment] Next, a second embodiment of the present invention will be described. This embodiment enables information about an object of interest of a user of a display device, which is a wearable terminal that can be worn by a person, to be provided to the outside. In the following configuration, components having the same functions as those of the first embodiment will be given the same reference numerals and names, and specific descriptions thereof will be omitted.

[0088] <Display Device Overview> The display device according to this embodiment is a wearable device such as VR (Virtual Reality) glasses or AR (Augmented Reality) glasses. The display device can provide a user wearing the display device with a display of a virtual space or a display of digital content overlaid on a real space. The display device can provide information about an object of interest of the user wearing the display device to an external device. The display device is used, for example, in a place where multiple people gather (a classroom, a conference room, etc.) to provide information about the object of interest of the user to the user wearing the display device and third parties who happen to be present. The object of interest can be, for example, any object (such as a person or object) existing in real space or any object (such as a person or object) existing in a virtual space displayed on the display device. The display device is an example of a "display system."

[0089] FIG. 14 is a diagram illustrating an example of a usage environment of the display device 3 according to this embodiment. In this example, a user H4 wearing the display device 3 and a user H5 not wearing the display device exist in the same space. An eye image EM of user H4 is displayed on the outer surface of the display device 3 worn by user H4 (e.g., the outer surface of the lens portion of VR glasses). The eye image EM is an image of the eye of user H4. The eye image EM may be an image captured in real time or may be an image captured in advance. Alternatively, the eye image EM may be an image simulating a human eye or the appearance of an eye. An image of an object that user H4 is focusing on (hereinafter referred to as the "object of interest") is superimposed on this eye image EM. In this example, an image of user H5 is superimposed as the object of interest. User H5, a third party, can identify the object of interest of user H4 (user H5 himself in this example) by checking the eye image EM displayed on the outer surface of the display device 3 worn by user H4.

[0090] FIG. 15 is a diagram showing another example of a usage environment of the display device 3 according to this embodiment. In this example, a user H4 wearing the display device 3 and a user H5 not wearing the display device are present in the same space. Furthermore, an external display device 5 communicably connected to the display device 3 via a network NW is also placed in this space. An image of user H4 including an eye image EM of user H4 is displayed on this external display device 5. The eye image EM is an image of the eyes of user H4. An image of an object of interest of user H4 is superimposed on this eye image EM. In this example, an image of user H5 is superimposed and displayed as the object of interest. User H5, a third party, can identify the object of interest of user H4 (user H5 himself in this example) by checking the eye image EM displayed on the external display device 5.

[0091] FIG. 16 is a diagram showing yet another example of a usage environment of the display device 3 according to this embodiment. In this example, a user H4 wearing a display device 3A and a user H5 wearing a display device 3B are present in the same space. The display device 3A and the display device 3B are communicably connected via a network NW. Here, an image of user H4 including an eye image EM of user H4 is displayed on the display unit of the display device 3B that user H5 is viewing. The eye image EM is an image of the eyes of user H4. An image of an object of interest of user H4 is superimposed on this eye image EM. In this example, an image of user H5 is superimposed and displayed as the object of interest. User H5, a third party, can ascertain the object of interest of user H4 (user H5 himself in this example) by checking the eye image EM displayed on such display device 3B.

[0092] <Example of display device configuration> Next, a configuration example of a display device according to this embodiment will be described. Fig. 17 is a functional block diagram showing a configuration example of a display device 3 according to this embodiment. The display device 3 includes, for example, an imaging unit 401, an input interface 402, a communication unit 403, a first display unit 404, a second display unit 405, a control unit 500, and a storage unit 600.

[0093] The image capturing unit 401 includes an external camera that captures the external environment viewed by a user wearing the display device 3, and an internal camera that captures the user's eyes. The image capturing unit 401 is, for example, a CMOS image capturing element or a CCD image capturing element. The image may be any one of a still image, a series of still images, or a moving image.

[0094] The input interface 402 accepts various operations from the user. The input interface 402 is a physical operation unit (such as a switch) provided on the display device 3. Alternatively, the input interface 402 may be a virtual operation unit (such as a button) provided in a virtual space displayed on the first display unit 404.

[0095] The communication unit 403 is a communication interface for wireless or wired communication with display devices worn by other users or external display devices (such as displays).

[0096] The first display unit 404 provides a display of digital content superimposed on the real world, a display of a virtual space, a display of a remote space, etc. to the user wearing the display device 3. The first display unit 404 is, for example, a display provided inside (on the user side, inside) the part that corresponds to the lens of VR glasses or AR glasses.

[0097] The second display unit 405 displays an eye image EM of the user wearing the display device 3 so that it can be seen from outside the user. The second display unit 405 is, for example, a display provided on the outside (the side opposite the user, the outer surface) of the lens of the VR glasses or AR glasses. The eye image EM displayed on the second display unit 405 is an image of the eyes of the user wearing the display device 3. An image of an object of interest of the user is superimposed on the eye image EM. A third party can ascertain the object of interest of the user by checking the eye image displayed on the second display unit 405.

[0098] The control unit 500 includes, for example, an acquisition unit 511, a focus object determination unit 513, and an image generation unit 515. Each of these functional units is realized by, for example, a hardware processor (computer) such as a CPU executing a program (software). Furthermore, some or all of these components may be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, GPU, or SOC, or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (storage device having a non-transitory storage medium) such as an HDD or flash memory of the display device 3, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the display device 3 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0099] The acquiring unit 511 acquires, for example, image information captured by the imaging unit 401, instruction information input via the input interface 402, information on the eye images of other users transmitted from other display devices, etc. The acquiring unit 511 may also acquire desired image information via a network such as the Internet.

[0100] The attention target determination unit 513 determines an attention target of the user wearing the display device 3 at each timing. For example, the attention target determination unit 513 determines the user's gaze direction based on an image of the user's eyes captured by the image capture unit 401, and determines an object existing in real space located in the determined gaze direction or an object existing in virtual space located in the gaze direction as an attention target. An object existing in real space is determined by associating an image captured by the image capture unit 401 (external camera) or a sensor value of a depth sensor, etc., with the determined gaze direction. An object existing in virtual space is determined by associating a virtual space image displayed on the first display unit 404 with the determined gaze direction. Alternatively, the attention target determination unit 513 may determine an attention target based on a user instruction input via the input interface 402. In addition to determining an attention target, the attention target determination unit 513 may determine a degree of attention to the attention target.

[0101] Image generation unit 515 generates an eye image to be displayed on second display unit 405 based on the determination result of the target of interest by target of interest determination unit 513, and causes the generated eye image to be displayed on second display unit 405. Image generation unit 515 also generates an image to be displayed on first display unit 404 (for example, a virtual space image), and causes the generated image to be displayed on first display unit 404.

[0102] An image corresponding to the target of interest is included in or above the eyes of the eye image generated by image generation unit 515. For example, if the target of interest is the face of a person existing in real space, an image of the person's face is superimposed and displayed in or above the eyes of the eye image. Alternatively, if the target of interest is an object existing in virtual space, an image of the object is superimposed and displayed in or above the eyes of the eye image.

[0103] Image generation unit 515 is an example of an "image generation unit." Image generation unit 515 displays the generated eye image on a display provided on the outer surface of the wearable device. Image generation unit 515 displays the generated eye image on an external display communicatively connected to the wearable device. Image generation unit 515 displays the generated eye image on a display of another wearable device communicatively connected to the wearable device.

[0104] The storage unit 600 stores programs, algorithms, learning models, predetermined values, thresholds, etc. required for various controls and processes of the display device 3. The storage unit 600 may be realized by the various storage devices described above, or an SSD, EEPROM, ROM, RAM, etc. The storage unit 600 may be located on the cloud or connected via a network.

[0105] <Processing flow> Next, a description will be given of the processing of the display device 3. Fig. 18 is a flowchart showing an example of the processing flow of the display device 3 according to this embodiment.

[0106] First, the acquisition unit 511 acquires image information captured by the image capturing unit 401 and instruction information input via the input interface 402 (step S201).

[0107] Next, the focus target determination unit 513 determines a focus target of the user wearing the display device 3 (step S203). For example, the focus target determination unit 513 determines the user's line of sight based on the image of the user's eyes captured by the image capture unit 401, and determines that an object existing in the real space and located in the determined line of sight or an object existing in the virtual space and located in the line of sight is the focus target.

[0108] Next, image generation unit 515 generates an eye image to be displayed on second display unit 405 (step S205), and causes second display unit 405 to display the generated eye image (step S207). An image according to the target of interest is superimposed and displayed inside or on the eye of the eye image generated by image generation unit 515. A third party can understand what the user wearing display device 3 is focusing on by checking the image of the target of interest displayed on second display unit 405.

[0109] 15 , the image generation unit 515 transmits the generated eye image to the external display device 5 via the communication unit 403. As a result, the eye image of the user wearing the display device 3 is displayed on the external display device 5. By checking the eye image displayed on the external display device 5, a third party can understand what the user wearing the display device 3 is focusing on.

[0110] 16, in a configuration in which communication is performed between display devices, image generation unit 515 transmits the generated eye image to the other display device via communication unit 403. As a result, the eye image of the user wearing display device 3 is displayed on the other display device. A third party can understand what the user wearing display device 3 is focusing on by checking the eye image displayed on such other display device (first display unit 404).

[0111] Next, the control unit 500 determines whether or not to terminate the operation of the display device 3 (step S209). For example, if an end instruction is received from the user, the control unit 500 determines to terminate the operation of the display device 3 (step S209; YES). On the other hand, if the control unit 500 determines not to terminate the operation of the display device 3 (step S209; NO), the process returns to step S201 and repeats the subsequent processes. By repeating such operations, a moving image of the target object is displayed on the second display unit 405. In this moving image, the eye displayed on the second display unit 405 follows the target object and moves so that the target object is included in the image of the eye. This completes the processing of this flowchart.

[0112] As described above, according to the second embodiment, information regarding the target and / or direction of attention of a user wearing the display device 3 can be provided in a manner suitable for communication with people while eliminating ambiguity as much as possible.

[0113] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0114] 1, 1A... communication robot, 102... photographing unit, 103... sound collection unit, 104... actuator, 111... display unit, 111a, 111b... eye display unit, 111c... mouth display unit, 112... speaker, 120... housing, 121... boom, 122... horizontal bar, 131... moving unit, 132, 133, 134... moving unit, 200... control unit, 211... acquisition unit, 213... image information processing unit, 215... sound Information processing unit, 217...attention object determination unit, 219...motion processing unit, 221...image generation unit, 223...sound generation unit, 225...drive unit, 300...storage unit, 3, 3A, 3B...display unit, 401...photographing unit, 402...input interface, 403...communication unit, 404...first display unit, 405...second display unit, 500...control unit, 511...acquisition unit, 513...attention object determination unit, 515...image generation unit, 600...storage unit

Claims

1. an attention object determination unit that determines an attention object; an image generating unit that generates an eye image by superimposing an image of the target of interest associated with the determined target of interest in or on the eye, and displays the generated eye image on a display device; Equipped with the image generation unit causes the display device to display the eye image, which is a moving image obtained by changing the image of the target of interest to be displayed by superimposing it in or on the eye according to the determination result of the target of interest. Display system.

2. the image generation unit is capable of adjusting the transparency of the target image; The display system of claim 1 .

3. the image generation unit is capable of adjusting at least one of a position and a size of the image of interest displayed in the eye image. The display system of claim 1 .

4. the image generation unit generates the image of the target object by modifying the captured image of the target object. The display system of claim 1 .

5. The target object is a person present in the surrounding environment of the display system, or a person present in another environment communicably connected to the display system. A display system according to any one of claims 1 to 4.

6. the eye image includes an image of a background of the target object that changes in accordance with a change in the target object; A display system according to any one of claims 1 to 4.

7. the image generation unit generates the eye image as a moving image combining movement of the display position of the eye in the eye image and movement of the background. The display system of claim 6.

8. the target-of-interest determination unit determines the target of interest based on a captured image of a surrounding environment of the display system or another environment communicably connected to the display system. A display system according to any one of claims 1 to 4.

9. the image generation unit generates the image of the target object by extracting a portion corresponding to the target object from the captured image. The display system of claim 8 .

10. the focus object determination unit determines the focus object based on an operating state of the display system. A display system according to any one of claims 1 to 4.

11. the target image is an image associated with an operating state of the display system; The display system of claim 10.

12. the display system is a communication system for communicating with people, The target object is an object that the communication system is focusing on. A display system according to any one of claims 1 to 4.

13. the display system is a wearable device that can be worn by a person; The target object is an object that the person is paying attention to. A display system according to any one of claims 1 to 4.

14. the image generation unit displays the generated eye image on a display device provided on an outer surface of the wearable device; 14. The display system of claim 13.

15. the image generation unit displays the generated eye image on an external display device communicably connected to the wearable device; 14. The display system of claim 13.

16. the image generation unit displays the generated eye image on a display device of another wearable device communicatively connected to the wearable device; 14. The display system of claim 13.

17. The display system computer Determine the target of interest, generating an eye image in which the image of the target of interest associated with the determined target of interest is superimposed in or on the eye; Displaying the generated eye image on a display device; displaying, on the display device, the eye image, which is a moving image obtained by changing the image of the target of interest that is displayed by being superimposed in or on the eye according to the determination result of the target of interest; Display method.

18. The display system computer Determine the target of attention, generating an eye image in which the image of the target of interest associated with the determined target of interest is superimposed in or on the eye; Displaying the generated eye image on a display device; displaying, on the display device, the eye image, which is a moving image obtained by changing the image of the target of interest that is displayed by being superimposed in or on the eye according to the determination result of the target of interest; program.

Images