Information processing method

Abstract

To solve a problem where in a conventional apparatus for displaying interviewee information on a portable information terminal held by a user, when the user actually faces an interviewee, it is not possible to quickly acquire information relating to the interviewee at timing immediately before an interview and such information is acquired after start of the interview, causing awkward situations where the user and the interviewee cannot be on the same topic at the start of the interview.SOLUTION: A portable information terminal according to a representative embodiment of the present invention determines whether or not a person in the vicinity of a user is a potential interviewee by performing behavior analysis on such a person, and when it is determined that the person is a potential interviewee, acquires information relating to the person to present it to the user. Consequently, the user is allowed to previously be aware of the information relating to the interviewee before the start of the interview or at timing at which the person is recognized as an interviewee.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a portable information terminal and an information processing method. [Background technology]

[0002] It is not rare that a person with whom you have had a face-to-face conversation in the past (hereinafter referred to as "the person you met") will not see you again until several years later, or that a person you met several times in the past but did not see frequently suddenly comes to visit you one day.

[0003] Then, the next time you meet someone you've met in the past, you may find that you've completely forgotten their information (such as their name), and when the conversation begins, they may be able to say their name, but you may not be able to say (remember) theirs, which can lead to an awkward situation.

[0004] In order to avoid the occurrence of the above-mentioned situation as much as possible, there is a method of recording information about many friends and associates in a user's notebook or other medium (whether paper or electronic). However, even if a user implements such a method, if there are a large number of people that the user has met in the past (in other words, information about people recorded in the notebook or other medium), the same problem as above may occur.

[0005] More specifically, the more information about people (people the user has met in the past) is recorded in a notebook, etc., the more the information about the recorded people gradually (for example, starting with the oldest information) is lost from the user's memory (conscious mind). For this reason, if a person the user has met in the past suddenly visits one day, the user may not be able to quickly recall the keywords about the person from their mind, which can lead to an awkward situation where the user is unable to say (remember) the other person's name when the conversation begins.

[0006] Recently, it has become common for users to carry around a mobile information terminal containing electronic information including a photograph of the user's face, and check the information of the person they are meeting with beforehand, thereby refreshing the user's memory and preparing for the meeting. A mobile information terminal storing such electronic information can be an effective tool when the person is known in advance. However, in a similar situation to the above, i.e., when a person whose electronic information, such as a photograph, is stored in the mobile information terminal suddenly visits the user and they meet face to face, the same problem as above can arise.

[0007] From another perspective, users of such tools must frequently check the information recorded in the tool and constantly refresh their memories so that they can quickly provide names and other information to unexpected visitors. However, this task becomes more tedious and time-consuming as the amount of recorded information increases. In addition, even if the number of unexpected visitors is not actually that high, many users may find the task of recalling information about a large number of people in preparation for the near future inefficient or even resistant to it.

[0008] In general, the above-mentioned portable information terminals are not considered to be effective tools because users would not be able to respond to unexpected visitors.

[0009] In recent years, advances in facial recognition technology and the widespread use of mobile information terminals equipped with small cameras have made it possible to identify people (interviewees) and obtain information about them.

[0010] For example, Patent Document 1 describes a technique relating to a method in which a user meets a person to be interviewed and acquires information about the person to be interviewed. [Prior art documents] [Patent documents]

[0011] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-106579 Summary of the Invention [Problem to be solved by the invention]

[0012] The method described in Patent Document 1 is carried out in the following procedure. (1) The user takes a picture of the person he or she is meeting with using a camera attached to a head mounted display (HMD), which is a portable information terminal. (2) By performing a facial image authentication process on the captured image, the facial image is identified and the person being interviewed is identified. (3) Obtain (obtain) information about the identified interviewee. (4) The acquired information about the interviewee is notified to the user.

[0013] However, with the technology described in Patent Document 1, the user needs to meet face-to-face with the person they are meeting, and after the meeting, information about the person they are meeting is acquired, resulting in a time lag before the information is acquired. In other words, with the technology described in Patent Document 1, the user needs to determine whether "this person is (likely) my person to meet." As a result, the user starts the meeting with no or insufficient information about the person, which can lead to inconveniences such as a lack of communication at the beginning of the meeting. In particular, if the person they are meeting recognizes information about the user (such as name and occupation) before the user does, in addition to the time lag described above, there is an additional lost time until the user recognizes "this person is (likely) my person to meet."

[0014] An object of the present invention is to provide a mobile information terminal and an information processing method that can provide information about an interviewee to a user more quickly. [Means for solving the problem]

[0015] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0016] A mobile information terminal according to a representative embodiment of the present invention determines whether a person around a user is a potential interviewee by analyzing the behavior of the person, and if it determines that the person is a potential interviewee, acquires additional information about the person. As a result, when the user recognizes the person as a potential interviewee, the user already knows the additional information about the interviewee. [Effects of the Invention]

[0017] The effects obtained by the representative inventions disclosed in this application can be briefly explained as follows.

[0018] That is, according to a representative embodiment of the present invention, the mobile information terminal automatically determines whether a person around the user is a potential interviewee, and if so, acquires additional information about that person, thereby enabling the user to more quickly receive information about the person. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram for explaining an overview of the present invention. [Figure 2] 1 is an external view showing an example of an HMD according to the first embodiment. [Figure 3] 1 is a system configuration diagram showing an example of the internal configuration of an HMD according to a first embodiment. [Figure 4] FIG. 2 is a functional block diagram showing an example of a functional block configuration according to the first embodiment. [Figure 5] 10 is a flowchart of new interviewee processing in the first embodiment. [Figure 6] 10 is a flowchart showing a subroutine of the face information detection process in the first embodiment. [Figure 7] 10 is an example of a table for storing interviewee information in the first embodiment. [Figure 8] 10 is a flowchart of interviewee identification and information acquisition processing in the first embodiment. [Figure 9] 10 is a flowchart showing a subroutine of interviewee information processing in the first embodiment. [Figure 10] FIG. 10 is a system configuration diagram showing an example of the internal configuration of an HMD 1 according to a second embodiment. [Figure 11] FIG. 10 is a functional block diagram showing an example of a functional block configuration according to a second embodiment. [Figure 12] 10 is a flowchart of new interviewee processing in the second embodiment. [Figure 13] 10 is a flowchart showing a subroutine of the voice information detection process in the second embodiment. [Figure 14] 10 is an example of a table for storing interviewee information in the second embodiment. [Figure 15] 10 is a flowchart of interviewee identification and information acquisition processing in the second embodiment. [Figure 16] 10 is a flowchart of interviewee information processing in the second embodiment. [Figure 17] FIG. 10 is a schematic diagram for explaining an overview of a third embodiment. [Figure 18] 13 is a flowchart of interviewee identification and information acquisition processing in the third embodiment. [Figure 19] 11 is a flowchart of processing audio information alone in the third embodiment. [Figure 20] FIG. 13 is an external view showing an example of an HMD used in the fourth embodiment. [Figure 21] 13 is a flowchart of interviewee identification and information acquisition processing in the fifth embodiment. [Figure 22A] 3 shows an example of information display in the first embodiment. [Figure 22B] 3 shows an example of information display in the first embodiment. [Figure 23] 13 is an example of information display in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Specific examples of embodiments to which the present invention is applied will be described in detail below with reference to the drawings. Each embodiment described below is an example for realizing the present invention, and does not limit the technical scope of the present invention. In the embodiments, components having the same function are given the same reference numerals, and repeated explanations thereof will be omitted unless particularly necessary.

[0021] <<Embodiment 1>> First, a first embodiment of the present invention will be described with reference to Figs. 1 to 3. Fig. 1 is a schematic diagram for explaining an overview of the first embodiment. Fig. 2 is an external view showing an example of an HMD in the first embodiment. Fig. 3 is a system configuration diagram showing an example of the internal configuration of the HMD in the first embodiment.

[0022] FIG. 1 shows a schematic view of a scene in which a person 15 is standing in front of a user 10 wearing a see-through HMD 1 in the shape of glasses. For ease of explanation, the see-through HMD 1 is shown at a distance from the user 10 in FIG. 1, but in reality, the user 10 is wearing the HMD 1 on the front of his or her head (where glasses are worn) and looking forward. In the following, the person 15 may also be referred to as the "interviewee 15."

[0023] The see-through HMD 1 of this embodiment is provided with a semi-transparent (transmissive) display screen 75 (display unit) at the lens position of glasses. The user can view the real space through the see-through display screen 75. The display screen 75 can also display an augmented reality AR object (interviewee information). Therefore, the wearer of the HMD 1 (user 10 in this example) can view both the augmented reality AR object (interviewee information) displayed on the display screen 75 and the situation in the real space at the same time.

[0024] 1 shows a state in which the interview partner 15 is not present in the line of sight 19 of the user 10, and the user 10 is not aware of the presence of the interview partner 15. More specifically, because the user 10 is wearing the HMD 1, the field of view of the real space outside the line of sight 19 of the user 10 is somewhat narrower than the field of view when viewed with the naked eye. In contrast, the interview partner 15 recognizes the presence of the user 10 before the user 10 recognizes the presence of the interview partner 15, and raises his right hand.

[0025] In this embodiment, for example, in the scene shown in Fig. 1, the HMD 1 acquires ambient information of the HMD 1 using an ambient information acquisition device immediately after startup. In one specific example, the "ambient information" is any one of video, distance measurement information, and audio, or a combination thereof.

[0026] 3, the ambient information acquisition device corresponds to the imaging unit 71 that acquires video of the surroundings of the HMD 1 (and thus the user 10 carrying the HMD 1), the distance measurement sensor 55 that acquires distance data (the distance between the user and an object), the sensor unit 5 that includes a human presence sensor 56 that detects the presence and approach of a person, and the audio input unit 81 such as a microphone that picks up (acquires) audio around the user. The HMD 1 recognizes people 15 that are present (or have appeared) around the user by analyzing the ambient information acquired through the ambient information acquisition device.

[0027] 1 according to the first embodiment shows an example in which only the imaging unit 71 is used as the ambient information acquisition device, and an image of the user's surroundings is acquired as ambient information by the imaging unit 71. In contrast, in a second embodiment described later, the imaging unit 71 and an audio input unit 81 are used as the ambient information acquisition device. In a third embodiment described later, only the audio input unit 81 is used as the ambient information acquisition device.

[0028] The HMD 1 determines whether the person 15 is a potential interview partner using the behavior analysis processing unit 74, and if it is determined that the person 15 is a potential interview partner, acquires additional information about the person 15. The acquired additional information is presented to the user through the information presentation unit.

[0029] This supplementary information is presented either visually or audibly, or both. When information is presented visually, the information presenting unit corresponds to the display screen 75 controlled by the display unit 72. On the other hand, when information is presented audio-wise, the information presenting unit corresponds to the audio output unit 82.

[0030] 1 shows an example in which supplementary information about a person 15 is displayed on a display screen 75. That is, in FIG. 1, a name 18 (in this example, "Yamada Taro") is displayed on the display screen 75 as supplementary information about the person 15.

[0031] In this specification, the terms "video" and "image" may refer to either moving images or still images.

[0032] As shown in Fig. 1, the HMD 1 is configured to be connectable to a network server 32 on a network 33. More specifically, the HMD 1 connects to an access point 31 by a communication processing unit 6, which will be described later in Fig. 2, and communicates with the network server 32 connected to the network 33 via the access point 31. The network server 32 shown in Fig. 1 includes various servers, such as a processing server that performs various types of calculation processing and a data server that stores various types of data. Therefore, the HMD 1 can utilize various external resources by communicating with the above servers as necessary.

[0033] 2 is an external view showing an example of the HMD 1 used in this embodiment. Display screens 75 are configured at the left and right lens positions of the glasses, and a right camera 711 is located at the edge of the right lens position of the glasses, and a left camera 712 is located at the edge of the left lens position of the glasses.

[0034] Although microphones are not shown, they are placed near the right camera 711 and the left camera 712. Furthermore, right speaker 821 and left speaker 822 are placed on the temples of the glasses.

[0035] The electronic components of the HMD 1, such as the circuits, are housed separately in a right housing 111 and a left housing 112.

[0036] Hereinafter, a specific method for realizing the problem in the present disclosure, that is, providing information about the person being interviewed to the user more quickly, will be described in more detail with reference to the drawings.

[0037] [HMD system configuration example] The main body of the HMD 1 used in the present invention is made up of various blocks as described below.

[0038] Fig. 3 is a system configuration diagram showing an example of the internal configuration of the above-mentioned HMD 1. As shown in Fig. 3, the HMD 1 is configured to include a main control unit 2, a system bus 3, a storage unit 4, a sensor unit 5, a communication processing unit 6, a video processing unit 7, an audio processing unit 8, and an operation input unit 9.

[0039] The main control unit 2 is a microprocessor unit that controls the entire HMD 1 according to a predetermined operation program. The system bus 3 is a data communication path for transmitting and receiving various commands and data between the main control unit 2 and each component block within the HMD 1.

[0040] The memory unit 4 is composed of a program section 41 that stores programs for controlling the operation of the HMD1, a data memory section 42 that stores various data such as operation setting values, detection values ​​from sensors, objects including content, and library information downloaded from libraries, and a rewritable program function section 43 that stores work areas used for various program operations.

[0041] The storage unit 4 can also store operation programs downloaded from a network and various data created by the operation programs. It can also store content such as moving images, still images, and audio downloaded from a network. It can also store data such as moving images and still images captured using the camera's photography function. The storage unit 4 can also store necessary information (setting values ​​such as thresholds, image data, etc.) in advance.

[0042] Furthermore, the storage unit 4 needs to retain the stored information even when no external power is supplied to the HMD 1. Therefore, the storage unit 4 may be, for example, a semiconductor memory such as a flash ROM or an SSD (Solid State Drive), or a magnetic disk drive such as an HDD (Hard Disk Drive). Note that the operating programs stored in the storage unit 4 can be updated and their functions expanded by downloading them from server devices on the network.

[0043] The sensor unit 5 is a group of various sensors (in other words, a "sensor device") for detecting the state of the HMD 1. The sensor unit 5 is composed of a GPS (Global Positioning System) receiving unit 51, a geomagnetic sensor 52, an acceleration sensor 53, a gyro sensor 54, a distance measurement sensor 55, a human presence sensor 56, and the like.

[0044] The sensor unit 5 can detect the position, tilt, direction, movement, etc. of the HMD 1 through the various sensors described above, and can also measure the distance to an object (an interviewee or various other objects). Therefore, the sensor unit 5 constitutes a part of a surrounding information acquisition device that acquires surrounding information including information about the interviewee.

[0045] Of the above, the distance sensor 55 is, for example, an optical ToF (Time of Flight) type, and measures the distance to objects (people and their accessories (e.g., glasses, hats, canes, flags, clothes, masks, etc.), as well as buildings, roads, etc.) around the HMD 1 and the user 10 (hereinafter, for simplicity, this may be simply referred to as the surroundings).

[0046] For simplicity, the "surroundings of the HMD 1 and the user 10" may be simply referred to as the "surroundings" below.

[0047] The human sensor 56 is, for example, an infrared type, and can selectively sense a person (human) among the various objects present in the surroundings.

[0048] Additionally, the GPS (Global Positioning System) receiver 51 can acquire the location of the HMD 1, in other words, the location where the surrounding information is acquired, by acquiring current location information using satellite communication. Alternatively, other systems, for example, other systems within the GNSS (Global Navigation Satellite System), may be used to acquire the current location information.

[0049] The sensor unit 5 may further include other sensors, such as detection or measurement devices such as an illuminance sensor or an altitude sensor, and these sensors may also be components of the surrounding information acquisition device.

[0050] The communication processing unit 6 is a communication device including a LAN (Local Area Network) communication unit 61, a telephone network communication unit 62, etc. Among these, the LAN communication unit 61 is connected to a network 33 such as the Internet (see FIG. 1 as appropriate) via an access point 31, etc., and transmits and receives data to and from each network server 32 on the network 33. The LAN communication unit 61 and the access point 31, etc. are connected by wireless communication such as Wi-Fi (registered trademark).

[0051] The main control unit 2 can cause an external server (network server 32) to perform at least a part of the characteristic processing performed by the HMD 1 via the communication processing unit 6 (communication device).

[0052] The telephone network communication unit 62 performs telephone communication (calls) and transmission and reception of data through wireless communication with base stations of a mobile telephone communication network, etc. Communication with base stations, etc. may be performed using the LTE (Long Term Evolution) method, the 5G method (a fifth-generation mobile communication system aiming for high speed, large capacity, low latency, and multiple simultaneous connections), or other communication methods.

[0053] The LAN communication unit 61 and the telephone network communication unit 62 each include an encoding circuit, a decoding circuit, an antenna, etc. Furthermore, the communication processing unit 6 may further include other communication units such as an infrared communication unit.

[0054] The video processing unit 7 includes an imaging unit 71 , a display unit 72 , a face information processing unit 73 , and a behavior analysis processing unit 74 .

[0055] The imaging unit 71 is a camera that inputs image data (video) of the surroundings or an object by converting light input from a lens into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor. In this embodiment, the imaging unit 71 includes a right camera 711, a left camera 712, etc.

[0056] The imaging unit 71 (right camera 711, left camera 712) constitutes a part (imaging device or image acquisition device) of a surrounding information acquisition device that acquires surrounding information including information about the person being interviewed.

[0057] The display unit 72 is, for example, a transmissive display device (liquid crystal display device) using semi-transparent liquid crystal. The display unit 72 forms a display screen 75 (see FIGS. 22A, 22B, and 23 as appropriate) and provides supplementary information about the person being interviewed to the user 10 of the HMD 1.

[0058] The face information processing unit 73 is a processing unit that extracts face information from the video of the interviewee captured by the imaging unit 71. Details of the processing that is mainly executed by the face information processing unit 73 will be described later.

[0059] The behavior analysis processing unit 74 is a processing unit that analyzes the behavior of a person from the video of the person captured by the imaging unit 71 and the distance to the person measured by the distance measurement sensor 55. Details of the processing that is mainly executed by the behavior analysis processing unit 74 will be described later.

[0060] In one specific example, the face information processing unit 73 and the behavior analysis processing unit 74 are each configured by separate processors. As another example, these processing units 73 and 74 may be configured by the same processor.

[0061] The audio processing unit 8 is made up of an audio input unit 81 and an audio output unit 82.

[0062] The audio input unit 81 is a microphone (hereinafter sometimes abbreviated as a microphone) that converts sounds in the real space, the user's voice, etc. into audio data and inputs the audio data. In this embodiment, a microphone is disposed near each of the right camera 711 and the left camera 712.

[0063] The voice input unit 81 constitutes a part (a voice pickup device or a voice acquisition device) of a surrounding information acquisition device that acquires surrounding information including information about the person being interviewed.

[0064] The audio output unit 82 is a speaker that outputs audio information required by the user. In this embodiment, the audio output unit 82 has a right speaker 821 and a left speaker 822 arranged near the user's ears. Although not shown, the audio output unit 82 may also include a wired or wireless terminal for connecting an external audio output device such as earphones or headphones. With an HMD 1 configured in this way, the audio output method or path can be appropriately selected depending on the application, etc.

[0065] The operation input unit 9 is a hardware device equipped with key switches and the like for inputting operation instructions and the like to the HMD 1, and outputs an operation input signal to the main control unit 2 in accordance with the operation content (input instruction) of the user.

[0066] In the present disclosure, the operation input unit 9 and the main control unit 2 function as a setting unit or a setting processing device for setting characteristic functions (e.g., ambient information acquisition, behavior analysis processing, information presentation, etc.) of the HMD 1. The display unit 72 described above may be included as another component of the setting unit or setting processing device.

[0067] 3 includes components that are less relevant to solving the above-described problems. Therefore, even if the configuration does not include components that are less relevant to solving the problems, the unique effects of this embodiment are not impaired. Furthermore, components not shown, such as an electronic money payment function, may be further added.

[0068] [Functional blocks of this embodiment] 4 is a functional block diagram showing an example of the functional block configuration of the HMD 1 of this embodiment. The control function 21 is a function for controlling the entire HMD 1, and as shown in FIG. 4, is mainly composed of a main control unit 2, and a program unit 41 and a program function unit 43 of the storage unit 4.

[0069] Of the above, the communication processing function 22 is a function that performs communication processing to connect to the network 33 using the LAN communication unit 61 of the communication processing unit 6 and the telephone network communication unit 62 of the communication processing unit 6 (see also Figures 1 and 3 as appropriate).

[0070] The photographed data acquisition function 23 is a function for photographing the interviewee using the imaging unit 71 (right camera 711, left camera 712) of the video processing unit 7 and acquiring photographed data.

[0071] The face information processing function 24 is a function for identifying the interviewee by analyzing face information from the video of the interviewee acquired by the photographic data acquisition function 23 using a face information processing unit 73. Details of face information processing will be described later.

[0072] The face information storage function 25 is a function for storing face information for identifying the interviewee obtained by the face information processing function 24 in the data storage unit 42 of the storage unit 4.

[0073] The interviewee information storage function 26 is a function for storing additional information about the interviewee in the data storage unit 42 of the storage unit 4.

[0074] The interviewee information output function 27 is a function that reads out additional information about the interviewee stored in the interviewee information storage function 26 and displays it on the display unit 72 of the video processing unit 7.

[0075] The behavior analysis processing function 30 is a function that analyzes the behavior of a person using the behavior analysis processing unit 74 based on the image of the person acquired by the photographing data acquisition function 23 and the distance between the person and the image acquired by the distance measurement data acquisition function 1000, and determines whether or not the person is a candidate for an interview. Details of the interview candidate determination process will be described later.

[0076] [Processing procedure of this embodiment] 5 is a flowchart showing the procedure of new interviewee processing (step S400) for acquiring information on a new interviewee in this embodiment. The processing procedure shown in FIG. 5 will be described below with reference to the functional block diagram of FIG.

[0077] Note that, in order to protect personal information, the new interviewee process (step S400) is performed on the premise that the consent of the new interviewee has been obtained in advance.

[0078] The new interviewee processing (step S400) shown in Figure 5 is executed, for example, by the following procedure. That is, after the start-up processing (step S401), such as starting up the software and resetting the memory, the new interviewee is first photographed (step S402). This corresponds to pre-processing for acquiring facial information from the video of the new interviewee.

[0079] Specifically, in step S402, the imaging unit 71 of the video processing unit 7 operates under the control of the main control unit 2 to capture an image of the background or subject in front of the user of the HMD 1. The following description is based on the assumption that a new interviewee is among the subjects in front of the user.

[0080] Next, a face information detection process (step S420), which is a predefined process (subroutine), is performed. This face information detection process (step S420) is a process for acquiring face information of the new interviewee. Specifically, in step S420, under the control of the main control unit 2, the face information processing unit 73 of the video processing unit 7 analyzes the image of the subject captured in step S402 to acquire face information of the new interviewee. This process obtains face information that identifies the new interviewee.

[0081] Here, the processing procedure of step S420 (face information detection processing) will be described in more detail with reference to Fig. 6. Fig. 6 is a flowchart showing the processing procedure of a subroutine of the face information detection processing (step S420).

[0082] The processing procedure shown in Fig. 6 will be explained below with reference to the block diagram of Fig. 3 and the functional block diagram of Fig. 4. In order to perform the face information processing function 24, the face information processing unit 73 reads out and sequentially executes the programs for the face recognition method stored in the program unit 41 of the storage unit 4.

[0083] Specifically, after the start-up process (step S421) such as software startup and memory reset, the face information processing unit 73 first performs a process to detect the face contour of the new interviewee in the captured frame using a face contour detection program (step S422).

[0084] In the following step S423, the face information processing unit 73 determines whether or not the face contour of the new interviewee has been detected by the face contour detection process (step S422).

[0085] Here, if the face information processing unit 73 determines that the face contour of the new interviewee has not been detected (step S423: NO), it proceeds to a face detection error setting process (step S428) in which a face detection error is set.

[0086] On the other hand, if the face information processing unit 73 determines that the face contour of the new interviewee has been detected (step S423: YES), the process proceeds to face element detection processing (step S424).

[0087] In the face element detection process (step S424), the face information processing unit 73 performs a process of detecting face elements such as the eyes, nose, and mouth inside the face outline using a face element detection program.

[0088] In the following step S425, the face information processing unit 73 determines whether or not the face elements of the new interviewee have been detected by the face contour detection process (step S424).

[0089] Here, if the face information processing unit 73 determines that the face elements of the new interviewee have not been detected by the face contour detection process (step S424) (step S425: NO), it proceeds to a face detection error setting process (step S428) in which a face detection error is set.

[0090] On the other hand, if the face information processing unit 73 determines that the face elements of the new interviewee have been detected by the face contour detection process (step S424), the process proceeds to the next face feature amount detection process (step S426).

[0091] In the face information feature amount detection process (step S426), the face information processing unit 73 performs a process of detecting face feature amounts such as the size and position of each element and the positional relationship between elements using a face feature amount detection program.

[0092] In the following step S427, the face information processing unit 73 determines whether or not the face feature amount of the new interviewee has been detected by the face information feature amount detection process (step S426).

[0093] Here, if the face information processing unit 73 determines that the face features of the new interviewee have not been detected by the face information feature detection process (step S426) (step S427: NO), it proceeds to a face detection error setting process (step S428) in which a face detection error is set.

[0094] On the other hand, if the face information processing unit 73 determines that the face elements of the new interviewee have been detected by the face information feature detection process (step S426) (step S427: YES), it terminates the face information detection process (step S420) (step S429).

[0095] Furthermore, in the face detection error setting process (step S428), the face information processing unit 73 clearly indicates at what stage the face detection error occurred, and ends the face information detection process (step S420) (step S429).

[0096] As another example, the face information detection process (step S420) described above may be performed by the network server 32. In this case, the main control unit 2 of the HMD 1 controls the communication processing unit 6 to transmit the video of the new interviewee captured by the video processing unit 7 (imaging unit 71) to the network server 32 that performs the face information detection process via the network 33. Subsequently, the main control unit 2 of the HMD 1 receives (only) the face information detection result executed by the network server 32 from the network server 32 via the network 33.

[0097] Returning now to the explanation of the processing procedure of the flowchart in Fig. 5, in step S403 following the face information detection process (step S420) described above, the face information processing unit 73 determines whether or not face information of the new interviewee has been acquired by the face information detection process (step S420).

[0098] Here, if the face information processing unit 73 determines that the face information of the new interviewee cannot be obtained through the face information detection processing (step S420) (step S403: NO), it determines that there is no face information to save and proceeds to new interviewee information processing (step S405).

[0099] On the other hand, if the face information processing unit 73 determines that the face information of the new interviewee has been acquired through the face information detection process (step S420) (step S403: YES), the process proceeds to face information storage process (step S404).

[0100] In the face information storage process (step S404), the face information processing unit 73 executes the face information storage function 25 (see FIG. 4) to store the face features of the interviewee related to the face identification of the interviewee in the data storage unit 42 of the storage unit 4. Next, the face information processing unit 73 proceeds to the new interviewee information acquisition process (step S405).

[0101] The new interviewee information acquisition process (step S405) is a process for acquiring additional information about the new interviewee. In this step S405, the face information processing unit 73 performs a process for acquiring additional information about the new interviewee, such as the name and age of the new interviewee.

[0102] Next, the face information processing unit 73 determines whether or not the additional information regarding the new interviewee has been acquired by the new interviewee information acquisition process (step S405) (step S406).

[0103] If the face information processing unit 73 determines that the additional information regarding the new interviewee has not been acquired (step S406: NO), it ends the new interviewee processing (step S400) shown in FIG. 5 (step S408).

[0104] On the other hand, if the face information processing unit 73 determines that the additional information about the new interviewee has been acquired (step S406: YES), it proceeds to step S407. In step S407, the face information processing unit 73 executes the interviewee information storage function 26 (see FIG. 4) to store the acquired additional information about the new interviewee in the data storage unit 42 of the storage unit 4, and then ends the new interviewee processing (step S400) shown in FIG. 5 (step S408).

[0105] If the additional information about the new interviewee is stored in the network server 32, the HMD 1 can also acquire the additional information about the new interviewee from the network server 32 via the network 33 under the control of the main control unit 2. In this case, the face information processing unit 73 stores the additional information about the new interviewee acquired from the network server 32 in the data storage unit 42 of the storage unit 4, and then terminates the new interviewee processing (step S400) (step S408).

[0106] FIG. 7 is a table showing an example of interviewee information stored in the data storage unit 42 of the storage unit 4.

[0107] The interviewee table (T840) shown in Figure 7 is configured to correspond to a person column 860 indicating the type of person (such as interviewee) and an item column 850 indicating information (items) about the user and each interviewee (interviewee 1, interviewee 2, ... interviewee n).

[0108] Of the above, the item column 850 is made up of two items: face information 851 and additional information 852 relating to the interviewee.

[0109] Meanwhile, in addition to the interviewees (862 to 864), a user 861 is registered in the person column 860. The presence of the user 861 in the person column 860, which essentially indicates the type of interviewee, is like a profile on a mobile phone, metaphorically speaking. Furthermore, the face information 851 of the user (861) can be obtained, for example, by taking a photo using a mirror (in which case a left-right reversed image is obtained) or by taking a so-called "selfie."

[0110] The interviewee table (T840) as shown in Figure 7 can also be stored in the network server 32 via the network 33. In particular, with regard to facial features, the process of identifying interviewees can be sped up by utilizing the network server 32 that specializes in the process of identifying interviewees from facial features.

[0111] In this embodiment, by using the processing described above in Figures 5 and 6 and a table with a data structure such as that shown in Figure 7, facial information of a new interviewee and additional information about the new interviewee can be obtained and stored in HMD1.

[0112] [Interviewee identification and information acquisition processing] Next, the process of identifying an interviewee and acquiring additional information about the interviewee will be described. Fig. 8 is a flowchart showing the procedure of the process (interviewee identification / information acquisition process) that determines whether or not a person can be an interviewee, and if so, acquires additional information about the person in advance, which is the gist of this embodiment.

[0113] The processing procedure in Fig. 8 will be described with reference to the block diagram in Fig. 3 and the functional block diagram in Fig. 4. In the following description, the main control unit 2 will be described as the subject of each process shown in Fig. 8, but some or all of the processes may be performed by the behavior analysis processing unit 74.

[0114] When the HMD 1 is started up, the main control unit 2 immediately starts processing to identify the interviewee, acquire information, and the like (step S431).

[0115] When the interviewee identification and information acquisition process is started (step S431), the main control unit 2 first performs surroundings photographing process (step S432). This is a process of photographing the surroundings of the HMD 1, i.e., the environment (scenery or scenery) around the user 10, using the photographing data acquisition function 23, and acquiring the photographing data.

[0116] Here, the captured image data to be acquired may be either video or still images. When the captured image data is video, it is expected that the accuracy of behavior analysis will be higher compared to when it is a still image. On the other hand, when the captured image data is a still image, it is expected that the power consumption of the HMD 1 will be reduced compared to when it is a video. Note that, from the viewpoint of ensuring a certain level of accuracy in behavior analysis, when the captured image data is a still image, it is preferable to capture images, i.e., acquire still images, at predetermined intervals.

[0117] In order to execute the above processing, the main control unit 2 controls the video processing unit 7 to start capturing images using the imaging unit 71. At this time, the video processing unit 7 captures images using the imaging unit 71 (camera), analyzes the captured images using the face information processing unit 73 and the behavior analysis processing unit 74, and outputs the analysis results to the main control unit 2.

[0118] In step S433 after receiving the analysis result from the video processing unit 7, the main control unit 2 determines whether or not a person is present in the acquired image (hereinafter referred to as "photographed data").

[0119] Here, if the main control section 2 determines that no person is present in the photographic data (step S433, NO), the process proceeds to an end instruction determination process in step S434.

[0120] On the other hand, if the main control unit 2 determines that a person is present in the photographed data (step S433, YES), the process proceeds to the interview candidate determination process of step S900.

[0121] In step S900 (interview candidate determination process), the main control unit 2 determines whether a person around the user is a potential interviewee. This determination is based on the behavior analysis results of the person in the photographed data by the behavior analysis processing unit 74, and determines that the person is a potential interviewee if their attention is directed toward the user. For this determination, additional video or still images may be taken.

[0122] Here, the conditions for determining that a person's attention is directed to the user include, for example, the following actions (person's behavior): (Condition 1) The person's line of sight is in the direction of the user. (Condition 2) A person greets the user by raising their hand or other such gesture. (Condition 3) A person is approaching the user. (Condition 4) A person calls the user by name or mentions the name of the person (or the name of the company or organization to which the person belongs).

[0123] Of the above, condition 4 is an action based on a person's vocalizations (voice), so it is not necessarily easy to acquire (extract) it from an image. More specifically, for example, by analyzing the lip movements of a person in a video image, it is possible to estimate what the person said (vocal sounds). On the other hand, in the current social situation, there has been an increase in the number of people wearing masks to prevent infection with various diseases (e.g., the new coronavirus), and in this case, it is thought that analyzing a person's lip movements will be difficult.

[0124] Considering the above circumstances, conditions 1 to 3 will be mainly considered in the first embodiment, and condition 4 will be explained in the second and third embodiments.

[0125] The above-mentioned conditions 1 to 3 are behaviors based on the physical movements of a person, and can generally be defined as "behaviors showing interest in the user." Therefore, when the behavior of a person included in the photographed data (ambient information) is behavior showing interest in the user 10, the main control unit 2 (behavior analysis processing device) determines that the person can be a potential interviewee (step S900, YES).

[0126] In the following, potential interviewees will be referred to as "interview candidates" as appropriate.

[0127] Note that the above-mentioned conditions 1 to 3 are merely examples of "behaviors that show interest in the user," and in actual operation, various other conditions (modes of a person's behavior) can be added.

[0128] Furthermore, as an exceptional process (criterion for judgment), the main control unit 2 may judge that the person is an interview candidate (step S900, YES) regardless of the above conditions when the distance between the user and a person is closer than a predetermined certain distance, for example, based on the detection result of the above-mentioned human presence sensor 56. This is because, for example, if the user 10 is wearing a mask, the other person may not realize the identity of the user 10 until they get close to the user 10.

[0129] As another exceptional processing (criterion for judgment), the main control unit 2 may not select a person as an interview candidate even if their attention is directed toward the user (for example, even if all of conditions 1 to 3 are met), i.e., may perform processing to judge NO in step S900.

[0130] This is in consideration of cases where, for example, a store clerk, a receptionist at a company or other facility attendant, or a security guard, etc., is only paying attention to the user as part of their professional duties. More specifically, no additional information is usually registered for these people, and performing a process to acquire additional information for these people could prevent the acquisition of additional information for people who truly need it.

[0131] Also, from the viewpoint of prioritizing the acquisition of supplementary information for truly necessary people, the process shown in Fig. 8 may not be performed (the function may be automatically stopped) in specific places such as home where the user only encounters people he or she knows. In this case, the main control unit 2 may determine whether or not the location is a "specific location" based on information received by the GPS receiving unit 51 (see Fig. 3).

[0132] Furthermore, from the viewpoint of prioritizing acquisition of supplementary information for truly necessary people, people who are frequently met, such as family members, may be excluded from the processing targets of steps S420 and S450 (setting of so-called "excluded people"). Also, for people who have undergone the processing of steps S420 and S450 once, the processing of steps S420 and S450 may not be performed for a certain period of time (setting of so-called "display suspension period").

[0133] The above-mentioned various exceptional processing settings (so-called exclusion settings) can be made by, for example, the user operating the operation input unit 9. By performing the above-mentioned various exclusion settings or exclusion processing, it is possible to suppress the presentation of unnecessary information, which contributes to the quick acquisition of supplementary information for truly necessary people, thereby improving convenience.

[0134] If the main control unit 2 determines in the interview candidate determination process of step S900 that the people in the vicinity are not interview candidates, the process proceeds to the end instruction determination process of step S434.

[0135] In step S434, the main control unit 2 monitors an input signal from, for example, the operation input unit 9, and determines whether or not the user 10 or the like has instructed to end the processing of this embodiment.

[0136] Here, if the main control unit 2 determines that an instruction to end the process has been issued (step S434: YES), it ends the routine (interview candidate determination and information acquisition process) shown in FIG. 8 (step S436).

[0137] On the other hand, if the main control unit 2 determines that the end of the process has not yet been instructed (step S434: NO), the process returns to the surroundings imaging process (step S432) of imaging the surroundings of the HMD 1 to continue the routine shown in FIG. 8.

[0138] Thus, if the main control unit 2 determines in the interview candidate determination process of step S900 that a person in the surrounding area is an interview candidate (step S900: YES), it performs a face information detection process (step S420), which is a predefined process (subroutine).

[0139] Note that even before acquiring the interview candidate information, the main control unit 2 may display 1100 (in the illustrated example, a message displaying "An interview candidate is present") indicating that the mobile information terminal has recognized the interview candidate, as shown in FIG. 22A. In this case, the main control unit 2 may superimpose a mark 1101 indicating the interview candidate to inform the user of the interview candidate's location. In the illustrated example, the mark 1101 displays a shape surrounding the person 15 who is the interview candidate, but it may also be another shape, such as an "arrow" pointing to the person 15.

[0140] Furthermore, details of the face information detection process (step S420) have been explained in the flowchart of FIG. 6, so explanation thereof will be omitted here.

[0141] After completing the face information detection process (step S420), the main control unit 2 performs interviewee information processing (step S450), which is a predefined process (subroutine). This interviewee information processing (step S450) is a process for identifying the interviewee and acquiring additional information about the interviewee.

[0142] The main control unit 2 waits for the interviewer information process (step S450) to end, and then ends the interviewer identification / information acquisition process of this embodiment (step S436).

[0143] Here, the specific contents of the interviewee information processing (the processing of the subroutine of step S450) will be described. Figure 9 is a flowchart showing the processing procedure of the interviewee information processing subroutine (step S450). The processing procedure shown in Figure 9 will be described with reference to the hardware block diagram of Figure 3 and the functional block diagram of Figure 4 as appropriate.

[0144] When the processing of step S450 (interviewee information processing) starts (step S451), the main control unit 2 first determines whether the face information detected in the face information detection processing (step S420) is face information of a known interviewee (step S452).

[0145] In this example, the main control unit 2 compares the facial information (facial features) detected in the facial information detection process (step S420) with the facial information (facial features) stored by the facial information storage function 25, and if they are very similar (for example, if the degree of similarity of the facial contours (contours) is within a preset threshold), it determines the person as a known interviewee. Note that in recent years, as the number of people wearing masks to prevent infectious diseases, etc. has increased, for mask wearers, the main control unit 2 determines whether the degree of similarity of the facial contours (contours) of the parts other than the mask is within the above threshold.

[0146] If the main control unit 2 judges NO in step S452, i.e., if the detected facial information (facial features) does not match the stored facial information (facial features), or if the facial information detection process (step S420) does not detect facial information sufficient to identify the person, it determines that the person is not a known interviewee and proceeds to step S400.

[0147] On the other hand, if the determination in step S452 is YES, i.e., if the detected face information (facial feature amount) matches the stored face information (facial feature amount), the main control unit 2 proceeds to step S453. In step S453, the main control unit 2 acquires additional information about the known interviewee stored by the interviewee information storage function 26, and proceeds to step S454.

[0148] In step S454, the main control section 2 determines whether or not the supplementary information relating to the known interviewee needs to be corrected.

[0149] Here, if the main control section 2 determines that the supplementary information relating to the known interviewee does not need to be corrected (step S454: NO), the process proceeds to the interviewee information output process (step S457).

[0150] On the other hand, if the main control unit 2 determines that the supplementary information related to the known interviewee needs to be corrected (step S454: YES), it proceeds to a process of saving corrected interviewee information (step S455) to save the corrected interviewee information.

[0151] In the interviewee modified information saving process (step S455), the main control unit 2 modifies the interviewee information saved in the interviewee information saving function 26 and saves the modified interviewee information. After completing the interviewee modified information saving process (step S455), the main control unit 2 proceeds to the interviewee information output process (step S457).

[0152] On the other hand, if the determination process in step S452 determines that the interviewee is not a known interviewee (step S452: NO), there is no information about the interviewee, so new information about the interviewee must be acquired. Therefore, the main control unit 2 performs the new interviewee processing (step S400) of this embodiment. Details of the new interviewee processing (step S400) have been explained in the flowchart of Figure 5, so explanation will be omitted here.

[0153] Next, the main control unit 2 determines whether or not interviewee information has been obtained in the new interviewee process (step S400) (step S456).

[0154] Here, if the main control unit 2 determines that new interviewee information has been obtained (step S456: YES), it proceeds to the interviewee information output process (step S457). Note that if additional information about the interviewee has been obtained in the new interviewee process (step S400), the new interviewee information has already been saved, so it is possible to proceed to the interviewee information output process (step S457).

[0155] In the interviewee information output process (step S457), the main control unit 2 outputs additional information about the interviewee to the outside using the interviewee information output function 27. In this embodiment, the main control unit 2 displays and outputs interviewee information 1102 on the display unit 72 of the video processing unit 7 (see FIG. 22B).

[0156] After the interviewee information output process (step S457) is completed, the main control unit 2 ends the interviewee information process (step S450) (step S458). Also, if the main control unit 2 determines in the determination process of step S456 that interviewee information has not been obtained, the main control unit 2 also ends the interviewee information process (step S450) (step S458).

[0157] In this embodiment, facial information of the new interviewee and additional information about the new interviewee are acquired and stored in advance. Then, before the user recognizes the person, the main control unit 2 determines whether a person in the vicinity is a potential interviewee by analyzing the person's behavior, and if it determines that the person is a potential interviewee, it presents the user 10 with text information about the person, such as their name, as additional information by displaying it on the display screen 75 (see FIG. 22B, etc.).

[0158] As another example, the additional information to be saved and presented may be graphic information such as illustrations, or audio information using the output from the right speaker 821 and the left speaker 822.

[0159] Thus, the HMD 1 (portable information terminal) of embodiment 1 comprises an ambient information acquisition device (sensor unit 5, imaging unit 71, audio input unit 81) that acquires ambient information about the terminal and the user 10, a behavior analysis processing device (main control unit 2, behavior analysis processing unit 74) that determines whether there is an interview candidate for the user 10 (a person who is about to interview the user) by analyzing the behavior of people included in the acquired ambient information, and an information presentation device (display unit 72) that, if it is determined that there is an interview candidate, presents additional information corresponding to that person to the user 10.

[0160] With this HMD1, additional information about the person being interviewed can be provided to the user 10 more quickly, and by the time the user 10 recognizes the person as the person being interviewed, the user 10 already knows the additional information about the person.

[0161] Therefore, the HMD1 disclosed herein can effectively prevent the time lag problem that was an issue with conventional devices, i.e., problems such as starting an interview with no or insufficient information about the other party, resulting in inconveniences such as a lack of communication at the beginning of the meeting.

[0162] <<Embodiment 2>> The following describes embodiment 2 of the present disclosure. Note that the basic hardware configuration and software configuration of embodiment 2 are the same as those of embodiment 1 described above, and the following mainly describes the differences between this embodiment (embodiment 2) and embodiment 1 described above, and omits explanations of common parts as much as possible to avoid duplication.

[0163] In the first embodiment described above, interview candidates are identified using facial information of the interviewee. In contrast, in this embodiment, interview candidates are identified by taking into account the person's voice information. This embodiment will be described below.

[0164] [System Configuration Example of Second Embodiment] Fig. 10 is a system configuration diagram showing an example of the internal configuration of the HMD 1 used in this embodiment. The system configuration diagram shown in Fig. 10 is almost the same as the system configuration diagram in Fig. 3, with an audio information processing unit 83 added to the system configuration diagram in Fig. 3. Here, the configuration of the audio information processing unit 83 will be mainly described.

[0165] The voice information processing unit 83 performs a function of performing processing to extract voice information from the voice of the interviewee input from the voice input unit 81. In one specific example, the voice information processing unit 83 uses a hardware processor separate from the main control unit 2, and performs the above function under the control of the main control unit 2. Details of the processing performed by the voice information processing unit 83 will be described later.

[0166] [Functional blocks of this embodiment] FIG. 11 is a functional block diagram showing an example of the functional block configuration of the HMD 1 according to the present embodiment.

[0167] The functional block diagram shown in Fig. 11 is almost the same as the functional block diagram shown in Fig. 4 already explained, but is obtained by adding a voice information processing function 28 and a voice information storage function 29 to the functional block diagram of Fig. 4. The added voice information processing function 28 and voice information storage function 29 will be described below.

[0168] The voice information processing function 28 is a function that analyzes voice information from the voice of the interviewee input from the voice input unit 81 using the voice information processing unit 83 and identifies the interviewee, and is one of the functions performed by the voice information processing unit 83 described above in Figure 10.

[0169] The voice information storage function 29 is a function for storing voice information for identifying the interviewee obtained by the voice information processing function 28 in the data storage unit 42 of the storage unit 4.

[0170] [Processing procedure of the second embodiment] 12 is a flowchart showing the procedure for new interviewee processing (step S460) in this embodiment for acquiring information that takes into account the voice information of a new interviewee. The processing procedure in FIG. 12 will be described below with reference to the functional block diagram in FIG. 11.

[0171] When executing this new interviewee process (step S460), it is desirable to obtain the new interviewee's consent in advance from the viewpoint of protecting personal information. However, such consent does not imply any technical restrictions.

[0172] The flowchart showing the steps of the new interviewee processing (step S460) in Figure 12 is almost the same as the flowchart showing the steps of the new interviewee processing (step S400) in Figure 5. The difference is that it adds a predefined subroutine, a voice information detection processing (step S470), a judgment processing (step S462) that judges the detection results of the voice information detection processing (step S470), and a storage processing (step S463) that saves the voice information obtained in the voice information detection processing (step S470). Here, only the processing added in Figure 12 will be explained.

[0173] When the new interviewee process (step S460) in this embodiment is started (step S461), the same process as in FIG. 5 (steps S402 to S404) is executed, and the process related to face information is completed.

[0174] Here, the processing of step S470 (audio information detection processing), which is a subroutine, will be described. Fig. 13 is a flowchart showing the processing procedure of the audio information detection processing (step S470), which is a subroutine. The processing procedure of Fig. 13 will be described below with reference to the functional block diagram of Fig. 11.

[0175] In order to perform the functions of the voice information processing function 28, the voice information processing unit 83, under the control of the main control unit 2, reads out a program for a voice recognition method stored in the program section 41 of the memory unit 4 (step S471), and sequentially executes the processes from step S472 onwards.

[0176] When the process of step S470 (audio information detection process) starts, first, the audio information processing unit 83 determines whether or not a sound has been detected (step S472).

[0177] If the audio information processing unit 83 determines that no sound has been detected (step S472: YES), it proceeds to audio detection error setting processing (step S477). On the other hand, if the audio information processing unit 83 determines that sound has been detected (step S472: NO), it proceeds to step S473 (sound source separation processing).

[0178] In step S473 (sound source separation process), the voice information processing unit 83 checks the direction of sound generation and identifies (separates) the position of the sound source. In this embodiment, the position of the mouth of the new interviewee uttering the voice is the sound source position.

[0179] In the next step S474, the audio information processing unit 83 determines whether the sound whose sound source has been identified (separated) is a human voice. Whether the sound is a human voice can be determined (identified) from, for example, the frequency band of the sound, waveform characteristics, etc. Such technology is well known, and therefore a detailed description thereof will be omitted.

[0180] If the voice information processing unit 83 determines that the voice is not human (step S474, NO), it proceeds to voice detection error setting processing (step S477). On the other hand, if the voice information processing unit 83 determines that the voice is human (step S474, YES), it proceeds to voice feature detection processing (step S475).

[0181] In the speech feature detection process (step S475), the speech information processing unit 83 extracts elements attributable to an individual (such as speaking style, habits, intonation, etc.) as speech features. Note that other methods may be used as long as they can identify the characteristics of an individual (for example, identifying a specific person when a specific, rare language is extracted).

[0182] In the following step S476, the voice information processing unit 83 determines whether or not an individual feature (voice feature in this example) has been detected from the processing result of the voice feature detection process (step S475).

[0183] Here, if the audio information processing unit 83 determines that an audio feature has not been detected (step S476, NO), the process proceeds to audio detection error setting processing (step S477).

[0184] On the other hand, if it is determined that the audio feature has been detected (step S476, YES), the audio information processing unit 83 ends this audio information detection process (step S470) (step S478).

[0185] In the voice detection error setting process (step S477), the voice information processing unit 83 displays at what stage the voice detection error occurred on the display unit 72. Thereafter, the voice information detection process (step S470) ends (step S478).

[0186] As another example of this voice information detection process (step S470), the HMD 1 can transmit the acquired voice of the new interviewee via the network 33 to the network server 32, which performs voice information detection processing, and the voice information detection processing can be performed by the network server 32. In this case, under the control of the main control unit 2, the communication processing unit 6 of the HMD 1 described above receives only the voice information detection results from the network server 32 via the network 33.

[0187] Furthermore, the main control unit 2 of the HMD 1 can also cause the face information detection process and the voice information detection process to be performed by separate network servers 32 via the communication processing unit 6.

[0188] 12, we will continue to explain the processing performed by the main control unit 2 or the voice information processing unit 83. After the voice information detection process (step S470), the main control unit 2 (or the voice information processing unit; the same applies hereinafter to the processing entities up to S464) determines whether voice information of the new interviewee has been acquired by the voice information detection process (step S470) (step S462).

[0189] If the main control unit 2 determines in the judgment process of step S462 that the voice information of the new interviewee could not be obtained through the voice information detection process (step S470), there is no voice information to save, so it proceeds to the new interviewee information acquisition process (step S405).

[0190] If the main control section 2 determines in the determination process of step S462 that the voice information of the new interviewee has been acquired through the voice information detection process (step S470), the process proceeds to the voice information storage process (step S463). In the voice information storage process (step S463), the main control unit 2 stores the voice features of the interviewee related to the voice recognition of the interviewee in the data storage unit 42 of the storage unit 4 using the voice information storage function 29. Next, the main control unit 2 proceeds to the new interviewee information acquisition process (step S405).

[0191] For the processing after the new interviewee information acquisition processing (step S405), the main control unit 2 performs processing equivalent to that shown in the flowchart of Figure 5 (steps S406, S407), and then ends the new interviewee processing (step S460) of this embodiment (step S464).

[0192] Figure 14 is a table (T870) showing an example of interviewee information stored in this embodiment. The interviewee table (T870) shown in Figure 14 is composed of interviewee (person) types 860 and information items 850 about each interviewee.

[0193] Each interviewee information item 850 is made up of three items: face information 851, voice information 853, and additional information about the interviewee 852. The interviewee (person) type 860 includes a user 861 in addition to interviewees (862-864).

[0194] The presence of user 861 in the interviewer type has the significance of a profile on the mobile phone and of separating the user's voice information from the voice of a new interviewer during a conversation with the new interviewer.

[0195] This interviewee table (T870) can also be transmitted from the HMD 1 to the network server 32 via the network 33 (see Figure 1) and stored in the storage medium of the network server 32. In particular, with regard to facial features and voice information, by utilizing the network server 32 that specializes in the process of identifying interviewees from facial features and voice features, the process of identifying interviewees in the HMD 1 can be sped up.

[0196] As described above, in this embodiment, the information of a new interviewee that takes into account voice information can be acquired and saved by the processes of FIGS. 12 and 13 and the table of FIG.

[0197] <Interviewee identification and information acquisition processing> Next, the process of identifying an interviewee and acquiring additional information about the interviewee will be described. Fig. 15 is a flowchart showing the procedure of the process (interviewee identification / information acquisition process) that determines whether or not a person can be an interviewee by taking into account voice information, and acquires additional information about the person in advance if they can be an interviewee, which is the gist of this embodiment. The process procedure in Fig. 15 will be explained with reference to the functional block diagram in Fig. 11.

[0198] The flowchart of the second embodiment shown in Figure 15 is almost the same as the flowchart of the first embodiment described above in Figure 8, except for the details of the interviewee information processing. To distinguish between the two, the interviewee information processing of the second embodiment is indicated by a different step number (step S490) from that of the first embodiment (step S450).

[0199] Also, the second embodiment differs from the first embodiment in that a predefined subroutine, voice information detection processing (step S470), is added. Details of the voice information detection processing (step S470) have been explained in the flowchart of Fig. 13, so a repeated explanation will be omitted.

[0200] Fig. 16 is a flowchart (subroutine) showing details of the processing procedure of interviewee information processing (step S490) according to embodiment 2. Details of the processing procedure shown in Fig. 16 will be described with reference to the hardware block diagram in Fig. 10 and the functional block diagram in Fig. 11.

[0201] The flowchart of the second embodiment shown in Figure 16 is almost the same as the flowchart of the first embodiment described above in Figure 9, except for the process of determining (discriminating) whether the interviewee is a known person. To distinguish this, the determination (discrimination) process of the second embodiment is indicated by a different step number (step S492) from that of the first embodiment (step S452).

[0202] Furthermore, in the second embodiment, the new interviewee processing differs from that in the first embodiment, and for the purpose of distinction, the new interviewee processing is indicated by a step number (step S460) that is different from the step number (step S400) in the first embodiment.

[0203] When the process of step S490 (interviewee information process) starts (step S491), first, it is determined whether the interviewee is a known person based on the face information detected in the face information detection process (step S420) (step S492).

[0204] More specifically, the face information processing unit 73 compares the face information (facial features) detected in the face information detection process (step S420 in Figure 6) with the face information (facial and voice features) stored by the face information storage function 25, and if there is a match within a predetermined threshold, it determines that the interviewee is a known person (step S492, YES).

[0205] In another example, if audio is acquired, the main control unit 2 compares the audio information (audio features) detected in the audio information detection process (step S470) with the audio information (audio features) stored by the audio information storage function 29, and if there is a match within a preset threshold, it determines that the interviewee is a known person (step S492, YES).

[0206] Here, a person can be determined to be a known interviewee based on a match in facial information or a match in audio information alone, or a match in both facial information and audio information. If neither facial information nor audio information matches, or if neither facial information nor audio information sufficient to identify the person has been detected, the person is determined not to be a known interviewee (step S492, NO), and the process proceeds to step S460.

[0207] The details of step S460 (new interviewee processing) have been explained in the flowchart of FIG. 12, so an explanation thereof will be omitted here.

[0208] Thus, if it is determined that the interviewee is a known person (step S492, YES), the processes of steps S453 to S457 described in FIG. 9 are executed, and the interviewee information processing (step S490) of this embodiment is terminated (step S493).

[0209] As described above, according to the configuration of this embodiment, it is possible to identify interviewee candidates taking into account voice information, thereby improving the accuracy of providing information about the interviewee.

[0210] <<Embodiment 3>> The following describes embodiment 3 of the present invention. The basic hardware and software configurations of embodiment 3 are the same as those of the above-described embodiments, and the following mainly describes the differences between this embodiment (embodiment 3) and the above-described embodiments, and omits explanations of common parts as much as possible to avoid duplication.

[0211] In the above-described embodiment, it is assumed that the user is wearing a glasses-shaped HMD, and that the interviewee must be in front of the user in order to be recognized. In this embodiment, while it is assumed that the user is wearing a glasses-shaped HMD, a case where it is difficult for the user to easily recognize the interviewee (candidate), such as when the interviewee is located behind the user, is considered. This embodiment will be described below.

[0212] [Outline of operation, etc.] Fig. 17 is a schematic diagram for explaining the background of this embodiment. As can be seen by comparison with Fig. 1 described above, Fig. 17 shows a state in which an interviewee is not present in the line of sight 19 (see the dotted arrow in the figure) of a user 10 wearing a glasses-shaped HMD 1 and in the field of view of the user 10. Fig. 17 also shows a state in which an interviewee 16 approaches from behind (behind) the user 10, and the interviewee 16 recognizes the presence of the user 10 before the user 10 recognizes the presence of the interviewee 16 and utters a voice 14, "Heyyy," shown in a speech bubble.

[0213] In this embodiment, the HMD 1 is configured to start up in response to a voice 14 saying "Hey" in a scene such as that shown in FIG. 17, and immediately after starting up, to analyze the video and audio around the HMD 1 and recognize a person 16.

[0214] Here, the HMD 1 analyzes the surrounding video and audio to determine whether or not the person 16 can be an interviewee (whether or not the person 16 is an interview candidate), and if the person 16 can be an interviewee, acquires additional information about the person 16 and displays the acquired additional information on the display screen 75. In the example shown in FIG. 17, the HMD 1 displays a name (Yamada Jiro) 17 as additional information about the person 16 on the display screen 75.

[0215] 17, the HMD 1 is connected to a network 33 to which a network server 32 is connected via an access point 31. The network server 32 includes a network server that performs various types of calculation processing and a network server that stores various types of data, and the HMD 1 can utilize these servers as needed.

[0216] In one specific example, the main control unit 2 can control the processing of the behavior analysis processing unit 74 (behavior analysis processing device) of the HMD 1 to be performed by an external server (network server 32) via the communication processing unit 6 (communication device).

[0217] Such a configuration allows the overall resources of the HMD 1 to be used efficiently, improving processing speed and ultimately leading to the prompt presentation of necessary information to the user.

[0218] [Process for acquiring additional information] Next, the process of identifying the interviewee and acquiring additional information about the interviewee will be described.

[0219] 18 is a flowchart showing the procedure of step S500 of the process of determining whether a person can be an interviewer based on voice information, and if so, acquiring supplementary information about the person in advance (interviewer identification and information acquisition process), which is the gist of this embodiment. The processing procedure of FIG. 18 will be explained with reference to the functional block diagram of FIG. 11.

[0220] The flowchart showing the processing procedure in FIG. 18 is almost the same as the flowchart showing the procedure in FIG. 15, except that a predefined subroutine, processing of audio information alone (step S510), is added.

[0221] When the interviewee identification and information acquisition process (step S500) starts (step S501), the same process as in the flowchart of Figure 15 is performed, but if the HMD1 determines from the image information that there is no one around (step S433, NO), or if it determines from the image information that there is no interview candidate around (step S900, NO), it executes a subroutine, processing only audio information (step S510).

[0222] Here, the processing of step S510 (single voice information processing) which is a subroutine will be described. Fig. 19 is a flowchart showing the processing procedure of the single voice information processing (step S510) which is a subroutine. The processing procedure of Fig. 19 will be described with reference to the functional block diagram of Fig. 11.

[0223] The voice-only processing (step S510) of embodiment 3 shown in Figure 19 is almost the same as the voice information detection processing (step S470) shown in Figure 13 (embodiment 2). The differences are that in embodiment 3, compared to Figure 13, there is no judgment processing in step S476, and an interview candidate judgment (step S901) and an interviewee information processing subroutine (step S490) have been added.

[0224] After starting (step S511) the audio-only processing (step S510), the HMD 1 performs the same processing as that described above in Fig. 13 from step S472 to step S474. The audio detection error setting processing in step S477 is also the same as that described above.

[0225] In step S901 following step S474, the HMD 1 determines whether the voice is that of the interview candidate. In one specific example, in step S901, the HMD 1 determines whether the content of the voice is likely to be a call to the user.

[0226] Here, examples of cases where the content of the voice may be a call to the user include: (1) If the user's name is included, (2) When the voice is addressing a person (for example, "Hey, you there," "Maybe it's you," etc.), Examples include:

[0227] Therefore, in the case of (1) or (2) above, the HMD 1 determines that the voice is mainly the voice of the interview candidate (step S901, YES). In this case, the HMD 1 proceeds to the voice feature detection process (step S475) described above in FIG. 13.

[0228] On the other hand, if it is determined in the interview candidate determination in step S901 that the voice does not belong to the interview candidate (step S901, NO), the process proceeds to the end of this routine (step S512).

[0229] After the voice feature detection process (step S475), the HMD 1 executes the interviewee information process (step S490) described above in the subroutine of FIG. 16, and then ends the voice-only process (step S510) (step S512).

[0230] According to the HMD1 of the third embodiment, which performs the above-mentioned audio-only processing (step S510), even if the interview candidate cannot be identified from the image information, the interview candidate can be identified using only the audio information, and additional information about the interview candidate can be obtained. Therefore, even if, for example, a complete image of a person cannot be acquired due to a crowd, or if the above-mentioned imaging unit 71 breaks down, information about the interview candidate can be obtained.

[0231] <<Fourth Embodiment>> The following describes embodiment 4 of the present invention. The basic hardware and software configurations of embodiment 4 are the same as those of the above-described embodiments 1 to 3, and the following mainly describes the differences between this embodiment (embodiment 4) and the above-described embodiments 1 to 3, and omits explanations of common parts as much as possible to avoid duplication.

[0232] In the above-described first to third embodiments, it is assumed that an eyeglass-shaped HMD is worn. In contrast, in the fourth embodiment, a case where an HMD other than an eyeglass-shaped HMD is worn will be considered. This embodiment will be described below.

[0233] Fig. 20 is an external view showing an example of an HMD used in embodiment 4. The HMD 100 shown in Fig. 20 has a goggle-shaped housing (hereinafter also simply referred to as "goggles") and an outer shape, and is equipped with an HMD wearing belt 180. As shown in Fig. 20, a user 101 wears the HMD 100 on the head of the user 101 by hanging the HMD wearing belt 180 around the back of the head.

[0234] The HMD 100 has a display screen (display for displaying images) 175 arranged on the front of the goggles, and a left camera 172 and a right camera 171 arranged on the left and right ends of the front of the goggles, respectively.

[0235] Furthermore, in the HMD 100, left and right speakers are arranged at positions corresponding to the ears of the user 101. Note that Fig. 20 shows the left speaker 182, and the right speaker is not shown because it is in the shadow of the user 101.

[0236] 20, a left side camera 173, which is a camera different from the above-mentioned left camera 172, is arranged near the left speaker 182. Although not shown, a rear camera, which is a camera different from the above-mentioned cameras (171, 172, 173), is arranged on the HMD wearing belt 180 at a position corresponding to the back of the head of the user 101. Furthermore, although not shown, a right side camera, which is a camera different from the above-mentioned right camera 171, is also arranged near the above-mentioned right speaker.

[0237] In this way, in this embodiment, by adding or expanding the number of cameras, it is possible to expand the range of images captured around user 101. In particular, by installing a rear camera behind user 101, it is possible to identify person 16 behind user 101 based on facial information, without user 101 having to turn around, even in the positional relationship described in Fig. 17 .

[0238] The fourth embodiment is characterized in that an additional camera is installed to expand the range of images captured around the user 101. In other words, the fourth embodiment includes a plurality of cameras that capture images as a surrounding information acquisition device, and each camera is arranged to capture images of a range wider than the field of view of the user 101.

[0239] In this way, by configuring the device as a surrounding information acquisition device so that it can acquire surrounding information outside the user's 101 field of vision, the probability of capturing an interview candidate who is in a position that the user 101 does not notice is improved, and convenience is also improved.

[0240] Furthermore, although an example in which a camera is used as a device for the surrounding information acquisition device has been given here, as another example, a configuration in which a plurality of distance measuring sensors 55 or human presence sensors 56 (see FIG. 3) are arranged may also be used.

[0241] <<Fifth Embodiment>> The following describes embodiment 5 of the present invention. Note that the basic hardware and software configurations of embodiment 5 are the same as those of the above-mentioned embodiments, and the following mainly describes the differences between this embodiment (embodiment 5) and the above-mentioned embodiments, and explanations of common parts will be omitted as much as possible to avoid duplication. In the above-mentioned embodiments 1 to 4, the case where there is one interview candidate is assumed. In contrast, embodiment 5 considers the case where there are multiple interview candidates. This embodiment will be described below.

[0242] Figure 21 is a flowchart showing the procedure for the process (interviewee identification / information acquisition process) which is the gist of this embodiment, determining whether a person can be an interviewee and, if so, acquiring additional information about the person in advance.

[0243] The interviewee identification and information acquisition process shown in Figure 21 is almost the same as the interviewee identification and information acquisition process described above in Figure 8, except that it includes the addition of a process to determine whether there are multiple potential interviewees (step S522) and a priority determination process (step S523).

[0244] When the interviewee identification and information acquisition process of this embodiment starts (step S521), the HMD1 performs step S432 (surrounding photographing process), step S433 (processing to determine whether a person is present), and step S900 (processing to determine whether the person is a candidate for interview) described in Figure 8.

[0245] If the HMD 1 determines that the person is an interview candidate (step S900, YES), the HMD 1 proceeds to step S522. In step S522, the HMD 1 determines whether the number of people detected as interview candidates is one or multiple.

[0246] In step S522, if only one person is detected (step S522, NO), HMD1 performs face information detection processing (step S420) and interview information processing (step S450) as in other embodiments, and then terminates this routine.

[0247] On the other hand, in step S522, if there are multiple people detected as interview candidates (hereinafter, may be simply referred to as "interview candidates"), the HMD 1 proceeds to priority determination processing (step S523).

[0248] Here, the significance of the priority determination process will be explained. If there is sufficient processing speed of the processor in the HMD 1 and resources such as RAM, it may be possible to accommodate all of the interview candidates, even if there are multiple candidates.

[0249] However, in reality, it is thought that there are often limited hardware resources, and particularly when the HMD 1 is performing its original function (for example, when playing a video of a specific content), attempting to obtain information on all interview candidates takes a long time to process. Such a long processing time may result in at least one of the interviewees meeting face-to-face with the user, and the interview (conversation, etc.) may begin, which could lead to the problems mentioned above (such as increased psychological burden on the user who cannot say the other person's name, etc.).

[0250] In consideration of the above-mentioned problems, the inventors came up with the idea that when there are multiple interview candidates, it would be effective to narrow down or rank the people from whom information is to be obtained, and have therefore created a configuration for determining priority.

[0251] Specifically, in the priority determination process (step S523), the HMD1 identifies the person who is most likely to be the interviewee or who is considered to be the most important person from among multiple people who could be the interviewee as the priority person.

[0252] More specifically, in step S523, the HMD 1 performs, for example, (A) The person's gaze direction is toward the user. (B) A person greets the user by raising their hand or other gesture. (C) A person is approaching the user. (D) The distance to the user is short. For each of the actions, a predetermined weighting is applied to determine the priority, and the person with the highest priority is identified as the priority person.

[0253] When the priority determination process (step S523) is executed, the number of candidates is narrowed down to one, resulting in a state similar to that of other embodiments. Therefore, the HMD 1 then sequentially performs face information detection process (step S420) and interview information process (step S450) as in other embodiments, and then ends this routine.

[0254] As an example of weighting settings, in step S523, the HMD1 identifies (D), i.e., the person who is closest, as the priority person among the above (A) to (D). By this process, the person who is most likely to start the interview (conversation) earliest among multiple interview candidates is identified as the priority person, so the user can quickly learn information about that person (see also step S450).

[0255] In another example of weighting settings, in step S523, the HMD 1 specifies, among the above (A) to (D), (B), that is, a person who is greeting the user by raising their hand, as a priority person. This is because, in a case where there are multiple interviewees and the interviewees are a superior and a subordinate, it is considered that the superior (the person with the higher rank) is usually the one who is greeting, and that the superior is not always in the lead, but rather the subordinate may take the lead.

[0256] The weighting described above may be set arbitrarily by the user through operation of the operation input unit 9 or the like in advance.

[0257] 21, the configuration is such that information on only one person is acquired. As another example, if there are multiple interview candidates, the processes of steps S420 and S450 may be performed sequentially in descending order of priority as determined through the priority determination process (step S523). By performing such processing, the information on all interview candidates can be presented to the user in descending order of priority (in other words, importance) while making effective use of the hardware resources of HMD1.

[0258] As yet another example, if there are multiple interview candidates, steps S420 and S450 may be performed in descending order of priority as determined through the priority determination process (step S523), and for a predetermined number of N (N is an integer greater than or equal to 1) candidates. Such processing is effective, for example, when there are a large number of interview candidates, and allows the information of a certain number of interview candidates to be presented to the user in descending order of priority (in other words, importance) while making effective use of the hardware resources of the HMD1.

[0259] Furthermore, as a variation of the prioritization process, as shown in Figures 22A, 22B, and 23, a simplified display of information may be performed when the interview candidate is far away (Figure 22A), and a detailed display may be performed when the interview candidate approaches (Figure 22B).

[0260] Referring to FIG. 23, the HMD1 displays simplified information 1103a and 1103c (in this example, only the name) for people 15a and 15c who are far away, and displays detailed information 1104b (in this example, the name and various other information) for person 15b who is close.

[0261] In this way, by configuring the system to change the level of detail of the information displayed (i.e., presented or notified to the user) depending on the distance of the interview candidate (target person), the user's attention can be focused on information about people about whom they want more information, thereby increasing convenience.

[0262] When processing such a modified example, the same processing as in other embodiments is performed for all interview candidates (target individuals), and then the interviewee identification and information acquisition processing of this embodiment is terminated (step S524).

[0263] As described above, the configuration of the fifth embodiment makes it possible to quickly respond even when there are multiple candidates who could be interviewees.

[0264] As described above in detail, the mobile information terminal (HMD1, 100) of the present disclosure comprises an ambient information acquisition device (sensor unit 5, imaging unit 71, audio input unit 81) that acquires ambient information about the terminal and the user 10, a behavior analysis processing device (main control unit 2, behavior analysis processing unit 74) that determines whether there is an interview candidate for the user 10 (a person who is about to interview the user) by analyzing the behavior of people included in the acquired ambient information, and an information presentation device (display unit 72) that, if it is determined that there is an interview candidate, presents additional information corresponding to that person to the user 10.

[0265] With a portable information terminal (HMD1, 100) having the above configuration, additional information about the person being interviewed can be provided to the user 10 more quickly, and by the time the user 10 recognizes the person as the person being interviewed, the user 10 already knows the additional information about the person.

[0266] In addition, the above-mentioned portable information terminal (HMD1, 100) is configured to acquire one or more of the following ambient information: images captured by the imaging unit 71, distance information measured by the distance sensor 55 to objects including people, and audio picked up by the audio input unit 81.

[0267] With this configuration, the ambient information acquisition device can acquire ambient information taking into account the advantages of various types of information and the resources of the HMD1 (100) (such as the execution status of its original functions), and ultimately can provide additional information about the person being interviewed to the user 10 more quickly.

[0268] Furthermore, the above-described portable information terminal (HMD 1, 100) is configured so that the behavior analysis processing unit 74 does not determine the interview candidate depending on the location where the surrounding information is acquired.

[0269] According to this configuration, it is possible to suppress the presentation of unnecessary information, and it contributes to the quick acquisition of supplementary information for people who really need it, thereby improving convenience.

[0270] Furthermore, in the above-mentioned mobile information terminal (HMD1, 100), when there are multiple people determined to be interview candidates, the behavior analysis processing unit 74 assigns a priority to each interview candidate according to the results of the behavior analysis, and determines the number or order of interview candidates related to the supplementary information presented by the information presentation device (display unit 72) according to the assigned priority.

[0271] Furthermore, in the above-mentioned portable information terminal (HMD1, 100), when the behavioral analysis processing device determines that the interview candidate exists, the information presentation device (display unit 72) gradually presents information indicating that the interview candidate exists and presents the additional information corresponding to the interview candidate.

[0272] According to this configuration, the user's attention can be focused on information about people about whom the user wants more information, which increases convenience.

[0273] Although examples of embodiments of the present invention have been described above using Embodiments 1 to 5, the configurations for realizing the technology of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or it is also possible to add the configuration of another embodiment to the configuration of one embodiment. All of these fall within the scope of the present invention. Furthermore, numerical values, messages, etc. appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0274] The functions of the present invention described above may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. They may also be realized by software, with a microprocessor unit or the like interpreting and executing a program that realizes each function. Hardware and software may also be used together. The software may be pre-stored in the program section 41 of the HMD 1 at the time of product shipment. It may also be obtained from various server devices on the Internet after product shipment. The software may also be provided on a memory card, optical disc, or the like.

[0275] Furthermore, the control lines and information lines shown in the diagram are those considered necessary for explanation, and do not necessarily represent all of the control lines and information lines on the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0276] 1...HMD (personal digital assistant), 2...main control unit, 3...system bus, 4...memory unit, 5...sensor unit, 6...communication processing unit (communication device), 7...video processing unit, 8...audio processing unit, 9...operation input unit, 10...user, 15...person (interview candidate), 16...person (interview partner), 32...network server, 42...data memory unit, 71...imaging unit (surrounding information acquisition device), 72...display unit (information presentation device), 73...face information processing unit, 74...behavior analysis processing unit (behavior analysis processing device), 75...display screen, 81...audio input unit (surrounding information acquisition device), 82...audio output unit, 83...audio information processing unit, 100...HMD (personal digital assistant), 171...right camera, 172...left camera, 173...left side camera, 175...display screen, 711...right camera, 712...left camera.

Claims

1. In a portable information terminal carried by a user, a surrounding information acquisition device that acquires surrounding information; a behavior analysis processing device that analyzes the behavior of people included in the acquired surrounding information and determines interview candidates who are about to interview the user; an information presentation device that presents supplementary information corresponding to the interview candidate to a user; Mobile information terminal.

2. 2. The portable information terminal according to claim 1, The surrounding information is at least one of a captured image, distance information obtained by measuring a distance to an object including the person, and a collected sound. Mobile information terminal.

3. 2. The portable information terminal according to claim 1, The behavior analysis processing device determines that the person is the interview candidate when the behavior of the person included in the surrounding information indicates interest in the user. Mobile information terminal.

4. 4. The portable information terminal according to claim 3, The behavior showing interest in the user is one or more of a gaze behavior, a greeting behavior, an approach behavior, and a calling behavior. Mobile information terminal.

5. 2. The portable information terminal according to claim 1, Furthermore, the behavior analysis processing device is configured not to determine the interview candidate depending on the type of person included in the surrounding information. Mobile information terminal.

6. 6. The portable information terminal according to claim 5, The type of person for which the determination is not made includes one or more of a facility attendant and a security guard, Mobile information terminal.

7. 2. The portable information terminal according to claim 1, Furthermore, the behavior analysis processing device is configured not to determine the interview candidate depending on the location where the surrounding information is acquired. Mobile information terminal.

8. 2. The portable information terminal according to claim 1, When there are a plurality of people who are determined to be the interview candidates, the behavior analysis processing device assigning a priority to each of the interview candidates according to the results of the behavioral analysis; determining the number or order of the interview candidates related to the supplementary information to be presented by the information presentation device according to the assigned priority; Mobile information terminal.

9. 2. The portable information terminal according to claim 1, When the behavior analysis processing device determines that the interview candidate exists, the information presentation device presents information indicating that the interview candidate exists and presents the supplementary information corresponding to the interview candidate in a stepwise manner. Mobile information terminal.

10. 2. The portable information terminal according to claim 1, When there are a plurality of people determined to be the interview candidates by the behavior analysis processing device, the information presentation device presents the supplementary information corresponding to the interview candidates to the user in such a manner that the closer the person is to the user, the more detailed the information to be presented. Mobile information terminal.

11. 2. The portable information terminal according to claim 1, the surrounding information acquisition device includes a plurality of cameras for acquiring images, The camera is positioned to capture the image over a range wider than the user's field of view. Mobile information terminal.

12. 2. The portable information terminal according to claim 1, A part of the processing performed by the behavior analysis processing device is performed by an external server via a communication device. Mobile information terminal.

13. 2. The portable information terminal according to claim 1, The portable information terminal is an HMD (Head Mounted Display), Mobile information terminal.

14. An information processing method in a mobile information terminal, Acquires information about the user's surroundings, Analyzing the behavior of people included in the acquired surrounding information to determine interview candidates who are about to interview the user; presenting supplementary information corresponding to the determined interview candidate to a user; Information processing methods.

Images