Information processing system and program

Abstract

This makes it easier for a speaker to identify the timing of an utterance that he or she is about to make, compared to a configuration in which the speaker is not notified about the utterance. [Solution] A CPU generates control information for causing a display image indicating that a nearby person 42 may speak to be displayed on a display unit 217 of the device 200 as control information for causing a notification indicating that a nearby person 42 may speak as shown in (A) to be performed in the device 200. This generated control information is transmitted to the device 200, and the device 200 controls the display of the display unit 217 based on this control information. As a result, a display image 45 indicating that a nearby person 42 may speak is displayed on the display unit 217 of the device 200 as shown in (B).

Description

[Technical field]

[0001] The present invention relates to an information processing system and a program. [Background technology]

[0002] Patent document 1 discloses a process for determining a user who is attracting the gaze of a majority of users in a conversation, excluding users who are directing their gaze at the current speaker, to be the next speaker. Patent Document 2 discloses an apparatus including a notification voice storage means for storing, for each conference participant, a notification voice for notifying the conference participants of the next speaker. Patent Document 3 discloses a configuration including chat text input means for accepting input of chat text, and voice synthesis means for voice-synthesizing the chat text into chat voice data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2006-338493 A [Patent Document 2] JP 2012-146072 A [Patent Document 3] JP 2023-112602 A Summary of the Invention [Problem to be solved by the invention]

[0004] When a speaker wishes to communicate with a third party other than himself / herself, he / she usually speaks at a timing other than that at which the third party is speaking. If a speaker speaks while a third party is speaking, the speaker may interrupt the third party's speech, or the speaker's speech may become difficult for the third party to recognize. An object of the present invention is to make it easier for a speaker to identify the timing of an utterance that he or she is about to make, compared to a configuration in which the speaker is not notified about the utterance. [Means for solving the problem]

[0005] The invention according to claim 1 further comprises a processor, the processor being configured to or the status of users who are nearby Acquire status information, and But the user made a predefined move. In the above circumstances, User Control information used to control a device owned by , made the above move The above User A notification indicating the possible utterance of Before the user begins speaking It is an information processing system that generates control information for causing the device to operate. In the invention described in claim 2, the processor performs the following steps based on the image in which the surrounding people are captured: The user The information processing system according to claim 1, further comprising: The invention described in claim 3 is an information processing system described in claim 1, in which the processor generates control information that causes the device to notify the user that there is no speech from the surrounding people when the user is in a situation where the surrounding people are not speaking. The invention described in claim 4 is an information processing system described in claim 1, in which the processor generates control information that causes the device to provide a notification that appeals to the subject's visual sense as the control information that causes the device to provide the notification indicating the possibility of the utterance. The invention described in claim 5 is an information processing system described in claim 1, in which the processor generates control information that causes the notification indicating the possibility of the speech to be made by the device, and that causes a display image to be displayed on the device, the image indicating the possibility of the speech of the nearby person, to be displayed on a display unit of the device. The invention described in claim 6 is an information processing system described in claim 5, in which the processor generates control information for erasing the display image displayed on the display unit of the device if there is no actual speech by the person in the vicinity after generating the control information for causing the display image to be displayed on the display unit of the device. The invention described in claim 7 is an information processing system described in claim 5, in which the processor generates control information that causes the display image to be displayed on the display unit of the device, and that causes the display image to be displayed in a corresponding manner to the surrounding people. In the invention described in claim 8, the processor is configured to: Performed a predetermined movement The information processing system of claim 7 further comprises generating control information for causing the display image to be displayed in a corresponding manner to each of the plurality of surrounding persons when the display image is displayed in a corresponding manner to the surrounding persons. The invention described in claim 9 is an information processing system described in claim 5, in which, when the nearby person who is likely to speak actually speaks, the speech content, which is the content of the utterance, is displayed on the display unit of the device, and the processor generates control information for causing the display image to be displayed on the display unit of the device in a manner corresponding to the display location on the display unit of the device where the speech content is displayed, as the control information for causing the display image to be displayed on the display unit of the device. The invention described in claim 10 is an information processing system described in claim 9, in which the processor generates control information that causes the display image, having a shape surrounding the display location, to be displayed on the display unit as the control information that causes the display image to be displayed in a manner associated with the display location. The invention described in claim 11 is characterized in that the processor detects the surrounding person who is out of the range of vision of the target person who sees the front of the target person through the device. Performed a predetermined movementThe information processing system of claim 1 generates control information for causing a display image indicating the possibility of speech from a nearby person outside the field of view to be displayed on a display unit of the device when the situation is such that the display image indicates the possibility of speech from a nearby person outside the field of view. The invention described in claim 12 is an information processing system described in claim 11, in which the processor generates control information that causes the display image of the surrounding person who is outside the field of view to be displayed on the display unit of the device, and that causes the display image to be displayed on a line connecting the position of the surrounding person when projected onto a virtual plane along the display unit and the center of the display unit. The invention described in claim 13 is the information processing system described in claim 1, wherein the processor, in addition to pre-speech situation information, which is the situation information acquired before the actual speech by the surrounding person, further acquires during-speech situation information, which is information about the situation of the surrounding person who has actually started speaking, and when the situation identified by the acquired during-speech situation information is a specific situation, generates control information used to control the device, which causes the device to issue an end suggestion notification, which is a notification indicating that the speech of the surrounding person may end, or generates control information to cause the device to issue an end notification, which is a notification indicating that the speech of the surrounding person has ended. The invention described in claim 14 is an information processing system described in claim 13, wherein the processor generates control information that causes a predetermined display image to be displayed on a display unit provided on the device when the situation identified by the pre-speech situation information is the specific situation, and generates control information that causes the display form of the display image displayed on the display unit to be changed as the control information that causes the end suggestion notification to be performed on the device. The invention described in claim 15 is an information processing system described in claim 14, in which the processor generates control information for causing the end notification to be performed by the device, the control information causing the display form of the display image to be changed, and the control information for causing the display form of the display image whose display form has been changed by the control information causing the display form to be changed, further changing the display form of the display image. The invention described in claim 16 is an information processing system described in claim 13, in which the processor acquires the pre-speech situation information based on video in which the surrounding people are shown, and acquires the speech situation information based on audio information, which is information about the audio of the surrounding people. The invention according to claim 17 further comprises a processor, the processor being configured to or Peripheral people A user who is of Living organism Obtaining information and Living organism affection News In certain circumstances, User Control information used to control a device owned by The acquired biometric information is in a specific situation. The above User A notification indicating the possible utterance of , before the user's speech begins It is an information processing system that generates control information for causing the device to operate. Claim 18 The invention described in the is provided with a processor, the processor acquiring situation information of surrounding people located in the vicinity of the target person, When the surrounding person is speaking, Based on the acquired status information, The person is speaking. The information processing system is characterized in that a notification is given indicating that the speech of the nearby person may end before the speech of the nearby person ends. Claim 19 The invention described in claim 2 is characterized in that the processor acquires the situation information of the surrounding person based on voice information that is information about the voice of the surrounding person. 18 The information processing system is as described above. Claim 20 The invention described in the above relates to a method for controlling a device held by the subject person when the situation specified by the acquired situation information is a specific situation, the method comprising: generating control information for causing the device to notify the subject person that the speech of the subject person may be terminated, the control information changing an image displayed on a display unit of the device. 18 The information processing system is as described above. Claim 21The invention described in claim 2 further comprises generating, as the control information, control information for changing a display image displayed on the display unit in association with a display portion of the speech content of the surrounding person. 20 The information processing system is as described above. Claim 22 The invention described in claim 2 further comprises generating, as the control information for changing the corresponding display image, which is the display image displayed in correspondence with the display location of the utterance content, control information for changing at least one of a shape, a thickness, and a color of the corresponding display image displayed in a manner surrounding the display location. 21 The information processing system is as described above. Claim 23 The invention described in or the status of users who are nearby A function for acquiring status information, and the acquired status information But the user made a predefined move. In the above circumstances, User Control information used to control a device owned by , made the above move The above User A notification indicating the possible utterance of Before the user begins speaking and a program for causing a computer to realize the function of generating control information for causing the device to perform the function. The invention described in claim 24 is a method for treating a subject or Peripheral people A user who is of Living organism A function for acquiring information and Living organism affection News In certain circumstances, User Control information used to control a device owned by The acquired biometric information is in a specific situation. The above User A notification indicating the possible utterance of , before the user's speech begins and a program for causing a computer to realize the function of generating control information for causing the device to perform the function. Claim 25 The invention described in the is a function of acquiring situation information of people around a subject, When the surrounding person is speaking, Based on the acquired status information, The person is speaking.The program causes a computer to realize a function of notifying a user that a nearby person may finish speaking before the nearby person finishes speaking. Effect of the Invention

[0006] Claim 1 、17 According to the invention, it is possible to more easily identify the timing of an utterance that a speaker is about to make, compared to a configuration in which the speaker is not notified about the utterance. According to the invention of claim 2, situation information can be acquired based on the movements of people in the vicinity. According to the invention of claim 3, it becomes easier for the subject person to determine whether or not people nearby are speaking, compared to a configuration in which the device does not notify the user that no speech is occurring. According to the invention of claim 4, it becomes easier for a hearing-impaired person to determine whether or not there is a possibility that someone nearby is speaking, compared to a configuration in which a notification that appeals to the subject's hearing is made by a device. According to the invention of claim 5, the subject can obtain information about the possibility of speech by people nearby by referring to the display unit of the device. According to the invention of claim 6, it is possible to inform the subject that the possibility of speech from nearby people has decreased. According to the invention of claim 7, the subject can easily identify a nearby person who may possibly speak, compared to a configuration in which a display image cannot be associated with a nearby person. According to the invention of claim 8, it becomes easier for the subject person to identify a nearby person who may be likely to speak, compared to a configuration in which a display image is not displayed in a form in which it is associated with each of the multiple nearby people. According to the invention of claim 9, it is easier for the subject to recognize the location where the speech content is displayed, compared to a configuration in which the display image is displayed without being associated with the display location where the speech content is displayed. According to the invention of claim 10, the target person can more clearly recognize the location where the speech content is displayed, compared to a configuration in which the display image does not have a shape surrounding the display location. According to the invention of claim 11, it is possible to display images corresponding to nearby persons on the display unit even if the nearby persons are located outside the field of view of the subject viewing the area ahead of the subject through the device. According to the invention of claim 12, it becomes easier for the subject to identify in which direction the surrounding person is located, compared to a configuration in which the display image is displayed at a location that is off the line connecting the position of the surrounding person when projected onto a virtual plane along the display unit and the center of the display unit. According to the invention of claim 13, it is possible to more easily identify the timing of an utterance that a speaker is about to make, compared to a configuration in which no notification regarding the end of utterance is given. According to the invention of claim 14, the subject person can determine whether or not the person nearby is likely to finish speaking by referring to the display image displayed on the display unit of the device. According to the invention of claim 15, the subject person can determine whether or not the person nearby has finished speaking by referring to the display image displayed on the display unit of the device. According to the invention of claim 16, the accuracy of judgment as to the possibility of a surrounding person speaking can be improved compared to a configuration in which pre-utterance situation information is obtained based on the voice information of the surrounding person, and the accuracy of judgment as to the end of a surrounding person's speech can be improved compared to a configuration in which speech-in-progress situation information is obtained based on video in which the surrounding person is shown. Claim 18 According to the invention, it is possible to more easily identify the timing of an utterance that a speaker is about to make, compared to a configuration in which the speaker is not notified about the utterance. Claim 19 According to this invention, it is possible to improve the accuracy of determining when a nearby person has finished speaking, compared to a configuration in which situation information about the nearby person who is speaking is obtained based on video in which the nearby person is shown. Claim 20 According to the invention, the subject can determine whether or not the person nearby is likely to finish speaking by referring to the display image displayed on the display unit of the device. Claim 21According to this invention, it is easier to identify a target person who is likely to finish speaking, compared to a configuration in which a display image that is not displayed is changed in correspondence with the display location of the speech content of a surrounding person. Claim 22 According to the invention, the subject can determine whether or not the person nearby is likely to finish speaking by referring to the display image displayed on the display unit of the device. Claim 23、24 According to the invention, it is possible to more easily identify the timing of an utterance that a speaker is about to make, compared to a configuration in which the speaker is not notified about the utterance. Claim 25 According to the invention, it is possible to more easily identify the timing of an utterance that a speaker is about to make, compared to a configuration in which the speaker is not notified about the utterance. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an overall configuration of an information processing system. [Diagram 2] FIG. 2 illustrates a configuration of a management server. [Diagram 3] FIG. 2 is a diagram showing the hardware configuration of the device. [Figure 4] 1A to 1D are diagrams illustrating processes executed in an information processing system. [Diagram 5] 2 is a diagram showing a database stored in an information storage unit of a management server. FIG. [Figure 6] 13A and 13B are diagrams showing other display examples on the display unit of the device. [Figure 7] 13(A) to 13(C) are diagrams showing other processing examples. [Figure 8] 13(A) to 13(C) are diagrams showing other processing examples. [Figure 9] 13(A) to 13(C) are diagrams showing other processing examples. [Figure 10] FIG. 11 is a diagram showing another processing example. [Figure 11]11 is a diagram showing the display unit and people nearby as viewed from the direction indicated by arrow XI in FIG. [Figure 12] 11 is a flowchart showing a flow of processes executed when a notification process is performed. [Figure 13] 13(A) to 13(D) are diagrams showing specific examples of processing. [Figure 14] 1A to 1I are diagrams showing a series of flow of the display process. [Figure 15] 13 is a flowchart showing a process flow in the case where a notification process is performed even at the end of an utterance. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to the present embodiment. The information processing system 1 is provided with a management server 300 as an example of an information processing device. Furthermore, the information processing system 1 is provided with devices 200 that are worn by each of the subjects, which will be described later. 1, only one device 200 is shown, but a plurality of devices 200 are provided according to the number of subjects. The device 200 is a pair of glasses-type devices 200, and is worn on the head of the subject. The subject visually recognizes his / her surroundings through the device 200.

[0009] Furthermore, in this embodiment, a whole view camera 500 is provided, which is a camera that captures an image of a subject wearing the device 200 and surrounding people (described later) around the subject. The overall camera 500 is provided for each location of a subject, and when there are multiple subjects, multiple overall cameras 500 are provided. Furthermore, in this embodiment, an individual microphone 600 is provided which is worn by each of the surrounding people, which will be described later. The individual microphone 600 picks up the voices of the surrounding people and generates voice information. An individual microphone 600 is provided for each nearby person, and when there are multiple nearby people, multiple individual microphones 600 are provided. Each of the device 200, the overall camera 500, and the individual microphone 600 is connected to the management server 300 via a communication line 400 such as the Internet.

[0010] [Management Server Configuration] 2 is a diagram showing the configuration of the management server 300. The management server 300 is realized by a computer. The management server 300 includes an arithmetic processing unit 111 that executes digital arithmetic processing according to a program, and an information storage unit 19 that stores information. The information storage unit 19 is realized by an existing information storage device such as an HDD (Hard Disk Drive), a semiconductor memory, or a magnetic tape.

[0011] The arithmetic processing unit 111 includes a CPU 11a as an example of a processor. The arithmetic processing unit 111 is also provided with a RAM 11b used as a working memory for the CPU 11a, and a ROM 11c in which programs executed by the CPU 11a are stored. The arithmetic processing unit 111 is also provided with a non-volatile memory 11d that is rewritable and can retain data even if the power supply is interrupted, and an interface unit 11e that controls each unit, such as a communication unit, connected to the arithmetic processing unit 111.

[0012] The non-volatile memory 11d is composed of, for example, a battery-backed SRAM, a flash memory, etc. In the information storage unit 19, various types of information such as programs executed by the arithmetic processing unit 111 are stored. In this embodiment, the CPU 11a provided in the arithmetic processing unit 111 reads programs stored in the ROM 11c or the information storage unit 19, whereby various processes performed in the management server 300 are executed.

[0013] The program executed by the CPU 11a may be provided to the management server 300 in a state in which it is stored in a computer-readable recording medium such as a magnetic recording medium (such as a magnetic tape or a magnetic disk), an optical recording medium (such as an optical disk), a magneto-optical recording medium, or a semiconductor memory. The program executed by the CPU 11a may also be provided to the management server 300 using a communication means such as the Internet.

[0014] [Equipment configuration] FIG. 3 is a diagram showing the hardware configuration of the device 200. As shown in FIG. The device 200 includes a processor 211 , an information storage unit 212 , a sensor 213 , a device camera 214 , a device microphone 215 , a speaker 216 , and a display unit 217 . The arithmetic processing unit 211 includes a CPU 21a as an example of a processor. The arithmetic processing unit 211 is also provided with a RAM 21c used as a working memory for the CPU 21a, and a ROM 21b in which programs executed by the CPU 21a are stored.

[0015] The information storage unit 212 is realized by an existing information storage device such as a semiconductor memory. Examples of the sensor 213 include a GPS sensor and a direction sensor. By referring to the output from the sensor 213, the current position of the device 200 and the orientation of the device 200 can be identified. The device camera 214 is a camera that captures the surroundings of the device 200 . When the subject wears the device 200, the device camera 214 faces the direction in front of the subject and captures an image of the forward direction. In other words, the device camera 214 faces the direction in which the subject faces and captures an image of the front of the subject.

[0016] The device microphone 215 captures the voice of the subject and generates voice information. The speaker 216 outputs sounds and voices to notify the person wearing the device 200. The display unit 217 is a so-called display, and displays various information. The display unit 217 is disposed in front of the subject's eyes when the subject wears the device 200. In this embodiment, an image obtained by the device camera 214 is displayed on the display unit 217. When the subject wears the device 200, an image showing the state in front of the subject is displayed on the display unit 217. In this embodiment, the subject visually checks what is ahead of him / her by referring to the image displayed on the display unit 217.

[0017] In addition, there is also a transmissive device 200, in which case a transparent display unit 217 that allows the subject to view what is behind the display unit 217 is provided as the display unit 217. The subject views the area behind the display unit 217 through the display unit 217. In other words, the subject views the area in front of the subject through the display unit 217. In the see-through device 200, when an image is displayed on the display unit 217, the user visually recognizes both the real space located behind the display unit 217 and the image displayed on the display unit 217.

[0018] The program executed by the CPU 21a may be provided to the device 200 in a state in which it is stored in a computer-readable recording medium such as a magnetic recording medium (such as a magnetic tape or a magnetic disk), an optical recording medium (such as an optical disk), a magneto-optical recording medium, or a semiconductor memory. In addition, the program executed by the CPU 21a may be provided to the device 200 using a communication means such as the Internet.

[0019] The device 200 is not limited to the glasses-type device 200, but may also be, for example, a smartphone, a tablet terminal, etc. The glasses-type device 200, the smartphone, the tablet terminal, etc. are all devices that can be carried by the subject person. A display unit and a device camera are also provided in the smartphone and tablet terminal. The subject can visually confirm the area ahead by referring to the image captured by the device camera and displayed on the display unit. In other words, in this case, the subject can visually recognize the space located in front of him / her and behind the smartphone or tablet device by referring to the display unit provided on the smartphone or tablet device placed in front of him / her. In this embodiment, the notification process described below is performed via the device 200, but this notification process can be performed not only via the glasses-type device 200, but also in the case of using a smartphone or a tablet terminal.

[0020] In this specification, the term "processor" refers to a processor in a broad sense, including general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). In addition, the operations of the processor may not only be performed by one processor, but may be performed by multiple processors in physically separate locations working together. The order of the operations of the processor is not limited to the order described in this embodiment, and may be changed.

[0021] [Description of Processing Executed in Information Processing System] 4(A) to (D) are diagrams illustrating the processing executed by the information processing system 1 of this embodiment. FIG. 4A shows a target person 41 to whom a notification process is to be performed, and surrounding people 42 who are people located in the vicinity of the target person 41. In this embodiment, as shown in Fig. 4(A), a target person 41 for whom a notification process is to be performed is wearing a device 200. In this embodiment, as shown in Fig. 4(A), a surrounding person 42 is displayed on a display unit 217 provided on the device 200. Furthermore, in this embodiment, as shown in FIG. 4(A), an individual microphone 600 is attached to the nearby person 42.

[0022] As described above, the device 200 of this embodiment is a glasses-type device. This glasses-type device 200 is worn on the head of the subject 41. The subject 41 visually recognizes a nearby person 42 located around the subject 41 through the device 200. In other words, the subject 41 visually recognizes a nearby person 42 located in front of the subject 41 through the device 200. As described above, the device 200 is provided with the device camera 214 (see FIG. 3) and the display unit 217 capable of displaying an image obtained by the device camera 214. The target person 41 visually recognizes the nearby person 42 by referring to the nearby person 42 photographed by the device camera 214 and displayed on the display unit 217. When the device 200 is the above-mentioned transmissive device, the target person 41 visually recognizes the surrounding person 42 positioned behind the transparent display unit 217 through the transparent display unit 217.

[0023] In this embodiment, in the state shown in Figure 4 (A), a CPU 11a as an example of a processor provided in a management server 300 (see Figure 2) acquires situation information, which is information about the situation of a nearby person 42 who is located in the vicinity of a target person 41. Specifically, the CPU 11a acquires situation information of the nearby person 42 based on a video in which the nearby person 42 is shown and audio information that is information about the audio of the nearby person 42.

[0024] When acquiring status information of the nearby person 42 based on an image showing the nearby person 42, the CPU 11a acquires the status information of the nearby person 42 based on an image acquired by the device camera 214 (see FIG. 3) provided on the device 200. In this embodiment, the image captured by the equipment camera 214 is transmitted to the management server 300 via a communication line 400 (see FIG. 1). The CPU 11a of the management server 300 analyzes this video and acquires status information of the nearby person 42. The CPU 11a acquires status information of the nearby person 42 based on the video in which the nearby person 42 is shown.

[0025] When acquiring status information of the surrounding person 42 based on the voice information of the surrounding person 42, the CPU 11a acquires the status information of the surrounding person 42 based on the voice information obtained by the individual microphone 600, which is a microphone worn by each of the surrounding persons 42. In this embodiment, the voice information obtained by the individual microphone 600 is transmitted to the management server 300 through the communication line 400. The CPU 11a of the management server 300 analyzes the voice information and obtains situation information of the nearby person .

[0026] [Database Description] FIG. 5 is a diagram showing a database stored in the information storage unit 19 (see FIG. 2) of the management server 300. As shown in FIG. In this embodiment, as shown in FIG. 5, information about each nearby person 42 is registered in a database stored in the information storage unit 19. In this embodiment, the following information is registered in advance in the database for each nearby person 42: an identification ID which is information used to identify each nearby person 42, microphone identification information which is identification information for the individual microphone 600 possessed by each nearby person 42, and facial information of the nearby person 42. In this embodiment, an image of the face of the nearby person 42 is captured in advance. Then, face information, which is information on the face of the nearby person 42, is registered in a database. As the face information, an image of the face of the nearby person 42 and information on the facial features of the nearby person 42 obtained by analyzing this image are registered.

[0027] In this embodiment, as described above, the video captured by the device camera 214 and the audio information obtained by the individual microphone 600 are transmitted to the management server 300. The CPU 11a of the management server 300 acquires this video and audio information, and acquires situation information of the nearby person 42 based on this video and audio information. Specifically, when the CPU 11a of the management server 300 acquires video captured by the equipment camera 214, it identifies the nearby person 42 appearing in the video based on the facial image of the nearby person 42 appearing in the video and the facial information stored in the database. Furthermore, the CPU 11a of the management server 300 analyzes this video and acquires situation information of the identified nearby person 42.

[0028] In addition, when the CPU 11a of the management server 300 acquires audio information acquired by the individual microphone 600, it identifies the nearby person 42 from whom the audio information was acquired by the individual microphone 600, based on the microphone identification information transmitted to the management server 300 together with the audio information and the microphone identification information pre-stored in the database. Furthermore, the CPU 11a of the management server 300 analyzes the voice information to obtain situation information of the identified nearby person 42.

[0029] Each of the individual microphones 600 stores microphone identification information for identifying the individual microphone 600 . The individual microphone 600 transmits to the management server 300 the audio information acquired by the individual microphone 600 and the microphone identification information. The CPU 11a of the management server 300 identifies the nearby person 42 from whom the voice information was acquired by the individual microphone 600, based on this microphone identification information and the microphone identification information registered in the database.

[0030] In this embodiment, an individual microphone 600 (see FIG. 4(A)) is provided for each nearby person 42. In this embodiment, the individual microphone 600 provided for each nearby person 42 acquires voice information of each nearby person 42 . In this embodiment, the audio information obtained by the individual microphone 600 is transmitted to the management server 300 together with the microphone identification information, as described above. The CPU 11a of the management server 300 identifies the nearby person 42 based on the microphone identification information, and further acquires situation information for the identified nearby person 42 based on the voice information.

[0031] More specifically, in this embodiment, when transmitting audio information from the individual microphone 600 to the management server 300, the individual microphone 600 selects audio information whose sound pressure exceeds a predetermined threshold from the audio information obtained by the individual microphone 600. Then, this selected audio information is transmitted from the individual microphone 600 to the management server 300 together with the microphone identification information. As a result, in this embodiment, voice information of nearby persons 42 other than the nearby person 42 wearing the individual microphone 600 is prevented from being transmitted to the management server 300 through this individual microphone 600.

[0032] The other surrounding persons 42 are located away from the microphone-wearing surrounding person 42, who is the surrounding person 42 to whom the individual microphone 600 is attached, and the sound pressure of the voice of the other surrounding person 42 picked up by the individual microphone 600 is usually small. In a configuration in which audio information whose sound pressure exceeds a predetermined threshold is selected and the audio information is transmitted to the management server 300, audio information of other nearby persons 42 is prevented from being transmitted to the management server 300 through the individual microphone 600 of the nearby person 42 wearing a microphone.

[0033] The management server 300 may select the audio information whose sound pressure exceeds a predetermined threshold value. In this case, the management server 300 selects, from the sound information transmitted from the individual microphones 600, sound information whose sound pressure exceeds a predetermined threshold value. Then, the management server 300 acquires this selected voice information as the voice information of the nearby person 42 wearing the microphone.

[0034] In addition, the CPU 11a of the management server 300 may acquire voice information of each of the surrounding persons 42 based on voice information obtained by a microphone installed on a terminal device owned by each of the surrounding persons 42, such as a smartphone or tablet terminal owned by each of the surrounding persons 42. Alternatively, a common microphone may be provided, and the CPU 11a of the management server 300 may acquire voice information of each of the nearby persons 42 based on the voice information acquired by this common microphone.

[0035] When a common microphone is used, characteristic information, which is information about the characteristics of the voices of each of the nearby people 42, is registered in advance in a database. The CPU 11a of the management server 300 identifies the voice information of each of the nearby persons 42 based on this characteristic information registered in the database, and acquires the situation information of each of the nearby persons 42 based on this voice information.

[0036] [Explanation of specific processing] In this embodiment, the CPU 11a of the management server 300 acquires situation information of the nearby person 42 based on the video captured by the equipment camera 214 and the audio information acquired by the individual microphone 600, as described above. Then, when the situation specified by the acquired situation information is a specific situation, the CPU 11a generates control information used to control the device 200 owned by the subject person 41 (see FIG. 4(A)). More specifically, the CPU 11a generates, as this control information, control information for allowing a predetermined notification to be given to the target person 41 via this device 200.

[0037] More specifically, the CPU 11a generates control information for notifying the target person 41 when the situation specified by the acquired situation information is one in which there is a possibility that the nearby person 42 will speak. More specifically, the CPU 11a generates control information for causing the device 200 to give a notification that there is a possibility that the nearby person 42 will speak.

[0038] The CPU 11a determines that there is a possibility that the nearby person 42 is speaking when the situation specified by the acquired situation information is, for example, any one of the following situations. If there is breathing noise from a nearby person 42 When a person nearby 42 makes a specific sound such as "hmm," "um," "ah," or "eh" When the facial expression of the person nearby 42 becomes specific, such as when the person nearby 42 opens his / her mouth wider. When a nearby person 42 visually recognizes the direction in which the target person 41 is located for more than a predetermined period of time. When the surrounding person 42 turns towards the target person 41 When the person 42 in the vicinity performs a specific predetermined motion, such as when the person 42 nods, moves his / her hand close to his / her face, or straightens his / her posture.

[0039] Alternatively, the status information of the nearby person 42 may be acquired based on biological information of the nearby person 42. Specifically, the CPU 11a may obtain situation information about the nearby person 42 based on biological information such as pulse, heart rate, and blood pressure obtained by a sensor (not shown) attached to the nearby person 42. When obtaining status information of the nearby person 42 based on the biometric information, the biometric information obtained by the sensors and the sensor identification information for each sensor are transmitted from the sensors to the management server 300 via a communication line (not shown).

[0040] The CPU 11a of the management server 300 identifies a nearby person 42 based on the sensor identification information, and if the situation identified by the transmitted biometric information is a specific situation, determines that there is a possibility that the identified nearby person 42 is speaking. Specifically, for example, when the values ​​of the pulse rate, heart rate, blood pressure, etc. increase, the CPU 11a of the management server 300 determines that there is a possibility that a nearby person 42 identified based on the sensor identification information is speaking.

[0041] In the processing example shown in Figure 4, the CPU 11a generates control information for causing the device 200 to issue a notification indicating that there is a possibility that nearby person 42 (see Figure 4 (A)) is about to speak, and for causing a display image indicating this to be displayed on the display unit 217 of the device 200. This generated control information is transmitted to the device 200, and the device 200 controls the display of the display unit 217 based on this control information. As a result, in this embodiment, as shown in FIG. 4(B), a display image 45 indicating that there is a possibility that the nearby person 42 will speak is displayed on the display unit 217 of the device 200.

[0042] This display image 45 is an image that represents a so-called "speech bubble." In this embodiment, when it is determined that there is a possibility of speech from nearby person 42, before the content of the speech, which will be described later, is displayed, this display image 45 consisting of an image representing a speech bubble, is displayed on display unit 217 of device 200, as shown in FIG. 4(B). This allows the target person 41 to recognize that there is a possibility that a nearby person 42 is speaking. Display image 45 is not limited to a 2D (2-dimensional) image, and may be a 3D (3-dimensional) image. When display image 45 is a 3D (3-dimensional) image, images with different angles for each eye are displayed on display unit 217. In other words, when display image 45 is a 3D (3-dimensional) image, a plurality of images with different viewing angles are displayed on display unit 217 as display image 45.

[0043] The control information may be generated by a device other than the management server 300 . In this embodiment, the management server 300 generates the control information, but this is not limiting, and the control information may be generated by a device other than the management server 300 . The control information may be generated by the device 200, for example. When device 200 generates control information, device 200 determines whether nearby person 42 is in a specific situation based on, for example, video of nearby person 42 obtained by device camera 214 (see FIG. 3) that device 200 has and audio information obtained by individual microphone 600.

[0044] Then, the device 200 generates control information for causing the device 200 to notify the user that there is a possibility that the nearby person 42 will speak when the nearby person 42 is in a particular situation. Specifically, the device 200 generates control information for causing the display image 45 to be displayed on the display unit 217 of the device 200 itself. As a result, the display image 45 is displayed on the display unit 217 of the device 200 in the same manner as when the CPU 11a of the management server 300 generates the control information.

[0045] The CPU 11a of the management server 300 generates control information for displaying the display image 45 (see FIG. 4(B)) on the display unit 217 in such a manner that the display image 45 is associated with the nearby person 42 appearing on the display unit 217. As a result, as shown in FIG. 4(B), display image 45 is displayed in a form in which display image 45 is associated with nearby person 42 appearing on display unit 217. More specifically, in this embodiment, the display image 45 is displayed in a form in which the display image 45 is associated with the periphery of the head of the nearby person 42.

[0046] The CPU 11a of the management server 300 identifies each of the nearby persons 42 shown on the display unit 217 based on the video captured by the equipment camera 214 (see FIG. 3). Specifically, the CPU 11a of the management server 300 identifies each of the nearby persons 42 shown on the display unit 217 of the device 200 based on the image of the nearby persons 42 shown in the image acquired by the device camera 214 and the facial information registered in the database.

[0047] Then, the CPU 11a of the management server 300 generates control information that allows the display image 45 to be associated with a nearby person 42 that is determined to have the potential to speak among the identified nearby people 42 (hereinafter, sometimes referred to as a "surrounding person 42 that may speak"). Specifically, when generating this control information, the CPU 11a generates the control information including position information that is information about the display position of the display image 45.

[0048] The CPU 11a determines the position of the nearby speech-possible person 42 captured in the video captured by the equipment camera 214 as the display position of the display image 45, and generates control information including position information that is information about this display position. More specifically, the CPU 11a determines the position around the head of the nearby person 42 who may speak on the video captured by the equipment camera 214 as the display position of the display image 45. Then, the CPU 11a generates control information including position information that is information about the determined display position.

[0049] In this embodiment, control information including this location information is transmitted to the device 200. Then, the device 200 performs display control so that the display image 45 is displayed at the position specified by this position information. As a result, as shown in FIG. 4B, on the display unit 217 of the device 200, the display image 45 is displayed in a form in which the display image 45 is associated with the speech possible nearby person 42 who is likely to speak.

[0050] In addition, when the device 200 is the above-mentioned transparent type device 200, the CPU 11a of the management server 300 generates control information for displaying the display image 45 on a portion of the display unit 217 of the device 200 that is located on a straight line connecting the eyes of the subject 41 and the surrounding person 42 who may be able to speak. In this case, the CPU 11a of the management server 300 first obtains the angle between the front direction of the device 200 and the direction from the device 200 toward the surrounding person 42 who may be able to speak. Specifically, the CPU 11a of the management server 300 analyzes the image captured by the device camera 214, and acquires the angle between the front direction and the direction toward the surrounding person 42 who may be able to speak.

[0051] Then, the CPU 11a determines the display position of the display image 45 on the display unit 217 based on this angle, and generates control information including information on this determined display position. As a result, even in the see-through device 200, the display image 45 is displayed in a form in which the display image 45 is associated with the nearby speech potential person 42 who is likely to speak. In this case, the subject 41 will see a nearby person 42 who may be able to speak and exists in real space, and a display image 45 displayed on the display unit 217 and positioned between the subject 41's eyes and the nearby person 42 who may be able to speak.

[0052] Thereafter, in this processing example, actual speech by the peripheral person 42 is started, as indicated by reference symbol 4C in Fig. 4(C) In other words, actual speech by the speech-possible peripheral person 42 is started. When the surrounding person 42 starts to actually speak, an image 46 indicating that the surrounding person 42 has started speaking is displayed inside a display image 45 displayed in correspondence with the surrounding person 42, as shown in FIG. 4(C). When an actual speech is made by a nearby person 42 with whom a display image 45 is associated, an image 46 is displayed inside the display image 45, indicating that speech by this nearby person 42 has begun.

[0053] In other words, in this embodiment, when an actual speech is made by a nearby person 42 with whom the display image 45 is associated, an image is displayed inside the display image 45 indicating that acquisition of audio information by the individual microphone 600 has begun. Whether or not there has been actual speech by the nearby person 42 with whom the display image 45 is associated is determined based on, for example, the output from the individual microphone 600 worn by this nearby person 42.

[0054] In this embodiment, within the area surrounded by the display image 45, an image 46 is displayed indicating that the nearby person 42 has started speaking. This image 46 indicating that the call has started is displayed on the display unit 217 of the device 200 until the CPU 11a of the management server 300 acquires the content of the speech of the nearby person 42.

[0055] It takes time for the CPU 11a to acquire the speech content. In this embodiment, an image 46 indicating that the nearby person 42 has started speaking is displayed on the display unit 217 until the CPU 11a acquires the speech content. Note that the display of image 46 indicating that processing has started is not essential, and the display shown in FIG. 4(D), which will be described next, may be displayed from the display shown in FIG. 4(B) without passing through the display in FIG. 4(C).

[0056] When the nearby person 42 actually speaks, the CPU 11a of the management server 300 acquires the speech content, which is the content of the nearby person 42's speech. The CPU 11a of the management server 300 analyzes the voice information transmitted from the individual microphone 600 worn by the nearby person 42 with whom the display image 45 is associated, and acquires the contents of the speech of the nearby person 42. Note that the acquisition of the contents of the speech based on the voice information may be performed using a known method.

[0057] Next, the CPU 11a of the management server 300 generates control information for displaying the acquired utterance content on the display unit 217 in a manner associated with the nearby person 42 who made the utterance with the utterance content. Then, the CPU 11a of the management server 300 transmits the generated control information to the device 200.

[0058] As a result, in this embodiment, as shown in FIG. 4(D), the speech content 48 of the nearby person 42 is displayed in a predetermined display area 47 of the display unit 217 of the device 200. In this embodiment, when the nearby person 42 who was likely to speak actually speaks, the display unit 217 of the device 200 displays the speech content 48 of this nearby person 42 . In this embodiment, when the spoken content 48 is displayed, the spoken content 48 is displayed inside the display image 45 made up of an image showing a speech bubble. In this embodiment, speech content 48 of a nearby person 42 who is determined to be likely to speak is displayed inside a display image 45 that is displayed in association with the nearby person 42.

[0059] In the processing example described above, first, when there is a possibility that the nearby person 42 will speak, the CPU 11a generates control information for causing the display image 45 to be displayed on the display unit 217 of the device 200 as described above. As a result, a display image 45 is displayed on the display unit 217, as shown in FIG. 4(B). The CPU 11a generates control information for displaying the display image 45 so that the display image 45 is displayed in a manner associated with a display location 47 on the display unit 217 where the speech content 48 (not shown in FIG. 4(B)) is scheduled to be displayed.

[0060] In this embodiment, the inside of the display image 45 (see FIG. 4(B)) is a display area 47 where the speech content 48 is to be displayed. The CPU 11a generates control information for causing the display image 45 to be associated with the display location 47 as control information for causing the display image 45 to be displayed. More specifically, the CPU 11a generates control information for associating the display image 45 with the display location 47, such that an image representing a speech bubble having a shape surrounding the display location 47 is displayed on the display unit 217, as shown in FIG. 4(B).

[0061] In this embodiment, after this control information is generated, when a nearby person 42 actually speaks, the speech content 48 of this nearby person 42 is displayed within an area surrounded by a display image 45 consisting of an image representing a speech bubble, as shown in FIG. 4(D). In other words, when the nearby person 42 actually speaks, the speech content 48 is displayed in a display area 47 located inside the display image 45 .

[0062] The CPU 11a of the management server 300 generates control information for causing the device 200 to notify the subject 41 of the possibility of speech, such as by displaying a display image 45. As a result, in this embodiment, the device 200 provides a notification that the subject person 41 can visually confirm.

[0063] [Notification processing format] Here, display image 45 is not limited to an image having a shape surrounding display location 47. The shape of display image 45 is not particularly limited, and any shape is acceptable as long as the image can be visually confirmed by subject 41. Another example of the display image 45 is a dot-shaped image. When this dot-shaped image is to be displayed on the display unit 217, the dot-shaped image is displayed in a form corresponding to the surrounding person 42, similar to the image having the above-mentioned surrounding shape. Furthermore, when this dot-like image is displayed on the display unit 217, the speech content 48 of the nearby person 42 is displayed around this dot-like image.

[0064] Additionally, as the display image 45 displayed on the display unit 217 of the device 200, for example, an image of text indicating the possibility of speech, such as "Possible speech", may be displayed. Furthermore, the notification that appeals to the visual sense of the subject person 41 is not limited to being made by an image, and may be made by turning on a light source (not shown) provided in the device 200. In addition, a notification that appeals to the visual sense of the subject 41 may be performed by changing the overall color of the display screen displayed on the display unit 217, or by changing the color of a part of the display screen, such as the edge of the display screen displayed on the display unit 217.

[0065] As another example of the control information for causing the device 200 to notify that there is a possibility that the surrounding person 42 is about to speak, control information for vibrating a vibration source (not shown) provided in the device 200 may be generated. In this case, the target person 41 recognizes the possibility that the surrounding person 42 is about to speak based on the vibration of the device 200. Additionally, for example, control information may be generated to cause a sound or voice to be emitted from a speaker 216 (see FIG. 3) provided in the device 200. If the subject 41 is hearing impaired, it is difficult to be notified by sound, but if the subject 41 is not hearing impaired, the subject 41 will recognize from the sound that a nearby person 42 may be speaking.

[0066] In the case where the subject 41 is hearing impaired, when the speech content 48 is displayed as shown in FIG. 4(D), the subject 41 can recognize that the surrounding person 42 is speaking. Here, the display of the speech content 48 is performed with a delay from the actual speech by the surrounding person 42. If the display of the speech content 48 is delayed, a situation may occur in which the speech content 48 has not yet been displayed even though the actual speech by the surrounding person 42 has already started. In this case, the target person 41 may mistakenly recognize that the surrounding person 42 is not speaking, and a situation may arise in which the target person 41 starts speaking while the surrounding person 42 is speaking.

[0067] In contrast, in this embodiment, the target person 41 is notified of the possibility of speech before the surrounding person 42 actually starts speaking, as described above. In this case, when the target person 41 is notified of the possibility of speaking, he / she will refrain from speaking. In this case, a situation where the target person 41 starts speaking after the surrounding person 42 starts speaking is unlikely to occur. The information processing system 1 of this embodiment also functions effectively when the target person 41 is someone other than a hearing-impaired person. Even if the subject 41 is not a person with a hearing impairment, if the subject 41 is notified of the possibility that the person nearby 42 will speak, the subject 41 will be less likely to speak after the person nearby 42 begins speaking.

[0068] [Other display examples] FIG. 6 is a diagram showing another example of display on the display unit 217 of the device 200. In FIG. FIG. 6 shows a display example in a situation where the surrounding person 42 is not speaking and there is no possibility that the surrounding person 42 will speak. The CPU 11a of the management server 300 may generate control information for causing the device 200 to notify the user that there is no speech when the nearby person 42 is not speaking. As a result, in this case, as shown in FIG. 6, an image 51 indicating that no speech is being made is displayed on the display unit 217 of the device 200.

[0069] 6, the image 51 indicating the absence of speech is an image composed of the characters "no speech." By displaying the image 51 indicating the absence of speech, the subject 41 recognizes that the surrounding person 42 is not speaking. Note that image 51 indicating no speech is not limited to an image made up of characters, and may be an image other than an image made up of characters, such as a symbol or a figure.

[0070] Here, it is assumed that, in the situation shown in FIG. 6, the possibility of the nearby person 42 speaking arises. In this case, the CPU 11a of the management server 300 generates control information for switching the display on the device 200 to the display shown in FIG. 4(B). More specifically, the CPU 11a generates control information for erasing the image 51 indicating the absence of speech and for displaying the display image 45 shown in FIG. When the display image 45 shown in FIG. 4(B) is displayed, the target person 41 recognizes that there is a possibility that a nearby person 42 is speaking.

[0071] [Other processing examples] 7(A) to (C) are diagrams showing another processing example. The process when there is no actual speech from the nearby person 42 will be described. In this processing example, similarly to the above, first, as shown in FIGS. 7(A) and (B), the possibility of nearby person 42 speaking arises, and in response, display image 45 is displayed on display unit 217 of device 200. After that, in this processing example, the situation is one in which there is no actual speech by the nearby person 42. In this case, in this processing example, as shown in FIG. 7(C), the display image 45 corresponding to this nearby person 42 is erased.

[0072] The CPU 11a generates control information for erasing the display image 45 displayed on the display unit 217 if there is no actual speech by the nearby person 42 after generating the above-mentioned control information for displaying the display image 45 on the display unit 217. More specifically, the CPU 11a generates control information for erasing the display image 45 if there is no actual speech by the nearby person 42 corresponding to the displayed display image 45 within a predetermined time period from when the above-mentioned control information for displaying the display image 45 is generated, or within a predetermined time period from when the display image 45 is displayed on the display unit 217. In response to this, device 200 erases display image 45. As a result, display image 45 that has been displayed on display unit 217 is erased, as shown in Figures 7(B) and (C).

[0073] [Other processing examples] 8(A) to (C) are diagrams showing another processing example. When multiple nearby persons 42 shown on the display unit 217 of the device 200 are in a specific situation, the CPU 11a of the management server 300 generates control information such that the display image 45 is displayed in a form in which the display image 45 is associated with each of the multiple nearby persons 42. As a result, in this case, as shown in FIG. 8(B), on the display unit 217 of the device 200, the display image 45 is displayed in a form in which the display image 45 is associated with each of the multiple nearby persons 42. In this case, the subject 41 looking at the display unit 217 of the device 200 recognizes that multiple nearby persons 42 may be speaking.

[0074] Thereafter, when a surrounding person 42 among the multiple surrounding people 42 actually speaks, the speech content 48 is displayed within an area surrounded by a display image 45 that is displayed in correspondence with the surrounding person 42 who made the actual speech, as shown by symbol 8D in Figure 8 (C). In addition, in this processing example shown in Figure 8, for another nearby person 42 indicated by symbol 8E in Figure 8 (C) who did not make an actual speech, the display image 45 that was displayed in association with this other nearby person 42 is erased, as shown in Figures 8 (B) and (C).

[0075] Although not shown, in the state shown in FIG. 8(C), if there is a possibility that another nearby person 42, indicated by the symbol 8E, may speak, the display image 45 corresponding to this other nearby person 42 will be displayed again. In this embodiment, when one of the surrounding persons 42, indicated by the symbol 8F, is making a statement and there arises a possibility that another surrounding person 42 may speak, a display image 45 and speech content 48 corresponding to this one of the surrounding persons 42 are displayed, while a new display image 45 corresponding to this other surrounding person 42 is displayed.

[0076] [Other processing examples] 9(A) to 9(C) and Fig. 10 are diagrams showing other processing examples. Fig. 10 shows a state in which the device 200, nearby person 42, and target person 41 are viewed from above in the vertical direction. In the processing example shown in Figures 9 and 10, as shown in Figure 10, some of the surrounding people 42 indicated by the symbol 10B are outside the shooting range 10A of the device camera 214 (not shown in Figure 10) provided on the device 200. The shooting range 10A can also be regarded as the range of vision of the subject 41 viewing what is ahead of him / her through the device 200, and in the processing example shown in Figures 9 and 10, some surrounding persons 42, indicated by the symbol 10B, are outside this range of vision. Hereinafter, this portion of the peripheral people 42 will be referred to as "non-display peripheral people 42B." As shown in FIG. 9(A), the display unit 217 of the device 200 displays two neighboring persons 42 indicated by the symbol 9D other than the non-displayed neighboring person 42B, and the non-displayed neighboring person 42B is not shown on the display unit 217 of the device 200.

[0077] Furthermore, in this processing example shown in Figures 9 and 10, this non-displayed peripheral person 42B who is not shown on the display unit 217 of the device 200 is in a specific situation, and there is a possibility that the non-displayed peripheral person 42B may speak. In this case, the CPU 11a of the management server 300 generates control information for causing the display unit 217 to display the display image 45 indicating that there is a possibility that the non-display nearby person 42B will speak. In this embodiment, control information is generated to cause the display image 45 to be displayed on the display unit 217 even when there is a possibility of speech from the non-display surrounding person 42B.

[0078] As a result, in this processing example, as shown in FIG. 9(B), a display image 45 corresponding to the non-display surrounding person 42B is displayed on the display unit 217 of the device 200. When the target person 41 refers to the display unit 217 in the state of FIG. 9(B), the target person 41 recognizes that there is a possibility that a speech is coming from a nearby person 42 who is located out of the target person's field of vision. When non-displayed peripheral person 42B actually speaks, as shown in FIG. 9(C), speech content 48 of non-displayed peripheral person 42B is displayed in correspondence with display image 45 corresponding to non-displayed peripheral person 42B. In this processing example as well, speech content 48 of non-display peripheral person 42B is displayed within an area surrounded by display image 45 corresponding to non-display peripheral person 42B. In the processing example shown in FIGS. 9 and 10, the speech content 48 of the non-display nearby person 42B, who is not shown on the display unit 217 of the device 200, is also displayed on the display unit 217 of the device 200.

[0079] In this processing example shown in Figures 9 and 10, when display image 45 corresponding to non-displayed surrounding person 42B is displayed on display unit 217 of device 200, a display is made so that subject person 41 can tell the direction in which non-displayed surrounding person 42B is located, as shown in Figure 9(B). In FIG. 9(B), non-display peripheral person 42B is located to the left of the front of device 200, and display image 45 corresponding to non-display peripheral person 42B is also located to the left of center 217C of display unit 217 in the figure. In this embodiment, the display position of the display image 45 corresponding to the non-display peripheral person 42B changes depending on the position of the non-display peripheral person 42B.

[0080] FIG. 11 is a diagram of the display unit 217 and the surrounding person 42 as viewed from the direction indicated by the arrow XI in FIG. The CPU 11a generates control information to display a display image 45 corresponding to the non-display peripheral person 42B on a portion of the display unit 217 that is located on a straight line 11L connecting the non-display peripheral person 42B and the center 217C of the display unit 217, as shown in FIG. 11. This will be explained in detail with reference to FIG. Here, assume an imaginary plane 10K along the display unit 217 of the device 200. Furthermore, assume a line 10H that connects the center of the target person 41 to the non-display peripheral person 42B.

[0081] Further, here, it is assumed that a non-display peripheral person 42B is projected onto the imaginary plane 10K. More specifically, it is assumed that non-display peripheral person 42B is projected from a location where non-display peripheral person 42B is located in the direction of line 10H described above and onto virtual plane 10K. In this case, on the imaginary plane 10K, the non-display peripheral person 42B is located at a position indicated by the reference symbol 10M. Hereinafter, the position of the non-display peripheral person 42B when the non-display peripheral person 42B is projected onto the imaginary plane 10K along the display unit 217 will be referred to as the "position on the plane 10M."

[0082] The CPU 11a generates control information for displaying a display image 45 (see FIG. 11) of the non-display surrounding person 42B on the display unit 217 in a direction on the straight line 11L connecting the planar position 10M and the center 217C of the display unit 217. As a result, the display image 45 is displayed on the display unit 217 of the device 200 at a location indicated by reference numeral 11X in FIG. By referring to the display unit 217 shown in FIG. 11, the target person 41 can specify in which direction the non-display surrounding person 42B who may speak is located.

[0083] When status information about non-displayed surrounding person 42B who is not shown on display unit 217 of device 200 is obtained based on an image showing this non-displayed surrounding person 42B, this status information about non-displayed surrounding person 42B is obtained based on an image obtained by overall camera 500 (see FIG. 10). Specifically, in this case, the non-displayed surrounding person 42B is identified based on the image obtained by the overall camera 500 and the face information registered in the database (see FIG. 5), and situation information of the identified non-displayed surrounding person 42B is obtained based on this image.

[0084] Furthermore, when displaying display image 45 on display unit 217 of device 200, it is necessary to identify the position of identified non-display surrounding person 42B. In this case, the CPU 11a of the management server 300, for example, analyzes the video captured by the overall camera 500 to identify the position of the non-display surrounding person 42B. Furthermore, in this case, the CPU 11a of the management server 300 analyzes the image captured by the panoramic camera 500 to identify the position of the center 217C of the display unit 217 of the device 200 worn by the subject person 41 and the orientation of the device 200.

[0085] Then, the CPU 11a of the management server 300 identifies the above-mentioned position 10M on the plane based on the position of the non-display surrounding person 42B, the position of the center 217C of the display unit 217 of the device 200, and the orientation of the device 200. Next, the CPU 11a of the management server 300 determines the display position of the display image 45 on the display unit 217 based on the specified on-plane position 10M and the position of the center part 217C of the display unit 217.

[0086] Then, the CPU 11a of the management server 300 generates control information including information about the determined display position. The device 200 controls the display on the display unit 217 in accordance with this control information. As a result, on the display unit 217 of the device 200, the display image 45 is displayed on a straight line 11L connecting the position 10M on the plane and the center 217C of the display unit 217, as shown in FIG.

[0087] In addition, the CPU 11a may generate control information that causes the device 200 to notify the user that there is a possibility that the other nearby person 42 will speak when the situation identified by the acquired situation information is one in which one nearby person 42 is looking at another nearby person 42. In the above, a determination is made as to whether or not there is a possibility that a surrounding person 42 will speak based on the situation information of the surrounding person 42. However, this is not limited to this, and a determination may be made as to whether or not there is a possibility that another surrounding person 42 will speak based on the situation information of one surrounding person 42.

[0088] When the CPU 11a determines that there is a possibility that the other nearby person 42 is speaking, the CPU 11a generates control information for associating the display image 45 with the other nearby person 42, for example. More specifically, the CPU 11a generates control information for associating the display image 45 with the other nearby person 42 appearing on the display unit 217, for example.

[0089] Specifically, for example, when a nearby person 42 captured in an image obtained by the equipment camera 214 continuously looks at another nearby person 42 captured in the image for more than a predetermined period of time, the CPU 11a determines that a situation exists in which there is a possibility that the other nearby person 42 will speak. In this case, the CPU 11a generates control information for displaying the display image 45 in association with the other nearby person 42.

[0090] [Explanation of processing flow] FIG. 12 is a flowchart showing the flow of the process executed when the above notification process is performed. The flow of the above-mentioned process will now be described. In this embodiment, first, the CPU 11a of the management server 300 determines whether the situation of each of the surrounding persons 42 located around the target person 41 is the above-mentioned specific situation (step S101). Then, when the CPU 11a determines that the situation of the nearby person 42 is in a specific situation, it identifies the nearby person 42 in this specific situation (step S102).

[0091] After that, the CPU 11a generates control information for causing the display image 45 corresponding to the nearby person 42 in the specific situation to be displayed on the display unit 217 (step S103). As a result, display unit 217 of device 200 displays display image 45 indicating that there is a possibility of speech. Thereafter, the CPU 11a judges, based on the voice information of the nearby person 42 identified as being in a specific situation, whether or not the identified nearby person 42 has actually spoken (step S104).

[0092] Then, when the CPU 11a does not determine that the identified nearby person 42 has actually spoken, the CPU 11a generates control information for erasing the display image 45 corresponding to the nearby person 42 (step S105). As a result, the display image 45 displayed on the display unit 217 of the device 200 is erased. On the other hand, when the CPU 11a determines that the identified nearby person 42 has actually spoken, it analyzes the voice information to obtain the speech content 48 corresponding to this nearby person 42 (step S106).

[0093] Next, the CPU 11a generates control information for displaying the speech content 48 within the display image 45 (step S107). In this case, the CPU 11a generates control information for displaying the speech content 48 within the display image 45 that is displayed in association with the identified nearby person 42. As a result, the spoken content 48 is displayed within the display image 45.

[0094] [Notification process for end of speech] Next, a process of notifying the end of speech will be described. The above describes the notification process regarding the possibility of utterance. Additionally, a notification may be sent to the target person 41 via the device 200 indicating that the speech may be coming to an end or that the speech has ended. In this embodiment, as described above, the CPU 11a of the management server 300 acquires situation information that is information about the situation of the nearby person 42 before actually speaking. Then, when there is a possibility that the nearby person 42 will speak, the CPU 11a performs a notification indicating the possibility of speaking, as described above. Hereinafter, in this specification, this situation information, which is information about the situation of the surrounding person 42 before an utterance is actually made, is referred to as "pre-utterance situation information."

[0095] Furthermore, in the process described below, after the nearby person 42 starts actual speech, the CPU 11a of the management server 300 acquires situation information that is information about the situation of the nearby person 42 who is speaking. Hereinafter, in this specification, this situation information regarding the situation of the surrounding person 42 who is speaking is referred to as "speech situation information." Then, the CPU 11a generates control information used for controlling the device 200 even when the situation specified by the acquired utterance situation information is a specific situation.

[0096] Specifically, the CPU 11a generates, as this control information, control information that causes the device 200 to issue a notification indicating that the speech of the nearby person 42 may end (hereinafter referred to as a "notification indicating possibility of end"). In other words, the CPU 11a generates, as the control information, control information for causing the device 200 to issue an end possibility indication notification indicating that the speech of the nearby person 42 who is the subject of the display image 45 may be ending.

[0097] In addition, when the situation identified by the acquired speech situation information is a specific situation, the CPU 11a generates control information that causes the device 200 to issue a notification indicating that the surrounding person 42 has finished speaking (hereinafter referred to as an "end notification"). In other words, the CPU 11a generates, as this control information, control information for causing the device 200 to perform an end notification indicating that the speech of the nearby person 42, who is the subject of the display image 45, has ended.

[0098] [Notice suggesting possibility of termination] The termination possibility indication notification will be explained. When the situation specified by the acquired speech situation information is one in which there is a possibility that the surrounding person 42 will finish speaking, the CPU 11a generates control information for causing the device 200 to issue a notification indicating the possibility of the end of speaking. The CPU 11a generates control information for making the device 200 perform an end possibility suggestion notification when the situation specified by the acquired utterance situation information is, for example, one of the following situations. When a situation is identified by audio information, such as a lower tone of voice or a lower pitch of speech of a nearby person 42, When the nearby person 42 performs a specific action that has been determined in advance, such as lowering a raised hand

[0099] [Notice of Completion] Next, the end notification will be described. When the situation specified by the acquired speech situation information indicates that the speech of the nearby person 42 has ended, the CPU 11a generates control information for causing the device 200 to notify the end of the speech. Specifically, the CPU 11a generates control information for causing the device 200 to perform an end notification when the situation specified by the acquired utterance situation information is, for example, one of the following situations: If audio information is no longer available When the facial expression of the person nearby 42 becomes a specific state, such as when the mouth of the person nearby 42 becomes closed

[0100] The CPU 11a acquires information on the speech state of the nearby person 42 based on the voice information of the nearby person 42 and the video in which the nearby person 42 is shown. More specifically, the CPU 11 a acquires information on the surrounding person 42 speaking based on the voice information obtained by the individual microphone 600 and the images obtained by the equipment camera 214 and the overall camera 500 . Then, the CPU 11a generates control information for making the device 200 give a termination possibility suggestion notice or a termination notice when the situation specified by the utterance situation information is a predetermined situation.

[0101] Note that the information used to acquire the pre-utterance situation information and the information used to acquire the utterance situation information may be different. Specifically, for example, pre-utterance situation information may be obtained based on video in which the surrounding person 42 is shown, and during-utterance situation information may be obtained based on audio information of the surrounding person 42.

[0102] When pre-utterance situation information is acquired, the surrounding person 42 often has not made a clear utterance, so it is easier to accurately judge the possibility of the surrounding person 42 making an utterance by acquiring pre-utterance situation information based on video in which the surrounding person 42 is shown. On the other hand, when the speech status information is acquired, the surrounding person 42 is actually speaking, so acquiring the speech status information based on audio information tends to improve the accuracy of determining the possibility of the speech ending and the end of the speech compared to acquiring the speech status information based on video.

[0103] The above control information for causing the device 200 to issue the termination possibility suggestion notice or termination notice is transmitted to the device 200 in the same manner as described above. In response to this, in this embodiment, the device 200 performs control based on this control information, and issues an end possibility suggestion notice or an end notice to the subject person 41. This allows the target person 41 to recognize that the speech of the surrounding person 42 is about to finish or that the speech of the surrounding person 42 has ended.

[0104] [Specific example of processing] 13(A) to (D) are diagrams showing a specific example of the process. FIG. 13(A) shows a situation in which a nearby person 42 is speaking. In this embodiment, in the situation shown in FIG. 13(A), utterance contents 48 are displayed within the area surrounded by the display image 45. The CPU 11a of the management server 300 acquires speaking situation information when the nearby person 42 is speaking as shown in FIG. 13(A). More specifically, the CPU 11a acquires speaking status information of the nearby person 42 to whom the display image 45 is associated, based on the images obtained by the overall camera 500 and the equipment camera 214, and the audio information obtained by the individual microphone 600.

[0105] Then, when the situation specified by the acquired speech situation information is a situation in which there is a possibility that the surrounding person 42 will finish speaking, the CPU 11a generates control information for causing the device 200 to issue a notification indicating a possibility of end. Furthermore, when the situation specified by the acquired speech situation information indicates that the speech of the nearby person 42 has ended, the CPU 11a generates control information for causing the device 200 to perform an end notification.

[0106] FIG. 13B shows the state of the display unit 217 of the device 200 when the situation specified by the speech situation information is a situation in which there is a possibility that the speech of the nearby person 42 will end. When there is a possibility that the speech of the nearby person 42 will end, the CPU 11a generates control information for making the device 200 give a notification indicating the possibility of end, as described above. In this processing example, the CPU 11a generates control information for changing the display image 45 displayed on the display unit 217 of the device 200 as the control information for causing the device 200 to perform the termination possibility suggestion notification. In the following, in this specification, the control information that causes the device 200 to issue a termination possibility suggestion notification is referred to as "first control information."

[0107] In this processing example, the CPU 11a generates, as the first control information, control information for changing a display image 45 displayed in association with a display location 47 of a speech content 48 (see FIG. 13(A)) of a nearby person 42. Here, the display image 45 can be regarded as a corresponding display image that is displayed in correspondence with the display location 47 of the speech content 48 of the nearby person 42. In this embodiment, the display image 45 showing a speech bubble is displayed as this corresponding display image. The CPU 11a generates, as first control information for causing the device 200 to perform a termination possibility suggestion notification, control information for changing the display image 45 representing the speech bubble, which is an example of a corresponding display image.

[0108] More specifically, CPU 11a generates control information for changing the shape of display image 45, which is displayed in a manner surrounding display location 47 and represents a speech bubble, as the first control information for changing the corresponding display image. Specifically, the CPU 11a generates control information for erasing a protrusion 45G (see FIG. 13(A)) provided as part of a display image 45 representing a speech bubble, as first control information for changing the corresponding display image.

[0109] As a result, in this embodiment, the protrusion 45G is eliminated as shown in FIGS. In this embodiment, the protrusion 45G provided on the display image 45 displayed in association with the nearby person 42 who is in a situation where there is a possibility that the speech will end is erased. By recognizing that the protruding portion 45G has been erased, the target person 41 recognizes that the speech of the nearby person 42 may be finished.

[0110] In this embodiment, as described above, when the situation identified by the pre-utterance situation information is a specific situation, the CPU 11a generates control information for causing a display image 45 representing a speech bubble to be displayed on the display unit 217 provided in the device 200. As a result, in this embodiment, first, the display image 45 showing the speech bubble is displayed on the display unit 217 of the device 200 in a manner associated with the nearby person 42, as shown in FIG. 13(A). The display image 45 is provided with a protruding portion 45G that protrudes toward the surrounding person . In this embodiment, a nearby person 42 who may be about to speak or who is currently speaking is located ahead of the protruding portion 45G in the protruding direction.

[0111] The CPU 11a generates, as first control information for causing the device 200 to perform an end suggestion notification, control information for causing the display form of the display image 45 displayed on the display unit 217 to be changed. Specifically, the CPU 11a generates, as the first control information, control information for making the protruding portion 45G of the display image 45 invisible. As a result, in this embodiment, as described above, protrusion 45G of display image 45 is hidden.

[0112] Furthermore, as described above, when the speech of the nearby person 42 has actually ended, the CPU 11a generates control information for causing the device 200 to issue an end notification, which is a notification indicating that the speech has ended. Hereinafter, this control information that causes the device 200 to issue an end notification is referred to as "second control information." The CPU 11a also generates, in this case, control information for changing the display image 45 displayed on the display unit 217 of the device 200, as the second control information. More specifically, the CPU 11a generates, as the second control information, control information for changing the display image 45 displayed in association with the display location 47 of the speech content 48 of the nearby person 42.

[0113] More specifically, the CPU 11a generates, as the second control information, control information for further changing the display form of the display image 45 after the display form has been changed by the above-mentioned first control information, which changes the display form of the display image 45. More specifically, the CPU 11a generates, as the second control information, control information for changing the thickness of the lines that form the display image 45.

[0114] As a result, in this embodiment, as shown in FIGS. 13(B) and (C), the lines constituting the display image 45 displayed on the display unit 217 of the device 200 become thinner. In other words, in this embodiment, the lines constituting the display image 45 displayed in association with the surrounding person 42 who is speaking become thinner. This allows the target person 41 to recognize that the surrounding person 42 has finished speaking.

[0115] In this embodiment, there is a time difference between the timing when the speech of the surrounding person 42 ends and the timing when the speech content 48 at the time when the speech of the surrounding person 42 ends is displayed on the display unit 217. In this case, in this embodiment, as shown in Figures 13 (C) and (D), the display process of the speech content 48 continues even after the timing when the surrounding person 42 finishes speaking, until all of the speech content 48 is displayed. In other words, in this embodiment, even if the lines constituting the display image 45 become thinner, the display process of the utterance content 48 does not end, but continues until the entire utterance content 48 is displayed.

[0116] The target person 41 can recognize the end of the speech of the nearby person 42 by recognizing the end of the display process of the speech content 48. Incidentally, in this embodiment, even though the speech of the nearby person 42 has already ended, the display process continues until the entire speech content 48 is displayed. In this case, the target person 41 is likely to mistakenly believe that the surrounding person 42 is still speaking, even though the surrounding person 42 has already finished speaking.

[0117] In this case, a blank period of time during which no speech is made is likely to occur between the end of speech by nearby person 42 and the start of speech by target person 41. In contrast, when an end possibility suggestion notice or end notice is given as in this embodiment, the target person 41 can recognize the end of the speech of the surrounding person 42 at an earlier stage. In this case, the target person 41 can speak immediately after the surrounding person 42 finishes speaking.

[0118] 14(A) to (I) are diagrams showing a series of flow of the display process. 14(A) shows a situation where the nearby person 42 is not speaking and there is no possibility that the nearby person 42 will speak, and in this case, no voice is detected. In this case, the CPU 11a does not generate control information for displaying the display image 45, and the display unit 217 of the device 200 does not display the display image 45. FIG. 14B shows a situation where there is a possibility of speech, in which case a display image 45 is displayed on the display unit 217 of the device 200.

[0119] 14(C) to (F) show a situation in which a nearby person 42 is in the middle of speaking. In this case, the CPU 11a acquires the speech content 48, and further generates control information for causing the speech content 48 to be displayed. As a result, as indicated by the reference symbol 13X, the utterance contents 48 are sequentially displayed on the display unit 217 of the device 200. More specifically, the utterance contents 48 are sequentially displayed inside the display image 45 displayed on the display unit 217 of the device 200. In this embodiment, there is a time lag between the timing at which the CPU 11a acquires the voice information and the timing at which the spoken content 48 is displayed on the display unit 217 of the device 200. Therefore, in this embodiment, as indicated by an arrow 14Y in FIG. 14, the speech content 48 is displayed after the CPU 11a acquires the voice information.

[0120] FIG. 14(F) shows a situation where the speech of the nearby person 42 may end. In this case, in this embodiment, the protrusion 45G (see FIG. 14(E)) that was displayed as part of the display image 45 is erased. 14(G) and subsequent figures show a situation in which the speech of the nearby person 42 has ended. In this case, as shown in Fig. 14(G) and (H), the lines constituting the display image 45 become thinner.

[0121] In this embodiment, when there is a possibility that the speech is about to end, at least one of the shape, thickness, and color of the display image 45 is changed, and when the speech has actually ended, at least one of the shape, thickness, and color of the display image 45 is further changed. In the above, an example has been described in which the shape of display image 45 is changed when there is a possibility that speech has ended, and the thickness of the lines that make up display image 45 is changed when speech has actually ended. In other words, in the above, an example has been described in which the shape of the display image 45 is first changed, and then the thickness of the lines constituting the display image 45 is changed.

[0122] As another example, first, the thickness of the lines constituting the display image 45 may be changed, and then the shape of the display image 45 may be changed. Alternatively, the shape of the display image 45 may be changed first, and then the shape of the display image 45 may be further changed. Alternatively, the lines constituting the display image 45 may be thinned first, and then the lines constituting the display image 45 may be further thinned. Alternatively, the lines constituting the display image 45 may be thickened first, and then the lines constituting the display image 45 may be further thickened. Alternatively, the color of the display image 45 may be changed first, and then the color of the display image 45 may be changed again.

[0123] [Notification processing format] In this embodiment, at the end of an utterance, a visual notification is given to the target person 41, similar to the start of an utterance. Specifically, in this embodiment, as a visual notification to the subject person 41, as described above, the display image 45 is changed a first time, and then the display image 45 is changed a second time. As another visual notification to the subject 41 when the speech ends, for example, a text image such as "Speech may end" or "Speech ended" may be displayed on the display unit 217 of the device 200.

[0124] Furthermore, when speech ends, the visual notification of the subject 41 may be given by, for example, turning on or off a light source (not shown) provided in the device 200. In addition, when an utterance ends, a visual notification to the subject 41 may be given by changing the overall color of the display screen displayed on the display unit 217, or by changing the color of a part of the display screen, such as the edge of the display screen displayed on the display unit 217.

[0125] Furthermore, the notification at the end of speech may be given, for example, by vibrating a vibration source (not shown) provided in device 200. Furthermore, the notification of the end of speech may be given, for example, by making a sound or voice be emitted from speaker 216 (see FIG. 3) provided in device 200. If the subject 41 is hearing impaired, it is difficult to notify the subject 41 by sound, but if the subject 41 is not hearing impaired, the subject 41 can be notified of the possibility of termination or termination by sound.

[0126] [Explanation of processing flow] Fig. 15 is a flowchart showing a process flow in the case where a notification process is performed even at the end of an utterance. Note that the processes in steps S201 to S207 in Fig. 15 are the same as the processes in steps S101 to S107 shown in Fig. 12. In this embodiment, first, similarly to the above, the CPU 11a determines whether the situation specified by the pre-utterance situation information for each of the surrounding persons 42 located around the target person 41 is the above-mentioned specific situation (step S201).

[0127] Then, when the CPU 11a determines that the situation specified by the pre-utterance situation information is a specific situation, it specifies the nearby person 42 who is in this specific situation (step S202). In other words, when the CPU 11a determines that the situation specified by the pre-utterance situation information is a situation in which utterance is likely to occur, it specifies a nearby person 42 who is likely to utter.

[0128] Next, the CPU 11a generates control information for causing the display image 45 corresponding to the nearby person 42 identified as being in the specific situation to be displayed on the display unit 217 (step S203). As a result, a display image 45 shown in FIG. 14(B) is displayed on the display unit 217 of the device 200 in a form corresponding to the nearby person 42 who may possibly speak. The displayed image 45 is provided with a protrusion 45G.

[0129] Thereafter, the CPU 11a judges, based on the voice information of the nearby person 42 who is the display target of the display image 45, whether or not this nearby person 42 has actually spoken (step S204). Then, when the CPU 11a does not determine that the nearby person 42 has actually spoken, the CPU 11a generates control information for erasing the display image 45 (step S205). As a result, the display image 45 displayed on the display unit 217 of the device 200 is erased. On the other hand, when the CPU 11a determines that the nearby person 42 has actually spoken, it analyzes the voice information to obtain the speech content 48 (step S206).

[0130] Next, the CPU 11a generates control information for displaying the utterance content 48 in the display image 45 (step S207). As a result, the utterance content 48 is displayed in the display image 45, as shown by reference numeral 13X in FIG. In the processing examples shown in Figures 13 and 14 above, a case has been described in which display image 45 composed of thick lines is displayed at a stage where there is a possibility that nearby person 42 will speak, but the representation form of display image 45 is not limited to this. At a stage where there is a possibility that the surrounding person 42 will speak, the display image 45 made up of thin lines may be displayed. Then, when the surrounding person 42 starts to actually speak, the lines constituting the display image 45 may be made thicker.

[0131] Next, in step S208, the CPU 11a determines whether or not there is a possibility that the speech of the nearby person 42 will end. Then, when the CPU 11a determines that the speech of the nearby person 42 is likely to end, the CPU 11a generates control information for erasing the protruding portion 45G (step S209). As a result, the protruding portion 45G is erased as described above. Next, the CPU 11a judges whether the speech of the surrounding person 42 has ended (step S210). If the CPU 11a judges that the speech of the surrounding person 42 has ended, the CPU 11a generates control information for thinning the lines constituting the display image 45 (step S211).

[0132] 〔others〕 The above describes three cases in which notification processing is performed: notification processing regarding the possibility of speaking before speaking, notification processing regarding the possibility of the end of speaking after speaking has started, and notification processing regarding the end of speaking after speaking has started. It is not essential that all of these three processes are performed, and only one of the processes may be performed, or two of the processes may be performed. In the above, we have described a case where two notification processes are performed when an utterance ends: a notification process about the possibility of the utterance ending, and a notification process when the utterance has actually ended.However, it is also possible to perform only one of these two notification processes when an utterance ends.

[0133] (Additional Note) (((1))) A processor is provided. The processor, Obtain situation information that is information about the situations of people who are located in the vicinity of the target person; generating control information used to control a device owned by the subject person when the situation specified by the acquired situation information is a specific situation, the control information causing the device to notify the subject person that there is a possibility of speech from the surrounding person; Information processing system. (((2))) The processor, acquiring the situation information of the surrounding person based on an image in which the surrounding person is captured; An information processing system according to (((1))). (((3))) The processor, generating control information for causing the device to notify the user of the absence of speech when the user is not speaking; 2. An information processing system according to claim 1, wherein the information processing system is a system for processing information according to any one of claims 1 to 10. (((4))) The processor, generating control information for causing the device to issue the notification indicating the possibility of the utterance, the control information causing the device to issue a notification that appeals to the subject's visual sense; 13. An information processing system according to any one of ((1))) to (((3))). (((5))) The processor, generating control information for causing the notification indicating the possibility of the speech to be performed by the device to display a display image, which is an image displayed on the device and indicates the possibility of the speech of the nearby person, on a display unit of the device; 13. An information processing system according to any one of ((1))) to (((4))). (((6))) The processor, generating control information for erasing the display image displayed on the display unit of the device when there is no actual speech by the person nearby after generating the control information for causing the display image to be displayed on the display unit of the device; The information processing system according to (((5))). (((7))) The processor, generating control information for displaying the display image to the nearby people in a corresponding manner as the control information for displaying the display image on the display unit of the device; The information processing system according to (((5))) or (((6))). (((8))) The processor, generating control information for displaying the display image in a manner associated with each of the plurality of surrounding persons when the plurality of surrounding persons are in the specific situation, as the control information for displaying the display image in a manner associated with the plurality of surrounding persons; An information processing system according to (((7))). (((9))) When the nearby person who may possibly make the utterance actually makes an utterance, the utterance content, which is the content of the utterance, is displayed on the display unit of the device, The processor, generating control information for displaying the display image on the display unit of the device in a manner that the display image is displayed in association with a display location on the display unit of the device where the speech content is displayed, as the control information for displaying the display image on the display unit of the device; 13. An information processing system according to any one of ((5))) to (((8))). (((10))) The processor, generating control information for causing the display image to be displayed on the display unit so as to have a shape surrounding the display location, as the control information for causing the display image to be displayed in a form corresponding to the display location; The information processing system according to (((9))). (((11))) The processor, generate control information for causing a display image, which is an image indicating that the surrounding person who is out of the field of view of the target person viewing the area ahead of the target person through the device, is likely to be speaking, to be displayed on a display unit of the device when the surrounding person who is out of the field of view of the target person is in the specific situation; 13. An information processing system according to any one of ((1))) to (((10))). (((12))) The processor, generating control information for causing the display image of the surrounding person who is out of the field of view to be displayed on the display unit of the device, the control information causing the display image to be displayed on a line connecting the position of the surrounding person when the surrounding person is projected onto a virtual plane along the display unit and a center of the display unit; The information processing system according to (((11))). (((13))) The processor, In addition to pre-utterance situation information, which is the situation information acquired before the actual utterance by the surrounding person, utterance situation information, which is information about the situation of the surrounding person who actually starts utterance, is further acquired; When the situation specified by the acquired speaking situation information is a specific situation, generate control information used for controlling the device, which causes the device to issue an end suggestion notification, which is a notification indicating that the surrounding person's speech may end, or generate control information for causing the device to issue an end notification, which is a notification indicating that the surrounding person's speech has ended. 13. An information processing system according to any one of ((1))) to (((12))). (((14))) The processor, generating control information for causing a predetermined display image to be displayed on a display unit provided on the device when the situation specified by the pre-utterance situation information is the specific situation; generating control information for changing a display form of the display image displayed on the display unit as the control information for causing the device to perform the end suggestion notification; The information processing system according to (((13))). (((15))) The processor, generating control information for further changing the display form of the display image whose display form has been changed by the control information for changing the display form of the display image, as the control information for causing the device to perform the end notification; The information processing system according to (((14))). (((16))) The processor, Acquire the pre-utterance situation information based on the video in which the surrounding people are captured; The speech situation information is acquired based on voice information, which is information about the voice of the surrounding person. 13. An information processing system according to any one of ((13))) to (((15))). (((17))) A processor is provided. The processor, Acquire situation information which is information about a situation of a person who is speaking and is located in the vicinity of the subject, generating control information used for controlling a device owned by the subject when the situation specified by the acquired situation information is a specific situation, the control information causing the device to issue a notification indicating that the speech of the surrounding person may end, and / or generating control information causing the device to issue a notification indicating that the speech of the surrounding person has ended; Information processing system. (((18))) The processor, acquiring the situation information of the surrounding person based on voice information which is information about the voice of the surrounding person; The information processing system according to (((17))). (((19))) The processor, The information processing system according to (((17))) or (((18))) generates control information for changing a display image being displayed on a display unit of the device as the control information for causing the device to issue a possibility suggestion notification, which is the notification indicating that the speech may end. (((20))) The processor, generating control information for changing a display image displayed on the display unit in association with a display location of the speech content of the surrounding person as the control information for causing the device to perform the possibility suggestion notification; The information processing system according to (((19))). (((twenty one))) The processor, An information processing system according to any one of (((17))) to (((20))), further comprising generating control information for changing a display image displayed on a display unit of the device as the control information for causing the device to issue an end notification, which is the notification indicating that the speech has ended. (((twenty two))) The processor, generating control information for changing a display image displayed on the display unit in association with a display portion of the speech content of the surrounding person as the control information for causing the device to perform the end notification; An information processing system according to (((21))). (((twenty three))) The processor, generating control information for changing at least one of a shape, a thickness, and a color of the corresponding display image that is displayed in a manner surrounding the display location of the utterance content, as the control information for changing the corresponding display image that is the display image that is displayed in correspondence with the display location of the utterance content; 2. An information processing system according to claim 1, wherein the information processing system is an information processing system according to claim 1. (((twenty four))) An acquisition function for acquiring situation information, which is information about the situation of a person nearby who is located near the subject; a generation function for generating control information used to control a device owned by the subject when the situation specified by the situation information acquired by the acquisition function is a specific situation, the control information being used to generate control information that causes the device to notify the subject that there is a possibility that the subject will speak; A program to make the above happen on a computer. (((twenty five))) An acquisition function for acquiring situation information, which is information about the situation of a person who is a person in the vicinity of the subject and is speaking; a generation function for generating control information used to control a device owned by the subject when the situation specified by the situation information acquired by the acquisition function is a specific situation, the control information causing the device to issue a notification indicating that the speech of the surrounding person may end, and / or generating control information causing the device to issue a notification indicating that the speech of the surrounding person has ended; A program to make the above happen on a computer.

[0134] According to the information processing system of (((1))), it is easier to identify the timing of an utterance that a speaker is about to make, compared to a configuration in which the speaker is not notified about the utterance. According to the information processing system of (((2))), situation information can be acquired based on the movements of people in the vicinity. According to the information processing system of (((3))), it becomes easier for the subject to determine whether or not those around him / her are speaking, compared to a configuration in which the device does not notify the subject of the absence of speech. The information processing system according to (((4))) makes it easier for a hearing-impaired person to determine whether or not there is a possibility that someone nearby is speaking, compared to a configuration in which a notification that appeals to the subject's hearing is sent by a device. According to the information processing system of (((5))), the subject can obtain information about the possibility of speech by people nearby by referring to the display unit of the device. According to the information processing system of (((6))), it is possible to inform a subject that the possibility of a person nearby speaking has decreased. According to the information processing system of (((7))), the subject can more easily identify nearby people who may be likely to speak, compared to a configuration in which a display image cannot be associated with a nearby person. According to the information processing system of (((8))), it is easier for a subject to identify a nearby person who may be speaking, compared to a configuration in which a display image is not displayed in a form associated with each of multiple nearby persons. According to the information processing system of (((9))), it is easier for the subject to recognize the location where the speech content is displayed, compared to a configuration in which a display image is displayed without being associated with the display location where the speech content is displayed. According to the information processing system of (((10))), the subject can more clearly recognize the location where the speech content is displayed, compared to a configuration in which the display image does not have a shape surrounding the display location. According to the information processing system of (((11))), it is possible to display on the display unit a display image corresponding to a nearby person who is located outside the field of vision of the subject viewing the area ahead of the subject through the device. According to the information processing system of (((12)), it is easier for the subject to identify in which direction the surrounding person is located, compared to a configuration in which the display image is displayed at a location that is off the line connecting the position of the surrounding person when the surrounding person is projected onto a virtual plane along the display unit and the center of the display unit. According to the information processing system of (((13))), it is easier to identify the timing of an utterance that the speaker is about to make, compared to a configuration in which no notification regarding the end of utterance is given. According to the information processing system of (((14))), the subject can determine whether or not the person nearby is likely to finish speaking by referring to the display image displayed on the display unit of the device. According to the information processing system of (((15))), the subject can determine whether or not the person nearby has finished speaking by referring to the display image displayed on the display unit of the device. According to the information processing system of (((16))), the accuracy of judgment as to the possibility of a surrounding person speaking can be improved compared to a configuration in which pre-utterance situation information is obtained based on the voice information of the surrounding person, and the accuracy of judgment as to the end of a surrounding person's speech can be improved compared to a configuration in which during-utterance situation information is obtained based on video in which the surrounding person is shown. According to the information processing system of (((17))), it is easier to identify the timing of an utterance that a speaker is about to make, compared to a configuration in which the speaker is not notified about the utterance. According to the information processing system of (((18))), it is possible to improve the accuracy of determining when a nearby person has finished speaking, compared to a configuration in which situational information about a nearby person who is speaking is obtained based on video in which the nearby person is shown. According to the information processing system of (((19))), a subject can determine whether or not a person nearby is likely to finish speaking by referring to the display image displayed on the display unit of the device. According to the information processing system of (((20))), it is easier to identify a target person who is likely to finish speaking, compared to a configuration in which a display image that is not displayed is changed in correspondence with the display location of the speech content of a nearby person. According to the information processing system of (((21))), a subject can determine whether or not a person nearby has finished speaking by referring to the display image displayed on the display unit of the device. According to the information processing system of (((22))), it is easier to identify a subject who has finished speaking, compared to a configuration in which a display image that is not displayed is changed in correspondence with the display location of the speech content of a nearby person. According to the information processing system of (((23))), a subject can determine whether or not there is a possibility that a person nearby will finish speaking, or whether or not a person nearby has finished speaking, by referring to the display image displayed on the display unit of the device. The program according to (((24))) makes it easier for the speaker to determine the timing of an utterance that he or she is about to make, compared to a configuration in which the speaker is not notified about the utterance. According to the program of (((25))), it is possible to make it easier for the speaker to identify the timing of an utterance that he or she is about to make, compared to a configuration in which the speaker is not notified about the utterance. [Explanation of symbols]

[0135] 1...information processing system, 10K...virtual plane, 11a...CPU, 11L...straight line, 41...target person, 42...surrounding person, 45...display image, 47...display location, 48...speech content, 200...device, 217...display unit, 217C...center unit

Claims

1. A processor is provided. The processor, Acquire situation information of the target person or a user who is a periphery of the target person; generating control information used to control a device owned by the user when the acquired situation information indicates a situation in which the user has made a predetermined movement, the control information causing the device to issue a notification indicating that the user who made the movement is likely to speak before the user starts speaking; Information processing system.

2. The processor, acquiring the situation information of the user based on an image showing the people nearby; The information processing system according to claim 1 .

3. The processor, generating control information for causing the device to notify the user of the absence of speech when the user is not speaking; The information processing system according to claim 1 .

4. The processor, generating control information for causing the device to issue the notification indicating the possibility of the utterance to a visual display of the subject; The information processing system according to claim 1 .

5. The processor, generating control information for causing the notification indicating the possibility of the speech to be performed by the device, so that a display image, which is an image displayed on the device and indicates the possibility of the speech of the nearby person, is displayed on a display unit of the device; The information processing system according to claim 1 .

6. The processor, generating control information for erasing the display image displayed on the display unit of the device when there is no actual speech by the person nearby after generating the control information for causing the display image to be displayed on the display unit of the device; 6. The information processing system according to claim 5.

7. The processor, generating control information for displaying the display image to the nearby people in a corresponding manner as the control information for displaying the display image on the display unit of the device; 6. The information processing system according to claim 5.

8. The processor, generating control information for displaying the display image in a manner associated with each of the plurality of surrounding persons as the control information for displaying the display image in a manner associated with each of the plurality of surrounding persons when the plurality of surrounding persons are in a situation in which they have performed the predetermined movement; The information processing system according to claim 7.

9. When the nearby person who may possibly make the utterance actually makes an utterance, the utterance content, which is the content of the utterance, is displayed on the display unit of the device, The processor, generating control information for displaying the display image on the display unit of the device in a manner that the display image is displayed in association with a display location on the display unit of the device where the speech content is displayed, as the control information for displaying the display image on the display unit of the device; 6. The information processing system according to claim 5.

10. The processor, generating control information for causing the display image to be displayed on the display unit so as to have a shape surrounding the display location, as the control information for causing the display image to be displayed in a form corresponding to the display location; The information processing system according to claim 9.

11. The processor, generate control information for causing a display image, which is an image indicating that the surrounding person out of the field of view of the target person viewing the area ahead of the target person through the device may be speaking, to be displayed on a display unit of the device when the surrounding person out of the field of view has made the predetermined movement; The information processing system according to claim 1 .

12. The processor, generating control information for causing the display image of the surrounding person who is out of the field of view to be displayed on the display unit of the device, the control information causing the display image to be displayed on a line connecting the position of the surrounding person when the surrounding person is projected onto a virtual plane along the display unit and a center of the display unit; The information processing system according to claim 11.

13. The processor, In addition to pre-utterance situation information, which is the situation information acquired before the actual utterance by the surrounding person, utterance situation information, which is information about the situation of the surrounding person who actually starts utterance, is further acquired; When the situation specified by the acquired speaking situation information is a specific situation, generate control information used for controlling the device, which causes the device to issue an end suggestion notification, which is a notification indicating that the surrounding person's speech may end, or generate control information for causing the device to issue an end notification, which is a notification indicating that the surrounding person's speech has ended. The information processing system according to claim 1 .

14. The processor, generating control information for causing a predetermined display image to be displayed on a display unit provided on the device when the situation specified by the pre-utterance situation information is the specific situation; The information processing system according to claim 13 , further comprising: generating control information for changing a display form of the display image displayed on the display unit as the control information for causing the device to issue the end suggestion notification.

15. The processor, generating control information for further changing the display form of the display image whose display form has been changed by the control information for changing the display form of the display image, as the control information for causing the device to perform the end notification; 15. The information processing system according to claim 14.

16. The processor, The pre-utterance situation information is acquired based on an image in which the surrounding people are captured; The speech situation information is acquired based on voice information, which is information about the voice of the surrounding person. The information processing system according to claim 13.

17. A processor is provided. The processor, Acquire biometric information of a user who is a target person or a nearby person; generating control information used to control a device owned by the user when the acquired biometric information is in a specific situation, the control information causing the device to notify the user that the acquired biometric information may be a speech of the user in the specific situation before the user starts speaking; Information processing system.

18. A processor is provided. The processor, Acquire situation information of people around the target person, When the surrounding person is speaking, a notification is given indicating that the surrounding person's speech may end before the surrounding person's speech ends based on the acquired situation information. An information processing system comprising:

19. The processor, acquiring the situation information of the surrounding person based on voice information which is information about the voice of the surrounding person; 20. The information processing system according to claim 18.

20. The processor, The information processing system of claim 18, further comprising: control information used to control a device possessed by the subject person when the situation identified by the acquired situation information is a specific situation, the control information causing the device to notify the subject that the speech of the person in the vicinity may end, the control information changing the display image displayed on a display unit of the device.

21. The processor, generating, as the control information, control information for changing a display image displayed on the display unit in association with a display portion of the speech content of the surrounding person; 21. The information processing system according to claim 20.

22. The processor, generating control information for changing at least one of a shape, a thickness, and a color of the corresponding display image that is displayed in a manner surrounding the display location of the utterance content, as the control information for changing the corresponding display image that is the display image that is displayed in correspondence with the display location of the utterance content; 22. The information processing system according to claim 21.

23. A function for acquiring situational information of a user who is a target or a surrounding person; a function for generating control information used to control a device owned by the user when the acquired situation information indicates a situation in which the user has made a predetermined movement, the control information being used to generate control information that notifies the device that the user who made the movement is likely to speak before the user starts speaking; A program to make the above happen on a computer.

24. A function for acquiring biometric information of a user who is a subject or a nearby person; a function for generating control information used to control a device owned by the user when the acquired biometric information is in a specific situation, the control information causing the device to notify the user that the acquired biometric information may be a speech of the user in the specific situation before the user starts speaking; A program to make the above happen on a computer.

25. A function of acquiring situation information of people around the target person; A function of notifying the user that the speech of the surrounding person may end before the speech of the surrounding person who is currently speaking ends based on the acquired situation information when the surrounding person is currently speaking; A program to make the above happen on a computer.

Images