Information processing system, operation device, and program

Abstract

To improve the accuracy of matching between speech and a speaker to which it corresponds, compared to when multiple potential speakers each independently declare their own speech using their own operation device.SOLUTION: (A) shows a first peripheral person 421 who is speaking. Further, in (A), an operation device 300 is facing the first peripheral person 421. In this case, the CPU of the operation device 300 identifies the first peripheral person 421 as a speaking peripheral person. Subsequently, as shown in (B), a second peripheral person 422 begins speaking. Accordingly, a selector 43 changes the orientation of the operation device 300, resulting in the state shown in (B). In the state of (B), the operation device 300 faces the second peripheral person 422. In state (B), the CPU of the operation device 300 identifies the second peripheral person 422.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an operation device, and a program. [Background technology]

[0002] Patent document 1 discloses a conference support system having a terminal used by each of multiple conference participants and a conference support device, in which the terminal has an operation unit for setting whether to make a speech and a self-speech notification unit for notifying other terminals of information indicating that a speech will be made. Patent document 2 discloses a process for generating a map image in which information about the content of audio is positioned according to the direction of the audio source relative to a multi-microphone device, and displaying the map image on the display unit of a display device. Patent Document 3 discloses a process for managing marking actions for assigning individual markers to audio in chronological order, and for linking the audio and the marking actions on the same chronological order. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2018-174439 [Patent Document 2] Japanese Patent Publication No. 2024-027122 [Patent Document 3] Japanese Patent Publication No. 2022-015775 Summary of the Invention [Problem to be solved by the invention]

[0004] By performing a process of associating an utterance with the speaker who made the utterance, it becomes possible to identify who made the utterance. Here, the utterance and the speaker can be identified based on the speaker's own declaration of the utterance. However, speakers tend to concentrate on their own utterances. In this case, it is possible that they forget to report their utterances or that the timing of their report differs from the timing of the utterances themselves. In this case, the accuracy of the correspondence between the utterances and the speakers associated with these utterances decreases. The object of the present invention is to improve the accuracy of matching between an utterance and the speaker to which it is associated, compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. [Means for solving the problem]

[0005] The invention described in claim 1 is an information processing system that includes a processor, the processor being configured to be able to select each of a plurality of candidates who may speak, acquires operation information of an operating device that a selector operates when selecting a speaker from the plurality of candidates, identifies a speaker from the plurality of candidates based on the acquired operation information, and associates the identified speaker with the utterance made when the selector operates the operating device. The invention described in claim 2 is the information processing system described in claim 1, wherein the processor acquires, as the operation information, orientation information that is information indicating the orientation of the operating device operated by the selector or the orientation of a movable part provided on the operating device, identifies the speaker based on the orientation information, and associates the identified speaker with an utterance made when the operating device or the movable part is facing in the direction identified by the orientation information used to identify the speaker. The invention described in claim 3 is an information processing system described in claim 2, further comprising a memory unit that stores the orientation of the operating device or the orientation of the movable part in association with each of the candidates, and the processor identifies the speaker based on the orientation identified by the acquired orientation information and the information stored in the memory unit. The invention described in claim 4 is an information processing system described in claim 2, in which the processor acquires, as the orientation information, information on the orientation of the operating device or the movable part when the movement speed of the operating device or the movable part is slower than a predetermined movement speed. The invention described in claim 5 is an information processing system described in claim 2, in which the processor, after identifying a speaker, identifies another speaker whose position around the vertical axis is different from that of the speaker when the operating device or the movable part rotates around the vertical axis, changing the orientation of the operating device or the movable part. The invention described in claim 6 is an information processing system described in claim 2, in which the processor, after identifying the speaker, identifies another speaker who is closer to the operating device or the movable part than the speaker, or another speaker who is farther from the operating device or the movable part than the speaker, when the operating device or the movable part rotates around a horizontal axis and the orientation of the operating device or the movable part changes. The invention described in claim 7 is an information processing system described in claim 2, in which the processor, after identifying a speaker, identifies another speaker located above or below the speaker when the operating device or the movable part rotates around a horizontal axis, changing the orientation of the operating device or the movable part. The invention described in claim 8 is an information processing system described in claim 1, in which the processor acquires, as the operation information, information on the selection operation of buttons provided on the operating device corresponding to each of the multiple candidates, identifies the speaker based on a selection button selected by the selector among the multiple buttons, and associates the identified speaker with the utterance made when the selector selected the selection button. The invention described in claim 9 is an information processing system described in claim 1, in which each operation of the operating device performed by the selector is associated with each of the candidates, and the processor further generates control information that causes candidate information, which is information about the candidate associated with the operation of the operating device performed by the selector, to be notified to the selector via the operating device. The invention described in claim 10 is an information processing system described in claim 9, in which the candidate information is displayed on a display unit of the operating device, and the processor further generates control information that changes the display of a non-display area of ​​the display unit other than the display area where the candidate information is displayed, each time the candidate information notified to the selector via the display unit changes. The invention described in claim 11 is an information processing system described in claim 10, in which the processor generates control information that changes the background color of the hidden area as the control information that changes the display of the hidden area. The invention described in claim 12 is an operating device configured to allow selection of each of a plurality of candidates who may speak and used when a selector selects a speaker, the operating device comprising a processor, the processor acquiring operation information of the operating device operated by the selector when selecting a speaker from the plurality of candidates, and identifying a speaker from the plurality of candidates based on the operation information. The invention described in claim 13 is the operating device described in claim 12, wherein the processor acquires, as the operation information, operation information for an operation to change the orientation of the operating device or operation information for an operation to change the orientation of a movable part provided on the operating device. The invention described in claim 14 is the operating device described in claim 12, in which each operation of the operating device performed by the selector is associated with each of the candidates, and the processor further generates control information that causes candidate information, which is information about the candidate associated with the operation of the operating device performed by the selector, to be notified to the selector via the operating device. The invention described in claim 15 is a program for causing a computer to realize an acquisition function for acquiring operation information of an operating device configured to allow selection of each of a plurality of candidates who may speak and operated by a selector when selecting a speaker from the plurality of candidates, an identification function for identifying a speaker from the plurality of candidates based on the operation information, and an association function for associating the identified speaker with the utterance made when the selector operates the operating device. The invention described in claim 16 is a program executed by a computer provided on an operating device configured to allow selection of each of a plurality of candidates who may speak and used by a selector to select a speaker, the program causing the computer to realize an acquisition function for acquiring operation information of the operating device operated by the selector when selecting a speaker from the plurality of candidates, and an identification function for identifying a speaker from the plurality of candidates based on the operation information. [Effects of the Invention]

[0006] According to the invention of claim 1, the accuracy of matching between an utterance and the speaker to which it is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. According to the invention of claim 2, it is possible to associate an utterance with a speaker based on an operation performed on an operating device that is at least partially changeable in direction. According to the invention of claim 3, it is possible to associate an utterance with a speaker based on information registered in advance in the storage unit. According to the invention of claim 4, the accuracy of associating an utterance with a speaker associated with the utterance can be improved compared to when information on the orientation of the operating device or the orientation of the movable part when the moving speed is greater than a predetermined moving speed is acquired as orientation information. According to the invention of claim 5, when an operation is performed on an operating device or a movable part to rotate the operating device or the movable part around a vertical axis, it becomes possible to identify a speaker other than the speaker identified before the rotation. According to the invention of claim 6, when an operation is performed on the operating device or the movable part to rotate the operating device or the movable part around a horizontal axis, it becomes possible to identify another speaker who is located closer to the operating device or the movable part than the speaker identified before the rotation, or another speaker who is located farther from the operating device or the movable part than the speaker identified before the rotation. According to the invention of claim 7, when an operation is performed on an operating device or a movable part to rotate the operating device or the movable part around a horizontal axis, it becomes possible to identify other speakers who are located above or below the speaker identified before the rotation. According to the invention of claim 8, it is possible to associate a speaker with an utterance based on an operation performed on an operating device having buttons provided corresponding to each of a plurality of candidates. According to the invention of claim 9, it is possible to notify the selector of information about the candidate identified based on the operation of the operating device performed by the selector. According to the invention of claim 10, it is possible to more clearly notify the selector that the information to be notified to the selector has been changed, compared to when only the display of candidate information is changed. According to the invention of claim 11, it is possible to more clearly notify the selector that the information to be notified to the selector has been changed, compared to when the background color of the non-displayed portion remains unchanged. According to the invention of claim 12, the accuracy of matching between an utterance and the speaker to which it is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. According to the invention of claim 13, it is possible to associate an utterance with a speaker based on an operation performed on an operating device that is at least partially changeable in direction. According to the invention of claim 14, it is possible to notify the selector of information about the candidate identified based on the operation of the operating device performed by the selector. According to the invention of claim 15, the accuracy of matching between an utterance and the speaker to which it is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. According to the invention of claim 16, the accuracy of matching between an utterance and the speaker to which it is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing the overall configuration of an information processing system. [Figure 2] FIG. 1 is a diagram showing a target person, surrounding people, and a selector. [Figure 3] FIG. 2 illustrates a configuration of a management server. [Figure 4] FIG. 2 is a diagram showing the hardware configuration of the device. [Figure 5] FIG. 2 is a diagram showing the hardware configuration of an operation device. [Figure 6] FIG. 3 is a diagram showing an example of information stored in a storage unit of a manipulation device. [Figure 7] 10A to 10C are diagrams illustrating the process of registering information in a storage unit of a manipulation device. [Figure 8] 8 is a flowchart showing a series of steps in the process shown in FIG. 7. [Figure 9] 4 is a diagram showing a database stored in an information storage unit of the management server shown in FIG. 3. FIG. [Figure 10] FIG. 10 is a diagram showing orientation information acquired by a CPU of a manipulation device. [Figure 11] FIG. 10 is a diagram showing an example of operation of an operation device. [Figure 12] 10A and 10B are diagrams showing another processing example. [Figure 13] 10A and 10B are diagrams showing another processing example. [Figure 14] FIG. 10 is a diagram showing another example of the configuration of the operation device. [Figure 15] FIG. 10 is a diagram showing another example of the configuration of the operation device. [Figure 16] FIG. 10 is a diagram showing a modified example of the operating device. [Figure 17] (A) and (B) are diagrams showing changes in non-display areas. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to this embodiment. The information processing system 1 is provided with a management server 100. Furthermore, the information processing system 1 is provided with a device 200 to be worn by a subject, which will be described later. Although only one device 200 is shown in FIG. 1, a plurality of devices 200 are provided according to the number of subjects. The device 200 is in the form of glasses. The device 200 is worn on the head of the subject. The subject visually recognizes his or her surroundings through the device 200.

[0009] The information processing system 1 is also provided with a microphone 600, which will be described later, for acquiring voices of nearby people. The microphone 600 acquires the voices of nearby people and generates voice information. Furthermore, an operation device 300 is provided for use in identifying the speaker. In this embodiment, the speaker is selected by a selector 43, which will be described later. When the selector 43 makes this selection, the selector 43 operates the operation device 300. The device 200 is connected to the management server 100 via a communication line 400 such as the Internet. The microphone 600 and the manipulation device 300 are also connected to the management server 100 via the communication line 400.

[0010] FIG. 2 is a diagram showing a target person 41, surrounding people 42, and a selector 43. In this embodiment, the device 200 is worn by the subject 41 as described above. A plurality of nearby persons 42 are located around the target person 41. Furthermore, a selector 43 who operates the operation device 300 is located around the target person 41. A microphone 600 is placed on the desk. In this embodiment, the target person 41 is notified of the speech content 48 of the nearby person 42 via the device 200. The selector 43 operates the operation device 300 to select the nearby person 42 who has spoken from the nearby people 42. In this embodiment, the speech content 48 of the nearby person 42 selected by the selector 43 is notified to the target person 41 via the device 200.

[0011] [Management Server Description] 3 is a diagram showing the configuration of the management server 100. The management server 100 is realized by a computer. The management server 100, which functions as the information processing system 1, has an arithmetic processing unit 111 that executes digital arithmetic processing in accordance with a program. Furthermore, the management server 100 has an information storage unit 19 that stores information. The information storage unit 19 is realized by an existing information storage device. Specifically, the information storage unit 19 is realized by, for example, an HDD (Hard Disk Drive). The information storage unit 19 is also realized by, for example, a semiconductor memory.

[0012] The arithmetic processing unit 111 is provided with a CPU 11a as an example of a processor. The arithmetic processing unit 111 is also provided with a RAM 11b that is used as a working memory for the CPU 11a, etc. The arithmetic processing unit 111 is also provided with a ROM 11c that stores programs executed by the CPU 11a, etc. The arithmetic processing unit 111 is also provided with a nonvolatile memory 11d that can retain data even if the power supply is interrupted.

[0013] The nonvolatile memory 11d is configured, for example, with a battery-backed SRAM, a flash memory, etc. The information storage unit 19 stores various types of information such as programs executed by the CPU 11a. The CPU 11a provided in the arithmetic processing unit 111 reads the programs stored in the ROM 11c or the information storage unit 19. As a result, various processes performed by the management server 100 are executed by this management server 100.

[0014] The program executed by the CPU 11a can be stored in a recording medium and provided to the management server 100. This recording medium is a computer-readable recording medium. Examples of recording media include magnetic recording media such as magnetic tapes and magnetic disks, and optical recording media such as optical disks. Other examples of recording media include magneto-optical recording media and semiconductor memories. Alternatively, the program executed by the CPU 11a may be provided to the management server 100 using a communication means, such as a communication means using an internet line.

[0015] [Device description] FIG. 4 is a diagram showing the hardware configuration of the device 200. The device 200 includes a processing unit 211, an information storage unit 212, a sensor 213, and a device camera 214. The device 200 also includes a device microphone 215, a speaker 216, and a display unit 217. The arithmetic processing unit 211 is provided with a CPU 21a as an example of a processor. The arithmetic processing unit 211 is also provided with a RAM 21c that is used as a working memory for the CPU 21a, etc. The arithmetic processing unit 211 is also provided with a ROM 21b that stores programs executed by the CPU 21a, etc.

[0016] The information storage unit 212 is realized by an existing information storage device such as a semiconductor memory. Examples of the sensor 213 include a GPS, a gyro sensor, an acceleration sensor, and a direction sensor. Based on the output from the sensor 213, the current position and orientation of the device 200 can be identified. The device camera 214 is a camera that captures images of the surroundings of the device 200. When the device 200 is worn by the subject 41, the device camera 214 faces in a forward direction of the subject 41 and captures an image of this forward direction. In other words, the device camera 214 faces in the direction in which the subject 41 is facing and captures an image of the front of the subject 41.

[0017] The device microphone 215 acquires the voice of the subject person 41 and generates voice information. The speaker 216 outputs sounds and voices to notify the target person 41 wearing the device 200. The display unit 217 is a so-called display that displays various information. The speech content 48 shown in Fig. 2 is displayed on the display unit 217. The display unit 217 is placed in front of the subject 41 when the device 200 is worn by the subject 41. It is also assumed that the target person 41 may be a person with a hearing impairment. When the speech content 48 is displayed, the hearing impaired person can also identify the speech content of the nearby person 42.

[0018] The display unit 217 displays an image obtained by the device camera 214. When the subject 41 is wearing the device 200, the display unit 217 displays an image showing the state in front of the subject 41. In this embodiment, the target person 41 looks at the image displayed on the display unit 217 to see what is in front of him or her.

[0019] In addition, there are also transmission-type devices 200. In this case, a transparent display unit 217 that allows the subject 41 to see what is behind the display unit 217 is installed. The target person 41 looks at what is behind the display unit 217 through the display unit 217. In other words, the target person 41 looks at what is in front of him or her through the display unit 217. In the see-through device 200, an image is displayed on the display unit 217. The user visually recognizes both the real space located behind the display unit 217 and the image displayed on the display unit 217.

[0020] The program executed by the CPU 21a may be stored in a recording medium and provided to the device 200. This recording medium is a computer-readable recording medium. As mentioned above, examples of the recording medium include magnetic recording media such as magnetic tapes and magnetic disks, and optical recording media such as optical disks. Other examples of the recording medium include magneto-optical recording media and semiconductor memories. Furthermore, the program executed by the CPU 21a may be provided to the device 200 using a communication means. An example of the communication means is a communication means using an internet line.

[0021] The device 200 is not limited to a glasses-type device, but may also be, for example, a smartphone, a tablet terminal, etc. The glasses-type device, the smartphone, the tablet terminal, etc. are all devices that can be carried by the target person 41. A display unit and a device camera are also provided on a smartphone or tablet terminal. In this case, the subject 41 can see what is ahead of him or her by referring to the image captured by the device camera and displayed on the display unit.

[0022] [Explanation of operating equipment] FIG. 5 is a diagram showing the hardware configuration of manipulation device 300. As shown in FIG. Manipulation device 300 includes a processing unit 311 , a storage unit 312 , a sensor 313 , and a display unit 314 . The arithmetic processing unit 311 is provided with a CPU 31a as an example of a processor. The arithmetic processing unit 311 is also provided with a RAM 31c that is used as a working memory for the CPU 31a, etc. The arithmetic processing unit 311 is also provided with a ROM 31b that stores programs executed by the CPU 31a, etc.

[0023] The storage unit 312 is realized by an existing information storage device such as a semiconductor memory. Storage unit 312 stores each of the orientations of manipulation device 300 and each of the candidates who may possibly speak in a state in which they are associated with each other. The candidates who may possibly speak refer to the above-mentioned peripheral persons 42. Storage unit 312 stores each of the orientations of manipulation device 300 and each of the peripheral persons 42 in a state in which they are associated with each other. Examples of sensor 313 include a gyro sensor, a direction sensor, and an acceleration sensor. CPU 31a of manipulation equipment 300 identifies the orientation of manipulation equipment 300 based on the output from sensor 313. The display unit 314 is a so-called display, and displays various types of information.

[0024] FIG. 6 is a diagram showing an example of information stored in storage unit 312 of manipulation device 300. As shown in FIG. Storage unit 312 stores each of the directions of manipulation device 300 and each of nearby persons 42 in association with each other. CPU 31a of manipulation device 300 identifies the speaker based on this information stored in storage unit 312. In response to an operation of the operation device 300, the CPU 31a refers to the information stored in the storage unit 312. Then, the CPU 31a identifies the nearby person 42 who made the utterance from among the multiple nearby people 42 stored in the storage unit 312.

[0025] Based on this information stored in the storage unit 312, the CPU 31a identifies the nearby person 42 who made the utterance. CPU 31a of manipulation device 300 identifies a speaker from among the candidates. CPU 31a of manipulation device 300 identifies nearby person 42 who has actually spoken from nearby people 42 who have the potential to speak. The details of identifying the nearby person 42 who made the utterance will be described later.

[0026] [Registration process of information in storage unit] 7(A) to 7(C) are diagrams illustrating the process of registering information in storage section 312 of manipulation device 300. FIG. As described above, storage unit 312 stores each orientation of manipulation device 300 and each nearby person 42 in association with each other. The information is registered in the storage unit 312 by the selector 43, for example. 7A. First, selector 43 points manipulation device 300 at first nearby person 42 among multiple nearby people 42.

[0027] Then, selector 43 performs a predetermined operation on manipulation device 300, such as selecting an input button. In response to this, CPU 31a of manipulation device 300 acquires information on the orientation of manipulation device 300. CPU 31a acquires this orientation information based on the output from sensor 313 (see FIG. 5) provided in manipulation device 300.

[0028] Thereafter, selector 43 operates operating device 300 to input information about this first nearby person 42. Hereinafter, in this specification, information about nearby person 42 may be referred to as "local person information." The selector 43 operates the operation device 300 to input the nearby person information of the first nearby person 42. Specifically, selector 43 inputs, as nearby person information, information such as the name of nearby person 42. In response to this, CPU 31a of manipulation device 300 acquires the nearby person information.

[0029] Next, CPU 31a of manipulation device 300 associates the acquired direction information with the acquired nearby person information. Next, CPU 31a registers the associated direction information and nearby person information in storage unit 312. Thereafter, the selector 43 is inquired through the display unit 314 as to whether or not registration of all nearby persons 42 has been completed.

[0030] Here, a case will be described in which registration is not yet complete and selector 43 continues to operate operation device 300. 7(B), selector 43 then points operating device 300 at another nearby person 42 different from first nearby person 42. Selector 43 points operating device 300 at second nearby person 42. Then, selector 43 performs a predetermined operation, such as selecting an input button, on manipulation equipment 300. In response to this, CPU 31a of manipulation equipment 300 acquires information on the orientation of this manipulation equipment 300. Furthermore, selector 43 inputs nearby person information of this second nearby person 42. In response to this, CPU 31a of manipulation device 300 acquires new nearby person information.

[0031] Then, CPU 31a of manipulation device 300 associates the acquired direction information with the acquired nearby person information. Next, CPU 31a of manipulation device 300 registers the associated direction information and nearby person information in storage unit 312. Thereafter, in this example, selector 43 points manipulation device 300 toward another nearby person 42, a third nearby person 42, as shown in FIG. 7(C). Next, selector 43 performs a predetermined operation, such as selecting an input button, on manipulation equipment 300. In response to this, CPU 31a of manipulation equipment 300 acquires information on the orientation of this manipulation equipment 300. Furthermore, selector 43 inputs nearby person information of this third nearby person 42. In response to this, CPU 31a of manipulation device 300 acquires new nearby person information.

[0032] Then, CPU 31a of manipulation device 300 associates the acquired direction information with the acquired nearby person information. Next, CPU 31a registers the associated direction information and nearby person information in storage unit 312. As described above, in this example, information on the orientation of three nearby persons 42 and nearby person information are registered in storage unit 312 of manipulation device 300.

[0033] FIG. 8 is a flowchart showing a series of steps in the process shown in FIG. In this embodiment, first, selector 43 points manipulation device 300 toward nearby person 42. Next, selector 43 performs a predetermined operation on manipulation device 300, such as selecting an input button. As a result, CPU 31a acquires information on the orientation of manipulation device 300 that is facing nearby person 42 (step S101). Next, the selector 43 inputs the nearby person information, and in response, the CPU 31a acquires the nearby person information (step S102). Thereafter, the CPU 31a associates the acquired direction information with the acquired nearby person information, and then the CPU 31a registers the associated direction information and nearby person information in the storage unit 312 (step S103).

[0034] Thereafter, the CPU 31a inquires of the selector 43 through the display unit 314 whether or not registration of all nearby persons 42 has been completed. In response, the selector 43 gives an answer through the display unit 314. Although not explained above, the display unit 314 is configured by a touch panel. Next, the CPU 31a determines whether or not the registration of all the nearby persons 42 has been completed based on the response from the selector 43 (step S104).

[0035] Then, the CPU 31a ends the process when it determines that the registration of all the nearby persons 42 has been completed. On the other hand, when the CPU 31a does not determine that the registration of all the nearby persons 42 has been completed, it performs the processes from step S101 onwards again. 7 and 8 are completed, storage unit 312 is in the state shown in Fig. 6. Storage unit 312 stores each orientation of manipulation device 300 and each nearby person 42 in a state where they are associated with each other.

[0036] Note that values ​​having upper and lower limits may be registered as information on orientations registered in storage unit 312. For example, although information of 45° is registered in Fig. 6, a value of 44° to 46° may be registered instead. When registering a value having a lower limit and an upper limit, the lower limit and upper limit are determined based on the value acquired in step S101. Specifically, the lower limit is determined by subtracting a predetermined value from the value acquired in step S101. The upper limit is determined by adding a predetermined value to the value acquired in step S101. The upper and lower limits thus determined are then registered in the storage unit 312.

[0037] In this specification, the term "processor" refers to a processor in the broad sense. An example of a processor is a general-purpose processor, such as a CPU (Central Processing Unit). Another example of a processor is a dedicated processor. An example of a dedicated processor is a GPU (Graphics Processing Unit). Another example of a dedicated processor is an ASIC (Application Specific Integrated Circuit). Another example of a dedicated processor is an FPGA (Field Programmable Gate Array) or a programmable logic device. The operations of the processor are not limited to being performed by a single processor. The operations of the processor may be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processor is not limited to the order described in this embodiment and may be changed.

[0038] [Database Description] FIG. 9 is a diagram showing a database stored in the information storage unit 19 of the management server 100 shown in FIG. As shown in FIG. 9, information about each nearby person 42 is registered in the database. The database registers, for each nearby person 42, an identification ID, which is information used to identify each nearby person 42. The database also registers, for each nearby person 42, the name of the nearby person 42.

[0039] Furthermore, facial information of each nearby person 42 is registered in the database. In this embodiment, an image of the face of the nearby person 42 is taken in advance. Then, face information, which is information about the face of the nearby person 42, is registered in a database. The facial information may include, for example, an image of the face of the peripheral person 42. The facial information may also include information about the facial features of the peripheral person 42. The information about the features can be obtained by analyzing the image of the face of the peripheral person 42.

[0040] [Processing by the management server] 2 is transmitted to the management server 100 (see FIG. 1). In particular, the video captured by the device camera 214 (see FIG. 4) provided in the device 200 is transmitted to the management server 100. In addition, audio information obtained by the microphone 600 shown in FIG. As a result, the CPU 11a (see FIG. 3) of the management server 100 acquires this video and audio information.

[0041] Furthermore, in this embodiment, information acquired by manipulation device 300 shown in Fig. 2 is also transmitted to management server 100. In response to this, CPU 11a of management server 100 acquires this information acquired by manipulation device 300. Specifically, CPU 11a acquires, from manipulation device 300, information on nearby person 42 identified by manipulation device 300.

[0042] Manipulation device 300 outputs information about nearby person 42 who has been selected by selector 43 operating manipulation device 300. This peripheral person 42 is the peripheral person 42 who made the utterance. This peripheral person 42 is the peripheral person 42 who the selector 43 identified as having made the utterance. Hereinafter, in this specification, the peripheral person 42 who made the utterance may be referred to as the "utterance peripheral person 42." In other words, the peripheral person 42 who was selected by the selector 43 may be referred to as the "utterance peripheral person 42." CPU 11a of management server 100 acquires information on utterance peripheral persons 42 from manipulation device 300.

[0043] Furthermore, the CPU 11 a of the management server 100 identifies nearby people 42 based on the video captured by the equipment camera 214. The CPU 11a identifies the nearby person 42 appearing in the video based on the video captured by the device camera 214. The CPU 11a identifies the nearby person 42 appearing in the video based on the video transmitted from the device 200. To identify the nearby persons 42, the CPU 11a acquires an image of the face of each of the nearby persons 42 appearing in the video. Then, the CPU 11a identifies the nearby persons 42 based on the image of the face and the face information stored in the database. Specifically, here, the CPU 11a identifies the name of each of the nearby persons 42.

[0044] Furthermore, the CPU 11 a acquires the speech content 48 of the nearby person 42 based on the voice information obtained by the microphone 600 . Furthermore, CPU 11a acquires information on utterance peripheral persons 42 transmitted from manipulation device 300 as described above. Then, the CPU 11a performs a process of associating the utterance content 48 with the utterance peripheral persons 42. The association between the utterance content 48 and the utterance peripheral persons 42 is performed based on time information, as will be described later.

[0045] By associating the utterance content 48 with the utterance surrounding person 42, it is possible to identify which surrounding person 42 has uttered the utterance content 48. Next, the CPU 11a generates control information for notifying the target person 41 of the utterance content 48. As a result, in this embodiment, the utterance content 48 is displayed on the display unit 217 in a state where it is associated with the utterance peripheral person 42.

[0046] As described above, the CPU 11 a of the management server 100 acquires the audio information acquired by the microphone 600 . The microphone 600 sequentially outputs the audio information acquired by the microphone 600. At this time, the microphone 600 also outputs time information output from a clock (not shown) provided in the microphone 600. In this embodiment, the audio information and the time information are output from the microphone 600 in a state where they are associated with each other.

[0047] Furthermore, as described above, CPU 11a of management server 100 acquires information on utterance peripheral persons 42 from manipulation device 300. Information on utterance peripheral persons 42 is information on peripheral persons 42 identified by CPU 31a of manipulation device 300. Information on utterance surrounding persons 42 is sequentially transmitted from manipulation device 300. Furthermore, time information indicating the current time is also sequentially output from manipulation device 300. A clock (not shown) is provided in manipulation device 300. Time information output from this clock is also output from manipulation device 300 to management server 100. Manipulation device 300 outputs information on utterance peripheral person 42 and time information in a correlated state.

[0048] As described above, the microphone 600 outputs audio information and time information in association with each other. Furthermore, manipulation device 300 outputs information on utterance peripheral person 42 and time information in a correlated state. In response to this, the management server 100 acquires the voice information and the time information, and also acquires the information of the utterance peripheral persons 42 and the time information.

[0049] When the CPU 11a of the management server 100 acquires the voice information, it acquires, based on the voice information, the speech content 48. Acquisition of the speech content 48 based on the voice information may be performed using a known method. Every time the CPU 31a acquires the utterance content 48, the CPU 31a associates time information with the utterance content 48. The CPU 31a associates time information transmitted from the microphone 600 together with the voice information with the utterance content 48. The CPU 31a associates the utterance content 48 with time information indicating the time when the voice information on which the utterance content 48 is based was obtained.

[0050] Thereafter, the CPU 11a associates the acquired information on the utterance peripheral persons 42 with the acquired utterance content 48 using the time information as a key. As a result, the utterance content 48 and the utterance peripheral person 42 who made the utterance with this utterance content 48 are associated with each other.

[0051] Next, the CPU 11a of the management server 100 generates control information for displaying the utterance content 48 on the display unit 217 of the device 200 (see FIG. 2). The CPU 11a associates the utterance content 48 with the peripheral people 42 displayed on the display unit 217. The CPU 11a associates the utterance content 48 with the utterance peripheral people 42 among the peripheral people 42 displayed on the display unit 217. The CPU 11a generates control information for displaying the speech content 48 in a state in which the speech content 48 is associated with the utterance peripheral person 42. In other words, the CPU 11a generates control information that allows the speech content 48 to be associated with the nearby person 42 who actually made the speech.

[0052] When generating this control information, first, the CPU 11a identifies each of the nearby persons 42. The CPU 11a identifies each of the nearby persons 42 displayed on the display unit 217. The CPU 11a identifies each of the nearby persons 42 based on the image captured by the equipment camera 214 (see FIG. 4). Specifically, the CPU 11a identifies each of the nearby people 42 based on the video and face information of the nearby people 42. The CPU 11a identifies each of the nearby persons 42 based on the image of the nearby persons 42 in the image and the facial information. The CPU 11a identifies each of the nearby persons 42 based on the image of the nearby persons 42 and the facial information registered in the database.

[0053] Thereafter, the CPU 11a identifies an utterance peripheral person 42 from among the identified peripheral people 42. The CPU 11a identifies an utterance peripheral person 42 who actually made an utterance from among the identified peripheral people 42. Specifically, CPU 11a identifies, from among the identified peripheral persons 42, a peripheral person 42 that matches with utterance peripheral person 42. CPU 11a identifies, from among the peripheral persons 42, a peripheral person 42 that matches with utterance peripheral person 42 transmitted from manipulation device 300. More specifically, CPU 11a identifies peripheral person 42 that matches utterance peripheral person 42 based on the name transmitted from manipulation device 300.

[0054] Next, the CPU 11a generates control information that allows the utterance content 48 to be associated with the identified nearby person 42. The CPU 11a generates control information for displaying the speech content 48 in a manner that correlates the speech content 48 with the nearby person 42. 2, utterance content 48 is associated with nearby person 42 who has made the utterance and is displayed on display unit 217. In other words, utterance content 48 is associated with nearby person 42 at whom manipulation device 300 is pointed.

[0055] As described above, CPU 11a identifies utterance content 48. Furthermore, CPU 11a identifies utterance peripheral persons 42 based on information from manipulation device 300, as described above. Furthermore, the CPU 11a identifies each of the nearby people 42 displayed on the display unit 217 of the device 200. Then, the CPU 11a associates the utterance content 48 with the utterance peripheral person 42 displayed on the display unit 217, using the information of the utterance peripheral person 42 as a key. The CPU 11a associates the utterance content 48 with the utterance peripheral person 42 displayed on the display unit 217, based on the name of the utterance peripheral person 42.

[0056] Next, the CPU 11a generates the above control information. This control information is control information that causes the utterance content 48 to be displayed on the display unit 217 of the device 200. In addition, this control information is control information that causes the utterance content 48 to be associated with the utterance peripheral person 42 displayed on the display unit 217. The CPU 11a generates this control information that enables the speech content 48 to be associated with the utterance peripheral person 42.

[0057] The CPU 11a generates control information including position information, which is information about the display position of the utterance content 48, as the control information. The CPU 11a determines the position of the utterance peripheral person 42 displayed on the display unit 217 as the display position of the utterance content 48. The CPU 11a determines the position around the head of the utterance peripheral person 42 as the display position of the utterance content 48. Then, the CPU 11a generates control information including position information, which is information about the display position. The CPU 11a also includes the identified utterance content 48 in the control information.

[0058] In this embodiment, the control information including the location information and the utterance content 48 is transmitted from the management server 100 to the device 200. Then, based on this control information, the CPU 21a (see FIG. 4) of the device 200 controls the display of the display unit 217. As a result, the utterance content 48 is displayed on the display unit 217 at the position specified by the position information. 2 is displayed on the display unit 217 of the device 200. In this display, the utterance content 48 is displayed in a state where the utterance content 48 is associated with the utterance peripheral person 42.

[0059] In this embodiment, as described above, the speech content 48 of the peripheral person 42 is displayed on the display unit 217 of the device 200. In this embodiment, when the peripheral person 42 speaks, the speech content 48 of this peripheral person 42 is displayed on the display unit 217. Furthermore, in this embodiment, the utterance content 48 is displayed in a state in which it is associated with the nearby person 42 who actually made the utterance. The utterance content 48 is displayed in a state in which it is associated with the nearby person 42 at whom the operation device 300 is pointed. In this embodiment, a display image 45 consisting of an image showing a speech bubble is displayed as shown in Fig. 2. When the speech content 48 is displayed, this display image 45 is displayed. The speech content 48 is displayed inside this display image 45.

[0060] [Explanation of operating equipment] Manipulation device 300 is configured to be able to select each of a plurality of candidates who may possibly make an utterance. In this embodiment, the plurality of nearby persons 42 correspond to the plurality of candidates. Manipulation device 300 is configured to be able to select each of these peripheral persons 42. Selector 43 operates manipulation device 300 to select a speaker from among the plurality of peripheral persons 42. When manipulation device 300 is operated, CPU 31a (see FIG. 5) of manipulation device 300 acquires operation information of manipulation device 300. CPU 31a acquires operation information which is information on the operation of manipulation device 300 by selector 43.

[0061] The CPU 31a identifies the speaker based on this operation information. In other words, the CPU 31a identifies the utterance peripheral persons 42 based on this operation information. The CPU 31a identifies some of the nearby persons 42 who have made the utterance from among a plurality of nearby persons 42 registered in advance in the storage unit 312. CPU 31a acquires, as operation information, orientation information that is information indicating the orientation of manipulation device 300. CPU 31a acquires, as operation information, operation information of an operation for changing the orientation of manipulation device 300.

[0062] Then, the CPU 31a identifies the speaker based on this operation information. In other words, the CPU 31a identifies the utterance peripheral persons 42 based on this operation information. Storage unit 312 of manipulation device 300 stores the orientation of manipulation device 300 and nearby persons 42. Storage unit 312 stores each orientation of manipulation device 300 and each nearby person 42 in a state where they are associated with each other.

[0063] The CPU 31a identifies the utterance peripheral person 42 based on the direction identified by the acquired operation information and the information stored in the storage unit 312. The CPU 31a identifies, from among the orientations stored in the storage unit 312, an orientation that matches the orientation identified by the acquired operation information. The CPU 31a identifies the peripheral person 42 that is associated with this matching orientation and stored in the storage unit 312 as the utterance peripheral person 42.

[0064] In this way, CPU 31a of manipulation device 300 identifies utterance peripheral person 42 who is the speaker. Note that CPU 11a of management server 100 may identify utterance peripheral person 42. When CPU 11a of management server 100 identifies utterance peripheral persons 42, operation information is transmitted from operation device 300 to management server 100. Then, the CPU 11a of the management server 100 identifies the utterance surrounding person 42 based on the transmitted operation information.

[0065] When the CPU 11a of the management server 100 identifies the utterance surrounding persons 42, information used for this identification is stored in the information storage unit 19 (see FIG. 3). Specifically, the information shown in FIG. 6 stored in storage unit 312 of manipulation device 300 is stored in information storage unit 19 of management server 100. CPU 11a identifies utterance peripheral persons 42 based on the operation information transmitted from operation device 300 and the information stored in information storage unit 19.

[0066] FIG. 10 is a diagram showing orientation information acquired by CPU 31a of manipulation device 300. As shown in FIG. As described above, CPU 31a acquires orientation information that is information indicating the orientation of manipulation device 300. 10, the CPU 31a acquires orientation information corresponding to 45° and orientation information corresponding to 90°, and further acquires orientation information corresponding to 135°. In this example, manipulation device 300 is operated to change from a state in which it faces a 45° direction to a state in which it faces a 90° direction. Also, in this example, manipulation device 300 is operated to change from a state in which it faces a 90° direction to a state in which it faces a 135° direction.

[0067] When these operations are performed, the CPU 31a acquires orientation information corresponding to 45°, 90°, and 135°. Then, the CPU 31a identifies the utterance surrounding persons 42 based on the acquired direction information. CPU 31a identifies utterance peripheral person 42 based on operation information, which is information on the operation of operation device 300 by selector 43. Here, "operation of operation device 300" refers to an operation of changing the orientation of operation device 300.

[0068] When identifying utterance peripheral persons 42, CPU 31a of manipulation device 300 identifies utterance peripheral persons 42 from among a plurality of candidates stored in advance. The CPU 31a identifies the utterance peripheral person 42 based on the direction identified by the acquired operation information and the information stored in the storage unit 312. Thereafter, the CPU 31a transmits the information of the identified utterance nearby person 42 to the management server 100. At this time, the time information is also transmitted to the management server 100, as described above.

[0069] CPU 31a may acquire, as the orientation information, information on the orientation of manipulation device 300 in a specific state. An example of the "specific state" is a state in which the movement speed of manipulation device 300 is slower than a predetermined movement speed. In this case, CPU 31a acquires information on the orientation of manipulation device 300 in a state where the moving speed is slower than a predetermined moving speed. In this case, information on the orientation of manipulation device 300 in a state where the movement speed is greater than a predetermined movement speed is not acquired as orientation information. It should be noted that "the moving speed is slower than a predetermined moving speed" also includes a state in which the moving speed of manipulation equipment 300 is zero.

[0070] 2, it is assumed that first peripheral person 421 first speaks. In this case, manipulation device 300 is directed toward first peripheral person 421. It is assumed that third peripheral person 423 then speaks. In this case, manipulation equipment 300 is directed from first peripheral person 421 to third peripheral person 423. At this time, manipulation device 300 temporarily turns toward second peripheral person 422. In this case, second peripheral person 422 may be identified as having spoken, even though second peripheral person 422 has not spoken. That is, there is a risk of erroneous identification of the speaker.

[0071] In contrast, in this embodiment, information on the orientation of manipulation device 300 that is moving at a high speed is not acquired as orientation information, which makes it less likely that the above-described erroneous identification will occur. When manipulation device 300 is directed from first peripheral person 421 to third peripheral person 423, the movement speed of manipulation device 300 increases. When manipulation device 300 is directed toward second peripheral person 422, the movement speed of manipulation device 300 increases. In this case, the orientation information when manipulation device 300 is facing second peripheral person 422 is not used to identify utterance peripheral person 42. In this case, second peripheral person 422 is not mistakenly identified as utterance peripheral person 42.

[0072] CPU 11a of management server 100 associates utterances made when manipulation device 300 is operated with utterance peripheral persons 42. CPU 11a associates utterance peripheral persons 42 identified by manipulation device 300 with the utterances. CPU 11a associates utterance peripheral person 42 with an utterance made when manipulation device 300 is facing the direction specified by the direction information. As described above, CPU 11a performs the association based on the time information from microphone 600 and the time information from manipulation device 300. The CPU 11a associates the utterance content 48 with the utterance peripheral person 42 who made the utterance in the utterance content 48 based on these two pieces of time information.

[0073] The association between the speech content 48 and the utterance peripheral persons 42 is not limited to that based on time information. The sequentially generated utterance contents 48 may be associated with sequentially generated information on the utterance peripheral persons 42. In this case, the CPU 11 a of the management server 100 sequentially generates the speech content 48 based on the voice information transmitted from the microphone 600 . Then, CPU 11a associates the information on utterance peripheral persons 42 with the sequentially generated utterance contents 48. CPU 11a associates the information on utterance peripheral persons 42 sequentially transmitted from manipulation device 300 with the utterance contents 48.

[0074] After associating the information on the utterance peripheral persons 42 with the utterance content 48, the CPU 11a generates control information as described above. The CPU 11 a generates control information used to control the display unit 217 of the device 200 . The CPU 11a generates control information for displaying the utterance content 48 in a state in which the utterance content 48 is associated with the utterance peripheral person 42. As a result, the speech content 48 is associated with the utterance surrounding person 42 displayed on the display unit 217.

[0075] FIG. 11 is a diagram showing an example of operation of manipulation device 300. As shown in FIG. In FIG. 11, the subject 41 is not shown. 11(A), first peripheral person 421 is speaking. Also, in FIG. 11(A), manipulation device 300 is facing first peripheral person 421. In this case, CPU 31a of manipulation device 300 identifies first peripheral person 421 as utterance peripheral person 42. 11(B), second nearby person 422 starts speaking. Accordingly, selector 43 changes the direction of manipulation device 300. In this case, manipulation device 300 rotates about vertical axis 92 passing through selector 43, changing the orientation of manipulation device 300. This results in the state shown in Fig. 11(B). In the state of Fig. 11(B), manipulation device 300 faces second nearby person 422. In the state shown in FIG. 11(B), CPU 31a of manipulation equipment 300 identifies second nearby person 422 who is another speaker.

[0076] The second peripheral person 422 is another speaker whose position around the vertical axis 92 is different from the position of the first peripheral person 421. Around the vertical axis 92, the position of the second peripheral person 422 and the position of the first peripheral person 421 are different. After identifying first peripheral person 421, CPU 31a identifies second peripheral person 422. When the orientation of manipulation device 300 changes and manipulation device 300 faces second peripheral person 422, CPU 31a identifies second peripheral person 422.

[0077] In this example, the manipulation device 300 rotates around the vertical axis 92. The orientation of the manipulation device 300 changes accordingly. In the state of FIG. 11(B), the CPU 31a identifies the second nearby person 422. The CPU 31a identifies a second peripheral person 422 whose position around the vertical axis 92 is different from the position of the first peripheral person 421.

[0078] After that, in this processing example, as shown in FIG. 11(C), the third nearby person 423 starts speaking. Accordingly, the selector 43 changes the orientation of the manipulation device 300. As a result, the manipulation device 300 further rotates around the vertical axis 92, and the orientation of the manipulation device 300 changes. This results in the state shown in Fig. 11(C) In the state of Fig. 11(C), manipulation device 300 faces third nearby person 423. In the state shown in FIG. 11C, the CPU 31a identifies a third peripheral person 423 who is another peripheral person 42 different from the second peripheral person 422.

[0079] The third peripheral person 423 is another speaker whose position around the vertical axis 92 is different from the position of the second peripheral person 422. Around the vertical axis 92, the position of the third peripheral person 423 and the position of the second peripheral person 422 are different. The CPU 31 a identifies the third nearby person 423 . When the orientation of manipulation device 300 changes and manipulation device 300 faces third nearby person 423, CPU 31a identifies this third nearby person 423.

[0080] 11(C), manipulation device 300 rotates about vertical axis 92, and the orientation of manipulation device 300 changes. In the state of FIG. 11(C), the CPU 31a identifies the third nearby person 423. The CPU 31a identifies a third peripheral person 423 whose position around the vertical axis 92 is different from the position of the second peripheral person 422.

[0081] Figures 12(A) and (B) are diagrams showing another processing example. Figure 12(B) is a diagram of manipulation device 300, etc., viewed from the direction indicated by arrow XIIB in Figure 12(A). In Figure 12(B), first peripheral person 421 and third peripheral person 423 are not shown. In this processing example, a fourth peripheral person 424 is located behind a second peripheral person 422 as shown in FIG. 12(B). In this processing example, first, second nearby person 422 makes an utterance. In this case, manipulation equipment 300 faces the direction indicated by arrow 12S in Fig. 12(B) In this case, CPU 31a of manipulation equipment 300 identifies second peripheral person 422 as utterance peripheral person 42.

[0082] After that, in this example, fourth nearby person 424 starts speaking. In response to this, selector 43 changes the orientation of manipulation device 300. As shown by arrow 12T in Fig. 12(B), selector 43 performs an operation to point manipulation device 300 upward. In this case, manipulation device 300 rotates about horizontal axis 93, which is an axis along the horizontal direction. This changes the orientation of manipulation device 300. As a result, manipulation device 300 enters the state indicated by reference symbol 12Y. When manipulation device 300 enters the state indicated by reference symbol 12Y, CPU 31a identifies fourth nearby person 424 who is another speaker.

[0083] When the orientation of manipulation device 300 changes as shown in FIG. 12(B), fourth nearby person 424 is identified. In this example, manipulation device 300 rotates about horizontal axis 93. Manipulation device 300 rotates so that it faces diagonally upward. Horizontal axis 93 extends in a direction perpendicular to the plane of the paper in FIG. 12(B). 12, manipulation device 300 rotates and faces a different direction from before the rotation. In this case, CPU 31a identifies fourth nearby person 424 who is different from second nearby person 422.

[0084] As shown in FIG. 12(B), fourth peripheral person 424 is located farther from manipulation device 300 than second peripheral person 422. When manipulation device 300 is facing diagonally upward, CPU 31a identifies fourth nearby person 424. In this example, orientation information of manipulation device 300 when manipulation device 300 faces diagonally upward is associated with fourth nearby person 424. This orientation information and fourth nearby person 424 are stored in storage unit 312 (see FIG. 5) in an associated state. In this case, when manipulation device 300 is facing diagonally upward, CPU 31a identifies fourth nearby person 424.

[0085] In this example, CPU 31a identifies fourth peripheral person 424 who is located farther from manipulation device 300 than second peripheral person 422. When manipulation device 300 is facing diagonally upward, CPU 31a identifies fourth nearby person 424, who is an example of another speaker. As described above, the fourth peripheral person 424 is a peripheral person 42 located farther from the manipulation device 300 than the second peripheral person 422. Here, the CPU 31a identifies the fourth peripheral person 424 located farther from the manipulation device 300 than the second peripheral person 422.

[0086] Additionally, it is also assumed that fourth peripheral person 424 is located closer to manipulation device 300 than second peripheral person 422. In other words, it is also assumed that fourth peripheral person 424 is located closer to selector 43 than second peripheral person 422. In this case, the selector 43 performs an operation on the manipulation device 300, for example, to point the manipulation device 300 downward. In this case as well, the selector 43 performs an operation on the manipulation device 300 to rotate the manipulation device 300 about the horizontal axis 93. In this case, CPU 31a identifies fourth peripheral person 424 who is located closer to manipulation device 300 than second peripheral person 422.

[0087] FIGS. 13A and 13B are diagrams showing another processing example. Figure 13(B) is a diagram of manipulation device 300 etc. viewed from the direction indicated by arrow XIIIB in Figure 13(A). In Figure 13(B), first peripheral person 421 and third peripheral person 423 are not shown. In this processing example, a fourth peripheral person 424 is positioned above a second peripheral person 422, as shown in FIG. 13(B).

[0088] In this processing example, first, manipulation equipment 300 faces the direction indicated by arrow 13S in Fig. 13(B). As a result, CPU 31a of manipulation equipment 300 identifies second nearby person 422. Thereafter, in this processing example, manipulation device 300 rotates about horizontal axis 93, changing the orientation of manipulation device 300. As a result, manipulation device 300 faces fourth nearby person 424, as shown in Fig. 13(B). In this example as well, selector 43 performs an operation to point manipulation device 300 upward.

[0089] In this case, the CPU 31a identifies the other speaker, the fourth peripheral person 424. The fourth peripheral person 424 is a speaker located above the second peripheral person 422, as described above. After identifying second nearby person 422, if the orientation of manipulation device 300 changes, CPU 31a identifies fourth nearby person 424. When the orientation of manipulation device 300 changes and faces upward, CPU 31a identifies fourth nearby person 424.

[0090] Although not shown in the drawings, it is also conceivable that fourth peripheral person 424 is positioned lower than second peripheral person 422. In this case, the selector 43 performs an operation of pointing the manipulation device 300 downward. In this case, the selector 43 also operates the operating device 300 so that the operating device 300 rotates around the horizontal axis 93. In this case, the CPU 31a identifies the fourth nearby person 424 who is located below the second nearby person 422.

[0091] 12 and 13, manipulation device 300 makes a rotational movement about horizontal axis 93. In response to this, the orientation of manipulation device 300 changes. In this case, as described above, another speaker located closer to or farther from manipulation device 300 than the first speaker is identified. Alternatively, in this case, as described above, other speakers positioned above or below the first speaker are identified.

[0092] FIG. 14 is a diagram showing another example of the configuration of manipulation device 300. In FIG. 14, a movable part 95 that is movable relative to the device main body 91 is provided separately from the device main body 91. The movable part 95 is rotatable around the point indicated by reference numeral 14A as the center of rotation. In the above, information on the overall orientation of manipulation device 300 is acquired as orientation information of manipulation device 300. In contrast, in manipulation device 300 shown in FIG. 14, information on the orientation of movable part 95 is acquired as orientation information of manipulation device 300.

[0093] 14, the selector 43 operates the movable part 95. The selector 43 operates the movable part 95 and points the rod-shaped movable part 95 toward each of the nearby persons 42. In this configuration example, CPU 31a of manipulation device 300 acquires information indicating the orientation of movable part 95 as orientation information.

[0094] 14 is provided with a sensor 313 that detects the rotation angle of the movable part 95. The CPU 31a obtains orientation information based on the output from the sensor 313. The processing after obtaining the orientation information is the same as that described above, and the description of the processing after obtaining the orientation information will be omitted.

[0095] FIG. 15 is a diagram showing another example of the configuration of manipulation device 300. In FIG. In this configuration example, operation device 300 is provided with a plurality of buttons 51. These plurality of buttons 51 are provided corresponding to each of the plurality of candidates. In other words, these plurality of buttons 51 are provided corresponding to each of nearby persons 42. In this operation device 300 as well, CPU 31 a of operation device 300 acquires operation information that is information on the operation performed by selector 43 . The CPU 31a acquires, as operation information, information about the button 51 selected by the selector 43. Then, the CPU 31a identifies the speaker based on this information.

[0096] 15 also includes a storage unit 312 (see FIG. 5). In this storage unit 312, buttons 51 and nearby persons 42 are registered in a state where they are associated with each other. The CPU 31a identifies the peripheral person 42 who made the utterance, based on the selected button 51 and the information stored in the storage unit 312. That is, the CPU 31a identifies the utterance peripheral person 42.

[0097] The process after identifying the utterance peripheral persons 42 is the same as that described above. When the utterance peripheral persons 42 are identified, the management server 100 associates the utterance peripheral persons 42 with the utterance content 48. Thereafter, control information is generated to associate the utterance content 48 with the utterance peripheral persons 42 displayed on the display unit 217 (see FIG. 2). 14 and 15 is used, utterance peripheral persons 42 may be identified by management server 100. Identification of utterance peripheral persons 42 may be performed by management server 100, not by manipulation device 300.

[0098] 2, 14, and 15 may be dedicated devices 200. Manipulation devices 300 may be dedicated devices 200 specialized in identifying utterance peripheral persons 42. Furthermore, manipulation device 300 may be realized by an information processing device such as a smartphone or a tablet terminal. Information processing devices such as smartphones and tablet terminals are provided with gyro sensors, acceleration sensors, direction sensors, etc. The orientation of the information processing device can also be determined by the information processing device itself.

[0099] 14 is realized by an information processing device, an image corresponding to the movable part 95 is displayed on a display unit provided in the information processing device. In this case, the display unit is configured by a touch panel. In this case, the selector 43 operates the movable part 95 displayed on the display unit to change the orientation of the movable part 95. In this case, manipulation device 300 identifies nearby person 42 who made the utterance based on the orientation of movable part 95.

[0100] Next, a case where manipulation equipment 300 shown in FIG. 15 is realized by an information processing device will be described. In this case, the display unit is configured to display an image corresponding to the button 51. In this case, the display unit is configured as a touch panel. In this case, the selector 43 selects the button 51 to identify the utterance peripheral person 42. The selector 43 selects the button 51 from among the plurality of buttons 51 displayed on the display unit to identify the utterance peripheral person 42. In this case, CPU 31a of manipulation device 300 identifies utterance peripheral persons 42 based on this selected button 51.

[0101] Alternatively, manipulation device 300 may be realized by a microphone with a built-in sensor. Manipulation device 300 may be realized by a microphone with a built-in gyro sensor, acceleration sensor, direction sensor, or the like. In this case, the selector 43 points the microphone towards the nearby person 42 who has spoken.

[0102] Alternatively, manipulation device 300 may be realized by device 200 (see FIG. 2) worn by subject person 41. The selector 43 may wear the device 200, and the operation device 300 operated by the selector 43 may be realized by this device 200. The operation device 300 may be realized by this device 200 worn by the selector 43.

[0103] The device 200 is also provided with sensors 213 (see FIG. 4) such as a gyro sensor, an acceleration sensor, and a direction sensor. The device 200 can also be used to acquire the orientation information described above. When using the device 200, the selector 43 wearing the device 200 faces the direction of the surrounding person 42 who made the utterance. In this way, the surrounding person 42 who made the utterance is identified.

[0104] 16 is a diagram showing a modified example of manipulation device 300. In FIG. 16, a modified example of manipulation device 300 shown in FIG. In this configuration example shown in FIG. 16, candidate information is displayed on display unit 314 of manipulation device 300. In this embodiment, as described above, each orientation of the operation device 300 is associated with each candidate who may possibly make an utterance. In other words, each operation of the operation device 300 performed by the selector 43 is associated with each of the nearby persons 42.

[0105] In this processing example, CPU 31a of operation device 300 acquires candidate information, which is information about the candidates. CPU 31a acquires candidate information, which is information about the candidates associated with the operation performed by selector 43. CPU 31a of operation device 300 acquires candidate information associated with the operation performed by selector 43 from storage unit 312. Then, the CPU 31a generates control information for notifying the selector 43 of the candidate information via the operation device 300.

[0106] 16 is displayed on display unit 314. On this screen, as indicated by reference symbol 16A, information on nearby person 42 corresponding to the direction in which manipulation device 300 is facing is displayed. In other words, this screen displays candidate information associated with the direction in which manipulation device 300 is facing. In this case, selector 43 can confirm which nearby person 42 is identified by manipulation device 300. This allows selector 43 to confirm whether manipulation device 300 is actually pointing toward nearby person 42 to which he or she wants manipulation device 300 to be pointed.

[0107] Furthermore, in this configuration example, the CPU 31a further generates control information for changing the display of non-display areas. The non-display area refers to an area of ​​the display unit 314 other than the display area where the candidate information is displayed. In this embodiment, the area inside the rectangle indicated by the reference numeral 16X is the display area, and the area outside the rectangle indicated by the reference numeral 16X is the non-display area.

[0108] The CPU 31a generates control information every time the candidate information notified to the selector 43 via the display unit 314 changes. The CPU 31a generates control information every time the candidate information displayed on the display unit 314 changes. As described above, the control information refers to control information that changes the display of non-displayed areas.

[0109] Specifically, the CPU 31a generates, as this control information, control information for changing the background color of the non-display area. In this case, the background color of the non-display area changes each time nearby person 42 located in the direction in which manipulation device 300 is directed changes. In other words, the color of the non-display area changes.

[0110] 17(A) and (B) are diagrams showing changes in non-display areas. In FIG. 17(A), display unit 314 of manipulation device 300 displays, as candidate information, "Peripheral person 1," which is information indicating the first peripheral person 42. In FIG. 17(B), display unit 314 of manipulation device 300 displays, as candidate information, "peripheral person 2," which is information indicating second peripheral person 42.

[0111] In this embodiment, when the direction in which manipulation device 300 is facing changes, the display changes as shown in (A) and (B) of Fig. 17. Specifically, as shown by reference numeral 17X, the display of information about nearby person 42 changes. Furthermore, in this embodiment, when the information of the nearby person 42 that is displayed changes, the color of the non-displayed portion changes.

[0112] When the color of the hidden portion changes, the selector 43 can more intuitively check the operation he or she is about to perform. The selector 43 can more intuitively check whether the operation device 300 is actually pointing toward the nearby person 42 to whom the selector 43 wants to point the operation device 300. This display on display unit 314 may also be performed in operation device 300 shown in Fig. 14. This display on display unit 314 may also be performed in operation device 300 shown in Fig. 15.

[0113] [Notification of speech content] The above describes the process of associating the utterance content 48 with the utterance peripheral person 42 displayed on the display unit 217 of the device 200. In this case, the target person 41 is notified of both the speaker and the utterance content 48 made by the speaker. The manner of notification to the target person 41 is not limited to this. In other words, the manner of display on the display unit 217 of the device 200 is not limited to this. It is not essential that the display unit 217 displays the image captured by the equipment camera 214. The notification to the target person 41 may be made using only text information.

[0114] For example, the utterance content 48 and the character information identifying the utterance peripheral person 42 may be displayed in a state where they are associated with each other. Specifically, the utterance content and the character information indicating the name of the utterance peripheral person 42 may be displayed in a state where they are associated with each other. Furthermore, the color and font of the characters representing the content of the speech may be different for each speaker. Furthermore, the display mode of the display image 45 shown in FIG. 2 may be different for each speaker. Specifically, at least one of the shape, line type, line thickness, and line color of the display image 45 may be made different for each speaker.

[0115] Furthermore, in the above, manipulation equipment 300 acquires information on the orientation of manipulation equipment 300, but information on the orientation of manipulation equipment 300 may be acquired by other methods. For example, information on the orientation of manipulation equipment 300 may be acquired based on an image showing manipulation equipment 300. In this case, for example, a general view camera is installed. This general view camera is a camera that takes pictures of nearby person 42, target person 41, selector 43, and operation device 300. In this case, the image obtained by the overall camera is analyzed to identify the direction in which the manipulation device 300 included in the image is facing.

[0116] Alternatively, for example, it may be possible to identify the destination of light emitted from a light source provided in manipulation device 300. Then, based on the identified destination, it may be possible to identify the direction in which manipulation device 300 is facing. In this case, a light source such as a laser pointer is provided for operation device 300. In this case, selector 43 operates operation device 300 so that nearby person 42 is irradiated with light from the light source. Furthermore, in this case, the image captured by the full-view camera is analyzed to identify the light emission destination, and based on this emission destination, the direction in which manipulation device 300 is facing is identified.

[0117] In the above description, a case has been described in which manipulation device 300 performs part of the processing executed in information processing system 1 of the present embodiment, and management server 100 performs the other part. In the above, the identification of utterance peripheral persons 42 based on the operation information is performed by operation device 300. Furthermore, the association of utterance content 48 with utterance peripheral persons 42 is performed by management server 100. However, the above-described processing executed in the information processing system 1 may be performed mainly by the management server 100. In this case, the management server 100 identifies the utterance peripheral persons 42 based on the operation information. Then, the management server 100 associates the utterance content 48 with the utterance peripheral persons 42. The processing of this embodiment may be realized by one device such as one management server 100, or may be realized by multiple devices. The information processing system 1 may be configured by a single device or may be configured by multiple devices.

[0118] (Addendum) (((1))) a processor; The processor: acquiring operation information for an operating device configured to be able to select each of a plurality of candidates who may speak, the operating device being operated by a selector when selecting a speaker from the plurality of candidates; Identifying a speaker from among the plurality of candidates based on the acquired operation information; Associating the identified speaker with an utterance made when the selector operated the operation device. Information processing system. (((2))) The processor: acquiring, as the operation information, orientation information that indicates an orientation of the operation device operated by the selector or an orientation of a movable part provided on the operation device; Identifying the speaker based on the orientation information; Associating the identified speaker with an utterance made when the operating device or the movable part is facing a direction identified by the direction information used to identify the speaker. The information processing system according to (((1))). (((3))) a storage unit that stores the orientation of the operating device or the orientation of the movable part in association with each of the candidates; The processor: Identifying the speaker based on the orientation identified by the acquired orientation information and the information stored in the storage unit. The information processing system according to (((2))). (((4))) The processor: acquiring, as the orientation information, information on the orientation of the control device or the movable part when the moving speed of the control device or the movable part is slower than a predetermined moving speed; The information processing system according to (((2))) or (((3))). (((5))) The processor: After identifying a speaker, when the operating device or the movable part rotates around a vertical axis to change the orientation of the operating device or the movable part, another speaker whose position around the vertical axis is different from that of the speaker is identified. An information processing system according to any one of (((2))) to (((4))). (((6))) The processor: After identifying the speaker, when the operating device or the movable part rotates around a horizontal axis and the orientation of the operating device or the movable part changes, another speaker who is closer to the operating device or the movable part than the speaker, or another speaker who is farther from the operating device or the movable part than the speaker, is identified. An information processing system according to any one of (((2))) to (((5))). (((7))) The processor: After identifying a speaker, when the operating device or the movable part rotates around a horizontal axis to change the orientation of the operating device or the movable part, another speaker located above or below the identified speaker is identified. An information processing system according to any one of (((2))) to (((5))). (((8))) The processor: As the operation information, information on the selection operation of buttons provided on the operation device and corresponding to each of the plurality of candidates is acquired; Identifying the speaker based on a selection button selected by the selector from among the plurality of buttons provided; Associating the identified speaker with an utterance made when the selector selected the selection button; The information processing system according to (((1))). (((9))) each operation of the operation device performed by the selector is associated with each of the candidates; The processor: Further, control information is generated so that candidate information, which is information about the candidate associated with the operation of the operation device performed by the selector, is notified to the selector via the operation device. An information processing system according to any one of ((1))) to (((8))). (((10))) The candidate information is displayed on a display unit of the operating device, The processor: Further, control information is generated to change the display of a non-display area of ​​the display unit other than the display area where the candidate information is displayed each time the candidate information notified to the selector via the display unit changes. The information processing system according to (((9))). (((11))) The processor: generating control information for changing the background color of the non-display area as the control information for changing the display of the non-display area; The information processing system according to (((10))). (((12))) An operation device configured to be able to select each of a plurality of candidates who may make an utterance and used when a selector selects a speaker, a processor; The processor: acquiring operation information of the operation device operated by the selector when selecting a speaker from the plurality of candidates; Identifying a speaker from among the plurality of candidates based on the operation information. Operation equipment. (((13))) The processor: As the operation information, operation information of an operation to change the orientation of the operation device or operation information of an operation to change the orientation of a movable part provided on the operation device is acquired. An operating device as described in (((12))). (((14))) each operation of the operation device performed by the selector is associated with each of the candidates; The processor: Further, control information is generated so that candidate information, which is information about the candidate associated with the operation of the operation device performed by the selector, is notified to the selector via the operation device. An operating device according to (((12))) or (((13))). (((15))) an acquisition function for acquiring operation information of an operation device configured to be able to select each of a plurality of candidates who may speak, the operation device being operated by a selector when selecting a speaker from the plurality of candidates; a function of identifying a speaker from among the plurality of candidates based on the operation information; a matching function for matching the specified speaker with an utterance made when the selector operates the operation device; A program to make the above happen on a computer. (((16))) A program executed by a computer provided in an operating device configured to allow selection of each of a plurality of candidates who may speak, and used when a selector selects a speaker, comprising: an acquisition function for acquiring operation information of the operation device operated by the selector when selecting a speaker from the plurality of candidates; a function of identifying a speaker from among the plurality of candidates based on the operation information; A program for causing the computer to realize the above.

[0119] According to the information processing system of (((1))), the accuracy of matching between an utterance and a speaker to which the utterance is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. According to the information processing system of (((2))), it is possible to associate a speaker with an utterance based on an operation performed on an operating device that is at least partially changeable in orientation. According to the information processing system of (((3))), it is possible to associate an utterance with a speaker based on information registered in advance in the storage unit. According to the information processing system of (((4))), the accuracy of matching between an utterance and a speaker associated with the utterance can be improved compared to when information on the orientation of the operating device or the orientation of the movable part when the moving speed is greater than a predetermined moving speed is acquired as orientation information. According to the information processing system of (((5))), when an operation is performed on an operating device or a movable part to rotate the operating device or the movable part around a vertical axis, it becomes possible to identify a speaker other than the speaker identified before the rotation. According to the information processing system of (((6))), when an operation is performed on an operating device or a movable part to rotate the operating device or the movable part around a horizontal axis, it becomes possible to identify another speaker who is located closer to the operating device or the movable part than the speaker identified before the rotation, or another speaker who is located farther from the operating device or the movable part than the speaker identified before the rotation. According to the information processing system of (((7))), when an operation is performed on an operating device or a movable part to rotate the operating device or the movable part around a horizontal axis, it becomes possible to identify other speakers who are located above or below the speaker identified before the rotation. According to the information processing system of (((8))), it is possible to associate a speaker with an utterance based on operations performed on an operating device having buttons corresponding to each of multiple candidates. According to the information processing system of (((9))), it is possible to notify the selector of information about the candidate identified based on the operation of the operating device performed by the selector. According to the information processing system of (((10))), it is possible to more clearly notify the selector that the information to be notified to the selector has changed, compared to when only the display of candidate information is changed. According to the information processing system of (((11))), it is possible to more clearly notify the selector that the information to be notified to the selector has changed, compared to when the background color of the hidden area does not change. According to the operating device related to (((12))), the accuracy of matching between an utterance and a speaker to be associated with that utterance can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. According to the operating device of (((13))), it is possible to associate a speaker with an utterance based on an operation performed on an operating device that is at least partially changeable in orientation. According to the operation device of (((14))), it is possible to notify the selector of information about the candidate identified based on the operation of the operation device performed by the selector. According to the program related to (((15))), the accuracy of matching between an utterance and a speaker to which the utterance is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. According to the program related to (((16))), the accuracy of matching between an utterance and a speaker to which the utterance is associated can be improved compared to when each of multiple candidates who may make an utterance declares their utterance by operating their own operating device. [Explanation of symbols]

[0120] 1...information processing system, 11a...CPU, 31a...CPU, 41...target person, 42...surrounding person, 43...selector, 51...button, 92...vertical axis, 93...horizontal axis, 95...movable part, 300...operation device, 312...storage part, 314...display part

Claims

1. a processor; The processor: acquiring operation information for an operating device configured to be able to select each of a plurality of candidates who may speak, the operating device being operated by a selector when selecting a speaker from the plurality of candidates; Identifying a speaker from among the plurality of candidates based on the acquired operation information; Associating the identified speaker with an utterance made when the selector operated the operation device. Information processing system.

2. The processor: acquiring, as the operation information, orientation information that indicates an orientation of the operation device operated by the selector or an orientation of a movable part provided on the operation device; Identifying the speaker based on the orientation information; Associating the identified speaker with an utterance made when the operating device or the movable part is facing a direction identified by the direction information used to identify the speaker. The information processing system according to claim 1 .

3. a storage unit that stores the orientation of the operating device or the orientation of the movable part in association with each of the candidates; The processor: Identifying the speaker based on the orientation identified by the acquired orientation information and the information stored in the storage unit. The information processing system according to claim 2 .

4. The processor: acquiring, as the orientation information, information on the orientation of the control device or the movable part when the moving speed of the control device or the movable part is slower than a predetermined moving speed; The information processing system according to claim 2 .

5. The processor: After identifying a speaker, when the operating device or the movable part rotates around a vertical axis to change the orientation of the operating device or the movable part, another speaker whose position around the vertical axis is different from that of the speaker is identified. The information processing system according to claim 2 .

6. The processor: After identifying the speaker, when the operating device or the movable part rotates around a horizontal axis and the orientation of the operating device or the movable part changes, another speaker who is closer to the operating device or the movable part than the speaker, or another speaker who is farther from the operating device or the movable part than the speaker, is identified. The information processing system according to claim 2 .

7. The processor: After identifying a speaker, when the operating device or the movable part rotates around a horizontal axis to change the orientation of the operating device or the movable part, another speaker located above or below the identified speaker is identified. The information processing system according to claim 2 .

8. The processor: As the operation information, information on the selection operation of buttons provided on the operation device and corresponding to each of the plurality of candidates is acquired; Identifying the speaker based on a selection button selected by the selector from among the plurality of buttons provided; Associating the identified speaker with an utterance made when the selector selected the selection button; The information processing system according to claim 1 .

9. each operation of the operation device performed by the selector is associated with each of the candidates; The processor: Further, control information is generated so that candidate information, which is information about the candidate associated with the operation of the operation device performed by the selector, is notified to the selector via the operation device. The information processing system according to claim 1 .

10. The candidate information is displayed on a display unit of the operating device, The processor: Further, control information is generated to change the display of a non-display area of ​​the display unit other than the display area where the candidate information is displayed each time the candidate information notified to the selector via the display unit changes. The information processing system according to claim 9 .

11. The processor: generating control information for changing the background color of the non-display area as the control information for changing the display of the non-display area; The information processing system according to claim 10.

12. An operation device configured to be able to select each of a plurality of candidates who may make an utterance and used when a selector selects a speaker, a processor; The processor: acquiring operation information of the operation device operated by the selector when selecting a speaker from the plurality of candidates; Identifying a speaker from among the plurality of candidates based on the operation information. Operation equipment.

13. The processor: As the operation information, operation information of an operation to change the orientation of the operation device or operation information of an operation to change the orientation of a movable part provided on the operation device is acquired. The operating device according to claim 12.

14. each operation of the operation device performed by the selector is associated with each of the candidates; The processor: Further, control information is generated so that candidate information, which is information about the candidate associated with the operation of the operation device performed by the selector, is notified to the selector via the operation device. The operating device according to claim 12.

15. an acquisition function for acquiring operation information of an operation device configured to be able to select each of a plurality of candidates who may speak, the operation device being operated by a selector when selecting a speaker from the plurality of candidates; a function of identifying a speaker from among the plurality of candidates based on the operation information; a matching function for matching the specified speaker with an utterance made when the selector operates the operation device; A program to make the above happen on a computer.

16. A program executed by a computer provided in an operating device configured to allow selection of each of a plurality of candidates who may speak, and used when a selector selects a speaker, comprising: an acquisition function for acquiring operation information of the operation device operated by the selector when selecting a speaker from the plurality of candidates; a function of identifying a speaker from among the plurality of candidates based on the operation information; A program for causing the computer to realize the above.

Images