Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program for displaying a map image according to a direction of a speech source.SOLUTION: In an information processing system that includes a display device, a controller and at least one multi-microphone device including a plurality of microphones, the controller obtains information indicating a direction of a speech source with respect to the at least one multi-microphone device; obtains information concerning details of speech emitted from the speech source and collected by the multi-microphone device; generates a map image in which the information concerning the details of the speech is disposed at a position corresponding to the direction of the speech source of the speech with respect to the multi-microphone device; and displays the map image on a display unit of the display device.SELECTED DRAWING: Figure 14

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] People with hearing loss may have a reduced ability to determine the direction from which a sound is coming due to a decline in their hearing function. When such people with hearing loss try to have a conversation with multiple people, it is difficult to accurately recognize who is saying what, which causes communication problems.

[0003] Patent document 1 discloses a conversation support device that sets display areas corresponding to multiple users in the image display area of ​​a display unit, and displays text that is the voice recognition result for one user's voice in the image display area set for another user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-129873 A Summary of the Invention [Problem to be solved by the invention]

[0005] In the conversation support device described in Patent Document 1, the utterances of other users are displayed in an aggregated state in an image display area set for a certain user. Therefore, particularly when there are three or more participants in a conversation, it is difficult to immediately grasp who made which utterance and who made what utterance.

[0006] An objective of the present disclosure is to enable users to intuitively associate speakers with what is being said based on visual information. [Means for solving the problem]

[0007] An information processing device of one embodiment of the present disclosure includes means for acquiring information indicating a direction of a sound source relative to at least one multi-microphone device, means for acquiring information relating to the content of sound emitted from the sound source and picked up by the multi-microphone device, means for generating a map image in which the information relating to the content of the sound is positioned in a position corresponding to the direction of the sound source of the sound relative to the multi-microphone device, and means for displaying the map image on a display unit of a display device. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a block diagram showing the configuration of a controller according to the present embodiment. [Diagram 3] 1A and 1B are diagrams illustrating the appearance of a multi-microphone device according to an embodiment of the present invention. [Figure 4] FIG. 1 is a diagram showing one aspect of the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a data structure of a sound source database according to the present embodiment. [Figure 6] 4 is a flowchart of audio processing according to the present embodiment. [Figure 7] FIG. 2 is a diagram for explaining sound collection by a microphone. [Figure 8] FIG. 2 is a diagram for explaining the direction of a sound source in a reference coordinate system. [Figure 9] FIG. 13 is a diagram illustrating an example of a map image. [Figure 10] 4 is a flowchart of a sound source setting process according to the present embodiment. [Figure 11] 11A to 11C are diagrams showing examples of screens displayed in the sound source setting process of the present embodiment. [Figure 12] FIG. 13 is a diagram showing one aspect of Modification 1. [Figure 13] FIG. 13 is a diagram showing a data structure of a comment database in the first modified example. [Figure 14] 13 is a flowchart of audio processing according to the first modified example. [Figure 15]FIG. 13 is a diagram showing an example of a map image of Modification 2. [Figure 16] FIG. 13 is a diagram showing another example of the map image of the second modification example. [Figure 17] FIG. 13 is a diagram showing an example of a map image of Modification 3. [Figure 18] FIG. 13 is a diagram showing an example of an image display in Modification 1. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In the drawings for explaining the embodiment, the same components are generally designated by the same reference numerals, and the repeated description will be omitted.

[0010] In the following explanation, a coordinate system (microphone coordinate system) based on the position and orientation of the multi-microphone device may be used. The microphone coordinate system has the position of the multi-microphone device (for example, the center of gravity of the multi-microphone device) as its origin, and the x-axis and y-axis intersect at the origin. In the microphone coordinate system, when the x+ direction is defined as the front of the multi-microphone device, the x-direction is defined as the rear of the multi-microphone device, the y+ direction is defined as the leftward direction of the multi-microphone device, and the y- direction is defined as the rightward direction of the multi-microphone device. Furthermore, a direction in a particular coordinate system refers to a direction relative to the origin of the coordinate system.

[0011] (1) Information Processing System Configuration The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to the present embodiment.

[0012] As shown in FIG. 1, the information processing system 1 includes a display device 10, a controller 30, and a multi-microphone device 50. The information processing system 1 is used by multiple users. At least one of the users may be hearing-impaired, but not all of the users may be hearing-impaired (that is, all of the users may have sufficient hearing for conversation).

[0013] The display device 10 and the controller 30 are connected via a communication cable or a wireless channel (eg, a Wi-Fi channel or a Bluetooth® channel). Similarly, the controller 30 and the multi-microphone device 50 are connected via a communication cable or a wireless channel (eg, a Wi-Fi channel or a Bluetooth channel).

[0014] The display device 10 includes one or more displays 11 (one example of a "display unit"). The display device 10 receives an image signal from the controller 30, and displays an image corresponding to the image signal on the display. The display device 10 is, for example, a tablet terminal, a personal computer, a smartphone, or a display device for a conference. The display device 10 may include an input device or an operation unit for acquiring instructions from a user.

[0015] The controller 30 controls the display device 10 and the multi-microphone device 50. The controller 30 is an example of an information processing device. The controller 30 is, for example, a smartphone, a tablet terminal, a personal computer, or a server computer.

[0016] The multi-microphone device 50 can be placed independently of the display device 10. That is, the position and orientation of the multi-microphone device 50 can be determined independently of the position and orientation of the display device 10.

[0017] (1-1) Controller configuration The configuration of the controller will now be described with reference to Fig. 2, which is a block diagram showing the configuration of the controller of this embodiment.

[0018] As shown in FIG. 2, the controller 30 includes a storage device 31, a processor 32, an input / output interface 33, and a communication interface .

[0019] The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of a Read Only Memory (ROM), a Random Access Memory (RAM), and a storage (for example, a flash memory or a hard disk).

[0020] The programs include, for example, the following programs: ·OS (Operating System) programs Application programs that perform information processing

[0021] The data includes, for example, the following data: Databases referenced in information processing Data obtained by performing information processing (i.e., the results of performing information processing)

[0022] The processor 32 is a computer that realizes the functions of the controller 30 by starting a program stored in the storage device 31. The processor 32 is, for example, at least one of the following. ·CPU(Central Processing Unit) ·GPU(Graphic Processing Unit) ·ASIC(Application Specific Integrated Circuit) ·FPGA(Field Programmable Array)

[0023] The input / output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the controller 30, and to output information (eg, an image signal) to an output device connected to the controller 30. The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof. The output device is, for example, a display.

[0024] The communication interface 34 is configured to control communication between the controller 30 and external devices (eg, the display device 10 and the multi-microphone device 50).

[0025] (1-2) Configuring a multi-microphone device The configuration of the multi-microphone device will now be described with reference to Fig. 3, which is a diagram showing the external appearance of the multi-microphone device of this embodiment.

[0026] The multi-microphone device 50 includes a plurality of microphones. In the following description, the multi-microphone device 50 includes five microphones 51-1, . . . , 51-5 (hereinafter, simply referred to as microphones 51 unless otherwise specified). The multi-microphone device 50 receives (collects) sound emitted from a sound source using the microphones 51-1, . . . , 51-5 to generate an audio signal. The multi-microphone device 50 also estimates the direction from which the sound comes in the microphone coordinate system (i.e., the direction of the sound source). The multi-microphone device 50 also performs beamforming processing, which will be described later.

[0027] The microphone 51 collects, for example, sounds around the multi-microphone device 50. The sounds collected by the microphone 51 include, for example, at least one of the following sounds. -Speech by a person - Sound of the environment in which the multi-microphone device 50 is used

[0028] The multi-microphone device 50 has a mark 50a on the surface of the housing thereof, for example, which indicates the reference direction of the multi-microphone device 50 (for example, forward (i.e., the x+ direction), but it may be another specified direction). This allows the user to easily recognize the orientation of the multi-microphone device 50 from visual information. Note that the means for recognizing the orientation of the multi-microphone device 50 is not limited to this. The mark 50a may be integrated with the housing of the multi-microphone device 50.

[0029] The multi-microphone device 50 further includes a processor for performing, for example, audio processing, which will be described later, a storage device, and a communication or input / output interface. The multi-microphone device 50 may also include an IMU (Inertial Measurement Unit) for detecting the movement and state of the multi-microphone device 50.

[0030] (2) One aspect of the embodiment An embodiment of the present invention will be described below with reference to Fig. 4, which is a diagram showing an embodiment of the present invention.

[0031] The controller 30 generates a map image and displays it on the display 11 of the display device 10 while a conversation (e.g., a conference) is taking place between multiple participants (i.e., users of the information processing system 1). The map image corresponds to a bird's-eye view of the sound source (speaker) environment around the multi-microphone device 50, and text based on the voice uttered by the speaker (one example of "information related to the content of the voice") is arranged at a position based on the direction of the speaker relative to the multi-microphone device 50. The controller 30 updates the map image in response to utterances by the participants. In this way, the map image plays the role of a UI (User Interface) for visually grasping the content of the most recent conversation (particularly, who is saying what) in real time.

[0032] Specifically, as shown in FIG. 4, the map image includes a microphone icon MI31, a circumference CI31, sound source icons SI31, SI32, SI33, and SI34, and text images TI32 and TI34.

[0033] The microphone icon MI31 represents the multi-microphone device 50. The microphone icon MI31 includes a mark MR31 that indicates the orientation of the microphone icon MI31. A viewer of the map image can recognize where the microphone icon MI31 is facing in the map image by checking the mark MR31. By making the appearances of the microphone icon MI31 and the mark MR31 similar to the appearances of the multi-microphone device 50 and the mark 50a, a viewer of the map image can easily associate participants in the real world with sound source icons in the map image. However, it is not essential that the appearances of the microphone icon MI31 and the mark MR31 are similar to the appearances of the multi-microphone device 50 and the mark 50a.

[0034] The circumference CI31 corresponds to a circumference with the microphone icon MI31 at the center. In the example of FIG. 4, the controller 30 arranges the sound source icons SI31, SI32, SI33, and SI34 corresponding to the participants of the conversation on the circumference CI31. Specifically, the controller 30 arranges each of the sound source icons SI31, SI32, SI33, and SI34 at a position on the circumference CI31 according to the direction of the sound source represented by the sound source icon with respect to the multi-microphone device 50. As an example, the controller 30 converts the microphone coordinate system into a coordinate system of a map image (hereinafter, "map coordinate system"). Then, the controller 30 arranges the sound source icon representing the sound source at the intersection of the circumference CI31 and a straight line extending from the display position of the microphone icon MI31 (an example of the "origin of the map coordinate system") in the (estimated) direction of the sound source represented in the map coordinate system.

[0035] The sound source icon SI31 represents a specific one of the multiple participants (e.g., a person with hearing loss who has more opportunities to view the map image than the other participants; hereinafter, sometimes referred to as "you"). The controller 30 may set a specific format (e.g., color, texture, optical effect, shape, size, etc.) for the sound source icon SI31 representing "you" that is different from the sound source icons representing other sound sources.

[0036] The sound source icon SI32 represents person D among the multiple participants. In the example of Fig. 4, person D is currently speaking. The controller 30 may set a format for the sound source icon SI32 representing the currently speaking speaker (sound source) that is different from that for sound source icons representing speakers (sound sources) in other states. In other words, the controller 30 can dynamically change the format of the sound source icon depending on the state of the sound source represented by the sound source icon.

[0037] The text image TI32 represents the most recent utterance of D (a voice recognition result for the voice uttered by D). The controller 30 arranges the text image TI32 on the map image in a form that allows a viewer of the map image to easily recognize the correspondence between the text image TI32 and the sound source icon SI32. As an example, the controller 30 arranges the text image TI32 at a predetermined position (e.g., the lower right) with respect to the sound source icon SI32. Furthermore, the controller 30 may set the text image TI32 to at least partially the same format as the sound source icon SI32. For example, the controller 30 may align the background or characters of the sound source icon SI32 and the text image TI32 in similar colors.

[0038] The sound source icon SI33 represents one of the multiple participants, Mr. T. In the example of Fig. 4, Mr. T is not speaking. The controller 30 may set a format for the sound source icon SI33 representing a speaker (sound source) who is not speaking, different from that for sound source icons representing speakers (sound sources) in other states.

[0039] The sound source icon SI34 represents one of the multiple participants, Mr. H. In the example of Fig. 4, Mr. H has just finished speaking. The controller 30 may set a format for the sound source icon SI34 representing a speaker (sound source) who has just finished speaking, different from that for sound source icons representing speakers (sound sources) in other states.

[0040] The text image TI34 represents the most recent statement made by Mr. H. The controller 30 arranges the text image TI34 on the map image in a form that allows a viewer of the map image to easily recognize the correspondence between the text image TI34 and the sound source icon SI34. As an example, the controller 30 arranges the text image TI34 at a predetermined position (e.g., lower right) with respect to the sound source icon SI34. The controller 30 may also set the text image TI34 to at least partially the same format as the sound source icon SI34. For example, the controller 30 may align the background or characters of the sound source icon SI34 and the text image TI34 in similar colors.

[0041] In this way, the controller 30 generates a map image by arranging text corresponding to the voice uttered by the speaker at a position according to the estimation result of the direction of the speaker relative to the multi-microphone device 50, and displays the map image on the display 11 of the display device 10. This allows a viewer of the map image to intuitively associate the speaker with the content of the speech.

[0042] (3) Database The databases of this embodiment are stored in the storage device 31.

[0043] (3-1) Sound source database The sound source database of this embodiment will now be described with reference to Fig. 5, which shows the data structure of the sound source database of this embodiment.

[0044] The sound source database stores sound source information. The sound source information is information about sound sources (typically, speakers) around the multi-microphone device 50 that are identified by the controller 30.

[0045] 5, the audio source database includes an "ID" field, a "name" field, an "icon" field, a "direction" field, a "recognition language" field, and a "translation language" field. Each field is associated with the others.

[0046] The "ID" field stores a sound source ID. The sound source ID is information for identifying a sound source. When the controller 30 detects a new sound source, the controller 30 issues a new sound source ID and assigns the sound source ID to the sound source.

[0047] The "Name" field stores sound source name information. The sound source name information is information about the name of a sound source. The controller 30 may automatically determine the sound source name information, or may set it according to a user instruction as described later. The controller 30 may assign some initial sound source name to a newly detected sound source according to a predetermined rule or randomly.

[0048] The "icon" field stores icon information. The icon information is information about the icon of the sound source. As an example, the icon information may include information that can specify an icon image (e.g., one of the preset icon images, or a photo or picture provided by the user) or the format of the icon (e.g., color, texture, optical effect, shape, etc.). The controller 30 may automatically determine the icon information or set it according to a user instruction. The controller 30 may assign some initial icon to a newly detected sound source according to a predetermined rule or randomly. However, when the icon of the sound source is not displayed on the map image as in the second modified example described later, the icon information can be omitted from the sound source information.

[0049] The "direction" field stores sound source direction information. The sound source direction information is information about the direction of a sound source relative to the multi-microphone device 50. As an example, the direction of a sound source is expressed as an angle of deviation from an axis having a reference direction (in this embodiment, the front (x+ direction) of the multi-microphone device 50) set as 0 degrees in a microphone coordinate system, the reference direction being determined based on the microphones 51-1 to 51-5.

[0050] The "recognition language" field stores recognition language information. The recognition language information is information about the language used by the sound source (speaker). Based on the recognition language information of the sound source, a voice recognition engine to be applied to the voice generated from the sound source is selected. The recognition language information may be set by a user operation, or may be automatically set based on the language recognition result by a voice recognition model.

[0051] The "translation language" field stores translation language information. The translation language information is information about a target language when machine translation is applied to a speech recognition result (text) for a speech generated from a sound source. Based on the translation language information of the sound source, a machine translation engine to be applied to the speech recognition result for the speech generated from the sound source is selected. The translation language information may be set collectively for all sound sources rather than for each individual sound source, or may be set for each display device 10.

[0052] Additionally, the sound source information may include sound source distance information. The sound source distance information is information relating to the distance from the multi-microphone device 50 to the sound source. The sound source direction information and the sound source distance information can also be expressed as sound source position information. The sound source position information is information relating to the relative position of the sound source with respect to the multi-microphone device 50 (i.e., the coordinates of the sound source in the coordinate system of the multi-microphone device 50).

[0053] (4) Information processing The information processing of this embodiment will be described.

[0054] (4-1) Audio processing The audio processing of this embodiment will be described. Fig. 6 is a flowchart of the audio processing of this embodiment. Fig. 7 is a diagram for explaining sound collection by a microphone. Fig. 8 is a diagram for explaining the direction of a sound source in a reference coordinate system. Fig. 9 is a diagram showing an example of a map image.

[0055] The audio processing shown in Fig. 6 is started after the display device 10, the controller 30, and the multi-microphone device 50 are powered on and the initial settings are completed. However, the start timing of the processing shown in Fig. 6 is not limited to this. The processing shown in Fig. 6 may be repeatedly executed at a predetermined cycle, for example, so that the user of the information processing system 1 can view a map image that is updated in real time.

[0056] The multi-microphone device 50 acquires an audio signal via the microphone 51 (S150). Specifically, the multiple microphones 51-1 to 51-5 included in the multi-microphone device 50 collect speech sounds emitted by a speaker. The microphones 51-1 to 51-5 collect the speech sounds that arrive via the multiple paths shown in Fig. 7. The microphones 51-1 to 51-5 convert the collected speech sounds into audio signals.

[0057] The processor included in the multi-microphone device 50 acquires, from the microphones 51-1 to 51-5, audio signals including speech sounds emitted from at least one of the speakers PR3, PR4, and PR5. The audio signals acquired from the microphones 51-1 to 51-5 include spatial information (e.g., delay and phase change) based on the path along which the speech sounds have traveled.

[0058] After step S150, the multi-microphone device 50 performs estimation of the direction of arrival (S151). An arrival direction estimation model is stored in a storage device included in the multi-microphone device 50. The arrival direction estimation model describes information for identifying a correlation between spatial information included in an audio signal and the arrival direction of an uttered sound.

[0059] The direction of arrival estimation method used in the direction of arrival estimation model may be any existing method, such as MUSIC (Multiple Signal Classification) using eigenvalue expansion of an input correlation matrix, a minimum norm method, or ESPRIT (Estimation of Signal Parameters via Rotational Invariance Techniques).

[0060] The multi-microphone device 50 estimates the arrival direction of the speech sound collected by the microphones 51-1 to 51-5 (i.e., the direction of the sound source of the speech sound relative to the multi-microphone device 50) by inputting the audio signals received from the microphones 51-1 to 51-5 to the arrival direction estimation model. At this time, the multi-microphone device 50 expresses the arrival direction of the speech sound, for example, in a microphone coordinate system, as an angle from an axis having a reference direction (in this embodiment, the front (x+ direction) of the multi-microphone device 50) set as 0 degrees, which is determined based on the microphones 51-1 to 51-5. In the example shown in FIG. 8, the multi-microphone device 50 estimates the arrival direction of the speech sound emitted from the speaker PR3 as a direction shifted leftward by an angle A2 from the x-axis. The multi-microphone device 50 estimates the arrival direction of the speech sound emitted from the speaker PR4 as a direction shifted leftward by an angle A3 from the x-axis. The multi-microphone device 50 estimates the arrival direction of the speech sound emitted by the speaker PR5 to be a direction shifted to the right from the x-axis by an angle A1.

[0061] After step S151, the multi-microphone device 50 executes extraction of an audio signal (S152). A beamforming model is stored in a storage device included in the multi-microphone device 50. The beamforming model describes information for identifying a correlation between a specific direction and parameters for forming a directivity having a beam in that direction. Here, forming a directivity is a process for amplifying or attenuating a sound from a specific arrival direction.

[0062] The multi-microphone device 50 inputs the estimated arrival direction to the beamforming model, and calculates parameters for forming a directivity having a beam in the arrival direction.

[0063] 8, the multi-microphone device 50 inputs the calculated angle A1 into the beamforming model and calculates parameters for forming a directivity having a beam in a direction shifted to the right from the x-axis by angle A1. The multi-microphone device 50 inputs the calculated angle A2 into the beamforming model and calculates parameters for forming a directivity having a beam in a direction shifted to the left from the x-axis by angle A2. The multi-microphone device 50 inputs the calculated angle A3 into the beamforming model and calculates parameters for forming a directivity having a beam in a direction shifted to the left from the x-axis by angle A3.

[0064] The multi-microphone device 50 amplifies or attenuates the audio signals acquired from the microphones 51-1 to 51-5 using parameters calculated for the angle A1. The multi-microphone device 50 synthesizes the amplified or attenuated audio signals to extract, from the acquired audio signals, an audio signal regarding a speech sound arriving from a sound source in a direction corresponding to the angle A1.

[0065] The multi-microphone device 50 amplifies or attenuates the audio signals acquired from the microphones 51-1 to 51-5 using parameters calculated for the angle A2. The multi-microphone device 50 synthesizes the amplified or attenuated audio signals to extract, from the acquired audio signals, an audio signal regarding a speech sound arriving from a sound source in a direction corresponding to the angle A2.

[0066] The multi-microphone device 50 amplifies or attenuates the audio signals acquired from the microphones 51-1 to 51-5 using parameters calculated for the angle A3. The multi-microphone device 50 synthesizes the amplified or attenuated audio signals to extract, from the acquired audio signals, an audio signal regarding a speech sound arriving from a sound source in a direction corresponding to the angle A3.

[0067] The multi-microphone device 50 transmits the extracted audio signal to the controller 30 together with information indicating the direction of the sound source corresponding to the audio signal estimated in step S151 (that is, the estimation result of the direction of the sound source relative to the multi-microphone device 50).

[0068] After step S152, the controller 30 performs sound source identification (S130). Specifically, the controller 30 identifies a sound source present around the multi-microphone device 50 based on the estimation result of the direction of the sound source acquired in step 151 (hereinafter referred to as the “target direction”).

[0069] As an example, the controller 30 determines whether the sound source corresponding to the target direction is the same as an identified sound source, and assigns a new sound source ID (FIG. 5) if the sound source corresponding to the target direction is not an identified sound source. Specifically, the controller 30 compares the target direction with the sound source direction information for the identified sound sources (FIG. 5). Then, when the controller 30 determines that the target direction matches any of the sound source direction information for the identified sound sources, the controller 30 treats the sound source corresponding to the target direction as a (identified) sound source having the matching sound source direction information. On the other hand, when the controller 30 determines that the target direction does not match any of the sound source direction information for the identified sound sources, the controller 30 detects that a new sound source exists in the target direction, and assigns a new sound source ID to the new sound source. Here, the fact that the target direction matches the sound source direction information includes at least that the target direction matches the direction indicated by the sound source direction information, and further includes that the difference or ratio of the target direction to the direction indicated by the sound source direction information is within an allowable range.

[0070] After step S130, the controller 30 executes a voice recognition process (S131). The storage device 31 stores a voice recognition model. The voice recognition model describes information for identifying a correlation between a voice signal and a text for the voice signal. The voice recognition model is, for example, a trained model generated by machine learning. Note that the voice recognition model may be stored in an external device (for example, a cloud server) accessible by the controller 30 via a network (for example, the Internet) instead of the storage device 31.

[0071] The controller 30 inputs the extracted voice signal into a voice recognition model to determine text corresponding to the input voice signal. The controller 30 may select a voice recognition engine based on recognition language information of a sound source corresponding to the voice signal.

[0072] In the example shown in FIG. 8, the controller 30 inputs the voice signals extracted for the angles A1 to A3 into a voice recognition model, respectively, to determine the text corresponding to the input voice signals.

[0073] After step S131, the controller 30 executes machine translation (S132). Specifically, when translation language information (FIG. 5) is set in the sound source of the voice corresponding to the text generated in step S131, the controller 30 performs machine translation of the text. As a result, the controller 30 obtains text in the language specified by the translation language information. The controller 30 may select a machine translation engine based on the translation language information of the sound source corresponding to the voice signal. On the other hand, when translation language information (FIG. 5) is not set in the sound source of the voice corresponding to the text generated in step S131 (i.e., when converting the voice into text without translating it), the controller 30 can omit this step.

[0074] After step S132, the controller 30 generates a map image (S133). Specifically, the controller 30 generates a text image representing text based on the result of the voice recognition process in step S131 or the result of the machine translation process in step S132. The controller 30 places a sound source icon representing the identified sound source around the microphone icon (on a circumference centered on the microphone icon, as an example) based on the direction of the sound source relative to the multi-microphone device 50 (i.e., the estimation result in step S151). The controller 30 places the above-mentioned text image at a predetermined position relative to the sound source icon representing the sound source of the corresponding sound.

[0075] As an example, the controller 30 generates a map image shown in FIG. 9. In the map image of FIG. 9, the microphone coordinate system is converted to the map coordinate system so that the front (x+ direction) of the microphone icon MI31 faces the upper direction of the map image. However, the controller 30 can change the correspondence relationship between the microphone coordinate system and the map coordinate system. As an example, the controller 30 may rotate the display position of each sound source icon around the display position of the microphone icon MI31 so that a specific sound source icon is located in a predetermined direction (for example, downward) of the map coordinate system in response to a user instruction. For example, the map image of FIG. 4 can be generated by rotating the display positions of each sound source icon SI31 to SI34 90 degrees counterclockwise around the display position of the microphone icon MI31 in the map image of FIG. 9 so that the sound source icon SI31 is located in the lower direction of the map image, and moving the text images TI32 and TI34 to a predetermined position (for example, "lower right") with respect to the rotated sound source icons SI32 and SI34.

[0076] Furthermore, the controller 30 may generate a map image so as to highlight a sound source icon representing the sound source or text related to the sound while the sound source is emitting the sound. The controller 30 may highlight the sound source icon or text by, for example, at least one of the following means: -Add animation Enlarged view · Changing the color, texture, optical effects, or shape

[0077] After step S133, the controller 30 executes information display (S134). Specifically, the controller 30 displays the map image generated in step S133 on the display 11 of the display device 10.

[0078] (4-2) Sound source setting process The sound source setting process of this embodiment will be described below. Fig. 10 is a flowchart of the sound source setting process of this embodiment. Fig. 11 is a diagram showing an example of a screen displayed in the sound source setting process of this embodiment.

[0079] The sound source setting process shown in Fig. 10 is started in response to an instruction from a user of the information processing system 1 after the start of the audio process shown in Fig. 6. However, the start timing of the sound source setting process shown in Fig. 10 is not limited to this. For example, the process in Fig. 10 may be executed as an initial setting process before the start of the audio process shown in Fig. 6.

[0080] As shown in FIG. 10, the controller 30 selects a sound source (S230). Specifically, the controller 30 displays a sound source setting UI for allowing the user to set sound source information on the display 11 of the display device 10. As an example, the controller 30 displays the screen of Fig. 11 on the display 11 of the display device 10. The screen of Fig. 11 includes a map image MP40 and a sound source setting UI (image) CU40.

[0081] The sound source setting UI CU40 includes display objects A41 and A42 and an operation object B43. The display object A41 displays information of the registered participant (for example, a sound source icon and a registered sound source name). Here, the registered participant means a sound source (speaker) identified in the sound source identification (S130) in FIG. 6, whose sound source name information is registered by the sound source setting process shown in FIG. 10.

[0082] The display object A42 displays information of an unregistered participant (for example, a sound source icon and an initial sound source name). Here, the unregistered participant means a sound source (speaker) identified in the sound source identification (S130) in FIG. 6, for which sound source name information is not registered (i.e., a sound source using the initial sound source name determined by the controller 30).

[0083] The operation object B43 accepts an operation to add a participant. Specifically, the user of the information processing system 1 selects the operation object B43, and further designates one of the unregistered participants. The controller 30 may present an input form (for example, a text field, a menu, a radio button, a check box, or a combination thereof) on the display device 10 to accept the designation of the unregistered participant.

[0084] The controller 30 selects a sound source (an unregistered participant) for which sound source information is to be set in response to a user instruction.

[0085] After step S230, the controller 30 executes acquisition of sound source information (S231). Specifically, the controller 30 acquires sound source information to be set for the sound source selected in step S230 in response to a user instruction. As an example, the controller 30 acquires sound source name information for the selected sound source. Furthermore, the controller 30 may acquire icon information, recognition language information, translation language information, or a combination thereof for the selected sound source. In order to acquire the sound source information, the controller 30 may display an input form (e.g., a text field, a menu, a radio button, a check box, or a combination thereof) on the display 11 of the display device 10. The controller 30 may acquire participant information of the conversation and generate elements of the input form (menu, radio button, or check box) based on the participant information. The participant information of the conversation may be manually set before the start of the conversation, or may be acquired from the account names logged in to the information processing system 1 or the linked conference system.

[0086] After step S231, the controller 30 executes updating of the sound source information (S232). Specifically, the controller 30 updates the sound source information by registering the sound source information acquired in step S231 in the sound source database (FIG. 5) in association with the sound source ID that identifies the sound source selected in step S230.

[0087] With step S232, the controller 30 may end the sound source setting process shown in Fig. 10. Alternatively, the controller 30 may repeatedly execute the sound source setting process until the user instructs the end of the sound source setting process or until sound source information is set for all of the unregistered participants.

[0088] (5) Summary As described above, the controller 30 of this embodiment obtains an estimation result indicating the direction of a sound source relative to the multi-microphone device 50, and obtains information regarding the content of the sound emitted from the sound source and collected by the multi-microphone device 50. The controller 30 generates a map image in which text is arranged at a position according to the direction of the sound source corresponding to the text relative to the multi-microphone device 50, and displays the map image on the display 11 of the display device 10. This allows a viewer of the map image to intuitively recognize the association between the sound source (e.g., a speaker) and the content of the sound (e.g., a statement) emitted from the sound source.

[0089] The controller 30 may identify individual sound sources present around the multi-microphone device 50 based on the estimation result of the direction of the sound source, and set sound source information regarding the identified sound source, for example, according to a user instruction. This makes it possible to appropriately set sound source information for a sound source corresponding to text displayed in a map image. The controller 30 may set at least one of sound source name information, recognition language information, and translation language information for the identified sound source. This makes it possible to clarify who spoke the text displayed in a map image, and to generate accurate text or text that is easy for the user to understand.

[0090] The controller 30 may generate a map image including a microphone icon representing the multi-microphone device 50 and a sound source icon representing a sound source, and the sound source icon is arranged at a position on a circumference centered on the microphone icon according to the direction of the sound source corresponding to the sound source icon with respect to the multi-microphone device. This allows a viewer of the map image to intuitively recognize which direction the text displayed on the map image corresponds to the sound emitted from the sound source located with respect to the multi-microphone device 50. Also, a viewer of the map image can intuitively recognize which sound source in the real space the sound source icon displayed on the map image corresponds to. Also, the controller 30 may display the map image so as to emphasize the sound source icon representing the sound source or information regarding the content of the sound while the sound source is emitting sound. This allows a viewer to easily distinguish the sound source and text (for example, the speaker who is speaking and the content of the speech) that should be noted even when multiple sound source icons and multiple texts are displayed on the map image. Also, the controller 30 may rotate the display positions of each sound source icon and each text around the display position of the microphone icon so that a specific sound source icon is located in a specific direction (for example, downward) on the map image. This allows a speaker (eg, a person with hearing impairment) corresponding to a specific sound source icon to easily understand the correspondence between other speakers (sound sources) and the sound source icons in the map image.

[0091] (6) Variations A modification of this embodiment will now be described.

[0092] (6-1) Variation 1 A description will be given of Modification 1. Modification 1 is an example in which minutes are generated in addition to a map image.

[0093] (6-1-1) One aspect of Modification 1 A description will be given of one aspect of Modification 1. Fig. 12 is a diagram showing one aspect of Modification 1.

[0094] The controller 30 generates a map image and minutes while a conversation between multiple participants is taking place, and displays them on the display 11 of the display device 10. The minutes correspond to a speech history in which speech contents by sound sources (speakers) around the multi-microphone device 50 are arranged in chronological order. The controller 30 updates the map image and minutes in response to speech by the participants. In this way, the minutes serve as a UI for visually grasping the flow of the conversation so far (particularly, who has said what) in real time.

[0095] 12, the controller 30 displays the map image MP50 and the minutes (image) MN50, for example, arranged on one screen, on the display 11 of the display device 10. The minutes MN50 includes a display object A51. Note that instead of arranging the map image MP50 and the minutes MN50 on one screen, the controller 30 may display only one of the map image MP50 and the minutes MN50 selected by the user on the display 11 of the display device 10.

[0096] The display object A51 displays information on the speaker's utterance (for example, the icon or name of the speaker (sound source), the utterance time, the utterance content, or a combination thereof). When a user of the information processing system 1 (for example, the speaker, but may be another user) finds an error in the utterance content arranged in the minutes MN50 (for example, an error in voice recognition or an error in machine translation), the user can select the display object A51 displaying the utterance content and edit the utterance content. The controller 30 acquires the edited utterance content from the user, for example, via an input form, and updates the display object A51 based on the utterance content. Furthermore, when text corresponding to the edited utterance content is included in the map image MP50, the controller 30 may update the text. Moreover, the controller 30 may display a screen shown in FIG. 18 on the display 11 instead of the screen shown in FIG. 12. In the example of FIG. 18, a mark on a circular arc is displayed on the speaker's icon to indicate the direction of the speaker with respect to the multi-microphone device 50. This allows the user to know in which direction the speaker of each comment is located relative to the multi-microphone device 50, simply by checking the minutes MN50, without having to check the map image MP50.

[0097] In this way, the controller 30 generates minutes equivalent to a history of utterances made by speakers present around the multi-microphone device 50, and displays the minutes on the display 11 of the display device 10. This allows viewers of the minutes to easily look back on the flow of the conversation.

[0098] (6-1-2) Database A description will be given of the database of the modification 1. The following database is stored in the storage device 31.

[0099] (6-1-2-1) Speech Database The following describes the voice source database of Modification 1. Fig. 13 is a diagram showing the data structure of the utterance database of Modification 1.

[0100] The utterance database stores utterance information. The utterance information is information related to the voice (utterance) collected by the multi-microphone device 50.

[0101] As shown in FIG. 13, the statement database includes a "statement ID" field, a "sound source ID field", a "statement date and time" field, and a "statement content" field. Each field is associated with another.

[0102] The "utterance ID" field stores an utterance ID. The utterance ID is information for identifying an utterance. When the controller 30 detects a new utterance from the speech recognition result or the machine translation result, it issues a new utterance ID and assigns the utterance ID to the utterance. The controller 30 divides the utterances according to changes in speakers. Furthermore, the controller 30 can divide a series of utterances by the same speaker according to phonetic boundaries (e.g., silent intervals) or semantic boundaries of the text.

[0103] The "sound source ID" field stores a sound source ID. The sound source ID is information for identifying the speaker (sound source) who made the utterance. The sound source ID corresponds to an external key for referencing the sound source database in FIG. 5 as a parent table.

[0104] The "statement date and time" field stores statement date and time information. Statement date and time information is information related to the date and time when a statement was made. Statement date and time information may be information indicating an absolute date and time, or may be information indicating the elapsed time from the start of the conversation.

[0105] The "statement content" field stores statement content information. Statement content information is information related to the content of a statement. Statement content information is, for example, the result of speech recognition of a statement, or the result of machine translation of the speech recognition result, or the result of editing these by a user.

[0106] In this embodiment, the utterance database can also be used to reproduce a map image at a specific time point.

[0107] (6-1-3) Information processing The information processing of the first modification will be described.

[0108] (6-1-3-1) Audio processing The following describes the audio processing of Modification 1. FIG 14 is a flowchart of the audio processing of Modification 1.

[0109] The audio processing shown in Fig. 14 is started after the display device 10, the controller 30, and the multi-microphone device 50 are powered on and the initial settings are completed. However, the start timing of the processing shown in Fig. 14 is not limited to this. The processing shown in Fig. 14 may be repeatedly executed at a predetermined cycle, for example, so that the user of the information processing system 1 can view the map image and minutes that are updated in real time.

[0110] As shown in FIG. 14, similarly to FIG. 6, the multi-microphone device 50 acquires an audio signal (S150), estimates an arrival direction (S151), and extracts an audio signal (S152).

[0111] After step S152, the controller 30 executes sound source identification (S130), voice recognition processing (S131), machine translation (S132), and map image generation (S133) in the same manner as in Fig. 6. Note that the controller 30 registers utterance information in the utterance database (Fig. 13) during steps S130 to S132.

[0112] After step S133, the controller 30 executes the generation of minutes (S334). Specifically, the controller 30 refers to the comment database (FIG. 13) and generates the minutes. As an example, the controller 30 may update the minutes generated during the previous execution of step S334 (hereinafter referred to as "previous minutes") based on the comment information (i.e., new comment information) registered in the comment database during steps S130 to S132.

[0113] After step S334, the controller 30 executes information display (S335). Specifically, the controller 30 displays, on the display 11 of the display device 10, the map image generated in step S133 and the minutes generated in step S334.

[0114] (6-1-4) Summary As described above, the controller 30 of the first modification generates minutes based on text (i.e., voice recognition results or machine translation results) related to utterances by sound sources (speakers) present around the multi-microphone device 50, and displays the minutes alongside the map image on the display 11 of the display device 10. This allows viewers of the map image and minutes to intuitively recognize the association between the speaker and the contents of the utterances by the speaker by viewing the map image, and to easily review the flow of the conversation up to that point by viewing the minutes. The controller 30 may also generate minutes by arranging text related to the utterances in chronological order of the utterance dates and times. This allows viewers of the minutes to intuitively recognize the flow of the conversation up to that point. The controller 30 may also edit the text arranged in the minutes in response to a user instruction. This allows smooth communication to be promoted, since even if a user (especially a person with hearing loss) misunderstands the contents of the utterance due to an error in voice recognition or machine translation, the person who made the utterance or users around him / her can quickly correct the error. It is also possible to leave accurate minutes for checking the contents of the utterances during the meeting after the meeting is over.

[0115] (6-2) Variation 2 Modification 2 will be described. Modification 2 is an example of generating a map image different from that of this embodiment. Fig. 15 is a diagram showing an example of a map image of Modification 2. Fig. 16 is a diagram showing another example of the map image of Modification 2.

[0116] The controller 30 generates a map image while a conversation between multiple participants is taking place, and displays it on the display 11 of the display device 10. The map image corresponds to a bird's-eye view of the sound source (speaker) environment around the multi-microphone device 50, and text based on the voice uttered by the speaker is placed at a position based on the direction of the speaker relative to the multi-microphone device 50. The controller 30 updates the map image in response to utterances by the participants. In this way, the map image serves as a UI for visually grasping the content of the most recent conversation (particularly, who is saying what) in real time.

[0117] Specifically, the map image shown in FIG. 15 includes a microphone icon MI61, a circumference CI61, display objects A61 and A62, and text images TI61a, TI61b, and TI62.

[0118] Like the microphone icon MI31 (FIG. 4), the microphone icon MI61 represents the multi-microphone device 50. The microphone icon MI61 includes a mark MR61 that indicates the orientation of the microphone icon MI61. The circumference CI61 corresponds to a circumference having the microphone icon MI61 at its center, similar to the circumference CI31 (FIG. 4).

[0119] Text image TI61a is a utterance by the first speaker, and corresponds to the utterance content with the second most recent utterance date and time among text images TI61a, TI61b, and TI62 displayed in Fig. 15. Text image TI61a is placed at a position according to the direction of the first speaker relative to multi-microphone device 50. Specifically, text image TI61a is placed along a straight line extending from the display position of microphone icon MI61 (an example of the "origin of the map coordinate system") toward the (estimated) direction of the first speaker.

[0120] Text image TI61b is a speech by the first speaker, and corresponds to the most recent speech content among text images TI61a, TI61b, and TI62 displayed in FIG. 15. Text image TI61b is placed at a position according to the direction of the first speaker relative to multi-microphone device 50. Specifically, text image TI61b is placed along a straight line extending from the display position of microphone icon MI61 toward the (estimated) direction of the first speaker. However, text image TI61b is placed at a position closer to the display position of microphone icon MI61 than text image TI61a corresponding to an older speech date and time.

[0121] The display object A61 displays the (estimated) direction of the first speaker (sound source) relative to the multi-microphone device 50. The display object A61 corresponds to a sector having a predetermined angular width centered on a straight line extending from the display position of the microphone icon MI61 toward the direction of the first speaker. The controller 30 may set a specific format for the display object A61 that differs from objects that display the directions of other speakers. The controller 30 may set the display object A61 to a format that is at least partially the same as that of the text images TI61a and TI61b. For example, the controller 30 may align the display object A61 to a similar color to the background or characters of the text images TI61a and TI61b.

[0122] Text image TI62 is a utterance by the second speaker, and corresponds to the utterance content with the oldest utterance date and time among text images TI61a, TI61b, and TI62 displayed in Fig. 15. Text image TI62 is placed at a position according to the direction of the second speaker relative to multi-microphone device 50. Specifically, text image TI62 is placed along a straight line extending from the display position of microphone icon MI61 in the (estimated) direction of the second speaker.

[0123] The display object A62 displays the (estimated) direction of the second speaker (sound source) relative to the multi-microphone device 50. The display object A62 corresponds to a sector having a predetermined angular width centered on a straight line extending from the display position of the microphone icon MI61 toward the direction of the second speaker. The controller 30 may set a specific format for the display object A62 that differs from objects that display the directions of other speakers. The controller 30 may set the display object A62 to at least partially the same format as the text image TI62. For example, the controller 30 may align the display object A62 with the same color as the background or characters of the text image TI62.

[0124] In response to a new comment by a participant, the controller 30 updates the map image shown in FIG. 15 to the map image shown in FIG.

[0125] Specifically, the map image shown in FIG. 16 includes a microphone icon MI61, a circumference CI61, a display object A61, and text images TI61a, TI61b, and TI61c.

[0126] Text image TI61a is a utterance by the first speaker, and corresponds to the utterance content with the oldest utterance date and time among text images TI61a, TI61b, and TI61c displayed in Fig. 16. Text image TI61a is arranged along a straight line extending from the display position of microphone icon MI61 toward the (estimated) direction of the first speaker, similar to Fig. 15. However, controller 30 moves the display position of text image TI61a in a direction away from the display position of microphone icon M61, compared to the map image shown in Fig. 15.

[0127] Text image TI61b is a speech by the first speaker, and corresponds to the second most recent speech content among text images TI61a, TI61b, and TI61c displayed in FIG. 16. Text image TI61b is arranged along a straight line extending from the display position of microphone icon MI61 toward the (estimated) direction of the first speaker, as in FIG. 15. However, controller 30 moves the display position of text image TI61b in a direction away from the display position of microphone icon MI61, compared to the map image shown in FIG. 15. As a result, text image TI61b is arranged at a position closer to the display position of microphone icon MI61 than text image TI61a corresponding to an older speech date and time, and farther from the display position of microphone icon MI61 than text image TI61c corresponding to a newer speech date and time.

[0128] Text image TI61c is a utterance by the first speaker, and corresponds to the utterance content with the most recent utterance date and time among text images TI61a, TI61b, and TI61c displayed in Fig. 16. Text image TI61c is arranged along a straight line extending from the display position of microphone icon MI61 toward the (estimated) direction of the first speaker. However, text image TI61c is arranged in a position closer to the display position of microphone icon MI61 than text images TI61a and TI61b corresponding to older utterance dates and times.

[0129] 16, the controller 30 does not place on the map image a text image TI62 corresponding to an older utterance position than the text image TI61a, and does not place a display object A62 on the map image, which allows the viewer of the map image to easily focus on the content and speaker of the most recent utterance.

[0130] In this way, the controller 30 generates a map image by arranging texts corresponding to voices uttered by the same speaker along the (estimated) direction of the speaker relative to the multi-microphone device 50 so that the texts are arranged in ascending order of the corresponding generation date and time, moving away from the origin of the map coordinate system (for example, the display position of the microphone icon MI61). This allows a viewer of the map image to intuitively recognize the association between the speaker and the utterance content, and to grasp the chronological order of the utterances based on the distance between the display position of the text corresponding to the utterance and the origin of the map coordinate system. Note that in the examples of Figs. 15 and 16, each text image is displayed rotated in a direction corresponding to the direction of the sound source, but this is not limiting, and each text image may be upright regardless of the direction of the sound source.

[0131] (6-3) Variation 3 Modification 3 will now be described. Modification 3 is an example in which a map image is generated for each of a plurality of multi-microphone devices installed in different locations. Fig. 17 is a diagram showing an example of a map image in Modification 3.

[0132] While a conversation is taking place between multiple participants in different locations (for example, different conference rooms, different business establishments, or different companies), the controller 30 generates a map image for each location and displays it on the display 11 of the display device 10. Each map image corresponds to an overhead view of the sound source (speaker) environment around the multi-microphone device 50 installed in the respective location, and text based on the voice uttered by the speaker is placed at a position based on the direction of the speaker relative to each multi-microphone device 50. The controller 30 updates the map image in response to the utterances of the participants. As a result, the map image serves as a UI for visually grasping the content of the most recent conversation at each location (particularly, who is saying what at which location) in real time.

[0133] 17, the controller 30 displays a map image MP71 of a first location and a map image MP72 of a second location, for example, side-by-side on one screen on the display 11 of the display device 10. Note that instead of arranging the map images MP71 and MP72 on one screen, the controller 30 may display only one of the map images MP71 and MP72 selected by the user on the display 11 of the display device 10.

[0134] In this way, the controller 30 generates a map image for each of the multiple multi-microphone devices 50 installed in different locations. As a result, even when multiple participants in different locations hold a remote conference, for example, a viewer of the map image can intuitively recognize the association between the location, the speaker, and the content of the speech. In particular, a participant in a first location has a harder time accurately grasping who is speaking in the second location than a participant in a second location, but by viewing the map image of the second location, the speaker in the second location can be easily identified. In other words, the decrease in the sense of realism caused by a remote conference can be compensated for.

[0135] (7) Other modifications The storage device 31 may be connected to the controller 30 via a network.

[0136] Each step of the above information processing can be executed by any of the display device 10, the controller 30, and the multi-microphone device 50. For example, the controller 30 may acquire a multi-channel audio signal generated by the multi-microphone device 50, estimate an arrival direction (S151), and extract the audio signal (S152).

[0137] In the above description, an example has been shown in which the display device 10 and the controller 30 are independent devices. However, the display device 10 and the controller 30 may be integrated together. For example, the display device 10 and the controller 30 can be implemented as one tablet terminal or personal computer. In addition, the multi-microphone device 50 and the display device 10 or the controller 30 may be integrated together. In addition, for example, the controller 30 may exist in a cloud server.

[0138] In the above description, an example was given in which the display device 10 is an electronic device that can easily share display contents with multiple users, such as a tablet terminal, a personal computer, a smartphone, or a display device for a conference. However, the display device 10 may be configured to be wearable on a human head. For example, the display device 10 may be a glass-type display device, a head-mounted display, a wearable device, or smart glasses. The display device 10 may be an optical see-through type glass-type display device, but the form of the display device 10 is not limited thereto. For example, the display device 10 may be a video see-through type glass-type display device. That is, the display device 10 may be equipped with a camera. The display device 10 may display a composite image obtained by combining a text image generated based on voice recognition with an image captured by a camera on the display 11. The captured image is an image captured in the front direction of the user and may include an image of the speaker. In addition, the display device 10 may perform AR (Augmented Reality) display by combining a text image generated based on voice recognition with an image captured by a camera, for example, in a smartphone, a personal computer, or a tablet terminal.

[0139] Furthermore, a plurality of display devices 10 may be connected to one controller 30. In this case, for example, the layout of the map image (for example, the correspondence between the microphone coordinate system and the map coordinate system) and the translation language information may be configured to be changeable for each display device 10.

[0140] There is no limitation on the implementation method of the display 11 as long as it can present an image to the user. The display 11 can be implemented, for example, by the following implementation methods. - HOE (Holographic optical element) or DOE (Diffractive optical element) using optical elements (for example, light guide plates) LCD display Retina projection display LED (Light Emitting Diode) display - Organic EL (Electro Luminescence) display Laser display A display that uses optical elements (such as lenses, mirrors, diffraction gratings, liquid crystals, MEMS mirrors, and HOEs) to guide light emitted from a light emitter. In particular, when a retinal projection display is used, even people with weak eyesight can easily observe the image, and therefore people who suffer from both hearing loss and weak eyesight can more easily recognize the direction from which speech sounds are coming.

[0141] Only a part (for example, the upper half) of the map image described above may be displayed on the display 11. This allows the visibility of text images and the like to be maintained even if the display area of ​​the display 11 is small. The part of the map image displayed on the display 11 may be switched in response to a user instruction or automatically.

[0142] In the above embodiment, the example in which the user's instructions are input from the input device of the controller 30 has been described, but the present invention is not limited to this. The user's instructions may also be input from an operation unit included in the display device 10.

[0143] In the voice extraction process by the multi-microphone device 50, as long as a voice signal corresponding to a specific speaker can be extracted, the method for achieving this is not important. The multi-microphone device 50 may extract a voice signal by, for example, the following method. Frost Beamformer Adaptive filter beamforming (for example, the generalized sidelobe canceller) - Audio extraction methods other than beamforming (e.g. frequency filters or machine learning)

[0144] In the present embodiment or each of the modified examples, an example has been shown in which text (images) related to comments made by a plurality of participants are arranged on a map image. The controller 30 may obtain text posted by chat participants in a chat associated with a conversation, and arrange the text (images) on the map image. The controller 30 may also arrange poster icons representing chat participants on the map image, similar to the sound source icons. This makes it easier for conversation participants to recognize the contents posted by the chat participants. In this case, the display position of the text posted by the chat participants (hereinafter referred to as "posted text") or the poster icon can be determined by various techniques. As a first example, the controller 30 may display the poster icon or posted text, for example, outside the circumference CI31 or CI61, to distinguish it from the sound source icon or text related to the remark. As a second example, when the controller 30 detects that a chat participant is the same person as any speaker, the controller 30 may aggregate the remarks and posted contents of the same person by displaying the posted text by the speaker according to the same rules as the text related to the remark by the speaker. As a third example, the controller 30 may determine the direction of the chat participant relative to the multi-microphone device 50 in response to a user instruction, and arrange the poster icon or posted text (for example, arrange it on the circumference CI31) based on the determined direction. That is, the controller 30 may move the display position of the poster icon or posted text in the map image in response to a user instruction. Thereby, even if the chat participant does not make any remarks and the direction of the chat participant relative to the multi-microphone device 50 cannot be estimated, the display position of the poster icon or posted text can be optimized (for example, displayed in the same way as the sound source icon and text image of the speaker).

[0145] In the first modification, an example was described in which minutes are generated and the user can edit the remarks placed in the minutes. The user may not only modify the remarks themselves, but also add supplementary explanations regarding the remarks. This makes it possible to prevent the meaning of the remarks from being misunderstood or misinterpreted by viewers of the minutes.

[0146] In the first modification, an example is shown in which minutes are generated in which text indicating remarks made in a conversation by a plurality of participants is arranged in chronological order. The controller 30 may acquire text posted by chat participants in a chat associated with the conversation, and generate minutes based on the text. In this case, the controller 30 generates minutes by arranging the posted text or the text indicating the remarks in chronological order of the posting date and time or the remarks date and time. For example, the posted text and the text indicating the remarks may be arranged in chronological order in the same window. This makes it easier for participants in the conversation to recognize the posted contents of the chat participants, and prevents them from overlooking the posted contents of the chat participants when looking back on the flow of the discussion.

[0147] In the second modification, an example has been shown in which text images corresponding to the three most recent utterances are arranged on a map image. However, the number of text images arranged on a map image may be two or less, or may be four or more. The number of text images arranged on a map image may be fixed, or may be variable depending on various conditions (e.g., the size of the map image, the number of characters included in the utterances), etc. The text images arranged on a map image may be determined based on whether the time elapsed from the utterance date and time corresponding to the text image is within a threshold value.

[0148] It is possible to combine the map image described in this embodiment with the map image described in Modification 2. As an example, in the map image described in Modification 2, instead of or in addition to the display objects A61 and A62 indicating the (estimated) direction of the speaker relative to the multi-microphone device 50, the sound source icon described in this embodiment may be displayed.

[0149] In the third modification, an example is shown in which a map image for two locations is generated. However, the controller 30 may generate a map image for three or more locations. In addition, the first and third modifications may be combined. As an example, the controller 30 may generate minutes by arranging the contents of remarks made by participants at a plurality of locations in chronological order. In this case, the controller 30 may consolidate the remarks of each participant into the same minutes, regardless of where the participant is located.

[0150] Although the embodiment of the present invention has been described in detail above, the scope of the present invention is not limited to the above embodiment. Furthermore, the above embodiment can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above embodiment and the modified examples can be combined. [Explanation of symbols]

[0151] 1: Information processing system 10: Display device 11: Display 30: Controller 31:Storage device 32 : Processor 33: Input / Output Interface 34: Communication interface 50: Multi-microphone device

Claims

1. means for obtaining information indicating a direction of a sound source relative to at least one multi-microphone device; means for acquiring information about the content of the sound emitted from the sound source and collected by the multi-microphone device; A means for acquiring posted text information posted by a user; means for generating a map image in which information about the content of the voice and the posted text information are arranged at positions corresponding to the direction of the sound source of the voice relative to the multi-microphone device; a means for generating minutes based on information about the content of the voice and the posted text information; means for displaying the map image on a display unit of a display device; An information processing device comprising:

2. means for identifying a sound source present around the multi-microphone device based on information indicating the direction of the sound source; and means for setting sound source information relating to the identified sound source in response to a user instruction. The information processing device according to claim 1 .

3. the setting means sets at least one of a name of the identified sound source, a language used by the sound source, or a target language of machine translation applied to information about the content of the sound emitted from the sound source; The information processing device according to claim 2 .

4. When a target language for machine translation is set by the setting means, the means for generating minutes includes a machine translation of information about the content of the speech in the target language in the minutes. The information processing device according to claim 3 .

5. The display means displays the minutes together with the map image on a display unit of the display device.

4. The information processing device according to claim 1.

6. the means for generating minutes generates the minutes by arranging the information related to the content of the acquired voice and the posted text information in chronological order.

7. The information processing device according to claim 1.

7. The system further includes means for editing the text arranged in the minutes in response to a user instruction.

7. The information processing device according to claim 5.

8. the map image generating means includes a microphone icon representing the multi-microphone device and a sound source icon representing the sound source, and generates the map image so that the sound source icon is positioned on a circumference centered on the microphone icon at a position corresponding to the direction of the sound source corresponding to the sound source icon relative to the multi-microphone device.

8. The information processing device according to claim 1.

9. the means for generating a map image generates the map image so as to highlight an icon representing the sound source or information relating to the content of the sound while the sound source is emitting the sound. The information processing device according to claim 8 .

10. the means for generating a map image generates the map image so as to include a poster icon representing the user and text posted by the chat participant.

10. The information processing device according to claim 8 or claim 9.

11. The method further includes a means for moving a display position of the poster icon on the map image in response to a user instruction. The information processing device according to claim 10.

12. the means for generating a map image rotates the display positions of the sound source icon and the information relating to the content of the sound around the display position of the microphone icon so that one of the sound source icons is positioned in a predetermined direction on the map image.

12. The information processing device according to claim 8.

13. the means for generating the map image arranges a plurality of texts relating to voices emitted from the same sound source on the map image in order of the oldest corresponding utterance date and time, so that the texts are arranged farther from the center of the coordinate system of the map image; 13. The information processing device according to claim 1.

14. the at least one multi-microphone device includes a first multi-microphone device and a second multi-microphone device installed in different locations; the means for acquiring information indicating a direction of a sound source acquires information indicating a direction of a sound source relative to the first multi-microphone device and information indicating a direction of a sound source relative to the second multi-microphone device; the means for acquiring information about the content of the audio acquires a first text about a first audio collected by the first multi-microphone device and a second text about a second audio collected by the second multi-microphone device; the map image generating means generates a first map image in which the first text is arranged at a position corresponding to a direction of a sound source of a first voice corresponding to the first text relative to the first multi-microphone device, and a second map image in which the second text is arranged at a position corresponding to a direction of a sound source of a second voice corresponding to the second text relative to the second multi-microphone device; the means for displaying the map image displays the first map image and the second map image on the display unit of the display device.

14. The information processing device according to claim 1.

15. obtaining information indicative of a direction of a sound source relative to at least one multi-microphone device; acquiring information about the content of the sound emitted from the sound source and collected by the multi-microphone device; acquiring posted text information posted by a user; generating a map image in which information about the content of the audio and the posted text information are arranged at positions according to the direction of a sound source of the audio relative to the multi-microphone device; generating minutes based on information about the content of the voice and the posted text information; displaying the map image on a display unit of a display device; An information processing method comprising:

16. A program for causing a computer to realize each means of the information processing device according to any one of claims 1 to 14.

Images