Information processing device, information processing method, and program

Abstract

An information processing device 1 according to one embodiment of the present invention comprises: an acquisition unit 121 that acquires audio data; an identification unit 123 that identifies a feature amount of a section generated by dividing the audio data along a time axis; and an extraction unit 124 that compares the feature amount of the section of the audio data with feature amounts which are stored in advance in a storage unit and which are feature amounts of each of a plurality of sections generated by dividing search target data indicating at least one of audio, text, or an image produced in the past, and that extracts, from among the plurality of sections of the search target data, a section of the search target data for which similarity to the section of the audio data is a threshold value or higher.

Description

Information processing device, information processing method, and program

[0001] The present invention relates to an information processing device, an information processing method, and a program for searching stored data.

[0002] Patent document 1 discloses a method for obtaining text information by performing image recognition processing or voice recognition processing on multimedia data including audio and images, automatically adding the text information that meets predetermined conditions to the multimedia data as meta information, and making it possible to search for desired multimedia data using the meta information.

[0003] JP 2008-176748 A

[0004] In recent years, it has become possible to store large amounts of data, such as voice data, document data, and image data, in storage devices, and thus there is a demand for systems that allow users to easily search for desired data from such large amounts of data. For example, a user may want to search for past data related to the participants or content of a specific conference from among the data stored in a storage device. The method described in Patent Literature 1 requires the user to specify meta-information to be used in the data search, which makes it time-consuming to search for past data related to a specific conference.

[0005] The present invention has been made in consideration of these points, and aims to make it easier to search for past data related to a specific conference.

[0006] An information processing device of a first aspect of the present invention has an acquisition unit that acquires audio data, an identification unit that identifies features of sections generated by dividing the audio data along a time axis, and an extraction unit that compares the features of the sections of the audio data with features pre-stored in a memory unit, and with features of each of multiple sections generated by dividing search target data indicating at least one of previously created audio, document, or image, to extract sections of the search target data from among the multiple sections of the search target data whose similarity with the sections of the audio data is greater than or equal to a threshold.

[0007] The identification unit may identify features of the section of the audio data indicating speech uttered in an ongoing conference, and the information processing device may further have an output unit that outputs section information indicating the features of the section extracted by the extraction unit while the ongoing conference is taking place.

[0008] The identification unit may identify features of a section specified by a user in the audio data, and the information processing device may further have an output unit that outputs section information indicating the features of the section extracted by the extraction unit to an information terminal used by the user.

[0009] The identification unit may identify features of a section of the audio data that indicate at least one of a person, organization, or content of a conversation specified by a user, and the information processing device may further have an output unit that outputs section information indicating the features of the section extracted by the extraction unit to an information terminal used by the user.

[0010] The output unit may output the section information indicating at least one of the content of the search target data in the section of the search target data, the person associated with the search target data, and the organization to which the person associated with the search target data belongs.

[0011] The identification unit may identify as the feature at least one of the following in a section of the audio data: the content of the audio data, the person associated with the audio data, information corresponding to the amount of speech of the person associated with the audio data, and information corresponding to speaker transitions that occurred among multiple people associated with the audio data.

[0012] The extraction unit may extract a search target section group including a plurality of sections of the search target data, the similarity to the section of the audio data being equal to or greater than a threshold value.

[0013] The extraction unit may extract the search target segments having a similarity to a search source segment group including a plurality of segments of the audio data that is equal to or greater than a threshold value.

[0014] The group of search target intervals may be generated by classifying one or more intervals of the search target data that are associated with the same person or organization from among the intervals of the plurality of search target data.

[0015] The information processing device may further have a generation unit that generates multiple sections of the search target data by dividing the search target data based on the amount of speech of each of the one or more people who uttered the voice indicated by the search target data.

[0016] The information processing device may further have a generation unit that generates multiple sections of the search target data by dividing the search target data based on the similarity in the time series of speaker transitions that occurred among multiple people who uttered the voice indicated by the search target data.

[0017] The section of the search target data indicating a document may be generated by dividing the search target data based on the chapter structure of the document or the similarity of the content of the document.

[0018] The section of the search target data indicating a moving image may be generated by dividing the search target data based on the similarity of the time series of people appearing in the moving image.

[0019] An information processing method of a second aspect of the present invention includes the steps of: acquiring audio data; identifying features of sections generated by dividing the audio data along a time axis; and extracting, from among the multiple sections of the search target data, sections of the search target data whose similarity to the sections of the audio data is equal to or greater than a threshold value, by comparing the features of the sections of the audio data with features pre-stored in a memory unit, which features are generated by dividing search target data indicating at least one of previously created audio, documents, or images.

[0020] A third aspect of the program of the present invention causes a processor to execute the steps of acquiring audio data, identifying features of sections generated by dividing the audio data along a time axis, and extracting sections of the search target data from among the multiple sections of the search target data whose similarity to the sections of the audio data is equal to or greater than a threshold by comparing the features of the sections of the audio data with features pre-stored in a memory unit and generated by dividing search target data indicating at least one of previously created audio, documents, or images.

[0021] The present invention provides an advantage in that it is possible to easily search for past data related to a specific conference.

[0022] FIG. 1 is a schematic diagram of an information processing system. FIG. 2 is a block diagram of an information processing system. FIG. 3 is a schematic diagram illustrating a method by which a generation unit generates intervals. FIG. 4 is a schematic diagram illustrating a method by which a generation unit generates intervals in search target data that is audio data. FIG. 5 is a schematic diagram illustrating a method by which a generation unit generates intervals in search target data that is document data or image data. FIG. 6 is a schematic diagram illustrating a method by which a generation unit generates interval groups. FIG. 7 is a schematic diagram illustrating feature amounts of intervals in search target data. FIG. 8 is a schematic diagram illustrating exemplary search conditions specified by a user. FIG. 9 is a schematic diagram illustrating a method by which an extraction unit compares feature amounts between search source data and search target data. FIG. 10 is a schematic diagram illustrating a method by which an extraction unit compares feature amounts between different types of data. FIG. 11 is a schematic diagram illustrating a method by which an extraction unit calculates similarity between an interval or interval group and an interval group. FIG. 12 is a schematic diagram of an information terminal displaying interval information. FIG. 13 is a schematic diagram of an information terminal displaying interval information on a map. FIG. 14 is a flowchart of an exemplary information processing method executed by an information processing device.

[0023] <First embodiment> [Outline of information processing system S] Fig. 1 is a schematic diagram of an information processing system S according to this embodiment. The information processing system S includes an information processing device 1, a sound collection device 2, and an information terminal 3. The number of sound collection devices 2 and information terminals 3 included in the information processing system S is not limited. The information processing system S may also include other devices such as servers and terminals.

[0024] The information processing device 1 is a computer that searches for a section that is at least a part of the search target data using a section that is at least a part of the search source data. The search source data and the search target data are each at least one of voice data, text data, and image data. The information processing device 1 outputs information indicating the search results to an information terminal 3. The information processing device 1 is connected to a sound collection device 2 and the information terminal 3 via a network such as a local area network or the Internet, either wired or wirelessly.

[0025] The sound collection device 2 is a device that captures sounds emitted in a conference attended by multiple participants. The sound collection device 2 includes, for example, a microphone array including a sound collection unit such as multiple microphones arranged in different orientations. The microphone array includes, for example, multiple microphones (e.g., eight microphones) arranged at equal intervals on the same circumference in a horizontal plane relative to the ground. By using such a microphone array, the information processing device 1 can identify which participant is the speaker (sound source) based on the sounds emitted by the multiple participants surrounding the sound collection device 2.

[0026] The sound collection device 2 acquires sound from each of the multiple sound collection sections and records it internally as sound for each channel corresponding to each sound collection section. The sound collection device 2 then transmits the acquired sound from the multiple channels as audio data to the information processing device 1. The sound collection device 2 may transmit the acquired sound sequentially, or may transmit a predetermined amount or duration of sound. The sound collection device 2 may also transmit all of the sound from the start to the end of acquisition all at once.

[0027] The information terminal 3 is a computer that outputs information, such as a smartphone, a tablet terminal, or a personal computer. The user of the information terminal 3 is, for example, at least some of the multiple participants in the conference. The user of the information terminal 3 may also be a person different from the participants in the conference. The information terminal 3 has, for example, a display unit such as a liquid crystal display. The information terminal 3 displays the information received from the information processing device 1 on the display unit.

[0028] Furthermore, the information terminal 3 may have a sound collection unit such as a microphone and function as the sound collection device 2. In this case, the information terminal 3 used by the conference participants acquires the voices uttered in the conference using the sound collection unit and transmits voice data indicating the acquired voices to the information processing device 1. For example, in the case of a conference held over a network (e.g., a web conference), the information terminal 3, which is the sound collection device 2, is associated with one of the multiple participants.

[0029] The following is an overview of the processing executed by the information processing system S according to this embodiment. The information processing device 1 generates multiple sections by dividing search target data, which is at least one of previously created voice data, document data, and image data.

[0030] The information processing device 1 identifies feature amounts for each of the multiple sections of the generated search target data. The feature amounts include, for example, the content of the search target data, the person associated with the search target data, the time period associated with the search target data, etc. The information processing device 1 stores information indicating the feature amounts identified for each of the multiple sections of the search target data in a storage unit in advance.

[0031] When a user desires to search for search target data, the information processing device 1 acquires voice data as search source data. The information processing device 1 may acquire voice data sequentially from the sound collection device 2, or may acquire voice data that meets conditions specified by the user from a storage unit.

[0032] The information processing device 1 generates a plurality of sections by dividing the audio data, which is the search source data, along the time axis, and identifies the feature amount of one of the generated sections to be used for search.

[0033] The information processing device 1 compares the feature amounts of the sections of the search source data (the sections used for the search) with the feature amounts of each of the multiple sections of the search target data pre-stored in the storage unit, and extracts, from the multiple sections of the search target data, sections of the search target data whose similarity to the sections of the search source data is equal to or greater than a threshold. The information processing device 1 outputs section information indicating the features of the extracted sections to the information terminal 3.

[0034] In this way, the information processing system S compares the feature values ​​of a section of the voice data, which is the search source data, with the feature values ​​of each of multiple sections of the voice data, document data, or image data, which is the search target data, to extract a section of the search target data that is similar to a specific section of the search source data. This makes it easier for the information processing system S to use a section of the voice data of a specific conference, for example, to search for past search target data related to that conference.

[0035] [Configuration of Information Processing System S] FIG. 2 is a block diagram of the information processing system S according to this embodiment. In FIG. 2, arrows indicate the main data flows, and data flows other than those shown in FIG. 2 may also exist. In FIG. 2, each block indicates a functional configuration rather than a hardware (device) configuration. Therefore, the blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between blocks via any means, such as a data bus, a network, or a portable storage medium.

[0036] The information processing device 1 includes a storage unit 11 and a control unit 12. The information processing device 1 may be configured by connecting two or more physically separated devices via a wired or wireless connection. The information processing device 1 may also be configured by a cloud, which is a collection of computer resources.

[0037] The storage unit 11 is a storage medium including a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk drive, etc. The storage unit 11 stores in advance a program to be executed by the control unit 12. The storage unit 11 also stores information indicating the feature amounts of each of a plurality of sections generated by dividing the search target data. The storage unit 11 may be provided outside the information processing device 1, in which case data may be exchanged between the storage unit 11 and the control unit 12 via a network.

[0038] The control unit 12 includes an acquisition unit 121, a generation unit 122, an identification unit 123, an extraction unit 124, and an output unit 125. The control unit 12 is a processor such as a CPU (Central Processing Unit), and functions as the acquisition unit 121, the generation unit 122, the identification unit 123, the extraction unit 124, and the output unit 125 by executing a program stored in the storage unit 11. At least some of the functions of the control unit 12 may be performed by an electric circuit. Furthermore, at least some of the functions of the control unit 12 may be realized by the control unit 12 executing a program executed via a network. The functions of the acquisition unit 121, the generation unit 122, the identification unit 123, the extraction unit 124, and the output unit 125 will be described later.

[0039] [Database Construction] The information processing device 1 constructs a database of search target data in the storage unit 11. The acquisition unit 121 acquires search target data that is at least one of previously created voice data, document data, and image data. The voice data is, for example, data representing voice spoken in a conference attended by multiple participants. The document data is, for example, data representing documents such as papers, articles, and minutes of meetings. The image data is, for example, data representing still images or moving images depicting one or more subjects.

[0040] The acquisition unit 121 may acquire, as the search destination data, for example, voice data, document data, or image data received via a network from a device (sound collection device 2 or another device) different from the information processing device 1. The acquisition unit 121 may acquire, as the search destination data, voice data, document data, or image data stored in advance in the storage unit 11, for example.

[0041] The generation unit 122 generates multiple sections of the search target data by dividing the search target data acquired by the acquisition unit 121. Fig. 3 is a schematic diagram for explaining a method for generating sections by the generation unit 122. Each of the multiple sections of the search target data represents voice that is at least a part of the voice data, text (character strings) that is at least a part of the document data, or an image that is at least a part of the image data.

[0042] 4 is a schematic diagram illustrating a method for generating sections of search target data that is speech data by the generation unit 122. The example in Fig. 4 shows a method for generating multiple sections of speech data by dividing the speech data based on the similarity in the time series of speaker transitions that occurred among multiple people who uttered the speech represented by the speech data.

[0043] The generation unit 122 estimates the direction of the sound source relative to the sound collection device 2 at each time interval (e.g., every 10 to 100 milliseconds) by performing sound source localization based on the multiple channel sounds indicated by the audio data received from the sound collection device 2. The generation unit 122 determines which participant has spoken (uttered) at each predetermined time interval (e.g., every 10 to 100 milliseconds) in the conference based on the estimated direction of the sound source. The generation unit 122 identifies the continuous period from when one participant starts speaking to when he or she finishes speaking as the speech period.

[0044] In the case of a conference held over a network (such as a web conference), the generation unit 122 estimates, for example, a participant associated with the information terminal 3, which is the sound collection device 2 that is the sender of the acquired audio, as the sound source, and identifies the speech period of each of the multiple participants based on the acquired audio and the estimated sound source.

[0045] The generation unit 122 acquires the amount of speech (also referred to as speech volume) of each of the multiple participants in the conference in a time series based on the identified speech period. For example, the generation unit 122 divides the conference into first frames of a predetermined window width w1 (e.g., 30 seconds). The first frames are shifted by a predetermined shift width s1 (e.g., 10 seconds) that is shorter than the window width w1, and adjacent first frames partially overlap each other in time series.

[0046] The generation unit 122 calculates the speech volume for each first frame by dividing the length of the speech period (total speech time) of each participant in the first frame by the window width w1. The generation unit 122 calculates the speech volume for each first frame from the start time to the end time of the conference for each of the multiple participants.

[0047] The upper diagram in Fig. 4 shows graph G of the amount of speech of multiple participants over time. Graph G represents the amount of speech of multiple participants as a stacked graph. The horizontal axis of graph G represents time, and the vertical axis represents the amount of speech. Different patterns are displayed in the areas of graph G depending on the multiple participants.

[0048] The generation unit 122 identifies a loudest speaker who has the largest amount of speech among the multiple participants for each first frame based on the acquired speech volume, and outputs an array of loudest speakers from the first first frame to the last first frame of the conference.

[0049] The middle diagram in Fig. 4 shows the time-series largest talker as band B. Band B of the time-series largest talker is generated based on graph G of the time-series amount of speech, and the horizontal axis corresponds to the time on graph G. Band B of the largest talker shows different patterns for each of the multiple participants, which correspond to the patterns of the areas on graph G.

[0050] The generation unit 122 detects speaker transitions that have occurred among multiple participants based on the identified change in the loudest speaker for each first frame. The generation unit 122, for example, divides the conference into second frames with a predetermined window width w2. The window width w2 of the second frames is longer than the window width w1 of the first frames. That is, the second frames include multiple first frames. The window width w2 may be defined as a predetermined multiple of the window width w1 (e.g., 100 times the window width w1) or as a predetermined time (e.g., 3000 seconds).

[0051] The second frames are shifted by a predetermined shift width s2 that is shorter than the window width w2, and two adjacent second frames overlap each other in time series. The shift width s2 may be defined as a predetermined multiple of the window width w1 (e.g., five times the window width w1) or as a predetermined time (e.g., 150 seconds).

[0052] The generation unit 122 detects a transition from a first participant who is the loudest speaker in one first frame to a second participant who is the loudest speaker in a first frame following the first frame when the first participant is different from the second participant who is the loudest speaker in the first frame following the first frame. The generation unit 122 repeatedly detects transitions for one second frame from the first first frame to the last first frame of the second frame, and generates a transition matrix indicating the number of transitions detected for each combination of participants (i.e., a combination of a first participant and a second participant). If the number of participants is D, the transition matrix is ​​a D×D matrix. Furthermore, the generation unit 122 repeatedly generates the transition matrix from the first second frame to the last second frame of the discussion.

[0053] The lower diagram of Fig. 4 shows an exemplary transition matrix M. In the example of Fig. 4, there are three participants, U1, U2, and U3, and multiple transition matrices M are generated based on the band B of the largest speaker in the time series. The rows of the transition matrix M indicate the participants from which the transition occurs, and the columns indicate the participants to which the transition occurs.

[0054] The generation unit 122 then divides the speech data based on the similarity of the time series of speaker transitions that occur among the multiple participants, thereby generating multiple sections of the speech data. The generation unit 122 generates the multiple sections of the speech data by, for example, clustering the transition matrix in time series using a known clustering method. The generation unit 122 then determines a pattern of transitions between participants (such as the names of participants included in major transitions) for each of the multiple sections of the speech data.

[0055] As another method, the generation unit 122 may generate multiple sections of the audio data by dividing the audio data based on the similarity in the time series of the speech volume of one or more people who uttered the voices represented by the audio data. For example, based on the time series of the speech volumes of each of the multiple participants, the generation unit 122 generates a predetermined number of frames or more as one section when the ratio of the speech volume of one or more participants to the total speech volume is equal to or greater than a threshold, based on the time series of the speech volumes of each of the multiple participants. Then, the generation unit 122 determines the loudest speaker (e.g., the name of one or more participants whose speech volume is equal to or greater than a threshold) for each of the multiple sections of the audio data.

[0056] As another method, the generation unit 122 may generate multiple sections of the voice data based on an operation by a user on the information terminal 3. For example, the generation unit 122 receives an operation on the information terminal 3 that specifies one or more timings (e.g., X minutes after the start of a conference) at which to divide the sections in the voice data, and generates multiple sections of the voice data by dividing the voice data at the specified one or more timings.

[0057] The generating unit 122 is not limited to the specific method shown here, and may generate multiple sections of audio data using other methods.

[0058] 5A is a schematic diagram illustrating a method for generating sections of search target data, which is document data, by the generation unit 122. The example in FIG. 5A illustrates a method for generating multiple sections of document data by dividing the document data based on the similarity of the content of the document indicated by the document data.

[0059] The generation unit 122 sequentially extracts multiple texts included in a document. The generation unit 122 calculates the semantic similarity between a pair of two consecutive texts from the multiple texts included in the document. The semantic similarity between the two texts is, for example, the cosine similarity between two vectors representing lists of words that appear in the two texts. The generation unit 122 generates multiple sections of the document data by, for example, dividing the document data between two texts whose similarity is equal to or less than a threshold.

[0060] Alternatively, the generation unit 122 may generate multiple sections of the document data based on the chapter structure of the document indicated by the document data. For example, the generation unit 122 identifies the chapter structure of the document based on metadata indicating information about the document added to the document data or based on headings included in the document, and generates multiple sections of the document data by dividing the document data into the identified chapters or sections.

[0061] The generating unit 122 is not limited to the specific method shown here, and may generate multiple sections of document data using other methods.

[0062] Fig. 5(b) is a schematic diagram for explaining a method for generating sections of search target data that is image data by the generation unit 122. The example of Fig. 5(b) shows a method for generating multiple sections of image data by dividing the image data based on the similarity of the time series of people appearing in the video when the image data indicates a video.

[0063] The generation unit 122 extracts people included in each frame of the video using a known person recognition method, and generates multiple sections of the image data by dividing the image data at the timing when one or more people included in a frame of the video changes.

[0064] When the image represented by the image data is a still image, the generating unit 122 may generate one image as one section. The generating unit 122 is not limited to the specific method shown here, and may generate multiple sections of the image data using other methods.

[0065] The generation unit 122 may generate a group of sections (search target section group) including multiple sections of the search target data. Figures 6(a) and 6(b) are schematic diagrams for explaining a method for the generation unit 122 to generate a group of sections. The generation unit 122 generates a group of sections from the multiple sections of the acquired search target data based on at least one of the person or organization associated with the search target data and the time period associated with the search target data.

[0066] 6A, the generation unit 122 generates a group of sections by classifying one or more sections of search target data that are associated with the same person and time period from among the sections of multiple search target data. The generation unit 122 identifies the person and time period associated with the search target data using a method similar to the method used by the identification unit 123 (described later) when identifying features.

[0067] 6(b), the generation unit 122 generates a group of sections by classifying one or more sections of search target data associated with the same organization from among the sections of multiple search target data. The generation unit 122 identifies the organization (department, company, etc.) associated with the search target data based on, for example, affiliation information of the person associated with the search target data, which is stored in advance in the storage unit 11.

[0068] The generation unit 122 is not limited to the specific criteria shown here, and may generate a segment group based on other information indicating the commonality of multiple segments of the search target data. The generation unit 122 stores segment group information indicating one or more segments included in the generated segment group in the storage unit 11.

[0069] The identification unit 123 identifies the feature amount of each of the multiple sections generated by dividing the search target data by the generation unit 122. Figures 7(a), 7(b), and 7(c) are schematic diagrams for explaining the feature amounts of the sections of the search target data.

[0070] 7A shows the feature amounts of a section of the audio data. The identification unit 123 identifies, as the feature amount of the section of the audio data that is the search target data, at least one of the following: the content of the audio data, the person associated with the audio data, information corresponding to the amount of speech of the person associated with the audio data (e.g., the loudest speaker), and information corresponding to speaker transitions that have occurred among multiple people associated with the audio data (e.g., the type of speaker transition).

[0071] The identification unit 123, for example, converts the speech of a section of the speech data into text using a known speech recognition method, and inputs the converted text into a known learning model for summarizing text, thereby identifying the output summary as the content of the speech data.

[0072] The identification unit 123, for example, acquires conference information that has been previously associated with a section of the voice data from the storage unit 11, and identifies one or more participants indicated by the acquired conference information as people associated with the voice data. Alternatively, the identification unit 123 may identify people who have uttered the voice in the section of the voice data using a known voice identification method, and identify the identified one or more people as people associated with the voice data.

[0073] For example, the identification unit 123 identifies one or more participants who are the largest speakers determined by the generation unit 122 as information corresponding to the speech volume of a person associated with the audio data (speech volume type in FIG. 7A). For example, the identification unit 123 identifies a pattern of transitions between participants determined by the generation unit 122 as information corresponding to speaker transitions that occurred among a plurality of people associated with the audio data (transition type in FIG. 7A).

[0074] Furthermore, the identification unit 123 may identify, for example, at least one of the time associated with the voice data, the position associated with the voice data, the type of human emotion associated with the voice data, and the type of content of the voice data in the section of the voice data that is the search target data as a feature of the section.

[0075] The identification unit 123, for example, acquires the start date and time and the end date and time of the section from the timestamp recorded in the section of the audio data, and identifies the acquired start date and time and end date and time as the time associated with the audio data. The identification unit 123, for example, identifies the location (coordinates, address, etc.) where the audio, indicated by metadata indicating information related to the audio added to the audio data, as the location associated with the audio data.

[0076] The identification unit 123 estimates the emotions of the participants who uttered the voices in the section of the voice data using, for example, a known emotion estimation method, and identifies the proportion of each estimated emotion (joy, anger, sadness, happiness, etc.) as the type of human emotion associated with the voice data. The identification unit 123 estimates the type of content (question, suggestion, doubt, etc.) based on, for example, words included in the content of the above-mentioned voice data, and identifies the estimated type of content as the type of content of the voice data.

[0077] 7B shows the feature amounts of a section of document data. The identification unit 123 identifies, for example, in a section of document data that is search target data, at least one of the following as the feature amount of the section: the content of the document data, the person associated with the document data, the time period associated with the document data, the type of emotion of the person associated with the document data, and the type of content of the document data.

[0078] The identification unit 123, for example, inputs the text of a section of the document data into a known learning model for summarizing text, and outputs a summary as the content of the document data. The identification unit 123, for example, identifies one or more authors indicated by metadata indicating information about the document added to the document data as people associated with the document data.

[0079] The identification unit 123, for example, identifies at least one of the creation date and publication date of the document indicated by metadata indicating information about the document added to the document data as the time period associated with the document data. The identification unit 123, for example, estimates the emotions of the author who wrote the text of the section of the document data using a known emotion estimation method, and identifies the proportion of each estimated emotion (joy, anger, sadness, happiness, etc.) as the type of human emotion associated with the document data.

[0080] The identification unit 123, for example, estimates the type of content (question, suggestion, doubt, etc.) based on words contained in the content of the above-mentioned document data, and identifies the estimated type of content as the type of content of the document data.

[0081] 7C shows the feature amounts of a section of image data. The identification unit 123 identifies, for example, in the section of image data that is the search target data, at least one of the following as the feature amount of the section: the content of the image data, the person associated with the image data, the time period associated with the image data, the location associated with the image data, and the type of emotion of the person associated with the image data.

[0082] The identification unit 123 may, for example, identify a caption (explanatory text) indicated by metadata indicating information about the image added to the image data as the content of the image data. Furthermore, the identification unit 123 may also identify an explanatory text output by inputting the image data into a known learning model for generating an explanatory text for the image data as the content of the image data. The identification unit 123 may, for example, identify one or more subjects appearing in a section of the image data using a known image recognition method, and identify the identified one or more subjects as people associated with the image data.

[0083] The identifying unit 123 identifies, for example, the date and time when the image indicated by the metadata indicating information about the image added to the image data was captured as the time associated with the image data. The identifying unit 123 identifies, for example, the location (coordinates, address, etc.) when the image indicated by the metadata indicating information about the image added to the image data was captured as the location associated with the image data.

[0084] The identification unit 123, for example, estimates the emotions of the subject appearing in a section of the image data using a known emotion estimation method, and identifies the proportion of each estimated emotion (joy, anger, sadness, happiness, etc.) as the type of human emotion associated with the image data.

[0085] The identification unit 123 is not limited to the specific feature amounts shown in Figures 7(a), 7(b), and 7(c), and may identify other feature amounts that indicate the characteristics of the section of the search target data. The identification unit 123 stores the feature amount identified for the section in the storage unit 11 in association with identification information for identifying the section.

[0086] The identification unit 123 may anonymize the feature quantities of the section of the search target data in accordance with a predetermined conversion rule stored in advance in the storage unit 11. The conversion rule is, for example, information indicating an anonymous ID (Identification) corresponding to a person's name. The identification unit 123 anonymizes the feature quantities by converting the person's name included in the feature quantities into an anonymous ID in accordance with the conversion rule, and stores the anonymized feature quantities in the storage unit 11.

[0087] [Executing a Search] When a user desires to search for search target data, the acquisition unit 121 accepts designation of search conditions from the user. Figures 8(a), 8(b), and 8(c) are schematic diagrams for explaining exemplary search conditions designated by the user.

[0088] 8A illustrates an example in which a user specifies voice data as a search criterion. The acquisition unit 121, for example, accepts voice data of an ongoing conference in which the user is participating as a search criterion. In this case, the acquisition unit 121 accepts, as a search criterion, a predetermined section (hereinafter, a current section) generated by dividing the voice data of the ongoing conference along a time axis in response to a user's operation to specify the ongoing conference on the information terminal 3. The acquisition unit 121, for example, accepts, as a search criterion, a section in the voice data of the ongoing conference from a time a predetermined time before the current time to the current time. The acquisition unit 121 may also accept, as a search criterion, a section in the voice data of the ongoing conference from the start time of the conference to the current time.

[0089] For example, the acquiring unit 121 may accept, as a search condition, a section specified by a user in audio data of a past conference (hereinafter, a designated section). In this case, in response to a user's operation on the information terminal 3 to specify a section (such as a start date and time and an end date and time) generated by dividing the audio data of the past conference data along a time axis, the acquiring unit 121 accepts the specified section as a search condition.

[0090] 8B shows an example in which a user specifies a person, an organization, or a content as a search condition. The acquisition unit 121 accepts, for example, the person, organization, or content specified by the user as a search condition. In this case, in response to the user performing an operation on the information terminal 3 to specify a person, organization, or content, the acquisition unit 121 accepts the specified person, organization, or content as a search condition.

[0091] 8C shows an example in which a user specifies a future conference as a search condition. The acquisition unit 121 accepts, for example, a conference scheduled for the future as a search condition. In this case, the acquisition unit 121 acquires conference information related to the future conference pre-stored in the storage unit 11 in response to a user's operation to specify a future conference on the information terminal 3. The acquisition unit 121 accepts, as search conditions, people (e.g., participants) and content (e.g., agenda) associated with the future conference indicated by the acquired conference information.

[0092] When the acquisition unit 121 receives the current section or the designated section as a search condition (in the example of FIG. 8( a)), it acquires, as search source data, audio data received from the sound collection device 2 (for example, audio data indicating audio uttered in an ongoing conference) or audio data stored in the storage unit 11 (for example, audio data indicating audio uttered in a past conference). The identification unit 123 identifies, from the search source data, the feature amounts of the current section or the designated section, which are the search conditions.

[0093] The identification unit 123 identifies the feature quantities of a section of the voice data that is the search source data in a manner similar to the manner in which the feature quantities of a section of the search target data are identified. For example, the identification unit 123 identifies, as the feature quantities of the section of the search source data, at least one of the following: the content of the voice data, the person associated with the voice data, information corresponding to the amount of speech of the person associated with the voice data, and information corresponding to speaker transitions that occurred among multiple people associated with the voice data. Furthermore, the identification unit 123 may identify, as the feature quantities of the section of the search source data, at least one of the time period associated with the voice data, the position associated with the voice data, the type of emotion of the person associated with the voice data, and the type of content of the voice data.

[0094] When the acquisition unit 121 receives a person, an organization, or a content as a search condition (as in the examples of FIGS. 8(b) and 8(c)), the identification unit 123 may identify, as a feature quantity of a section of the search source data, one or more feature quantities that include at least one of the person, organization, or content that are the search conditions, from among the feature quantities of multiple sections of the search target data stored in the storage unit 11. This enables the information processing system S to reduce the calculation cost for newly identifying feature quantities from the search source data.

[0095] When the features of the search target data have been anonymized, the identification unit 123 anonymizes the features of the sections of the search source data in accordance with predetermined conversion rules pre-stored in the storage unit 11, in the same way as the features of the sections of the search target data. The identification unit 123 anonymizes the features by converting people's names included in the features into anonymous IDs in accordance with the conversion rules, and the extraction unit 124, described below, extracts sections of the search target data that are similar to the sections of the search target data using the anonymized features. This allows the information processing system S to extract sections of the search target data that are similar to the sections of the search target data while protecting personal information such as people's names.

[0096] The extraction unit 124 extracts a section of the search target data that is similar to the section of the search source data by comparing the feature amount of the section of the voice data that is the search source data identified by the identification unit 123 with the feature amount of each of multiple sections of at least one of the voice data, document data, or image data that is the search target data stored in the storage unit 11. The section of the search target data extracted by the extraction unit 124 is, for example, a section of the search target data that has a similarity to the section of the search source data that is equal to or greater than a threshold value, out of the multiple sections of the search target data.

[0097] 9 is a schematic diagram illustrating a method by which the extraction unit 124 compares feature quantities between the search source data and the search target data. FIG. 9 shows feature quantities for a section of speech data, which is the search source data, and feature quantities for a section of speech data, which is the search target data. The extraction unit 124 calculates an individual similarity between the search source data and the search target data for each of a plurality of feature quantity items. The individual similarity is defined so that the more similar the feature quantity items are, the larger the value, and the less similar the value is, the smaller the value.

[0098] For example, the extraction unit 124 calculates the cosine similarity of a vector indicating the content or a vector indicating the emotion type between the search source data and the search destination data as the individual similarity for each of the content and the emotion type. For example, the extraction unit 124 calculates the inverse of the distance (Euclidean distance, etc.) between the search source data and the search destination data as the individual similarity for each of the position.

[0099] The extraction unit 124 calculates the degree of match (number or percentage of matches, etc.) between the search source data and the search target data as individual similarities for each of, for example, participants, information corresponding to speech volume (speech volume type in Figure 9), information corresponding to speaker transitions (transition type in Figure 9), and content type.

[0100] The individual similarities shown in Fig. 9 are merely examples, and may be calculated according to other rules. In the example of Fig. 9, the extraction unit 124 does not calculate individual similarities for the periods (start date and time, end date and time), but the proximity of the periods may be reflected in the similarity. In this case, the extraction unit 124 calculates the individual similarities so that, for example, the smaller the difference in periods, the larger the value, and the larger the difference in periods, the smaller the value.

[0101] While Fig. 9 shows an example in which both the search source data and the search target data are voice data, there are cases in which the search source data is voice data and the search target data is document data or image data. Fig. 10 is a schematic diagram for explaining a method in which the extraction unit 124 compares feature quantities between different types of data.

[0102] The storage unit 11 pre-stores correspondence information indicating the correspondence between a plurality of feature items for sections of voice data, document data, and image data. Fig. 10 shows mutually comparable items among the plurality of feature items for sections of voice data, document data, and image data, arranged side by side. Fig. 10 also shows "none" when there is no comparable item in the feature items of document data or image data for an item of voice data.

[0103] Based on the correspondence information pre-stored in the memory unit 11, the extraction unit 124 calculates individual similarities between each of the multiple feature items of the audio data section and the feature items of the document data or image data section corresponding to that item.

[0104] The extraction unit 124 calculates the similarity as a weighted sum of the individual similarities for each of the calculated feature items. The weight of each feature item is stored in advance in, for example, the storage unit 11. The weight of each feature item may also be specified by the user on the information terminal 3. The similarity is defined so that the more similar the feature items of the section of the search source data and the section of the search target data are, the larger the value is, and the more dissimilar they are, the smaller the value is.

[0105] The extraction unit 124 is not limited to the specific method shown here, and may calculate the similarity between the section of the search source data and the section of the search target data using other methods.

[0106] The extraction unit 124 may calculate the similarity between a section of the search source data and a section group including multiple sections of the search target data. Fig. 11(a) is a schematic diagram illustrating a method in which the extraction unit 124 calculates the similarity between a section of the search source data and a section group of the search target data.

[0107] In this case, the extraction unit 124 identifies a search target section group including multiple sections of the search target data based on the section group information stored in the storage unit 11. The extraction unit 124 calculates the similarity between pairs of sections of the search source data and each of the multiple sections of the search target data included in the search target section group. The extraction unit 124 calculates a statistical value (average, median, maximum, minimum, etc.) of the similarity for each of the multiple pairs as the similarity between the section of the search source data and the search target section group. This allows the information processing system S to extract a section group formed by grouping multiple sections of the search target data that are similar to one section of the search source data.

[0108] The extraction unit 124 may calculate the similarity between a group of sections including multiple sections of the search source data and a group of sections including multiple sections of the search target data. Fig. 11(b) is a schematic diagram illustrating a method in which the extraction unit 124 calculates the similarity between a group of sections of the search source data and a group of sections of the search target data.

[0109] In this case, the extraction unit 124 identifies a search target section group including multiple sections of the search target data based on the section group information stored in the storage unit 11. The generation unit 122 also generates a search source section group including multiple sections of the search source data using a method similar to the method for generating the search target section group. The extraction unit 124 calculates the similarity between each of the multiple sections of the search source data included in the search source section group and each of the multiple sections of the search target data included in the search target section group.

[0110] The extraction unit 124 calculates the similarity between the search source interval group and the search target interval group by calculating the statistical value (average, median, maximum, minimum, etc.) of the similarity for each of the multiple pairs. This allows the information processing system S to extract interval groups formed by grouping multiple intervals in the search target data that are similar to interval groups formed by grouping multiple intervals in the search source data.

[0111] The extraction unit 124 extracts, from among a plurality of sections or section groups of the search target data, a section or section group of the search target data whose similarity to the calculated section or section group of the audio data is equal to or greater than a threshold. The similarity threshold is, for example, stored in advance in the storage unit 11. The similarity threshold may also be specified by the user on the information terminal 3.

[0112] When audio data of an ongoing conference is specified as a search condition, the information processing system S can extract sections of past search target data related to the ongoing conference without the user having to specify specific content or people, making it easier for the user to search for past data related to the ongoing conference.

[0113] When a person, organization, or content is specified as a search condition, the information processing system S can extract sections of past search target data that are similar to sections of audio data that include the specified person, organization, or content, and can therefore provide the user with search results that include more diverse data than when searching based solely on keyword matches.

[0114] When a future meeting is specified as a search condition, it is possible to extract sections of past search target data that are similar to sections of audio data containing people and content associated with the future meeting, thereby providing the user with past data related to the future meeting in advance.

[0115] [Output of Search Results] The output unit 125 outputs section information indicating the characteristics of the section or section group extracted by the extraction unit 124 to the information terminal 3 used by the user who specified the search conditions. The output unit 125 generates section information for a section based on the feature amount of the section. The output unit 125 also generates section information for a section group based on the feature amount of any section included in the section group.

[0116] The section information indicates, for example, at least one of the content of the search target data, the person associated with the search target data, and the organization to which the person associated with the search target data belongs, which are contained in the feature of the section or section group extracted by the extraction unit 124. The section information may also indicate other information contained in the feature of the section or section group extracted by the extraction unit 124. The output unit 125 transmits the generated section information to the information terminal 3. The information terminal 3 displays the section information received from the information processing device 1 on the display unit.

[0117] 12 is a schematic diagram of the information terminal 3 displaying section information. The information terminal 3 displays section information 31 for a section and section information 32 for a group of sections based on the section information received from the information processing device 1. This enables the information processing system S to provide the user with past data related to a specific conference, making it easier for the user to utilize the past data.

[0118] When audio data of an ongoing conference is used as a search condition, the output unit 125 may output section information while the ongoing conference is taking place. This allows the information processing system S to provide the user with past data related to the conference while the conference is taking place, making it easier for the user to proceed with the conference.

[0119] When a future conference is used as a search condition, the output unit 125 may output the section information during the start of the future conference. This allows the information processing system S to provide the user with past data related to the conference before the conference takes place, making it easier for the user to check information about the conference in advance.

[0120] The output unit 125 may display, as section information, the locations (locations where audio or images were captured) included in the features of the section or group of sections extracted by the extraction unit 124 on a map displayed on the information terminal 3.

[0121] 13 is a schematic diagram of an information terminal 3 displaying section information on a map. Based on the section information received from the information processing device 1, the information terminal 3 displays a position image 34 superimposed on the map 33, which indicates a position included in the feature amount of a section or a group of sections. The position image 34 is, for example, an image of a predetermined color or pattern, and is placed at a position on the map 33 corresponding to the position where the sound or image was captured. This allows the information processing system S to visualize the position associated with the search target data corresponding to the section or group of sections extracted by the extraction unit 124.

[0122] 14 is a flowchart of an exemplary information processing method executed by the information processing device 1 according to this embodiment. Feature quantities for each of a plurality of sections generated by dividing search target data indicating at least one of previously created speech, document, or image are stored in advance in the storage unit 11 of the information processing device 1. Furthermore, in the subsequent flow, a section group including a plurality of sections may be used instead of or in addition to a section.

[0123] When the user desires to search for search target data, the acquiring unit 121 receives a search condition from the user (S11). The acquiring unit 121 receives, as the search condition, for example, one of a section in the voice data and a person, an organization, or content.

[0124] The acquisition unit 121 acquires, as search source data, audio data received from the sound collection device 2 or audio data stored in the storage unit 11 (S12). The identification unit 123 identifies feature quantities of sections generated by dividing the search source data along the time axis and that satisfy the search conditions (S13). The identification unit 123 may newly identify feature quantities from the search source data. Alternatively, the identification unit 123 may identify, as feature quantities of the search source data, feature quantities that satisfy the search conditions, from feature quantities of sections of search target data pre-stored in the storage unit 11.

[0125] The extraction unit 124 calculates (S14) the similarity between the feature amount of the section of the voice data that is the search source data identified by the identification unit 123 and the feature amount of each of the multiple sections of the search target data stored in the storage unit 11. The similarity is defined so that the value is larger the more similar the feature amount of the section of the search source data and the feature amount of the section of the search target data are, and the value is smaller the more dissimilar they are.

[0126] The extraction unit 124 extracts, from among the multiple sections of the search target data, sections of the search target data whose similarity to the calculated section of the audio data is equal to or greater than a threshold (S15). The output unit 125 outputs section information indicating the characteristics of the sections extracted by the extraction unit 124 to the information terminal 3 used by the user who specified the search conditions (S16).

[0127] [Effects of this embodiment] According to the information processing system S of this embodiment, the information processing device 1 compares the feature amounts of a section of voice data, which is the search source data, with the feature amounts of each of multiple sections of voice data, document data, or image data, which is the search target data, to extract a section of search target data that is similar to a specific section of the search source data. This allows the information processing system S to easily search for past search target data related to a conference, using a section of voice data of the conference that meets search conditions specified by the user.

[0128] In the above embodiment, an example has been described in which audio data representing speech spoken by a person in a meeting is used as the search source data and the search destination data, but audio data representing speech spoken by a non-human animal may also be used. In this case, the storage unit 11 stores in advance feature quantities identified for each of a plurality of sections of audio data recorded in a specific area, for example.

[0129] The extraction unit 124 calculates the similarity between the feature amount of a section of audio data representing a sound made by a specific animal, which is the search source data, and the feature amount of each of the multiple sections of the search target data stored in the storage unit 11. The extraction unit 124 extracts, from the multiple sections of the search target data, sections of the search target data whose similarity to the calculated section of audio data is equal to or greater than a threshold value.

[0130] This allows the information processing system S to provide the user with information on sections of search target data that contain sounds made by a specific animal, thereby assisting the user in searching for times and locations where a specific animal's cries are most prevalent, or in searching for other animals that make sounds together with a specific animal.

[0131] <Second Embodiment> While the information processing system S according to the first embodiment extracts a section of audio data, text data, or image data that is similar to a section of audio data, the information processing system S according to this embodiment extracts a section of audio data that is similar to a section of audio data, text data, or image data. Below, differences from the first embodiment will be mainly described.

[0132] The information processing device 1 builds a database of search target data in the storage unit 11 in the same manner as in the first embodiment. The search target data used in this embodiment includes at least audio data, but does not necessarily include document data or image data. As a result, the storage unit 11 stores in advance feature quantities for each of a plurality of sections generated by dividing the audio data, which is the search target data, along the time axis.

[0133] The acquisition unit 121 receives search criteria from a user when the user desires to search for search target data. The search source data used in this embodiment is data indicating at least one of speech, documents, and images. The acquisition unit 121 receives, for example, one of a section in the search source data and a person, organization, or content as a search criterion. The identification unit 123 identifies, in the same manner as in the first embodiment, the feature amount of the section generated by dividing the search source data acquired by the acquisition unit 121.

[0134] The extraction unit 124 extracts a section of the search target data that is similar to a section of the search source data by comparing the feature amount of at least one section of the voice data, document data, or image data that is the search source data identified by the identification unit 123 with the feature amount of each of the multiple sections of the voice data that is the search target data stored in the storage unit 11. The section of the search target data extracted by the extraction unit 124 is, for example, a section of the search target data that has a similarity to the section of the search source data that is equal to or greater than a threshold value among the multiple sections of the search target data. The extraction unit 124 may also extract a group of sections that includes multiple sections of the search target data.

[0135] The output unit 125 outputs the section information indicating the characteristics of the section or section group extracted by the extraction unit 124 to the information terminal 3 used by the user who specified the search conditions.

[0136] According to the information processing system S of this embodiment, the information processing device 1 compares the feature values ​​of a section of the voice data, document data, or image data that is the search source data with the feature values ​​of each of multiple sections of the voice data that is the search target data, thereby extracting sections of the search target data that are similar to a specific section of the search source data. This allows the information processing system S to easily search for voice data of past meetings, etc. that is related to the data, using sections of the voice, document, or image data that meet search conditions specified by the user.

[0137] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating in any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

[0138] The processor of the information processing device 1 executes each step (process) included in the information processing method shown in Fig. 14. That is, the processor of the information processing device 1 reads a program for executing the information processing method shown in Fig. 14 from the storage unit 11, and executes the read program to execute the information processing method shown in Fig. 14. In the information processing method shown in Fig. 14, some steps may be omitted, the order of steps may be changed, or multiple steps may be executed in parallel.

[0139] S Information Processing System 1 Information Processing Device 2 Sound Collection Device 3 Information Terminal 11 Storage Unit 12 Control Unit 121 Acquisition Unit 122 Generation Unit 123 Identification Unit 124 Extraction Unit 125 Output Unit

Claims

1. an acquisition unit that acquires voice data; an identification unit that identifies feature quantities of sections generated by dividing the audio data along a time axis; an extraction unit that compares feature amounts of the sections of the voice data with feature amounts pre-stored in a storage unit, the feature amounts being for each of a plurality of sections generated by dividing search target data indicating at least one of documents or images created in the past, to extract sections of the search target data whose similarity to the sections of the voice data is equal to or greater than a threshold value from among the plurality of sections of the search target data; An information processing device having the above.

2. The feature quantity of the section of the search target data is at least one of the content of the document data that is the search target data, the person associated with the document data, the time associated with the document data, the type of emotion of the person associated with the document data, and the type of content of the document data. The information processing device according to claim 1 .

3. The feature quantity of the section of the search target data is at least one of the content of the image data that is the search target data, the person associated with the image data, the time associated with the image data, the location associated with the image data, and the type of emotion of the person associated with the image data. The information processing device according to claim 1 .

4. the identifying unit identifies a feature of the section of the voice data indicating a voice uttered in an ongoing conference; an output unit that outputs, while the ongoing conference is being held, section information indicating the characteristics of the section extracted by the extraction unit; The information processing device according to claim 1 .

5. the specifying unit specifies a feature amount of a section specified by a user in the audio data; an output unit that outputs section information indicating the characteristics of the section extracted by the extraction unit to an information terminal used by the user; The information processing device according to claim 1 .

6. the identifying unit identifies a feature of a section of the voice data that indicates at least one of a person, an organization, or a content of a conversation designated by a user; an output unit that outputs section information indicating the characteristics of the section extracted by the extraction unit to an information terminal used by the user; The information processing device according to claim 1 .

7. the output unit outputs the section information indicating at least one of the content of the search target data, a person associated with the search target data, and an organization to which the person associated with the search target data belongs, in the section of the search target data. The information processing device according to claim 4 .

8. the identification unit identifies, as the feature, at least one of information corresponding to the content of the voice data, a person associated with the voice data, an amount of speech of the person associated with the voice data, and information corresponding to speaker transitions occurring among a plurality of people associated with the voice data, in a section of the voice data; The information processing device according to claim 1 .

9. the extraction unit extracts a search target section group including a plurality of sections of the search target data, the similarity to the section of the audio data being equal to or greater than a threshold value; The information processing device according to claim 1 .

10. the extraction unit extracts the search target segments, each of which has a similarity to a search source segment that includes a plurality of segments of the audio data equal to or greater than a threshold value; The information processing device according to claim 9 .

11. The search target interval group is generated by classifying one or more sections of the search target data associated with the same person or organization from among the sections of the plurality of search target data. The information processing device according to claim 9 .

12. The section of the search target data indicating the document is generated by dividing the search target data based on the similarity of the chapter structure of the document or the content of the document. The information processing device according to claim 1 .

13. The section of the search target data indicating a video is generated by dividing the search target data based on the similarity of the time series of people appearing in the video. The information processing device according to claim 1 .

14. The processor executes acquiring audio data; Identifying feature quantities of sections generated by dividing the audio data along a time axis; a step of comparing a feature amount of the voice data section with a feature amount of each of a plurality of sections generated by dividing search target data representing at least one of documents or images created in the past, thereby extracting, from the plurality of sections of the search target data, sections of the search target data whose similarity with the voice data section is equal to or greater than a threshold value; An information processing method comprising:

15. The processor acquiring audio data; Identifying feature quantities of sections generated by dividing the audio data along a time axis; a step of comparing a feature amount of the voice data section with a feature amount of each of a plurality of sections generated by dividing search target data representing at least one of documents or images created in the past, thereby extracting, from the plurality of sections of the search target data, sections of the search target data whose similarity with the voice data section is equal to or greater than a threshold value; A program that executes.