Information processing device, information processing method, and information processing program

Abstract

To provide an information processor capable of generating a useful summary for understanding the current situation, an information processing method, and an information processing program.SOLUTION: An information processor includes: an acquisition unit that acquires audio data; a recognition unit that recognizes the audio data acquired by the acquisition unit and converts the recognized result into text data; a summary unit that generates summary data summarizing the content of the text data each time the text data converted by the recognition unit is obtained; and a transmission unit that transmits the summary data generated by the summary unit to the users.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and an information processing program that perform text summarization processing. [Background technology]

[0002] In recent years, with the increase in web conferences and video distribution, technologies for recording and summarizing these contents have been widely used.

[0003] For example, there is known a technology that extracts the start of a specific agenda item in a meeting and automatically generates a summary for each agenda item (for example, Patent Document 1 and Patent Document 2). There is also known a technology that generates a summary not only from audio but also from text obtained by performing character recognition on an image (for example, Patent Document 3). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-105741 [Patent Document 2] Japanese Patent Application Publication No. 2019-139572 [Patent Document 3] Japanese Patent Application Publication No. 2019-101754 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional techniques, problems may arise in terms of actually utilizing summaries. For example, when summaries are generated for each interval, a new summary may not be generated until the end of a certain interval is detected.

[0006] A person joining a web conference midway or a person taking over a conversation at a call center or the like often wants to be able to immediately refer to the content of the conversation that took place up until that point, rather than the content divided into sections. However, if a summary is generated by dividing the content into sections, a person joining midway or the like may not be able to understand the content of the previous conversation or the content immediately before joining until the summary is generated.

[0007] Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that are capable of generating a summary that is useful for understanding the current situation. [Means for solving the problem]

[0008] In order to solve the above problems, the information processing device according to the present disclosure includes an acquisition unit that acquires voice data, a recognition unit that recognizes the voice data acquired by the acquisition unit and converts the recognition result into text data, a summarization unit that generates summary data summarizing the content of the text data converted by the recognition unit each time the text data is obtained, and a transmission unit that transmits the summary data generated by the summarization unit to a user. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an overview of information processing according to an embodiment. [Figure 2] FIG. 2 is a block diagram schematically illustrating the flow of information processing according to the embodiment. [Figure 3] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a voice data storage unit according to the embodiment. [Figure 5] FIG. 4 is a diagram illustrating an example of a generation result storage unit according to the embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a summary setting storage unit according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating a conversion process according to the embodiment. [Figure 8]10 is a flowchart illustrating a procedure of information processing according to the embodiment. [Figure 9] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system that realizes information processing according to an embodiment. [Figure 10] FIG. 1 is a diagram illustrating an application example of information processing according to an embodiment. [Figure 11] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] (1. Embodiment) [1-1. Example of information processing according to the embodiment] Fig. 1 is a diagram schematically illustrating a flow of information processing according to an embodiment. The information processing according to the embodiment is executed by an information processing system 1 shown in Fig. 1. The information processing system 1 includes an information processing device 100, which is an example of an information processing device according to the present disclosure, and a user terminal 200. The devices included in the information processing system 1 are capable of transmitting and receiving data to and from each other via wireless communication or the like.

[0012] The information processing device 100 is an information processing device that recognizes any voice data, converts it into text, and generates a summary of the text. For example, the information processing device 100 recognizes voice from a web conference held on the web and summarizes the content. For example, the information processing device 100 is a cloud server or the like.

[0013] The user terminal 200 is a terminal device used by the user 10. For example, the user terminal 200 is a smartphone, a tablet terminal, etc. The user 10 is, for example, one of the participants in a web conference for which the information processing device 100 generates a summary.

[0014] In events involving speech, such as web conferences, speech recognition is sometimes used to create meeting minutes. Furthermore, speech-recognized content may be summarized by agenda item and made available to participants. However, such summaries are typically processed on a certain amount of text data, such as by agenda item. Generating summaries in this way, in chunks that follow the flow of the meeting, is useful in that it makes it easier to understand the content of each agenda item. However, for those who join the meeting midway through, there is a risk that they will not be able to access these summaries until a certain amount of time has passed, or that the information they want to know may not be included in the summaries.

[0015] In other words, when a participant joins a web conference midway, they need to instantly understand the content of the conversation that has taken place up until that point, rather than the content divided into sections. Current summarization processes have difficulty meeting this need.

[0016] Therefore, the information processing device 100 according to the present disclosure solves the above problem by performing the information processing described below. Specifically, the information processing device 100 according to the present disclosure recognizes the speech of a speaker and converts it into text data, and each time converted text data is obtained, generates summary data that summarizes the content of the text data. This allows the information processing device 100 to provide summary data that is useful for understanding the current status of a web conference, etc. Below, an overview of information processing by the information processing system 1 will be described using FIG. 1.

[0017] 1, user 10 is one of the participants in a web conference. User 10 accesses information processing device 100, which is a server that generates a summary of the conference, via user terminal 200 (step S1). This allows user terminal 200 to receive the summary of the web conference generated by information processing device 100 in real time.

[0018] The information processing device 100 acquires audio from an ongoing web conference. In this case, the information processing device 100 may acquire audio from the terminals of the participants accessing the web conference, or may acquire audio from a server hosting the web conference.

[0019] The information processing device 100 recognizes and converts the acquired speech into text using a known real-time speech-to-text conversion function. A conversation example 15 shown in Fig. 1 shows the contents of the meeting that were converted into text by the information processing device 100 and the time when the speech was made.

[0020] Then, every time the information processing device 100 obtains text data such as the conversation example 15, it summarizes the content obtained so far and generates summarized text data (hereinafter referred to as "summary data") (step S2). For example, the information processing device 100 generates the summary data by analyzing the text data, converting the data into features, and inputting the resulting data into a trained model. Note that the information processing device 100 may generate the summary data by using an API (Application Programming Interface) or the like provided by an arbitrary service, rather than performing all processing on its own device.

[0021] The information processing device 100 transmits the generated summary data to the user terminal 200 (step S3). The user terminal 200 displays the received summary data on its screen as appropriate. A display example 20 shown in Fig. 1 illustrates a situation in which the summary data generated by the information processing device 100 is displayed on the user terminal 200.

[0022] 1, the user terminal 200 receives summary data every minute and displays the received data. Note that the user terminal 200 does not receive summary data at predetermined time intervals, but can receive and display summary data at any timing, such as when requested by the user 10 or when the user 10 logs into the web conference. The user terminal 200 may also display newly received summary data while leaving past summary data intact, or may display only the most recent summary data.

[0023] According to the summary data included in display example 20, even if the topic of the web conference remains the same, the content of the discussion is continuously summarized sequentially, allowing user 10 to grasp the progress of the discussion in real time. That is, information processing device 100 generates a summary each time text data is obtained, without detecting specific intervals such as breaks in the discussion or when the speaker changes. Therefore, user 10 can immediately grasp the current state of the conference even if he or she joins the conference midway or temporarily leaves and returns from the conference.

[0024] The user 10 can also configure various settings related to summaries. For example, the user 10 can set the timing for receiving the summary (i.e., displaying it on the screen of the user terminal 200) and the length of the summary. Furthermore, if the user 10 is an administrator of a web conference, the user 10 can configure settings to adjust the length of the summary according to the attributes of the participants. For example, the user 10 can request the information processing device 100 to generate summary data with a relatively small number of characters for managers such as his or her superiors, so that the content can be understood in short sentences. Alternatively, the user 10 can request the information processing device 100 to generate summary data with a relatively large number of characters for people who need to understand the content in detail, such as practitioners.

[0025] Next, an overview of the flow of information processing executed by the information processing device 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram that schematically shows the flow of information processing according to the embodiment.

[0026] First, the information processing device 100 acquires an arbitrary media stream 30 (step S11). The media stream includes, for example, a video file, a live video, a web conference, and audio from radio. The media stream 30 is not limited to the example shown in FIG. 2 and may be in any format as long as it is a file that includes audio.

[0027] The information processing device 100 extracts audio information from the acquired media stream 30 (step S12). The audio information may include audio data converted into a predetermined format and meta-information (speaker information, recording information, media information, etc.) associated with the audio data. Furthermore, the information processing device 100 converts the extracted audio information into text by performing voice recognition (step S13).

[0028] The information processing device 100 stores the converted text data (step S14). Then, the information processing device 100 generates summary data at any timing (step S15). For example, the information processing device 100 may generate summary data every time it acquires a sentence of the text data, or may generate summary data when it receives a request from the user 10.

[0029] Thereafter, the information processing device 100 transmits the generated summary data to the user terminal 200, and causes the summary data to be displayed on the screen of the user terminal 200 (step S16).

[0030] 1 and 2, the information processing device 100 according to the embodiment acquires voice data, recognizes the acquired voice data, and converts it into text data. Furthermore, every time converted text data is acquired, the information processing device 100 generates summary data that summarizes the contents of the text data, and transmits the generated summary data to the user 10 (i.e., the user terminal 200).

[0031] In this way, the information processing device 100 generates summary data each time it obtains text data without extracting a specific section such as the agenda of a web conference, and provides the generated summary data to the user 10. This allows the user 10 to instantly grasp the current situation at the time of joining an event such as a web conference, where it is difficult to view the situation going back. In other words, the information processing device 100 can generate summary data that is useful for grasping the current situation.

[0032] [1-2. Configuration of the information processing device according to the embodiment] Next, a description will be given of the configuration of the information processing device 100 that executes information processing according to the embodiment. Fig. 3 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment.

[0033] 3, the information processing device 100 includes a communication unit 110, a storage unit 120, and a control unit 130. The information processing device 100 may also include an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator who manages the information processing device 100, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

[0034] The communication unit 110 is realized by, for example, a network interface controller or the like. The communication unit 110 is connected to a network N (for example, the Internet) by wire or wirelessly, and transmits and receives information to and from the user terminal 200 or the like via the network N. For example, the communication unit 110 may transmit and receive information using a communication standard or communication technology such as Wi-Fi (registered trademark), SIM (Subscriber Identity Module), or LPWA (Low Power Wide Area).

[0035] The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has an audio data storage unit 121, a generation result storage unit 122, and a summary setting storage unit 123.

[0036] The audio data storage unit 121 stores audio data that is the source of generating summary data. Fig. 4 illustrates an example of information stored in the audio data storage unit 121. Fig. 4 is a diagram illustrating an example of the audio data storage unit 121 according to the embodiment. In the example shown in Fig. 4, the audio data storage unit 121 has items such as "data ID," "acquisition date and time," and "audio data."

[0037] "Data ID" indicates identification information for identifying streaming data, etc., including audio data received by the information processing device 100. "Acquisition date and time" indicates the date and time when the data was acquired. "Audio data" indicates the content of the audio data included in the streaming data, etc. Note that the audio data may also include the recognition results obtained by converting the audio data into text. Also, in Figures 4 and 5, the information stored in the storage unit 120 may be conceptually described as "B01," but in reality, specific information such as audio data and text data is stored in the storage unit 120.

[0038] The generation result storage unit 122 stores information related to the summary data generated by the information processing device 100. Fig. 5 illustrates an example of information stored in the generation result storage unit 122. Fig. 5 is a diagram illustrating an example of the generation result storage unit 122 according to the embodiment. In the example shown in Fig. 5, the generation result storage unit 122 has items such as "summary ID," "original data," "generation date and time," and "text data."

[0039] "Summary ID" indicates identification information for identifying summary data. "Original data" indicates identification information for the data that is the source of the summary data. "Generation date and time" indicates the date and time the summary data was generated. "Text data" indicates the text that makes up the summary data.

[0040] The summary setting storage unit 123 stores information related to settings when the information processing device 100 generates summary data. Fig. 6 shows an example of information stored in the summary setting storage unit 123. Fig. 6 is a diagram showing an example of the summary setting storage unit 123 according to the embodiment. In the example shown in Fig. 6, the summary setting storage unit 123 has items such as "setting ID," "designated item," "designated content," and "setting content."

[0041] "Setting ID" indicates identification information that identifies the settings when generating summary data. "Specified item" indicates the items that are specified as settings. For example, "viewer attributes" specifies the attributes of the user who views the summary data. "Speaker attributes" specifies the attributes of the user who spoke the speech that is the source of generating the summary data. "Specified content" indicates the specific content that is set in the specified item. "Setting content" indicates the specific content that is set when generating the summary data.

[0042] For example, if the specified item is "viewer attribute" and "management position" is specified in that item, when generating summary data for a user with the attribute of "management position," the information processing device 100 generates summary data of "50 characters or less."

[0043] Furthermore, if the specified item is "speaker attribute" and "section staff" is specified in that item, the information processing device 100 generates summary data of "50 characters or less" when summarizing the content spoken by a user who has the attribute of "section staff."

[0044] Returning to Fig. 3, the explanation will be continued. The control unit 130 is realized by, for example, a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), or the like executing a program stored inside the information processing device 100 using a random access memory (RAM) or the like as a working area. The control unit 130 is also a controller, and is realized by, for example, an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0045] As shown in FIG. 3, the control unit 130 includes an acquisition unit 131, a recognition unit 132, a summary unit 133, and a transmission unit 134.

[0046] The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 connects to a server or the like that hosts the media stream 30 and acquires the audio data included in the media stream 30.

[0047] Furthermore, the acquisition unit 131 acquires various setting information related to summarization from the user 10, etc. For example, the acquisition unit 131 acquires setting information specifying the timing of summarization, setting information specifying the number of characters in the summary data, etc.

[0048] The acquisition unit 131 may also acquire attributes of the users 10 participating in a web conference or the like along with the setting information. For example, the acquisition unit 131 acquires information about the attributes of the users 10 who receive the summary data, via a user interface such as user registration. For example, the acquisition unit 131 acquires attributes such as whether the user 10 is a manager or a section employee in a company. Alternatively, the acquisition unit 131 acquires attributes such as whether the participants (users who may speak) are a manager or a section employee in a company from an administrator of the web conference or the like.

[0049] Furthermore, when images and the like are presented in a web conference, the acquisition unit 131 may acquire not only audio data but also image data. In this case, the subsequent processing units (the recognition unit 132 and the summarization unit 133) input the image data into a trained model to convert the image data into text data that describes the content of the image, and generate summary data that includes the converted text data.

[0050] The above processing will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining the conversion processing according to the embodiment.

[0051] 7, the information processing device 100 acquires image data 40. Then, the information processing device 100 inputs the image data 40 into a text conversion model 50 that has been pre-trained using the image data and text data including a description of the content of the image as training data.

[0052] Through this processing, the information processing device 100 obtains text data 60 indicating the content of the image corresponding to the image data 40. Specifically, the information processing device 100 obtains text data 60 saying "There are three red apples" from the image data 40 of an image showing three red apples.

[0053] In this way, the information processing device 100 can obtain text data from image data. That is, the information processing device 100 can obtain not only text data obtained by OCR processing of characters written on a whiteboard, but also the content of the image itself as text data. This allows the information processing device 100 to generate a summary that more accurately reflects the content of the web conference.

[0054] The recognition unit 132 recognizes the voice data acquired by the acquisition unit 131 and converts the recognition result into text data. For example, the recognition unit 132 converts the voice data into text data using a predetermined trained language processing model, an API that performs voice recognition processing, or the like.

[0055] Furthermore, as described above, the recognition unit 132 can also convert image data into text data that describes the content of the image by inputting the image data into a trained model.

[0056] Each time text data converted by the recognition unit 132 is obtained, the summarization unit 133 generates summary data that summarizes the contents of the text data.

[0057] The summarization unit 133 can also generate a summary at any timing. For example, when the user 10 participates in an event for which audio data is continuously acquired by the acquisition unit 131, the summarization unit 133 generates summary data (hereinafter referred to as "user summary data") that summarizes the content of text data corresponding to the audio data acquired up until the user 10 participated. Note that the event is not limited to a web conference or live video, but also broadly includes a predetermined media stream, a stream or file containing audio data, etc.

[0058] The user summary data is generated for each user 10 who participates in the event. Therefore, even if a user 10 joins the event midway, for example, the user 10 can immediately refer to the current situation, such as how the conference is progressing.

[0059] In addition, in an event where audio data is continuously acquired by the acquisition unit 131, the summarization unit 133 may generate summary data (hereinafter referred to as ``summary data for the second user'') that summarizes the content of the text data corresponding to the audio data acquired up to the time of the request by the user 10, at the time when the user 10 requests summary generation.

[0060] The second user summary data is generated instantly in response to a request from the user 10. Therefore, for example, if the user 10 temporarily leaves the event or wants to understand what has happened so far, the user 10 can immediately refer to the current situation.

[0061] Furthermore, if the summarization unit 133 has received advance settings regarding summarization, it generates summary data in accordance with the settings. For example, the summarization unit 133 generates summary data of a number of characters specified by rules linked to the user's attributes. Alternatively, the summarization unit 133 may generate summary data at specified intervals. Alternatively, the summarization unit 133 may generate summary data in accordance with various settings, such as summarizing the content up to that point when the speaker changes.

[0062] Furthermore, when the recognition unit 132 obtains text data from image data as described above, the summarization unit 133 generates summary data including text data that explains the content of the image.

[0063] The transmitting unit 134 transmits various types of information to the user terminal 200, etc. For example, the transmitting unit 134 transmits the summarized data generated by the summarizing unit 133 to the user 10.

[0064] The transmission unit 134 may transmit the summary data generated by the summarization unit 133 to the user 10 at predetermined time intervals. For example, the transmission unit 134 transmits the summary data to the user 10 at predetermined time intervals, such as every minute.

[0065] Alternatively, the transmission unit 134 may transmit the summary data generated by the summarization unit 133 to the user 10 at predetermined time intervals specified by the user 10. This allows the user 10 to view the summary data at any timing, at time intervals according to their own desires, for example, if they do not want to view summary data that has been updated too frequently.

[0066] Furthermore, when user summary data or second user summary data is generated for each user 10, the transmission unit 134 transmits the user summary data or second user summary data to the target user 10. For example, the transmission unit 134 transmits the user summary data or second user summary data to the user 10 at the timing when the summarization unit 133 generates the data. This allows the transmission unit 134 to transmit appropriate summary data to the user 10, which is a summary of the text data up to that point, in accordance with the timing when the user 10 wishes to view it.

[0067] [1-3. Information Processing Procedure According to the Embodiment] Next, a processing procedure according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the information processing procedure according to the embodiment.

[0068] 8, the information processing device 100 determines whether or not voice data has been received (step S21). If voice data has not been received (step S21; No), the information processing device 100 waits until voice data is received.

[0069] On the other hand, if voice data is received (step S21; Yes), the information processing device 100 converts the voice data into text (step S22). Note that if image data is received, the information processing device 100 also converts the image data into text.

[0070] Furthermore, the information processing device 100 determines whether settings for generating a summary have been accepted (step S23). If the settings for generating a summary have been accepted (step S23; Yes), the information processing device 100 reflects the settings, such as by limiting the number of characters in the summary data (step S24).

[0071] After the summarization settings are reflected, or if the summarization settings have not been accepted (step S23; No), the information processing device 100 summarizes the text data and generates summary data without making any particular settings (step S25).The information processing device 100 then transmits the summary results to the user 10 (step S26).

[0072] (2. Application Examples of the Embodiments) [2-1. Information Processing System Configuration] In the above embodiment, an example has been shown in which summarization processing is performed between the information processing device 100 according to the present disclosure and the user terminal 200. Here, the information processing according to the embodiment may be performed by a plurality of server devices or the like working together. This point will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the configuration of an information processing system 2 that realizes the information processing according to the embodiment. FIG. 9 shows an application example in which the information processing according to the embodiment is performed by the information processing system 2 having a plurality of server devices.

[0073] 9, for example, a user 10 uploads a media stream including predetermined audio data (step S31) via a user terminal 200. In FIG. 9, summary data is generated by a real-time summary generation module including multiple servers.

[0074] When the relay server 300 receives the media stream, it returns the URL of the page displaying the summary data to the user terminal 200 (step S32). The relay server 300 also sends the received media stream to the processing server 310 (step S33).

[0075] The processing server 310 functions as an audio stream conversion module. For example, the processing server 310 inputs audio data of a media stream into a conversion model, Speech2Text311, to obtain text data (step S34). The processing server 310 also inputs image data into a conversion model, Image2Text312, to obtain text data (step S35).

[0076] The processing server 310 sums up all the text data that has been processed up to this point and stores the text data in the storage 320 (step S36). At this time, the monitoring server 330 constantly monitors the updates to the storage 320 (step S37). If an update has been made to the storage 320, the monitoring server 330 notifies the summary server 340 of this fact (step S38).

[0077] Upon receiving the notification, summary server 340 acquires the data from storage 320 (step S39). Then, summary server 340 generates summary data (step S40). That is, summary server 340 functions as a summary generation module. Summary server 340 stores the generated summary data in storage 320 (step S41).

[0078] Relay Server 300 acquires the relay result when there is an update in Storage 320. Then, Relay Server 300 periodically communicates with User Terminal 200 and controls the display of the relay result (i.e., summary data) on User Terminal 200.

[0079] In this way, the information processing according to the present disclosure may be realized by the information processing system 2 in which a plurality of servers cooperate. That is, the information processing according to the present disclosure may be processing executed not only by the information processing device 100 but also by a plurality of devices working together.

[0080] [2-2.Example of application using GUI] An application example when the information processing according to the embodiment is actually used will be described with reference to Fig. 10. Fig. 10 is a diagram showing an application example of the information processing according to the embodiment.

[0081] In the example shown in Fig. 10, it is assumed that the user 10 watches a live video via the user terminal 200. In the example of Fig. 10, the user terminal 200 accesses the service server 400 and watches the live video.

[0082] First, the user terminal 200 starts up live video in a GUI (Graphical User Interface) module 410 provided by the service server 400 (step S51). The GUI module 410 selects a video for which a summary is to be generated, for example, in accordance with a selection by the user 10, and sends the selected video to the real-time summary generation module 420 (step S52). The real-time summary generation module 420 may have the same configuration as the real-time summary generation module shown in FIG. 9.

[0083] The real-time summary generation module 420 continues the process of generating the summary in real time, and sends a URL for obtaining the summary result to the GUI module 410 (step S54).

[0084] When the GUI module 410 receives the summary result from the real-time summary generation module 420, it reflects it in the GUI. That is, the GUI module 410 provides the summary result to the user 10 in real time in accordance with the progress of the live video (step S55).

[0085] With this configuration, the user 10 can view a summary of the content of the live video in real time while watching the live video, making it easy to understand the current situation even if the user leaves the live video temporarily, etc. The audio captured by the GUI module 410 is not limited to live video, etc., but may also be audio captured by the user terminal 200 using a microphone or predetermined software.

[0086] This configuration can also be applied to processes that summarize call content in real time at call centers, etc. This allows call center staff to summarize the main points of the conversation, improve their response capabilities, and ensure smooth handovers. This configuration also makes it easier for call center staff to create a record of the call content after the call.

[0087] [2-3. Modifications] The information processing according to the embodiment may be modified in various ways. For example, the information processing device 100 may identify speakers in a web conference or the like and generate summary data when a predetermined number of speakers are recognized.

[0088] Although the information processing according to the embodiment is performed by the information processing device 100 connected to the user terminal 200 via the network N, the user terminal 200 may perform the processing independently. In this case, the user terminal 200 has the same configuration as the information processing device 100 shown in FIG.

[0089] (3. Other embodiments) The processing according to the above-described embodiment may be implemented in various different forms other than the above embodiment.

[0090] For example, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. Furthermore, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0091] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0092] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content.

[0093] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0094] (4. Effects of the information processing device according to the present disclosure) As described above, the information processing device according to the present disclosure (information processing device 100 in the embodiment) has an acquisition unit (acquisition unit 131 in the embodiment), a recognition unit (recognition unit 132 in the embodiment), a summarization unit (summarization unit 133 in the embodiment), and a transmission unit (transmission unit 134 in the embodiment). The acquisition unit acquires voice data. The recognition unit recognizes the voice data acquired by the acquisition unit and converts the recognition result into text data. Each time text data converted by the recognition unit is obtained, the summarization unit generates summary data summarizing the content of the text data. The transmission unit transmits the summary data generated by the summarization unit to a user.

[0095] In this way, the information processing device according to the present disclosure generates summary data each time text data is obtained without extracting a specific section such as an agenda, and transmits the generated summary data to the user. This allows the information processing device to allow the user to instantly grasp the current status of a web conference, etc. In other words, the information processing device can generate summary data that is useful for grasping the current situation.

[0096] The transmitting unit transmits the summarized data generated by the summarizing unit to the user at predetermined time intervals.

[0097] In this way, the information processing device according to the present disclosure can assist the user in understanding the current content of the meeting by periodically sending summary data to the user at predetermined time intervals, rather than dividing the meeting by agenda or the like.

[0098] The transmitting unit also transmits the summarized data generated by the summarizing unit to the user at predetermined time intervals defined by the user.

[0099] In this way, the information processing device according to the present disclosure can transmit summary data at intervals defined by the user, thereby transmitting the amount of summary data that meets the user's request.

[0100] The summarizing unit generates user summary data that summarizes the content of the text data corresponding to the voice data acquired up to the time the user participates in an event for which voice data is continuously acquired by the acquiring unit, when the user participates in the event. The transmitting unit transmits the user summary data to the user when the summarizing unit generates the user summary data.

[0101] In this way, the information processing device according to the present disclosure can detect user participation (such as login) and generate a summary specifically for the user at that timing, thereby providing summary data optimized for the user to understand the current situation.

[0102] Furthermore, in an event in which voice data is continuously acquired by the acquisition unit, the summarization unit generates second user summary data that summarizes the content of the text data corresponding to the voice data acquired up to the user's request when the user requests summary generation, and the transmission unit transmits the second user summary data to the user when the summarization unit generates the second user summary data.

[0103] In this way, the information processing device according to the present disclosure can provide summary data that meets the user's needs by generating summary data at the timing requested by the user.

[0104] The acquisition unit acquires user attributes, and the summarization unit generates summary data having a number of characters specified by a rule associated with the user attributes.

[0105] In this way, the information processing device according to the present disclosure generates summary data according to the user's attributes, and can generate summaries that meet a variety of needs, such as users who are busy and want a short summary, or users who want to know the details of the content.

[0106] The acquisition unit acquires image data along with the audio data. The recognition unit converts the image data into text data describing the content of the image by inputting the image data into a trained model. The summarization unit generates summary data including the text data describing the content of the image.

[0107] In this way, the information processing device according to the present disclosure generates summary data that also includes text data that describes the image, and therefore can generate summary data that expresses the content in more detail.

[0108] (5. Hardware Configuration) Information devices such as the information processing device 100 and user terminal 200 according to the above-described embodiments are realized by a computer 1000 having a configuration as shown in FIG. 11, for example. The information processing device 100 according to the embodiment will be described below as an example. FIG. 11 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, an HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The components of the computer 1000 are connected by a bus 1050.

[0109] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0110] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000 .

[0111] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a program that executes information processing according to the present disclosure, which is an example of program data 1450.

[0112] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0113] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disk), magneto-optical recording media such as an MO (Magneto-Optical disk), tape media, magnetic recording media, and semiconductor memories.

[0114] For example, when the computer 1000 functions as the information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200, thereby realizing functions of the control unit 130 and the like. The HDD 1400 stores programs for executing information processing according to the present disclosure and data in the storage unit 120. The CPU 1100 reads and executes program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from another device via an external network 1550.

[0115] The above describes the embodiments of the present application in detail based on the drawings, but these are merely examples, and the present invention can be implemented in other forms that include the embodiments described in the Disclosure of the Invention section and that have been modified and improved in various ways based on the knowledge of those skilled in the art. [Explanation of symbols]

[0116] 1. Information Processing Systems 100 Information processing device 110 Communications Department 120 Storage section 121 Audio data storage unit 122 Generation result storage unit 123 Summary setting memory unit 130 Control Unit 131 Acquisition Department 132 Recognition part 133 Summary 134 Transmitter 200 user terminals

Claims

1. an acquisition unit that acquires audio data from video data provided together with a GUI (Graphical User Interface); a recognition unit that recognizes the voice data acquired by the acquisition unit and converts the recognition result into text data; a summarization unit that generates summary data that summarizes the content of the text data each time a sentence of the text data converted by the recognition unit is newly obtained; a transmitting unit that transmits the summary data generated by the summarizing unit to a user; Equipped with The summary section generating real-time summary data corresponding to the progress of the video data being viewed by the user as the summary data to be displayed together with the video data on the GUI; 1. An information processing device comprising:

2. The transmission unit transmitting the summary data generated by the summarization unit to a user at predetermined time intervals; 2. The information processing apparatus according to claim 1, wherein:

3. The transmission unit transmitting the summary data generated by the summarization unit to the user at predetermined time intervals defined by the user; 2. The information processing apparatus according to claim 1, wherein:

4. The acquisition unit Acquire attributes of the user; The summary section generating the summary data having a number of characters specified by a rule associated with the attribute of the user; 4. The information processing device according to claim 1, wherein the information processing device is a computer.

5. The acquisition unit Acquire image data along with audio data, The recognition unit The image data is input into a trained model to convert the image data into text data that describes the content of the image; The summary section generating summary data summarizing the content including text data that is a result of recognizing the voice data and text data that describes the content of the image; 5. The information processing device according to claim 1, wherein the information processing device is a computer.

6. The computer Acquire audio data from video data provided together with a GUI (Graphical User Interface); Recognizing the acquired voice data and converting the recognition result into text data; generating summary data summarizing the content of the text data each time a sentence of the converted text data is newly obtained; transmitting the generated summary data to a user; An information processing method comprising: generating real-time summary data corresponding to the progress of the video data being viewed by the user as the summary data to be displayed together with the video data on the GUI; Information processing methods.

7. Computer, an acquisition unit that acquires audio data from video data provided together with a GUI (Graphical User Interface); a recognition unit that recognizes the voice data acquired by the acquisition unit and converts the recognition result into text data; a summarization unit that generates summary data that summarizes the content of the text data each time a sentence of the text data converted by the recognition unit is newly obtained; a transmitting unit that transmits the summary data generated by the summarizing unit to a user; An information processing program characterized by causing the program to function as The summary section generating real-time summary data corresponding to the progress of the video data being viewed by the user as the summary data to be displayed together with the video data on the GUI; Information processing program.

Images