Information processing device, information processing method, and information processing program

Abstract

To more appropriately support a user who views a lecture.SOLUTION: An information processing device 100 includes: an acquisition unit 131 that obtains voice data of a lecture which is viewed by a user; a conversion unit 132 that converts the voice data obtained by the acquisition unit 131 into text data; a determination unit 133 that determines whether the user is viewing the lecture;; a generation unit 135 that generates a summary of the text data in a period which is determined by the determination unit 133 to be a period for which the user is not viewing, by using a learned model that outputs a summary of text data in a period for which a user is not viewing according to an input of both text data and a viewing state of a user, with an input of the converted text data obtained by the conversion unit 132 and a determination result obtained by the determination unit 133; and an output unit 136 that outputs the summary generated by the generation unit 135.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] BACKGROUND ART Conventionally, speech recognition techniques for converting speech data into text data are known, such as speech recognition techniques using various language models (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-110087 Summary of the Invention [Problem to be solved by the invention]

[0004] However, conventional technologies may not be able to adequately support users who are watching a lecture. For example, while conventional technologies can convert the content of a speaker into text data as it is spoken, if a user leaves their seat and then resumes watching, it may be difficult for them to catch up on the current content of the lecture by rereading all that was said while they were not watching.

[0005] The present invention has been made in view of the above, and has an object to provide an information processing device, an information processing method, and an information processing program that more appropriately support users who watch lectures. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the information processing device of the present invention is characterized by having an acquisition unit that acquires audio data of a lecture watched by a user, a conversion unit that converts the audio data acquired by the acquisition unit into text data, a judgment unit that judges whether the user is watching the lecture, a generation unit that uses the text data converted by the conversion unit and the judgment result by the judgment unit as input, and generates a summary of the text data for a period in which it is judged by the judgment unit that the user is not watching, using a trained model that outputs a summary of the text data for a period in which the user is not watching in accordance with the input of the text data and the user's viewing status, and an output unit that outputs the summary generated by the generation unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide more appropriate support to users who are viewing a lecture. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of an output process performed by the output unit according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of an output process performed by the output unit according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of processing by the information processing device according to the embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of the flow of processing by the information processing device according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a computer that executes an information processing program. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, with reference to the drawings, an information processing device, an information processing method, and an information processing program according to the present application will be described in detail. Note that the present invention is not limited to these embodiments. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals, and duplicated explanations will be omitted.

[0010] (Embodiment) [1. Information Processing System Configuration] An information processing system 1 shown in Fig. 1 will be described. Fig. 1 is a diagram showing an example of the configuration of the information processing system 1 according to the embodiment. As shown in Fig. 1, the information processing system 1 includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication.

[0011] 1 is a device such as smart glasses, a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, smart glasses, etc. Furthermore, the terminal device 10 may be, for example, a terminal equipped with augmented reality (AR) technology.

[0012] The terminal device 10 is a device used for viewing and listening to lectures. The terminal device 10 can also record, record, and play back lectures. The terminal device 10 is equipped with a display, an imaging device, a speaker, a microphone, an acceleration sensor, and the like. For example, the terminal device 10 records audio data using the microphone provided therein and transmits the recorded audio data to the information processing device 100. The terminal device 10 can also receive output from the information processing device 100 and display text data.

[0013] 1 is a computer that can communicate with any terminal device 10 connected by wire or wirelessly. The information processing device 100 processes voice data acquired from the terminal device 10 and outputs the processed information to the terminal device 10.

[0014] The information processing device 100 is a computer intended to provide appropriate support to users who are viewing a lecture, and may be any device that can implement the processes in the embodiments.

[0015] For example, the information processing device 100 acquires audio data of a lecture being viewed by a user, converts the acquired audio data into text data, determines whether the user is viewing the lecture, and uses the voice-recognized audio data and the user's viewing status, which is the determination result, as input, to generate a summary of the text data for the period during which it was determined that the user was not viewing, using a trained model that outputs a summary of the text data for the period during which it was determined that the user was not viewing, and outputs the generated summary.

[0016] As a result, the information processing device 100 can generate a summary of the text data for the period when the user is not watching the lecture from the text data converted from the audio data of the lecture that the user is watching and the user's viewing status, and display the summary on the terminal device 10, thereby providing more appropriate support to the user watching the lecture.

[0017] In other words, the information processing device 100 displays text data summarizing the content of the lecture during the period when the user was not watching the lecture on the terminal device 10 used by the user, making it easy for the user to review the content of the lecture when they resume watching.

[0018] Viewing refers to the act of watching or listening to a lecture, regardless of the location of the lecture. In other words, lectures that users view include not only streaming or distributed lectures, but also lectures given by speakers who are present in the same venue as the user.

[0019] 2. Configuration of Information Processing Device Next, the configuration of the information processing device 100 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 2, the information processing device 100 includes a communication unit 110, a control unit 130, and a storage unit 120. Note that these units may be held in a distributed manner in multiple devices. The processing of these units will be described below.

[0020] (Communication unit 110) The communication unit 110 is realized by a NIC (Network Interface Card) or the like, and enables communication between an external device and the control unit 130 via an electric communication line such as a LAN or the Internet. For example, the communication unit 110 enables communication between the external device and the control unit 130.

[0021] (Storage unit 120) The storage unit 120 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 has a user information storage unit 121, a model storage unit 122, a voice information storage unit 123, and a text information storage unit 124. Each unit of the storage unit 120 will be described below.

[0022] (User information storage unit 121) The user information storage unit 121 stores information about users, such as the user's name, gender, age, fields and phrases requiring annotation, viewing status, summary generation speed setting, summary accuracy setting, etc.

[0023] (Model storage unit 122) The model storage unit 122 stores a trained model. The trained model is a model that has been trained to generate a summary of text data during a period when the text data is not being viewed, using text data and a viewing situation as input. An example of the trained model is a generation AI.

[0024] The generating AI is, for example, Tsuzumi (registered trademark). Tsuzumi is a natural language processing model fine-tuned to a specific field. Examples of specific fields include finance, medicine, semiconductors, IT (Information Technology), academia, factories (plants), law, and office services. Tsuzumi was built with an emphasis on low power consumption and has a faster processing speed than ChatGPT. The model storage unit 122 stores multiple Tsuzumis, each corresponding to a specific field such as finance, medicine, semiconductors, IT, academia, factories, law, or office services. The specific fields are not limited to the above.

[0025] The generating AI is, for example, ChatGPT (registered trademark). ChatGPT is a large-scale natural language processing model that is slower than Tsuzumi but more accurate.

[0026] (Audio information storage unit 123) The voice information storage unit 123 stores voice data. For example, the voice information storage unit 123 stores voice data acquired by the acquisition unit 131. Note that the format of the voice data stored in the voice information storage unit 123 is not particularly limited.

[0027] (Text information storage unit 124) The text information storage unit 124 stores text data. For example, the text information storage unit 124 stores text data converted into speech by the conversion unit 132 (described later). Also, for example, the text information storage unit 124 stores text data of a summary generated by the generation unit 135 (described later).

[0028] The text data stored in the text information storage unit 124 includes information about the playback time of the audio data. For example, the text data stored in the text information storage unit 124 includes, for each sentence, information such as "uttered at xx minutes xx seconds in the audio data."

[0029] (Annotation information storage unit 125) The annotation information storage unit 125 stores text data that the output unit 136 adds as an annotation. For example, the annotation information storage unit 125 stores information explaining a predetermined phrase as an annotation. More specifically, for the term "open competitive bidding," the annotation information storage unit 125 stores the following information as an annotation: "A competitive bidding method in which no restrictions are placed on bidders and a contract is concluded with the bidder who offers the most favorable terms to the local government."

[0030] (control unit 130) The control unit 130 is realized using a CPU (Central Processing Unit), an NP (Network Processor), an FPGA (Field Programmable Gate Array), or the like, and executes a processing program stored in memory. As shown in Fig. 2, the control unit 130 has an acquisition unit 131, a conversion unit 132, a determination unit 133, a selection unit 134, a generation unit 135, and an output unit 136. Each unit of the control unit 130 will be described below.

[0031] (Acquisition part 131) The acquisition unit 131 acquires audio data of a lecture viewed by a user. For example, the acquisition unit 131 acquires audio data of a lecture viewed by a user that is recorded by a microphone provided in the terminal device 10 or connectable to the terminal device 10. Furthermore, for example, the acquisition unit 131 acquires audio data of a lecture viewed by a user that is played back by the terminal device 10. Note that the format of the audio data acquired by the acquisition unit 131 does not matter.

[0032] The acquiring unit 131 further acquires information about the lecture, a setting of a speed for generating a summary, and information about the accuracy of the summary. For example, the acquiring unit 131 acquires information about the lecture, such as "field, summary, content, and confidentiality (to be disclosed)," from a server that provides information about the lecture (for example, a server that distributes videos of the lecture, or a server that publishes a website on which information about the lecture is posted). The acquiring unit 131 may also acquire information about the lecture by accepting input from a user.

[0033] Furthermore, the acquiring unit 131 refers to the user information storage unit 121 to acquire information on the setting of the speed at which the summary is generated and the setting of the accuracy of the summary.

[0034] (Conversion unit 132) The conversion unit 132 converts the voice data acquired by the acquisition unit 131 into text data. For example, the conversion unit 132 converts the voice data acquired by the acquisition unit 131 into text data using a voice recognition technique. More specifically, the conversion unit 132 converts the voice data acquired by the acquisition unit 131 into text data by inputting the voice data into a voice recognition model. At this time, the conversion unit 132 performs voice recognition so that information on the playback time of the voice data is included in the text data.

[0035] (Judgment unit 133) The determination unit 133 determines whether or not the user is viewing the lecture. For example, the determination unit 133 determines whether or not the user is viewing the lecture based on one or more of the user's seating status, the user's line of sight, and the status of the window in which the lecture is displayed.

[0036] First, an example of determining whether a user is watching a lecture based on the user's seating status will be described. For example, the determination unit 133 detects that the user has removed the terminal device 10 using an acceleration sensor provided in the terminal device 10, and determines that the user is not watching a lecture. On the other hand, the determination unit 133 detects that the user is wearing the terminal device 10 using an acceleration sensor provided in the terminal device 10, and determines that the user is watching a lecture. Note that in the above example of determining whether a user is watching a lecture based on the detection of the wearing or removal of the terminal device 10, the terminal device 10 is assumed to be wearable on the user's body, such as smart glasses.

[0037] As another example, the determination unit 133 determines that the user is not seated and is not watching the lecture because the user is no longer visible in the field of view of the imaging device included in the terminal device 10. On the other hand, the determination unit 133 determines that the user is seated and watching the lecture because the user is visible in the field of view of the imaging device included in the terminal device 10.

[0038] As another example, the determination unit 133 determines that the user is not seated and is not watching the lecture, based on the fact that it has received a press of an "Away" button displayed on an application installed on the terminal device 10. On the other hand, the determination unit 133 determines that the user is seated and is watching the lecture, based on the fact that it has received a press of an "At Seat" button displayed on an application installed on the terminal device 10.

[0039] Furthermore, the determination unit 133 can determine whether the user is watching a lecture based on the user's utterance. For example, the determination unit 133 determines that the user is not seated and is not watching a lecture based on speech recognition of the user's utterance (e.g., "I'm leaving my seat"), which is input from the microphone provided in the terminal device 10. On the other hand, the determination unit 133 determines that the user is seated and is watching a lecture based on speech recognition of the user's utterance (e.g., "I'm back"), which is input from the microphone provided in the terminal device 10, which is input from the microphone provided in the terminal device 10.

[0040] Next, an example of determining whether a user is watching a lecture based on the direction of the user's line of sight will be described. For example, the determination unit 133 detects the direction of the user's line of sight from an imaging device included in the terminal device 10, and determines that the user is not watching a lecture if the user's line of sight is not in the direction for watching a lecture. On the other hand, the determination unit 133 detects the direction of the user's line of sight from an imaging device included in the terminal device 10, and determines that the user is watching a lecture if the user's line of sight is in the direction for watching a lecture.

[0041] Next, an example will be described in which it is determined whether or not the user is watching a lecture based on the state of a window playing the lecture. The determination unit 133 determines that the user is not watching a lecture because the state of the window playing the video of the lecture is inactive on the terminal device 10. On the other hand, the determination unit 133 determines that the user is watching a lecture because the state of the window playing the video of the lecture is active on the terminal device 10.

[0042] The determination unit 133 can determine whether or not the user is viewing the lecture at predetermined intervals. For example, the determination unit 133 determines whether or not the user is viewing the lecture every five seconds.

[0043] (Selection unit 134) The selection unit 134 selects, as the trained model, either a first trained model which is a natural language processing model fine-tuned to a specific field, or a second trained model which is a large-scale natural language processing model, based on information about the lecture acquired by the acquisition unit 131, a setting for the speed at which the summary is generated, and a setting for the accuracy of the summary.

[0044] More specifically, the selection unit 134 selects a first trained model (e.g., Tsuzumi) when the lecture is in a specific field (e.g., finance) and the speed at which the set summary is generated is faster than standard. Furthermore, the selection unit 134 selects the first trained model when the level of confidentiality of the lecture is equal to or higher than standard. In this case, the selection unit 134 selects the first trained model corresponding to the determined field of the lecture.

[0045] On the other hand, if the lecture does not fall into a specific field and the accuracy of the set summary is higher than the standard, the selection unit 134 selects the second trained model (e.g., ChatGPT). Also, if the confidentiality of the lecture is lower than the standard, the selection unit 134 selects the second trained model.

[0046] (Generation unit 135) The generation unit 135 generates a summary of the text data for the period when it is determined that the user did not watch the lecture. For example, the generation unit 135 receives as input the text data speech-recognized by the conversion unit 132 and the user's viewing status determined by the determination unit 133, and generates a summary of the text data for the period when it is determined by the determination unit 133 that the user did not watch, using a trained model that outputs a summary of the text data for the period when the user did not watch, in accordance with the input of the text data and the user's viewing status.

[0047] For example, the generation unit 135 inputs text data speech-recognized by the conversion unit 132 and the viewing situation determined by the determination unit 133 into a model that has been trained to generate a summary of text data for periods when the user is not viewing, and generates a summary of the text data for periods when the determination unit 133 determines that the user is not viewing.

[0048] The generation unit 135 can generate a summary using the first trained model or the second trained model selected by the selection unit 134.

[0049] The generation unit 135 generates a summary when the user resumes viewing after suspending viewing. For example, the generation unit 135 generates a summary of the text data when the determination unit 133 determines that the user is not viewing a lecture and then determines that the user is viewing a lecture.

[0050] For example, when the determination unit 133 determines that the user is not watching a lecture, and then the determination unit 133 determines that the user is watching a lecture, the generation unit 135 inputs the text data speech-recognized by the conversion unit 132 and the user's viewing status, which is the determination result by the determination unit 133, into a trained model that outputs a summary of the text data for the period when the user was not watching in response to the input of the text data and the user's viewing status, and generates a summary of the text data for the period when the determination unit 133 determined that the user was not watching.

[0051] In this case, the generation unit 135 may generate a summary if the period from when the user stops viewing to when they resume viewing exceeds a predetermined period. The generation unit 135 generates a summary of the text data if the period from when the determination unit 133 determines that the user is not viewing the lecture to when the determination unit 133 determines that the user is viewing the lecture exceeds a predetermined period. For example, if the period from when the determination unit 133 determines that the user is not viewing the lecture to when the determination unit 133 determines that the user is viewing the lecture exceeds three minutes, the generation unit 135 generates a summary of the text data for the period during which the determination unit 133 determined that the user was not viewing the lecture.

[0052] (output unit 136) The output unit 136 outputs the summary generated by the generation unit 135. For example, the output unit 136 displays text data of the summary generated by the generation unit 135 on the display of the terminal device 10. Here, a case where a lecture is being held at a location different from the location where the user is present will be described with reference to FIG. 3. FIG. 3 is a diagram for explaining an example of output processing by the output unit 136 according to the embodiment. FIG. 3 shows the display of the terminal device 10 when the user returns from leaving their seat and presses a take seat button displayed on the display of the terminal device 10 in a situation where a lecture is being held at a location different from the user's location, and the display shows a video of the lecture, the leave / take seat button, and a summary.

[0053] For example, as shown in Figure 3, the output unit 136 displays a summary of the content of the lecture that the user is watching while away from his or her desk, in a location on the display of the terminal device 10 that plays the video of the lecture that the user is watching, that does not overlap with the video of the lecture that the user is watching.

[0054] Next, the processing of the output unit 136 when a lecture is being held in the same place as the user is present will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining an example of output processing of the output unit 136 according to the embodiment. Fig. 4 shows the field of view of a user who has returned from leaving their seat and is wearing smart glasses, which are the terminal device 10, in a situation where a lecture is being held in the same place as the user is present, and the field of view includes the smart glasses, the lecture venue, the speaker, and the summary.

[0055] For example, as shown in FIG. 4, the output unit 136 displays a summary of the content of a lecture given while the user is away from their seat in a location on the display portion of the smart glasses, which are the terminal device 10, that does not interfere with viewing (for example, an edge of the display).

[0056] The output unit 136 outputs an annotation for a predetermined wording included in the summary generated by the generation unit 135. For example, the output unit 136 uses the predetermined wording included in the summary generated by the generation unit 135 to refer to the text information storage unit 124 to acquire an annotation corresponding to the predetermined wording, and outputs the acquired annotation.

[0057] Here, examples of the predetermined wording include wording set in advance by the user, technical terms in a predetermined field, alphabetic abbreviations, phrases containing kanji other than common kanji, etc. More specifically, when the summary generated by the generation unit 135 includes the term "open competitive bidding" in the field of finance, the output unit 136 displays on the display of the terminal device 10 an annotation that reads "a competitive bidding method in which no bidders are limited and a contract is concluded with the bidder who offers the most favorable terms to the local government" together with the summary of the text data.

[0058] [3. Processing] Next, an example of processing realized by the information processing device 100 according to the present embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating an example of processing by the information processing device 100 according to the embodiment. Fig. 5 shows an example of processing in which acquired voice data is speech-recognized as text data, a summary of the text data for a period when the user is not listening to the lecture is generated, and the summary is output to the terminal device 10.

[0059] First, the information processing device 100 acquires audio data listened to by the user. Next, the information processing device 100 converts the acquired audio data into text data. At this time, the information processing device 100 determines whether the user is listening to a lecture in parallel with acquiring the audio data and converting it into text data.

[0060] After it is determined that the user is not watching the lecture, the information processing device 100 inputs the converted text data and the user's viewing status, which is the determination result, into a model that outputs a summary of the text data for the period when the user is not watching, at the time when it is determined that the user is watching the lecture, in response to input of the text data and the user's viewing status, and generates a summary of the text data for the period when it is determined that the user is not watching.

[0061] Then, the information processing device 100 causes the terminal device 10 to display the generated summary.

[0062] [4. Flowchart] Next, the flow of processing by the information processing system 1 will be described with reference to Fig. 6. Note that the steps below may be executed in a different order, and some processing may be omitted.

[0063] First, the acquisition unit 131 acquires audio data of a lecture that the user is listening to (step S101). For example, the acquisition unit 131 acquires audio data of a lecture that the user is listening to, which is recorded by a microphone that the terminal device 10 includes or that can be connected to the terminal device 10.

[0064] Next, the conversion unit 132 converts the voice data acquired by the acquisition unit 131 into text data (step S102). For example, the conversion unit 132 converts the voice data acquired by the acquisition unit 131 into text data using a voice recognition technique.

[0065] Next, the selection unit 134 selects the first trained model or the second trained model based on information about the lecture, a setting for the speed at which the summary is generated, and a setting for the accuracy of the summary (step S103).

[0066] Next, the determination unit 133 determines whether the user is viewing the lecture (step S104). For example, the determination unit 133 determines whether the user is viewing the lecture based on one or more of the user's seating status, the user's line of sight, and the status of the window in which the lecture is displayed.

[0067] Here, if the determination unit 133 determines that the user is watching the lecture (step S104: Yes), the process of step S103 is performed again. On the other hand, if the determination unit 133 determines that the user is not watching the lecture (step S104: No), the determination unit 133 then determines whether the user is watching the lecture (step S105).

[0068] Here, if the determination unit 133 determines that the user is not viewing the lecture (step S105: No), the process of step S105 is performed again. On the other hand, if the determination unit 133 determines that the user is viewing the lecture (step S105: Yes), the generation unit 144 then generates a summary of the text data for the period during which it was determined that the user was not viewing the lecture (step S106).

[0069] For example, the generation unit 135 receives as input the text data speech-recognized by the conversion unit 132 and the user's viewing status, which is the result of the determination by the determination unit 133, and generates a summary of the text data for the period in which the user was determined not to be viewing by the determination unit 133, using a trained model that outputs a summary of the text data for the period in which the user was not viewing, based on the input text data and the user's viewing status.

[0070] Then, the output unit 136 outputs the summary generated by the generation unit 135 (step S107). For example, the output unit 136 causes the display of the terminal device 10 to display text data of the summary generated by the generation unit 135.

[0071] [5. Effects] The information processing device 100 according to the embodiment includes an acquisition unit 131 that acquires audio data of a lecture being viewed by a user, a conversion unit 132 that performs speech recognition on the audio data acquired by the acquisition unit 131 as text data, a determination unit 133 that determines whether or not the user is viewing the lecture, a generation unit 135 that receives as input the text data speech-recognized by the conversion unit 132 and the determination result by the determination unit 133, and generates a summary of the text data for a period during which the determination unit 133 determines that the user is not viewing, using a trained model that outputs a summary of the text data for a period during which the user is not viewing, in accordance with the input of the text data and the user's viewing status, and an output unit 136 that outputs the summary generated by the generation unit 135.

[0072] As a result, the information processing device 100 can generate a summary of the text data for the period when the user is not watching the lecture from the text data converted from the audio data of the lecture that the user is watching and the user's viewing status, and display the summary on the terminal device 10, thereby providing more appropriate support to the user watching the lecture.

[0073] In other words, the information processing device 100 can display text data summarizing the content of the lecture during the period when the user is not watching the lecture on the terminal device 10 used by the user, making it easy for the user to review the content of the lecture and providing more appropriate support to users watching the lecture.

[0074] In addition, the acquisition unit 131 of the information processing device 100 according to the embodiment further acquires information about the lecture, a setting for the speed at which a summary is generated, and a setting for the accuracy of the summary, and further has a selection unit 134 that selects, as the trained model, a first trained model which is a natural language processing model fine-tuned to a specific field, or a second trained model which is a large-scale natural language processing model, based on the information about the lecture acquired by the acquisition unit 131, the setting for the speed at which a summary is generated, and the setting for the accuracy of the summary, and the generation unit 135 generates a summary using the first trained model or the second trained model selected by the selection unit 134.

[0075] As a result, the information processing device 100 generates summaries of lectures during periods when the user is not watching them using a model selected according to the field, content, confidentiality, summary generation speed, accuracy, etc. of the lecture being watched by the user, and displays the summaries on the terminal device 10, making it easier for the user to review the content of the lecture and providing more appropriate support to users watching the lectures.

[0076] The determination unit 133 of the information processing device 100 according to the embodiment determines whether the user is watching the lecture based on one or more of the user's seating status, the user's line of sight, and the status of the window in which the lecture is displayed.

[0077] This allows the information processing device 100 to more accurately determine whether the user is watching the lecture based on information on the user's seating status, line of sight, and the status of the window in which the lecture is displayed, and by generating a summary of the text data for the period when the user is not watching the lecture and displaying it on the terminal device 10, it can more appropriately support the user watching the lecture.

[0078] The generation unit 135 of the information processing device 100 according to the embodiment generates a summary of the text data when the judgment unit 133 determines that the user is not watching the lecture, and then the judgment unit 133 determines that the user is watching the lecture.

[0079] This allows the information processing device 100 to generate a summary of the text data when the user resumes watching the lecture, making it possible to display the summary at the time when the user wants to review the lecture, thereby providing more appropriate support to users watching the lecture.

[0080] The generation unit 135 of the information processing device 100 according to the embodiment generates a summary when the period from when the judgment unit 133 determines that the user is not watching the lecture to when the judgment unit 133 determines that the user is watching the lecture exceeds a predetermined period.

[0081] This allows the information processing device 100 to generate a summary only if the period during which the user has interrupted viewing is long enough to require a summary, and display the summary on the terminal device 10, thereby providing better support to users viewing a lecture.

[0082] The output unit 136 of the information processing apparatus 100 according to the embodiment outputs an annotation for a predetermined phrase included in the summary generated by the generation unit 135.

[0083] As a result, the information processing device 100 can add annotations to phrases in the summary that require supplemental information and display them on the terminal device 10, thereby more appropriately supporting users who are watching the lecture.

[0084] [6. Program] It is also possible to create a program written in a computer-executable language that executes the processes executed by the information processing device 100 described in the above embodiment. In this case, the same effects as those of the above embodiment can be achieved by having a computer execute the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to achieve the same processes as those of the above embodiment.

[0085] Fig. 7 is a diagram showing an example of a computer that executes an information processing program. As shown in Fig. 7, a computer 1000 includes, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0086] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0087] 7, the hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. The tables described in the above embodiments are stored in the hard disk drive 1090 or the memory 1010, for example.

[0088] The information processing program is stored in the hard disk drive 1090 as a program module in which instructions to be executed by the computer 1000 are written. Specifically, the hard disk drive 1090 stores a program module 1093 in which each process executed by the computer 1000 described in the above embodiment is written.

[0089] Furthermore, data used for information processing by the information processing program is stored as program data, for example, in the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 and program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as necessary, and executes each of the above-described procedures.

[0090] The program module 1093 and program data 1094 related to the information processing program are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100, etc. Alternatively, the program module 1093 and program data 1094 related to the control program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0091] [7. Other] Although various embodiments have been described in detail herein with reference to the drawings, these embodiments are merely examples and are not intended to limit the present invention. The features described herein can be realized in various ways, including various modifications and improvements based on the knowledge of those skilled in the art.

[0092] Furthermore, the above-mentioned "module (-er suffix, -or suffix)" can be read as a unit, means, circuit, etc. For example, a communication module, a control module, and a storage module can be read as a communication unit, a control unit, and a storage unit, respectively. [Explanation of symbols]

[0093] 1. Information Processing Systems 100 Information processing device 110 Communications Department 120 Storage section 121 User information storage unit 122 Model Memory Unit 123 Voice information storage unit 124 Text information storage unit 125 Annotation information storage unit 130 Control Unit 131 Acquisition Department 132 Conversion Unit 133 Judgment section 134 Selection Section 135 Generation part 136 Output section

Claims

1. an acquisition unit that acquires audio data of a lecture that a user is listening to; a conversion unit that converts the voice data acquired by the acquisition unit into text data; a determination unit that determines whether the user is watching the lecture; a generation unit that receives as input the text data converted by the conversion unit and the determination result by the determination unit, and generates a summary of the text data for a period in which the user is not viewing, using a trained model that outputs a summary of the text data for a period in which the user is not viewing, in accordance with the input of the text data and the user's viewing situation; and an output unit that outputs the summary generated by the generation unit; An information processing device comprising:

2. The acquisition unit further acquires information about the lecture, a setting of a speed for generating the summary, and a setting of accuracy of the summary; a selection unit that selects, as the trained model, a first trained model that is a natural language processing model fine-tuned to a specific field or a second trained model that is a large-scale natural language processing model, based on information about the lecture acquired by the acquisition unit, a setting for a speed at which the summary is generated, and a setting for accuracy of the summary. and The generation unit generates the summary using the first trained model or the second trained model selected by the selection unit.

2. The information processing apparatus according to claim 1, wherein:

3. The determination unit Whether the user is watching the lecture is determined based on at least one of the seating status of the user, the direction of the user's line of sight, and the status of the window in which the lecture is displayed.

2. The information processing apparatus according to claim 1, wherein:

4. The generation unit After the determination unit determines that the user is not viewing the lecture, if the determination unit determines that the user is viewing the lecture, a summary of the text data is generated.

2. The information processing apparatus according to claim 1, wherein:

5. The generation unit If a period from when the determination unit determines that the user is not viewing the lecture to when the determination unit determines that the user is viewing the lecture exceeds a predetermined period, a summary of the text data is generated.

2. The information processing apparatus according to claim 1, wherein:

6. The information processing apparatus according to claim 4 , wherein the output unit outputs an annotation for a predetermined phrase included in the summary generated by the generation unit.

7. 1. A computer-implemented information processing method, comprising: an acquisition step of acquiring audio data of a lecture that a user is listening to; a conversion step of converting the voice data acquired by the acquisition step into text data; a determining step of determining whether the user is watching the lecture; a generation process for generating a summary of the text data for a period determined by the determination process to be non-viewing by the determination process using a trained model that receives the text data converted by the conversion process and the determination result by the determination process as input and outputs a summary of the text data for a period when the user is not viewing in accordance with the input of the text data and the user's viewing situation; an output step of outputting the summary generated by the generation step; An information processing method comprising:

8. an acquisition step of acquiring audio data of a lecture to be viewed by a user; a conversion step of converting the voice data acquired by the acquisition step into text data; a determining step of determining whether the user is viewing the lecture; a generation step of generating a summary of the text data for a period determined by the determination step not to be viewed by the user using a trained model that receives the text data converted by the conversion step and the determination result by the determination step as input, and outputs a summary of the text data for a period not viewed by the user in accordance with the input of the text data and the user's viewing situation; an output step of outputting the summary generated by the generation step; An information processing program characterized by causing a computer to execute the above.

Images