Information processing method, information processing device, and program
The method addresses the challenge of students missing audio content by calculating similarity between written and audio data to generate support data, enhancing note-taking efficiency and review capabilities.
Patent Information
- Application Number
- JP2023532903
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-07-05
AI Technical Summary
Students often miss important parts of audio content while focusing on writing down what is being said, leading to incomplete notes and difficulty in reviewing the missed content.
An information processing method that includes acquiring user data from a terminal, calculating the similarity between written and audio content, and generating support data to highlight missed audio segments for review.
Effectively extracts and highlights missed audio content for review, allowing students to efficiently organize and summarize their notes.
Smart Images

Figure 0007593498000001 
Figure 0007593498000002 
Figure 0007593498000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing method, an information processing device, and a program. [Background technology]
[0002] Services that allow students to learn by viewing and listening to content consisting of image and audio content are becoming more and more common. While students are viewing and listening to such content, they often write down the content for later review.
[0003] While listening to the voices spoken in the audio content (hereinafter referred to as spoken voices), students may organize the contents of what is being spoken in their own way and write them down using words other than the words that were spoken. There is a high possibility that the students have a deep understanding of the parts that are written down in this way.
[0004] However, it is not easy for students to do this while watching the content, and as a result, their attention is likely to be drawn to simply writing down what is being said (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] Pam A. Mueller and Daniel M. Oppenheimer. “The Pen Is Mightier Than the Keyboard: Advantages of Longhand Over Laptop Note Taking”, Psychological Science published online 23 April 2014 Summary of the Invention [Problem to be solved by the invention]
[0006] While the student is concentrating on writing down what is being said, he / she may miss what is being said. The student is not able to write down what he / she has missed. Therefore, the student is not able to review what he / she has missed.
[0007] The present invention has been made in light of the above circumstances, and aims to provide a technique for extracting data relating to spoken audio that a user may have missed while recording what is being spoken in audio content. [Means for solving the problem]
[0008] In order to solve the above problem, an information processing method of one embodiment of the present invention is executed by an apparatus having a hardware processor and a memory, and includes an acquisition process for acquiring first data from a user terminal, a calculation process for calculating a similarity between a first portion of a first document content related to the first data and a second portion of a second document content related to a spoken voice, a generation process for generating third data in which the first portion of data is associated with second data including at least one of the second portion of data and a third portion of data following the second portion in the second document content based on the similarity exceeding a first threshold, and an output process for outputting the third data to the user terminal. Effect of the Invention
[0009] According to one aspect of the present invention, it is possible to provide a technique for extracting data relating to spoken audio that a user may have missed while recording what was spoken in audio content. [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a server device and a user terminal used as an information processing device according to the first embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a server device and a user terminal used as an information processing device according to the first embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram showing an example of a software configuration of a control unit of a server device used as an information processing device according to the first embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram showing an example of a software configuration of a control unit of the user terminal according to the first embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram showing a structure of data stored in the user document data storage unit of the user terminal according to the first embodiment of the present invention. [Figure 6] FIG. 6 is a diagram showing a structure of data stored in the content data storage unit of the server device used as the information processing device according to the first embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing the structure of data stored in the assistance data storage unit of the server device used as the information processing device according to the first embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart showing an example of the operation executed by the server device and the user terminal used as the information processing device according to the first embodiment of the present invention. [Figure 9] FIG. 9 is a diagram for explaining details of the operation performed by the server device used as the information processing device according to the first embodiment of the present invention under the control of the assistance data generating unit. [Figure 10] FIG. 10 is a diagram for explaining details of the operation performed by the user terminal according to the first embodiment of the present invention under the control of the display data output unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that it is not essential that each functional block shown in the drawings is distinguished as shown. For example, some functions may be executed by a functional block other than the illustrated functional block. Furthermore, the illustrated functional block may be further divided into smaller functional sub-blocks. Also, the names of each functional block and each component in the following description are for convenience and do not limit the configuration and operation of each functional block and each component.
[0012] [First embodiment] (Configuration example) (1) Schematic diagram FIG. 1 shows a schematic configuration of a server device SV and a user terminal UT according to the first embodiment of the present invention.
[0013] The user terminal UT is, for example, a smartphone, a mobile terminal (for example, a tablet terminal), a personal computer, etc. The user terminal UT is capable of transmitting and receiving data to and from the server device SV. For example, a display unit DP is connected to the user terminal UT. In this specification, the description will be given assuming that the display unit DP is connected to the user terminal UT, but the user terminal UT and the display unit DP may be integrated. The display unit DP is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electroluminescence) display, etc.
[0014] For example, using the user terminal UT, the user watches the content that he / she wants to study (hereinafter, also referred to as the study content). More specifically, the user terminal UT displays the image content on the display screen of the display unit DP by, for example, playing the content data CTD related to the study content, and outputs the audio content from an output unit (not shown). The user studies through the image content and the audio content. The user terminal UT generates the user document data UD under the control of the user document data generating unit 51, for example, based on the input by the user studying via a keyboard or the like. In this embodiment, the study content is described as including the image content and the audio content, but the study content may be, for example, only the audio content. The image content may be, for example, a slide content or a video content, or may be a combination of these. The audio content includes spoken voice.
[0015] The user terminal UT can display document content related to the user document data UD on the display unit DP. Through the document content displayed on the display unit DP, the user can review the study content that he or she has studied in the past.
[0016] Prior to such a review, the user terminal UT may transmit the user document data UD to the server device SV. Under the control of the comparison unit 133, the server device SV compares the document content related to the user document data UD with the contents of the spoken voice related to the content data CTD, and generates assistance data AD based on the comparison. The assistance data AD includes data of the document content. The document content relates to, for example, a part of the contents of the spoken voice that the user may have missed while studying. The server device SV transmits the assistance data AD to the user terminal UT.
[0017] The user terminal UT displays the document content related to the assistance data AD on the display unit DP together with the document content related to the user document data UD. Through these document contents displayed on the display unit DP, the user can review the contents that he or she has studied in the past.
[0018] (2) Hardware configuration of the server device and user terminal 2 is a block diagram showing an example of the hardware configuration of the server SV and the user terminal UT. In this specification, the combination of the server SV and the user terminal UT is also referred to as a system SYS.
[0019] The server device SV includes a control unit 1, a program storage unit 2, a data storage unit 3, an input / output interface (input / output I / F) 4, and a bus BUS 1. Each of the program storage unit 2, the data storage unit 3, and the input / output interface 4 is connected to the control unit 1 via the bus BUS 1.
[0020] The control unit 1 includes a hardware processor such as a central processing unit (CPU).
[0021] The program storage unit 2 is a combination of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM) as a storage medium. The program storage unit 2 stores middleware such as an operating system (OS) as well as programs used to execute various control processes according to this embodiment.
[0022] The data storage unit 3 is a combination of a non-volatile memory such as an HDD or SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium. The data storage unit 3 is used as a working area for the hardware processor of the control unit 1, temporarily holds data, and functions as a buffer and cache.
[0023] The input / output interface 4 transmits and receives data to and from the user terminal UT using a communication protocol defined by the communication network NW under the control of the control unit 1. The input / output interface 4 is configured by an interface compatible with, for example, a wired LAN or a wireless LAN.
[0024] The user terminal UT includes a control unit 5, a program storage unit 6, a data storage unit 7, an input / output interface 8, and a bus BUS2. Each of the program storage unit 6, the data storage unit 7, and the input / output interface 8 is connected to the control unit 5 via the bus BUS2.
[0025] The same explanation as given for control unit 1 applies to control unit 5. The same explanation as given for program storage unit 2 applies to program storage unit 6. The same explanation as given for program storage unit 2 applies to data storage unit 7, as in the explanation given for data storage unit 3, with control unit 1 replaced with control unit 5. The same explanation as given for input / output interface 8, as in the explanation given for input / output interface 4, with control unit 1 replaced with control unit 5 and user terminal UT replaced with server device SV.
[0026] For example, an input unit IP and a display unit DP are connected to the input / output interface 8. The input unit IP is, for example, a keyboard, a mouse, a touch pad, etc. The input unit IP and the display unit DP may be realized by a touch panel.
[0027] (3) Software configuration related to the server device Fig. 3 is a block diagram showing an example of the software configuration of the control unit 1 of the server device SV. Fig. 3 also shows the configuration related to transmission and reception of data related to each process performed under the control of the software configuration. The same applies to the following similar drawings.
[0028] The control unit 1 includes, for example, a content request acquisition unit 11, a content data output unit 12, an assistance data generation unit 13, and an assistance data output unit 14. The processing functions of each unit included in the control unit 1 are realized by the control unit 1 causing a hardware processor of the control unit 1 to execute a program stored in a program storage unit 2. Although the above description assumes that a program stored in the program storage unit 2 is used, the program used may be one provided via a communication network NW.
[0029] The input / output interface 4 receives a content request transmitted from the user terminal UT via the communication network NW, and inputs the content request to the control unit 1. The input / output interface 4 receives data output from the control unit 1, and transmits the data to the user terminal UT via the communication network NW according to instructions from the control unit 1.
[0030] The data storage unit 3 includes, for example, a user document data storage unit 31, a content data storage unit 32, a user document divided data storage unit 33, a spoken document divided data storage unit , and an assistance data storage unit .
[0031] The user document data storage unit 31 stores user document data UD and identifiers of study content related to the user document data UD.
[0032] The content data storage unit 32 stores the content data CTD and an identifier of the study content related to the content data CTD. The content data storage unit 32 will be described as being included in the server device SV, but may be provided on a cloud computing system, for example.
[0033] The user document divided data storage unit 33 and the spoken document divided data storage unit 34 store data in the middle of processing by the assistance data generating unit 13 .
[0034] The assistance data storage unit 35 stores the assistance data AD.
[0035] The content request acquisition unit 11 performs a process of acquiring a content request transmitted from the user terminal UT via the input / output interface 4. The content request includes the user document data UD and an identifier of the learning content related to the user document data UD. The content request acquisition unit 11 performs a process of storing, for example, the user document data UD and the identifier in the user document data storage unit 31. In the storage process, the user document data UD is associated with the identifier.
[0036] The content data output unit 12 reads out the content data CTD associated with the identifier from the content data storage unit 32, outputs the content data CTD to the outside of the server device SV via the input / output interface 4, and performs processing to transmit the content data CTD to the user terminal UT.
[0037] The assistance data generator 13 performs a process of generating assistance data AD based on the user document data UD and the content data CTD. More specifically, the process is as follows.
[0038] The assistance data generating unit 13 includes, for example, a user document data dividing unit 131 , a content data processing unit 132 , and a comparison unit 133 .
[0039] The user document data division unit 131 reads out the user document data UD from the user document data storage unit 31, and performs processing to generate data obtained by dividing the user document data UD (hereinafter, also referred to as user document divided data). More specifically, the processing is as follows. The user document data division unit 131 performs processing to generate user document divided data, which is data for each part obtained by dividing the document content related to the user document data UD, into several parts. The user document data division unit 131 performs processing to store the generated user document divided data in the user document divided data storage unit 33.
[0040] The content data processing unit 132 reads out the content data CTD from the content data storage unit 32, and performs processing to generate data of document content in which the contents of the uttered voice related to the content data CTD are expressed in text form (hereinafter, also referred to as spoken document data).
[0041] The content data processing unit 132 performs a process of generating data obtained by dividing the spoken document data (hereinafter, also referred to as spoken document divided data). More specifically, the process is as follows. The content data processing unit 132 performs a process of generating spoken document divided data, which is data for each part obtained by dividing the document content related to the spoken document data, into several parts. The content data processing unit 132 performs a process of storing the generated spoken document divided data in the spoken document divided data storage unit 34.
[0042] The comparison unit 133 reads out certain user document segmentation data from the user document segmentation data storage unit 33, reads out certain spoken document segmentation data from the spoken document segmentation data storage unit 34, and performs a process of comparing the document content related to the user document segmentation data with the document content related to the spoken document segmentation data. In the comparison process, a process of calculating the similarity between the two document contents is performed. The comparison unit 133 performs a process of determining whether or not the spoken document segmentation data corresponds to the user document segmentation data based on the similarity. When the comparison unit 133 determines that the spoken document segmentation data corresponds to the user document segmentation data, it performs the process described below. The spoken document segmentation data for which the determination has been performed is hereinafter also referred to as corresponding spoken document segmentation data.
[0043] The comparison unit 133 performs a process of reading out certain spoken document segmentation data from the spoken document segmentation data storage unit 34. The spoken document segmentation data is data of a portion subsequent to a portion related to the corresponding spoken document segmentation data among document contents related to the original spoken document data (hereinafter, also referred to as subsequent spoken document segmentation data). The comparison unit 133 performs a process of storing the user document segmentation data, the corresponding spoken document segmentation data, and the subsequent spoken document segmentation data in the assistance data storage unit 35. In the storage process, the corresponding spoken document segmentation data and the subsequent spoken document segmentation data are associated with the user document segmentation data.
[0044] The comparison unit 133 may also carry out the same processing as described above for other combinations of user document segmentation data and spoken document segmentation data. As a result, combinations of user document segmentation data and corresponding spoken document segmentation data and subsequent spoken document segmentation data associated with the user document segmentation data are sequentially stored in the assistance data storage unit 35. The data stored in the assistance data storage unit 35 in this manner is the assistance data AD used in this embodiment.
[0045] The assistance data output unit 14 reads out the assistance data AD from the assistance data storage unit 35, and outputs the assistance data AD to the outside of the server device SV via the input / output interface 4, and transmits the same to the user terminal UT.
[0046] (4) Software configuration related to user terminals FIG. 4 is a block diagram showing an example of the software configuration of the control unit 5 of the user terminal UT.
[0047] The control unit 5 includes, for example, a user document data generating unit 51, a content request transmitting unit 52, a content data acquiring unit 53, an assistance data acquiring unit 54, and a display data output unit 55. The processing functions of each unit included in the control unit 5 are realized by the control unit 5 causing a hardware processor of the control unit 5 to execute a program stored in the program storage unit 6. Although the above description assumes that a program stored in the program storage unit 6 is used, the program used may be one provided through the communication network NW.
[0048] The input / output interface 8 receives a content request output from the control unit 5, and transmits the content request to the server SV via the communication network NW in accordance with an instruction from the control unit 5. The input / output interface 8 receives content data CTD and assistance data AD transmitted from the server SV via the communication network NW, and inputs the content data CTD and assistance data AD to the control unit 5.
[0049] The data storage unit 7 includes, for example, a user document data storage unit 71, a content data storage unit 72, and an assistance data storage unit 73.
[0050] The user document data storage unit 71 stores user document data UD and identifiers of study content related to the user document data UD.
[0051] The content data storage unit 72 stores the content data CTD.
[0052] The assistance data storage unit 73 stores the assistance data AD.
[0053] The user document data generating unit 51 generates user document data UD, and performs a process of storing the user document data UD and an identifier of the learning content related to the user document data UD in the user document data storage unit 71. In the storage process, the user document data UD is associated with the identifier.
[0054] The process of generating the user document data UD is based on, for example, input via the input unit IP by a user who is studying the study content. The user document data UD generated in this manner may include information on the time at which the user made input related to some parts of the document content related to the data UD.
[0055] The content request sending unit 52 performs the following process in response to, for example, a user input via the input unit IP. That is, the content request sending unit 52 reads out the user document data UD and an identifier of the study content related to the user document data UD from the user document data storage unit 71, and generates a content request including the user document data UD and the identifier. The content request sending unit 52 performs a process of transmitting the content request to the server device SV via the input / output interface 8.
[0056] The content data acquisition unit 53 performs a process of acquiring content data CTD transmitted from the server device SV via the input / output interface 8. The content data CTD is transmitted by the server device SV in response to the server device SV acquiring the above content request. The content data CTD has been stored by the server device SV in association with the above identifier. The content data acquisition unit 53 performs a process of storing the content data CTD in the content data storage unit 72.
[0057] The assistance data acquisition unit 54 performs a process of acquiring the assistance data AD transmitted from the server device SV via the input / output interface 8. The assistance data AD is transmitted by the server device SV in response to the server device SV receiving the content request. The assistance data AD is generated by the server device SV based on, for example, the user document data UD. The assistance data acquisition unit 54 performs a process of storing the assistance data AD in the assistance data storage unit 73.
[0058] The display data output unit 55 performs processing to read the user document data UD from the user document data storage unit 71, read the content data CTD from the content data storage unit 72, and read the assistance data AD from the assistance data storage unit 73. The display data output unit 55 performs processing to output information based on the user document data UD, the content data CTD, and the assistance data AD to the display unit DP via the input / output interface 8. More specifically, this is as follows.
[0059] The display data output section 55 includes, for example, a content data output section 551 , a summary candidate information output section 552 , and a summary support information output section 553 .
[0060] The content data output unit 551 performs a process of outputting the content data CTD to the display unit DP.
[0061] The summary candidate information output unit 552 performs a process of outputting information based on the user document data UD and the user document division data in the assistance data AD to the display unit DP.
[0062] The summary assistance information output unit 553 performs processing to output the speech document segmentation data in the assistance data AD to the display unit DP.
[0063] (5) Data Structure FIG. 5 shows the structure of data stored in the user document data storage unit 71. As shown in FIG. The user document data storage unit 71 has a column of identifiers of study contents (hereinafter also referred to as content identifiers) and a column of user document data. Data related to one study content is stored as each record in the user document data storage unit 71. FIG. 5 shows examples of some records stored in the user document data storage unit 71, as described below.
[0064] When storing certain user document data UD in the user document data storage unit 71, an identifier of the learning content related to the user document data UD is stored in the content identifier column as a certain record, and the user document data UD is stored in the user document data column.
[0065] The identifier is used to uniquely identify the learning content.
[0066] The user document data UD is generated based on, for example, input via an input unit IP such as a keyboard by a user studying the learning content, but is not limited to this. For example, if the user terminal UT is equipped with a camera, such as a smartphone, the data UD may be generated by photographing character information written down by the user in a physically existing notebook using the camera. Alternatively, if the user terminal is a tablet or laptop personal computer, the data UD may be generated by saving character information written down with a stylus pen or a finger. The file format of the data UD generated in this way is, for example, represented by the extension ".doc", the extension ".pdf", the extension ".png", etc., but is not limited to these.
[0067] In this manner, the user document data storage unit 71 stores the identifier and the user document data UD in association with each other as one record.
[0068] In this embodiment, as an example, a case where the user document data UD is related to document content consisting of text information, for example, is described. However, this embodiment is not limited to this. When the user document data UD is related to image content, the user document data division unit 131 performs a process of generating document content data by extracting character information from the image content, for example, using an optical character recognition (OCR) technique, before the user document division data generation process. The user document data division unit 131 performs the process described as being performed on the user document data UD on the data.
[0069] FIG. 6 shows the structure of data stored in the content data storage unit 32. As shown in FIG. The content data storage unit 32 has a column of content identifiers and a column of content data. Data related to one study content is stored as each record in the content data storage unit 32. Figure 6 shows examples of some records stored in the content data storage unit 32, as described below.
[0070] As one record in content data storage unit 32, an identifier for a certain study content is stored in the content identifier column, and content data CTD relating to that study content is stored in the content data column.
[0071] The identifier is used to uniquely identify the learning content. In the example of Figure 6, the same identifier is used as the identifier of a certain learning content as was used in the example of Figure 5.
[0072] The content data CTD may be composed of one file or multiple files. The files constituting the content data CTD may be related to image content such as slide content and video content, or may be related to audio content. The file formats of such files are, for example, those represented by the extensions ".mp4", ".mov", ".pdf", and ".pptx", but are not limited to these.
[0073] In this manner, the content data storage unit 32 stores the identifier and the content data CTD in association with each other as one record.
[0074] FIG. 7 shows the structure of the data stored in the assistance data storage unit 35. As shown in FIG. The assistance data storage unit 35 has, for example, a column of user document segmentation data, a column of corresponding spoken document segmentation data, and a column of subsequent spoken document segmentation data. In Fig. 7, examples of some records stored in the assistance data storage unit 35 are shown, as will be described next.
[0075] When storing certain user document division data UDD in the assistance data storage unit 35, the data UDD is stored in a column of user document division data as a certain record, spoken document division data CDD corresponding to the data UDD is stored in a column of corresponding spoken document division data, and subsequent spoken document division data CDD related to the data UDD is stored in a column of subsequent spoken document division data.
[0076] In this manner, in the assistance data storage unit 35, the data UDD, the corresponding spoken document segment data CDD, and the subsequent spoken document segment data CDD are stored in association with each other as one record.
[0077] In this embodiment, it is assumed that the corresponding spoken document segmentation data CDD and the subsequent spoken document segmentation data are stored as assistance data AD in association with the user document segmentation data UDD in the assistance data storage unit 35. However, only one of the corresponding spoken document segmentation data CDD and the subsequent spoken document segmentation data may be stored as assistance data AD. In this case, the assistance data storage unit 35 may not have one of the column of the corresponding spoken document segmentation data and the column of the subsequent spoken document segmentation data described above.
[0078] (Example of operation) Next, an example of the operation of the system SYS configured as above will be described.
[0079] (1) Operation flow 8 is a flowchart showing an example of the operation executed by the server device SV and the user terminal UT. The operation described below is merely an example, and the operation according to this embodiment is not limited to this.
[0080] Prior to this operation, the user, for example, watches learning content using the user terminal UT. For example, based on input by the user studying via the input unit IP, the control unit 5 of the user terminal UT generates user document data UD under the control of the user document data generation unit 51, and stores the user document data UD and an identifier of the learning content in the user document data storage unit 71. When reviewing the learning content, the user, for example, inputs information into the user terminal UT via the input unit IP. For example, the operation shown in the flowchart of FIG. 8 is started by the input from the user.
[0081] The control unit 5 of the user terminal UT, under the control of the content request transmission unit 52, reads out the user document data UD from the user document data storage unit 71, and transmits a content request including the user document data UD to the server device SV (ST01). The content request also includes the identifier.
[0082] The control unit 1 of the server device SV acquires the user document data UD via the content request under the control of the content request acquisition unit 11 (ST02). In response to the content request, the control unit 1 may read out the content data CTD associated with the identifier from the content data storage unit 32 under the control of the content data output unit 12, and transmit the content data CTD to the user terminal UT.
[0083] The control unit 1, under the control of the user document data division unit 131, generates user document division data UDD, which is data for each part obtained by dividing the document content related to the user document data UD, into several parts (ST03). The user document division data UDD generated in this way are referred to as user document division data UDD0, user document division data UDD1, user document division data UDD2, ..., user document division data UDD(p-1) (p is an integer of 2 or more) in order from the part recorded first by the user in the document content related to the original user document data UD. The control unit 1, under the control of the user document data division unit 131, stores the generated user document division data UDD in the user document division data storage unit 33.
[0084] Next, the control unit 1 generates speech document division data CDD under the control of the content data processing unit 132 (ST04). More specifically, the process is as follows.
[0085] The control unit 1 reads out the content data CTD associated with the identifier from the content data storage unit 32, and generates spoken document data, which is data of document content that expresses the contents of the spoken voice related to the content data CTD in a document. The generation process of the spoken document data is performed using, for example, some kind of voice recognition technology. The control unit 1 generates spoken document division data CDD, which is data of each part obtained by dividing the document content related to the spoken document data into several parts. Among the spoken document division data CDD generated in this way, the spoken document division data CDD related to the part that is spoken first in the spoken voice related to the original content data CTD are called spoken document division data CDD0, spoken document division data CDD1, spoken document division data CDD2, ..., spoken document division data CDD(q-1) (q is an integer equal to or greater than 2) in order. The control unit 1 stores the generated spoken document division data CDD in the spoken document division data storage unit 34.
[0086] Note that the operation described above with ST03 (hereinafter also referred to as the operation of ST03. The same applies to other similar descriptions) and the operation of ST04 may be performed in the reverse order or with partial overlap.
[0087] Next, the control unit 1 generates assistance data AD as follows.
[0088] First, under the control of the comparison unit 133, the control unit 1 sets the value of the variable i to 0 (ST05).
[0089] Next, the control unit 1 reads out the user document division data UDDi from the user document division data storage unit 33 under the control of the comparison unit 133. At this time, the user document division data UDD0 is read out. The control unit 1 reads out, for example, the spoken document division data CDD0 from the spoken document division data storage unit 34 under the control of the comparison unit 133. The control unit 1 calculates the similarity between the document content related to the user document division data UDDi and the document content related to the spoken document division data CDD0 under the control of the comparison unit 133. The calculated similarity becomes higher as the two target document contents are more similar. Similarly, the control unit 1 reads out, for example, the spoken document division data CDD1 from the spoken document division data storage unit 34 and calculates the similarity between the document content related to the user document division data UDDi and the document content related to the spoken document division data CDD1. The same applies below. Under the control of the comparison unit 133, the control unit 1 calculates the similarity between the document content related to the user document segment data UDDi and the document content related to the spoken document segment data CDDj for each case in which the integer j is an integer from 0 to (q-1) (ST06).
[0090] The control unit 1 performs the following operation for, for example, the highest similarity among the similarities calculated in this way.
[0091] The control unit 1, under the control of the comparison unit 133, determines whether or not the similarity exceeds the threshold value VSH (ST07).
[0092] If it is determined that the similarity exceeds the threshold value VSH (ST07, Yes), the control unit 1, under the control of the comparison unit 133, determines that the spoken document segmentation data CDD used to calculate the similarity is the spoken document segmentation data CDD corresponding to the user document segmentation data UDDi, and performs the operation described below.
[0093] The control unit 1 reads out the subsequent spoken document division data CDD from the spoken document division data storage unit 34. The subsequent spoken document division data CDD is data of a portion subsequent to the portion related to the corresponding spoken document division data CDD in the document content related to the original spoken document data. The control unit 1 stores the user document division data UDDi, the corresponding spoken document division data CDD, and the subsequent spoken document division data CDD in the assistance data storage unit 35. In the storage process, the corresponding spoken document division data CDD and the subsequent spoken document division data CDD are associated with the data UDDi. The data stored in the assistance data storage unit 35 is assistance data AD used in this embodiment, and the assistance data AD is generated or updated by the storage process (ST11).
[0094] If it is determined that the similarity does not exceed the threshold value VSH (ST07, No), the control unit 1, under the control of the comparison unit 133, determines whether the similarity exceeds the threshold value VSM (ST08). The threshold value VSM is lower than the threshold value VSH.
[0095] If it is determined that the similarity exceeds the threshold value VSM (ST08, Yes), the control unit 1, under the control of the comparison unit 133, calculates the degree of approximation between the position of the document content related to the user document division data UDDi in the document content related to the original user document data UD and the position of the speech voice related to the spoken document division data CDD used in calculating the similarity in the speech voice related to the content data CTD (ST09). The closer the two target positions are, the lower the calculated degree of approximation becomes. Under the control of the comparison unit 133, the control unit 1 determines whether the degree of approximation is below the threshold value VN (ST10).
[0096] If it is determined that the approximation is below the threshold value VN (ST10, Yes), the control unit 1, under the control of the comparison unit 133, determines that the spoken document division data CDD is the spoken document division data CDD corresponding to the user document division data UDDi, and performs the operation of ST11.
[0097] Following the operation of ST11, or if it is determined that the similarity does not exceed the threshold value VSM (ST08, No), or if it is determined that the approximation does not fall below the threshold value VN (ST10, No), the control unit 1, under the control of the comparison unit 133, determines whether or not the process has been completed for all i (ST12). At this point, no process has been performed for cases where i is other than 0. If the process has not been completed for all i in this way (ST12, No), the control unit 1, under the control of the comparison unit 133, increments the value of the variable i by 1 (ST13).
[0098] Next, the control unit 1 repeats the process related to the next user document division data UDD, starting from the operation in ST06. The control unit 1 continues to repeat the operations from ST06 to ST13 in this manner, and updates the assistance data AD stored in the assistance data storage unit 35 every time the control unit 1 performs the operation in ST11.
[0099] The control unit 1 determines that the processing for all i has been completed in the operation of ST12 following the processing related to the user document division data UDD(p-1). In this case (ST12, Yes), the control unit 1, under the control of the assistance data output unit 14, reads the assistance data AD from the assistance data storage unit 35 and transmits the assistance data AD to the user terminal UT (ST14).
[0100] The control unit 5 of the user terminal UT acquires the assistance data AD under the control of the assistance data acquisition unit 54 (ST15).
[0101] The control unit 5 outputs information based on the assistance data AD to the display unit DP under the control of the display data output unit 55 (ST16). Through the content displayed on the display screen of the display unit DP, the user can review the content that he or she has previously studied.
[0102] The above-mentioned operation example is merely one example of the operations executed by the server device SV and the user terminal UT. In the above, the operations from ST07 to ST11 are performed for the highest similarity among the calculated similarities. However, these operations may be performed for each of the calculated similarities. In this case, after these operations are performed for each of the calculated similarities, the operation proceeds to ST12.
[0103] Furthermore, the above-mentioned operations of ST07 and ST08 may be performed in the reverse order. In this case, the operations are performed as follows. First, it is determined whether the similarity exceeds the threshold value VSM. If it is determined that the similarity does not exceed the threshold value VSM, the operation proceeds to ST12. If it is determined that the similarity exceeds the threshold value VSM, it is determined whether the similarity exceeds the threshold value VSH. If it is determined that the similarity exceeds the threshold value VSH, the operation proceeds to ST11. If it is determined that the similarity does not exceed the threshold value VSH, the operation proceeds to ST09.
[0104] Furthermore, the above-mentioned operation of ST09 may be performed, for example, before the operation of ST07 and the operation of ST08. For example, following the operation of ST06, the operation of ST09 is performed on the spoken document segmentation data CDD used to calculate the highest similarity. Following this operation, the operations from ST07 onwards are performed.
[0105] Furthermore, as described above, when the operations ST07 to ST11 are performed for each of the calculated similarities, the operation ST09 may be performed for each spoken document segmentation data CDD before or in parallel with the operation ST06.
[0106] (2) Details of the assistance data generation process by the server device FIG. 9 is a diagram for explaining the details of the operation of each ST that is performed under the control of the support data generating unit 13 of the server device SV, which has been described with reference to FIG.
[0107] First, the operation of ST03 will be described. The document contents related to the user document data UD are grouped using a plurality of regions UR as shown in Fig. 9. More specifically, the document contents are divided into document contents in region UR0, document contents in region UR1, document contents in region UR2, document contents in region UR3, ..., document contents in region UR(p-1), in the order of first recorded by the user.
[0108] Such grouping is performed, for example, as follows. For example, the document content may be grouped such that the character information included in each line of the document content constitutes one group. Alternatively, the document content may be grouped based on the frequency of co-occurrence between adjacent words in the document content. Alternatively, the document content may be grouped based on the white space in the document content. Alternatively, the document content may be grouped based on some topic analysis technique. Alternatively, the document content may be grouped such that the areas of the multiple regions UR are substantially the same.
[0109] In the operation of ST03 in the example of Figure 8, under the control of the user document data division unit 131, user document split data UDD0, which is the data of the document content in area UR0, is generated, user document split data UDD1, which is the data of the document content in area UR1, user document split data UDD2, which is the data of the document content in area UR2, user document split data UDD3, which is the data of the document content in area UR3, ..., user document split data UDD(p-1), which is the data of the document content in area UR(p-1), is generated.
[0110] Next, the operation of ST04 will be described. Spoken document data is generated based on the content data CTD. The document contents related to the spoken document data are also grouped using a plurality of regions CR as shown in Fig. 9. More specifically, the document contents are divided into document contents in region CR0, document contents in region CR1, document contents in region CR2, document contents in region CR3, ..., document contents in region CR(q-1) in order of the first utterance in the speech voice related to the original content data CTD.
[0111] Such grouping is performed, for example, as follows. For example, the document content may be grouped such that each line of the document content is composed of character information contained in the line. Alternatively, the document content may be grouped based on the frequency of co-occurrence between adjacent words in the document content. Alternatively, the document content may be grouped based on white space in the document content. The white space corresponds to, for example, a section where there is no speech. Alternatively, the document content may be grouped based on some topic analysis technique. Alternatively, the document content may be grouped such that the areas of the multiple regions UR are substantially the same. The area of each region UR may be proportional to, for example, the speech time of the document content in the region.
[0112] In the operation of ST04 in the example of Figure 8, under the control of the content data processing unit 132, spoken document segmentation data CDD0, which is the data of the document content in area CR0, spoken document segmentation data CDD1, which is the data of the document content in area CR1, spoken document segmentation data CDD2, which is the data of the document content in area CR2, spoken document segmentation data CDD3, which is the data of the document content in area CR3, ..., spoken document segmentation data CDD(q-1), which is the data of the document content in area CR(q-1), is generated.
[0113] Next, some operations of the ST performed under the control of the comparison unit 133 will be described with reference to examples in which these operations are performed on the user document divided data UDD1.
[0114] In the operation of ST06 in the example of Fig. 8, for example, for each case where the integer j is an integer from 0 to (q-1), the similarity between the document content related to the user document segment data UDD1 and the document content related to the spoken document segment data CDDj is calculated. Fig. 9 shows the case where the integer j is 2.
[0115] The similarity calculation process may be based on how similar the character strings and / or topics are between the two target document contents. The similarity calculation process may be based on a technique that varies depending on the granularity of the grouping. For example, the similarity calculation process may be based on the length of the longest common character string between the two target document contents. Alternatively, the similarity calculation process may be based on the co-occurrence rate of words between the two target document contents. The co-occurrence rate of words is based on, for example, (number of co-occurring characters) / (number of characters in the document contents). Alternatively, the similarity calculation process may be based on some topic analysis technique.
[0116] Of the similarities calculated in this way, the following description will be given assuming that the similarity between the document content related to the user document divided data UDD1 and the document content related to the spoken document divided data CDD2 is the highest.
[0117] In the operations ST07 and ST08 in the example of FIG. 8, the highest similarity is compared with a threshold value.
[0118] If the similarity exceeds the threshold value VSH, it is determined that the spoken document segmented data CDD2 used to calculate the highest similarity corresponds to the user document segmented data UDD1, and the operation of ST11 is performed.
[0119] If the similarity does not exceed the threshold value VSH but exceeds the threshold value VSM in the operation of ST08 in the example of FIG. 8, the process described below is performed in the operation of ST09 in the example of FIG.
[0120] The degree of similarity between the position of the document content relating to the user document split data UDD1 in the document content relating to the original user document data UD and the position of the spoken voice relating to the spoken document split data CDD2 in the spoken voice relating to the content data CTD is calculated.
[0121] The position of the document content related to the user document divided data UDD1 in the document content related to the user document data UD is indicated by, for example, a numerical value of (area of region UR0 preceding region UR1 related to data UDD1) / (total area of regions UR).
[0122] Alternatively, when the user document data UD includes information on the time when the user was making input related to some parts of the document content related to the data UD, the position may be obtained as follows: That is, the position may be indicated by a numerical value of (the time from when the user starts inputting for the document content related to the data UD to when the user starts inputting for the document content related to the data UDD1) / (the time from when the user starts inputting for the document content related to the data UD to when the user finishes inputting for the document content related to the data UD).
[0123] The position of the speech sound related to the spoken document division data CDD2 in the speech sound related to the content data CTD is indicated, for example, by a numerical value of (the time from the start of the speech sound related to the content data CTD to the start of the speech sound related to data CDD2) / (the time from the start to the end of the speech sound related to the content data CTD).
[0124] Alternatively, the position may be indicated by a numerical value of, for example, (the sum of the areas of the regions CR0 and CR1 preceding the region CR2 related to the data CDD2) / (the total area of the region CR).
[0125] The degree of approximation is calculated from the absolute value of the difference between these two numbers.
[0126] If the degree of approximation is below the threshold value VN, it is determined that the spoken document segment data CDD2 corresponds to the user document segment data UDD1, and the operation of ST11 is performed.
[0127] In the operation of ST11 in the example of Fig. 8, the data UDD1, the corresponding spoken document segment data CDD2, and the subsequent spoken document segment data CDD3 are stored as data in the assistance data AD. In the assistance data AD, the data UDD1 is associated with the corresponding spoken document segment data CDD2 and the subsequent spoken document segment data CDD3.
[0128] The user document segmentation data UDD1, spoken document segmentation data CDD2, and spoken document segmentation data CDD3, which are associated in this way in the assistance data AD, are used in the operation of ST16 in the example of Fig. 8. More specifically, the user document segmentation data UDD1 is used under the control of the summary candidate information output unit 552, and the spoken document segmentation data CDD2 and CDD3 are used under the control of the summary assistance information output unit 553. This will be explained in more detail with reference to the next figure.
[0129] (3) Details of information output processing based on assistance data by the user device Fig. 10 is a diagram for explaining details of the operation of ST16 performed under the control of display data output unit 55 of the user terminal UT, which was described with reference to Fig. 8. The dashed line frames and dashed line frames in the diagram are shown for the purpose of explanation and are not images displayed on the display screen of display unit DP.
[0130] On the display screen of the display unit DP connected to the user terminal UT, for example, a display area DR1 and a display area DR2 are displayed. Display area DR1 displays document content related to user document data UD under the control of summary candidate information output section 552. Display area DR2 displays, for example, image content related to content data CTD under the control of content data output section 551.
[0131] In the display area DR1, under the control of the summary candidate information output unit 552, of the document contents related to the user document data UD, the document contents related to the user document divided data UDD in the assistance data AD are highlighted. In Fig. 10, the highlighted document contents are surrounded by << >>. In Fig. 10, the highlighted document contents related to the user document divided data UDD1 in the assistance data AD are shown surrounded by a dashed frame. Note that in the display area DR1, under the control of the summary candidate information output unit 552, the document contents related to the user document data UD may not be displayed, and only the document contents related to the user document divided data UDD in the assistance data AD may be displayed.
[0132] For example, when the cursor displayed on the display screen of the display unit DP is placed on a certain document content that is highlighted based on an operation by the user, for example, a display area DR3 or a display area DR4 is displayed on the display screen above, below, to the left or right of the document content under the control of the summary support information output unit 553. Fig. 10 shows an example in which the cursor is placed on a document content related to the highlighted user document division data UDD1.
[0133] In the display region DR3, under the control of the summary assistance information output unit 553, document contents related to the corresponding spoken document segmentation data CDD2 associated with the data UDD1 in the assistance data AD are displayed.
[0134] In the display region DR4, under the control of the summary assistance information output unit 553, document contents related to the subsequent spoken document segmentation data CDD3 associated with the data UDD1 in the assistance data AD are displayed.
[0135] (effect) The server device SV according to the first embodiment acquires user document data UD from a user terminal UT. The user document data UD is generated based on an input by a user who is studying a certain learning content. The server device SV generates a certain user document division data UDD based on the user document data UD. The user document division data UDD is a certain part of the document content related to the original user document data UD. Meanwhile, the server device SV reads out content data CTD related to the learning content, and generates spoken document data which is data of document content expressing the contents of the spoken voice related to the content data CTD in a document. The server device SV generates a certain spoken document division data CDD based on the spoken document data. The spoken document division data CDD is a certain part of the document content related to the original spoken document data. The server device SV calculates a similarity between the document content related to the user document division data UDD and the document content related to the spoken document division data CDD.
[0136] The server device SV judges whether the spoken document division data CDD corresponds to the user document division data UDD based on the similarity. When the server device SV judges that the spoken document division data CDD corresponds to the user document division data UDD, the server device SV stores the user document division data UDD, the corresponding spoken document division data CDD, and the subsequent spoken document division data CDD in the assistance data storage unit 35. The subsequent spoken document division data CDD is data of a portion following the portion related to the corresponding spoken document division data CDD in the document content related to the original spoken document data. In the storage process, the corresponding spoken document division data CDD and the subsequent spoken document division data CDD are associated with the user document division data UDD. In this manner, assistance data AD is generated in the assistance data storage unit 35. The server device SV transmits the assistance data AD to the user terminal UT.
[0137] It is highly likely that the document content related to the user document division data UDD in the assistance data AD is a direct recording of the contents spoken in the learning content by the user. The user may not have a deep understanding of the parts recorded in this way. Therefore, when the document content related to the data UDD is highlighted on the display screen of the display unit DP as described with reference to FIG. 10, the user can easily see the parts where the contents of the spoken voice were recorded as is, and can efficiently review the contents of the learning content. For example, the user can organize the contents of the highlighted document in his / her own way, summarize the contents, and re-record them in a different expression.
[0138] Furthermore, as described with reference to FIG. 10, the document content related to the spoken document division data CDD associated with the data UDD in the assistance data AD may be displayed on the display screen. Specifically, it is as follows. The document content related to the corresponding spoken document division data CDD may be displayed. Furthermore, the document content related to the subsequent spoken document division data CDD may be displayed. The document content displayed in this manner is a document representation of the spoken voice that the user may have missed because the user was too focused on recording the spoken content as it was. Therefore, by using the document content displayed in this manner, the user does not need to rewatch the learning content to listen to the part that he or she missed, and therefore can review efficiently. The user can summarize the content of the document that is highlighted as described above while referring to the document content displayed in this manner. When the granularity of the grouping described with reference to FIG. 9 is large, for example, the display of the document content related to the corresponding spoken document division data CDD may be more useful to the user. On the other hand, when the granularity of the grouping is small, for example, displaying the document content related to the subsequent spoken document segmentation data CDD may be more useful to the user.
[0139] More specifically, the server device SV judges whether the spoken document division data CDD corresponds to the user document division data UDD as follows. If the similarity exceeds a threshold value VSH, the server device SV judges that the data CDD corresponds to the data UDD. If the similarity does not exceed the threshold value VSH but exceeds the threshold value VSM, the following process is performed. The server device SV calculates the degree of approximation between the position of the document content related to the data UDD in the document content related to the original user document data UD and the position of the speech voice related to the data CDD in the speech voice related to the content data CTD. If the degree of approximation is below a threshold value VN, the server device SV judges that the data CDD corresponds to the data UDD.
[0140] For example, when a user records spoken contents as they are, it is difficult to record the spoken contents word for word. A part recorded partially differently from the spoken contents may not be extracted by a simple string comparison with the spoken contents. This is because such a string comparison is to find a completely matching string among the target character information. On the other hand, as described above, according to the server device SV of the first embodiment, whether or not the spoken document division data CDD corresponds to the user document division data UDD can be determined based on the above-mentioned approximation in addition to the similarity between the document content related to the data UDD and the document content related to the data CDD. Moreover, the calculation process of the similarity is not limited to one based on the string comparison. Therefore, according to the server device SV of the first embodiment, it is possible to extract data UDD that would not be extracted by conventional techniques and include it in the assistance data AD.
[0141] The present invention is not limited to the above-mentioned embodiment, and can be modified in various ways without departing from the gist of the present invention. The embodiments may be combined as appropriate, and in that case, the combined effect can be obtained. Furthermore, the above-mentioned embodiment includes various inventions, and various inventions can be extracted by combinations selected from the multiple components disclosed. For example, if the problem can be solved and the effect can be obtained even if some components are deleted from all the components shown in the embodiment, the configuration from which the components are deleted can be extracted as an invention. [Explanation of symbols]
[0142] SYS…System SV: Server device UT: User terminal NW: Communications network 1,5…Control section 2,6…Program memory section 3,7…Data storage section 4,8... Input / Output Interface BUS…bus IP…Input section DP...Display section 11...Content request acquisition unit 12...Content data output section 13...Assistance data generation unit 131...User document data division unit 132...content data processing unit 133…Comparison section 14...Assistance data output section 31...User document data storage unit 32...Content data storage unit 33...User document division data storage unit 34... Speech document segmentation data storage unit 35...Assistance data storage unit 51...User document data generation unit 52...Content request transmission unit 53...Content data acquisition unit 54…Assistance data acquisition section 55...Display data output section 551...Content data output unit 552...Summary candidate information output unit 553...Summary support information output unit 71...user document data storage unit 72...Content data storage unit 73...Assistance data storage unit CTD: Content data UD: User document data AD: Support data UDD: User document division data CDD: Speech document segmentation data UR,CR…area DR…display area
Claims
1. An information processing method executed by an apparatus having a hardware processor and a memory, comprising: an acquiring step of acquiring first data from a user terminal; a calculation step of calculating a similarity between a first portion of a first document content related to the first data and a second portion of a second document content related to a speech voice; a generating step of generating third data by associating the first portion of data with second data including at least one of the second portion of data and a third portion of data following the second portion in the second document content, based on the similarity exceeding a first threshold value; an output step of outputting the third data to the user terminal; An information processing method comprising:
2. The information processing method according to claim 1 , wherein the second data includes the third portion of data.
3. a calculation step of calculating a degree of similarity between a position of the first portion in the first document content and a position of a speech related to the second portion in the speech sound when the degree of similarity does not exceed the first threshold but exceeds a second threshold; generating the third data based on the degree of approximation being lower than a third threshold; The information processing method according to claim 1 , further comprising:
4. a calculation step of calculating a degree of similarity between a position of the first portion in the first document content and a position of a speech sound related to the second portion in the speech sound; a generating step of generating the third data based on the fact that the degree of approximation is below a third threshold when the degree of similarity does not exceed the first threshold but exceeds a second threshold; The information processing method according to claim 1 , further comprising:
5. 5. The information processing method according to claim 1, further comprising a generating step of generating data of the second document content based on fourth data related to audio content including the spoken voice, the fourth data being stored in a storage unit.
6. An acquisition unit that acquires first data from a user terminal; a calculation unit that calculates a similarity between a first portion of a first document content related to the first data and a second portion of a second document content related to a speech voice; a generating unit that generates third data by associating the first portion of data with second data including at least one of the second portion of data and a third portion of data following the second portion in the second document content, based on the similarity exceeding a first threshold value; an output unit that outputs the third data to the user terminal; An information processing device comprising:
7. 6. A program for causing said hardware processor to implement each step of the information processing method according to claim 1.
Citation Information
Patent Citations
Learning-aid device, learning-aid program, and recording medium with the same recorded therein
JP2004037849A
Speech output device and method therefor
JP2007086592A
Electronic note creation support device and program for electronic note creation support device
JP2014085998A
