Electronic note-taking system and program

Abstract

This system and program provides an electronic note-taking system and program that can reduce the burden on students and is useful for efficient review of lecture content. [Solution] Processors 10 and 20, which process video data of a board 6 on which the lecture content is written, extract still image data from the video data, extract text, figures, and table objects from the still image data, group related objects into object groups Gr1 and Gr2, generate text data from the audio data included in the video data, divide the text data into sentences Txt1-Txt14, associate sentences related to the object groups with the object groups, and display the object groups and the sentences associated with the object groups as a single unit on the display device 23.

Description

Technical Field

[0001] The present invention relates to an electronic note creation system and a program.

Background Art

[0002] In lectures conducted at educational institutions, typically, an instructor writes lecture content on a board such as a blackboard or a whiteboard, and in parallel, verbally explains the matters written on the board. On the other hand, lecture attendees create notes during the lecture to review the lecture content.

[0003] The matters written on the board by the instructor are generally the key points of the lecture content, and detailed explanations and supplements are verbally explained. When the lecture attendees review the lecture content, in order to understand and deepen the understanding of the lecture content, not only the matters written on the board but also the instructor's oral explanations are important. Therefore, lecture attendees who create notes are required to copy the matters written on the board into their notes while listening to the instructor's explanations, and write down the instructor's explanations or summaries as needed. When the lecture progresses quickly, the burden on the lecture attendees is relatively large.

[0004] As a system that can reduce the burden on lecture attendees, for example, the lecture recording system described in Non-Patent Document 1 records lectures and analyzes and converts the audio of the recorded lectures into text by AI. Lecture attendees can review the recorded lecture videos at a later date, and can also perform a word search on the text data and play the lecture videos from the searched word portions.

[0005] Furthermore, the electronic note-taking support device described in Patent Document 1 acquires video and audio data of lectures, extracts still image data from the video data as needed, and automatically generates text data from the audio data. The still image data and text data are then displayed on the display screen of the information terminal held by the student. The student can create notes by selecting still image data displayed on the information terminal and adding desired text selected from the text data displayed on the information terminal, or any text entered by hand or keyboard, to the still image data. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2014-85998 [Non-patent literature]

[0007] [Non-Patent Document 1] Tomisaki, Orie, et al., "Operation of a Lecture Recording System Utilizing AI Speech Recognition Technology - An Initiative at Nagoya University Law School," [online], December 13, 2023, Proceedings of the Annual Conference of the Japan Council for the Promotion of University ICT, [Retrieved October 31, 2025], Internet.<https: / / www.jstage.jst.go.jp / article / axies / 2023 / 0 / 2023_313 / _pdf / -char / ja> [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] According to the lecture recording system described in Non-Patent Document 1, students can use word search to jump to specific sections of lecture videos. However, if the search word is not specified, it is necessary to watch the lecture video from beginning to end. Even if the search word is specified, the information on the board is updated sequentially as the lecture progresses, so watching only the specific section is insufficient to grasp the entire content of the board; it is necessary to watch the lecture video that follows the specific section. Watching lecture videos of a certain length each time for review is not efficient.

[0009] According to the electronic note-taking support device described in Patent Document 1, students can select text to add to still image data from text data automatically generated from the lecture's audio data. While this eliminates the need to manually input text, it introduces new operations such as selecting the desired text from the text data and adding the selected text to the still image data. Students must perform these operations while listening to the instructor's explanations. Therefore, it cannot necessarily be said that the burden on students is reduced.

[0010] One of the objectives of the present invention is to provide an electronic note-taking system and program that can reduce the burden on students and is useful for efficient review of lecture content. [Means for solving the problem]

[0011] An electronic note-taking system according to one aspect of the present invention is: It is equipped with a processor that processes video data of a board on which lecture content is written, The aforementioned processor, Still image data is extracted from the aforementioned video data. From the aforementioned still image data, text, figures, and tables are extracted, and groups of related objects are grouped together as object groups. Text data is generated from the audio data contained in the aforementioned video data. The aforementioned text data is divided into sentences, The sentence derived from the audio data related to the object group is associated with the object group, The object group and the statement associated with the object group are displayed on the display device as a single unit.

[0012] Furthermore, a program according to another aspect of the present invention is: The processor that processes the video data of the board on which the lecture content is written performs the following: The steps include extracting still image data from the aforementioned video data, The steps include extracting text, figures, and tables from the still image data and storing related object groups as object groups, The steps include generating text data from audio data contained in the aforementioned video data, The steps include dividing the aforementioned text data into sentences, The steps include: associating the sentence derived from the audio data, which is related to the object group, with the object group; The steps include displaying the object group and the statement associated with the object group as a single unit on a display device, The processor is made to execute an electronic notebook creation method that includes the following. [Effects of the Invention]

[0013] According to the present invention, it is possible to provide an electronic note-taking system and program that can reduce the burden on students and is useful for efficiently reviewing lecture content. [Brief explanation of the drawing]

[0014] [Figure 1] This figure schematically shows an example configuration of an electronic note-taking system for illustrating embodiments of the present invention. [Figure 2] Figure 1 is a block diagram of the electronic notebook creation system. [Figure 3]It is a flowchart showing an example of an electronic note creation method for explaining an embodiment of the present invention. [Figure 4] It is a diagram schematically showing the processing of a method for creating an electronic note for explaining an embodiment of the present invention. [Figure 5] It is a diagram schematically showing the processing of a method for creating an electronic note following FIG. 4. [Figure 6] It is a diagram schematically showing the processing of a method for creating an electronic note following FIG. 5. [Figure 7] It is a diagram schematically showing the processing of a method for creating an electronic note following FIG. 6. [Figure 8] It is a diagram schematically showing an example of the display of an electronic note for explaining an embodiment of the present invention.

Mode for Carrying Out the Invention

[0015] FIG. 1 and FIG. 2 show a configuration example of an electronic note creation system for explaining an embodiment of the present invention.

[0016] The electronic note creation system 1 includes a camera 2 and a microphone 3 for recording lectures, a server 4 for storing the video data of the lectures acquired by the camera 2 and the microphone 3, and an information terminal 5 possessed by a student attending the lectures.

[0017] The camera 2 is, for example, a digital camera using a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal - Oxide - Semiconductor) image sensor. The camera 2 photographs the board 6 on which the lecture content is written, converts it into an electrical signal, and transmits it to the server 4. The microphone 3 detects the voice spoken by the lecturer, converts it into an electrical signal, and transmits it to the server 4. The transmission and reception of the electrical signal between the camera 2 and the microphone 3 and the server 4 are performed by wire or wirelessly.

[0018] Server 4 comprises a processor 10, memory 11 such as ROM (Read Only Memory) and RAM (Random Access Memory) for storing programs executed by the processor 10 and various data used when the processor 10 executes programs, storage devices 12 such as HDD (Hard Disk Drive) and SSD (Solid State Drive) for storing various data, and a communication interface 13.

[0019] The processor 10 synchronizes the video data transmitted from the camera 2 and the audio data transmitted from the microphone 3 using common time information such as a timestamp or timecode, generates video data by combining the video data and audio data, and saves the generated video data to the storage device 12. The microphone 3 may be built into the camera 2, and if the microphone 3 is built into the camera 2, the video data may be generated by the camera 2.

[0020] Server 4 transmits video data to the student's information terminal 5 in real time during the lecture, and also transmits it to the information terminal 5 after the lecture ends in response to requests from the information terminal 5. Data is transmitted between Server 4 and the information terminal 5 via wired or wireless connection.

[0021] The information terminal 5 is, for example, a tablet terminal and comprises a processor 20, a memory 21 that stores programs executed by the processor 20 and various data used when the processor 20 executes programs, a storage device 22 that stores various data, a display device 23 such as a liquid crystal display or an OLED (Organic Electro Luminescence Diode) display, and a communication interface 24. The display device 23 is composed of a touch panel display including a touch panel and may also serve as an input means.

[0022] The display device 23 displays video based on video data received from, for example, the server 4, under the control of the processor 20, and also displays electronic notebooks created by the electronic notebook creation system 1. An example of how to create an electronic notebook using the electronic notebook creation system 1 is described below.

[0023] Figure 3 shows an example of the process for creating an electronic notebook.

[0024] The processor 10 of server 4 extracts still image data from the video data (step S1). The images represented by the still image data include objects such as text, diagrams, and tables written on the board. The processor 10 extracts objects from the extracted still image data and groups related to each other into object groups (step S2).

[0025] The information on the board is updated sequentially as the lecture progresses. Depending on the timing of still image data extraction from video data, the objects that can be extracted from still image data (text, figures, tables, etc.) change. For example, processor 10 can determine the timing of still image data extraction based on changes in the information on the board shown in the video. Specifically, processor 10 applies video analysis such as the interframe difference method to the video data to detect changes in the information on the board shown in the video. Then, if there are no changes in the information on the board for a predetermined time, it extracts still image data. Note that the lecturer may also be visible in the video. When detecting changes in the information on the board, it is desirable to use a person detection algorithm to appropriately exclude areas in the video where the lecturer is visible, so that the lecturer's movements do not have an effect. For such video analysis processing, for example, Intel's computer vision library "OpenCV" can be used.

[0026] In the process of extracting objects such as text, figures, and tables from still image data, processor 10 identifies the objects in the still image data, detects the position of each object within the image, and recognizes the text (words) contained in each object. For such object extraction processing, for example, the "Cloud Vision API," an image recognition API provided by Google, can be used.

[0027] In the process of grouping related objects into an object group, the processor 10 can determine whether or not objects are related based, for example, on the distance between them. Specifically, if the distance between objects is below a threshold, that is, if the two objects are close together in the image, the processor 10 can determine that these two objects are related. On the other hand, if the distance between objects exceeds a threshold, that is, if the two objects are far apart in the image, the processor 10 can determine that these two objects are not related.

[0028] The threshold used to determine whether or not objects are related can be a fixed value or it can be changed dynamically. For example, the threshold can be set lower when the variation in distance between objects is relatively small, and higher when the variation in distance between objects is relatively large. By changing the threshold according to the balance of object placement in this way, it is possible to determine whether or not objects are related while taking into account the instructor's specific tendencies regarding how the board is used, and to generate object groups more appropriately.

[0029] If a first object group is generated based on the first still image data, and the second still image data following the first still image data includes other objects in addition to the objects that make up the first object group, the processor 10 determines whether or not to add the other objects to the first object group based on the distance between the other objects and the objects that make up the first object group. That is, if the distance between the other objects and any of the objects in the first object group is below a threshold, the processor 10 adds the other objects to the first object group. On the other hand, if the distance between the other objects and all the objects in the first object group exceeds a threshold, the processor 10 treats the other objects as objects belonging to a second object group independent of the first object group.

[0030] In parallel with extracting still image data and generating object groups, processor 10 generates text data from audio data contained in video data and divides the generated text data into sentences (step S3). For the process of generating text data and dividing it into sentences, for example, the "Speach-to-Text API" provided by Google for speech recognition and speech-to-text conversion can be used.

[0031] Next, the processor 10 associates the sentence derived from the audio data generated in step S3 with the object group generated in step S2 (step S4).

[0032] In the process of associating sentences derived from audio data with object groups, the processor 10 can use, for example, the time information of the object group and the time information of the sentences derived from the audio data. The time information of the object group is the time information of the still image data from which the group of objects constituting the object group was extracted, and the time information of the still image data represents the position on the time axis in the video data. When an object is added to an already generated object group and the object group is updated, the time information of that object group is the time information of the original object group. The time information of the sentences derived from audio data represents the position on the time axis in the video data. The processor 10 identifies sentences whose position on the time axis corresponds to the position on the time axis of the object group, and associates the identified sentences with the object group.

[0033] While the position of an object group on the time axis is fixed at a single point in time, a sentence derived from audio data has a certain length on the time axis. The statement that the position of a sentence derived from audio data on the time axis "corresponds" to the position of an object group on the time axis means that the position of the object group on the time axis is included within the range of the length of the sentence derived from audio data on the time axis.

[0034] Furthermore, in the process of associating sentences derived from audio data with object groups, processor 10 can use the words contained in both the object groups and the sentences derived from audio data. For example, if at least one object belonging to the object group contains the word "ABC", processor 10 recognizes this. Also, processor 10 recognizes the words contained in the sentences derived from audio data. Then, it associates the sentences derived from audio data that contain the word "ABC" with the object groups.

[0035] Here, a first object group is generated based on the first still image data, and a second object group is generated based on the second still image data following the first still image data, and both the first and second object groups contain the phrase "ABC". The second time information of the second object group is temporally later than the first time information of the first object group. In this case, whether a sentence derived from audio data containing the phrase "ABC" is associated with the first or second object group can be determined based on the time information of that sentence. Specifically, if the time information of a sentence derived from audio data containing the phrase "ABC" is earlier than the second time information, the sentence should be associated with the first object group. On the other hand, if the time information of the sentence is later than or equal to the second time information, the sentence should be associated with the second object group.

[0036] Sentences derived from audio data that do not fall under either the temporal information-based or word-based linking methods described above, and whose object group to which they belong is undetermined, may be linked to an appropriate object group in relation to other sentences linked to that object group. For example, if natural language processing techniques such as Sentence-BERT, which evaluate the degree of relevance between sentences, are applied and the undetermined sentence is evaluated as being related to the preceding and succeeding sentences linked to the object group, then that undetermined sentence can be linked to the same object group. Furthermore, if an undetermined sentence is sandwiched between two sentences linked to the same object group on the time axis, that undetermined sentence can also be linked to the same object group.

[0037] Processor 10 creates note data describing each object group, with the object group and the statements associated with the object group as a single dataset (step S5).

[0038] The note data created in step S5 is sent from server 4 to information terminal 5. The processor 20 of information terminal 5 displays images and text on the display device 23 based on the note data (step S6). On the display device 23, object groups and the sentences associated with these object groups are displayed as a single unit according to the description in the note data.

[0039] A student operating the information terminal 5 can edit the display on the display device 23, i.e., the electronic notebook. Examples of edits include changing the layout of object groups and sentences associated with object groups, changing text within sentences, and adding text and shapes entered via input means such as a touch panel. Changing text within sentences includes adding highlights and underlines, and also includes summarizing the text. Text summarization may be done by the student themselves, or it may be an automatic summary using a generation AI such as ChatGPT provided by OpenAI. The processor 20 saves the notebook data reflecting the edits as the student's personal electronic notebook in the storage device 22 of the information terminal 5 (step S7).

[0040] The transmission of note data from server 4 to information terminal 5 may be performed automatically after the lecture ends, or all at once upon request from information terminal 5. Alternatively, during the lecture, the differential data may be sent from server 4 to information terminal 5 each time the note data is updated. When note data (differential data) is sent from server 4 to information terminal 5 during the lecture, students can edit the electronic notes displayed on the display device 23 of information terminal 5 as needed.

[0041] Figures 4 to 7 schematically illustrate the process of creating the electronic notes described above, using a mathematics lecture as an example. In Figures 4 to 7, the video data includes time t nThe frame images (n=1,2,3,...) are arranged in chronological order. The audio data is represented as a waveform, and the position of each frame image of the video data is indicated on its time axis. The text data is generated from the audio data and is divided into sentences.

[0042] Figure 4 shows the process from when the instructor writes the problem statement and diagram on the board, then explains the guidelines for solving the problem to the students, and finally when the students begin solving the problem. Frame image Frm1 at time t1 shows the problem statement Obj1, and frame image Frm2 at time t2 shows the problem statement Obj1 in addition to diagram Obj2. During this time, the information on the board shown in the video changes sequentially, and the processor 10 of server 4 does not extract still image data.

[0043] Meanwhile, from time t2 onward, while the instructor explains the solution guidelines to the students, the information on the board displayed in the video remains unchanged. At time t3, a predetermined time (e.g., 10 seconds) has elapsed since time t2, the processor 10 extracts the frame image Frm3 from time t3 as still image data. From the extracted still image data, the processor 10 extracts the problem statement Obj1 and figure Obj2, and recognizes the text (words) contained in the problem statement Obj1 and figure Obj2.

[0044] Furthermore, the processor 10 determines the relationship between the problem statement Obj1 and the figure Obj2 based on the distance between the problem statement Obj1 and the figure Obj2. The distance between the problem statement Obj1 and the figure Obj2 can be defined using the bounding boxes of rectangles surrounding each object. Except when one bounding box encloses or partially overlaps the other, the distance between the two opposing sides of the two bounding boxes can be used as the distance between the problem statement Obj1 and the figure Obj2. If one bounding box encloses or partially overlaps the other, the processor 10 can determine that the problem statement Obj1 and the figure Obj2 are related to each other. In the example in Figure 4, the distance between the problem statement Obj1 and the figure Obj2 is below the threshold, so the processor 10 determines that the problem statement Obj1 and the figure Obj2 are related to each other and generates a first object group Gr1 consisting of the problem statement Obj1 and the figure Obj2.

[0045] In parallel with the extraction of still image data and the generation of object groups, processor 10 generates text data from audio data and divides the generated text data into sentences. In the example in Figure 4, three sentences Txt1-Txt3 are generated. When processor 10 generates the first object group Gr1 at time t3, it appropriately associates sentence Txt1, which was generated before time t3, with the first object group Gr1 using time information and words as described above.

[0046] In the example in Figure 4, the position on the time axis indicated by the time information of the first object group Gr1 (time t3) falls within the length range on the time axis indicated by the time information of statement Txt1. In other words, the time information of statement Txt1 corresponds to the time information of the first object group Gr1. Therefore, according to the time information, statement Txt1 is associated with the first object group Gr1.

[0047] Furthermore, in the example in Figure 4, problem statement Obj1, belonging to the first object group Gr1, contains the words "ABC", "circumcenter", "O", "angle", and "α", and figure Obj2 contains the words "A", "B", "C", "O", "70°", "20°", and "α". Sentence Txt1 contains the word "circumcenter". Therefore, based on the words, sentence Txt1 is associated with the first object group Gr1. Note that in Figure 4, words in sentences derived from audio data that are common with words in the first object group Gr1 are shown in white text.

[0048] Figure 5 shows the process that follows Figure 4, while the instructor is explaining the problem. Frame image Frm4 at time t4 shows auxiliary lines Obj3 added to figure Obj2 of the generated first object group Gr1. Frame image Frm5 at time t5 shows the solution text Obj4. During this time, the writing on the board shown in the video changes sequentially, and processor 10 does not extract still image data.

[0049] After the time t5 in which the answer statement Obj4 is written, the writing on the board shown in the video does not change. At time t6, a predetermined time (for example, 10 seconds) has elapsed since time t5, the processor 10 extracts the frame image Frm6 from time t6 as still image data. Then, from the extracted still image data, it extracts the generated first object group Gr1, the auxiliary line Obj3, and the answer statement Obj4.

[0050] Since part of the bounding box of the auxiliary line Obj3 overlaps with the bounding box of the first object group Gr1, the processor 10 adds the auxiliary line Obj3 to the first object group Gr1. Furthermore, since the distance between the auxiliary line Obj3 and the solution statement Obj4 is below a threshold, the processor 10 also adds the solution statement Obj4 to the first object group Gr1.

[0051] When processor 10 updates the first object group Gr1 at time t6, it appropriately associates sentences derived from audio data generated before time t6 with the first object group Gr1. Here, sentence Txt1 that is already associated with the first object group Gr1 may be excluded from the association process.

[0052] In the example in Figure 5, statements Txt4-Txt8 each contain words that are included in the first object group Gr1. Therefore, the processor 10 associates statements Txt4-Txt8 with the first object group Gr1.

[0053] Figure 6 shows the process following Figure 5, from when the instructor writes the problem statement and diagram of a similar problem on the board until the student begins solving the problem. Frame image Frm7 at time t7 shows the first object group Gr1 in addition to the problem statement Obj5, and frame image Frm8 at time t8 shows the first object group Gr1 and the problem statement Obj5 in addition to the diagram Obj6. During this time, the writing on the board shown in the video changes sequentially, and processor 10 does not extract still image data.

[0054] After time t8, when Figure Obj6 is completed, the writing on the board shown in the video does not change. At time t9, a predetermined time (e.g., 10 seconds) has elapsed since time t8, processor 10 extracts the frame image Frm9 from time t9 as still image data. From the extracted still image data, it then extracts the generated first object group Gr1, the problem statement Obj5, and Figure Obj6.

[0055] Since the problem statement Obj5 and the first object group Gr1 are far apart, and the figure Obj6 and the first object group Gr1 are also far apart, the processor 10 does not add the problem statement Obj5 and the figure Obj6 to the first object group Gr1. On the other hand, the problem statement Obj5 and the figure Obj6 are close to each other. Therefore, the processor 10 generates a second object group Gr2 consisting of the problem statement Obj5 and the figure Obj6, separately from the first object group Gr1.

[0056] When processor 10 generates the second object group Gr2 at time t9, it appropriately associates sentences derived from audio data generated before time t9 with the first object group Gr1 and the second object group Gr2. Here, sentences Txt1 and Txt4-Txt8, which are already associated with the first object group Gr1, may be excluded from the association process.

[0057] In the example in Figure 6, the time information of statements Txt2, Txt3, and Txt9 does not correspond to the time information of the first object group Gr1 and the second object group Gr2. Furthermore, statements Txt2, Txt3, and Txt9 do not contain any words included in the first object group Gr1 or the second object group Gr2. Therefore, the processor 10 does not associate statements Txt2, Txt3, and Txt9 with the first object group Gr1, nor with the second object group Gr2.

[0058] Figure 7 shows the process that follows Figure 6, while the instructor is explaining a similar problem. The frame image Frm10 at time t10 shows auxiliary lines Obj7 added to Figure Obj6 of the generated second object group Gr2. The frame image Frm11 at time t11 shows the solution text Obj8. During this time, the writing on the board shown in the video changes sequentially, and processor 10 does not extract still image data.

[0059] After the time t11 when the answer statement Obj8 is finished writing, the writing on the board shown in the video does not change. At time t12, a predetermined time (for example, 10 seconds) has elapsed from time t11, the processor 10 extracts the frame image Frm12 at time t12 as still image data. Then, from the extracted still image data, it extracts the generated first object group Gr1, the generated second object group Gr2, the auxiliary line Obj7, and the answer statement Obj8.

[0060] Since part of the bounding box of the auxiliary line Obj7 overlaps with the bounding box of the second object group Gr2, processor 10 adds the auxiliary line Obj7 to the second object group Gr2. Furthermore, since the distance between the auxiliary line Obj7 and the solution statement Obj8 is below a threshold, processor 10 also adds the solution statement Obj8 to the second object group Gr2.

[0061] When processor 10 updates the second object group Gr2 at time t12, it appropriately associates sentences derived from audio data generated before time t12 with the first object group Gr1 and the second object group Gr2. Here, sentences Txt1 and Txt4-Txt8, which are already associated with the first object group Gr1, may be excluded from the association process.

[0062] In the example in Figure 7, statements Txt10-Txt14 each contain words included in the second object group Gr2. However, for example, the word "circumcenter" in statement Txt10 is also included in the first object group Gr1. The processor 10 determines which object group to associate statement Txt10 with based on the time information of the first object group Gr1, the time information of the second object group Gr2, and the time information of statement Txt10.

[0063] As described above, the time information of an object group is the time information of the still image data from which the objects constituting the object group were extracted. When an object is added to an already generated object group and the object group is updated, the time information of that object group is the time information of the original object group. Therefore, the time information of the first object group Gr1 is time t3 (Figure 4), and the time information of the second object group Gr2 is time t9 (Figure 6). Since the time information of statement Txt10 is from time t9 onwards, statement Txt10 is associated with the second object group Gr2. The other statements Txt11-Txt14 are similarly associated with the second object group Gr2.

[0064] As a result, note data is created in which statements Txt1 and Txt4-8 are associated with the first object group Gr1, and statements Txt10-Txt14 are associated with the second object group Gr2.

[0065] Figure 8 shows an example of how note data is displayed.

[0066] On the display device 23 of the information terminal 5, object groups and the statements associated with these object groups are displayed as a single unit, according to the description in the note data. In the example in Figure 8, the first object group Gr1 and the statements Txt1, Txt4-Txt8 associated with the first object group Gr1 are displayed on a single screen using images and text. It is preferable that the statements Txt1, Txt4-Txt8 are arranged in chronological order using their time information. By turning the page, the second object group Gr2 and the statements Txt10-Txt14 associated with the second object group Gr2 are also displayed in the same manner.

[0067] In addition to object groups and sentences associated with object groups, text data generated from audio data may also be displayed on the display device 23. When a sentence associated with an object group is selected, the time information of the selected sentence is used to display the corresponding portion of the text data as well as the surrounding portions, allowing learners to efficiently confirm the meaning of the selected sentence in accordance with the flow of the lecture. The text data may be displayed on the display device 23 at all times, or it may be displayed, for example, when a sentence associated with an object group is selected.

[0068] Students operating the information terminal 5 can check the display on the display device 23 and make edits such as changing the layout of object groups and sentences, changing text within sentences, and adding text and shapes entered via input means such as a touch panel. The edited note data is saved on the information terminal 5 as the student's personal electronic notebook.

[0069] In this way, by generating object groups from the video data of the lecture, taking into account the relationships between the items written on the board, and by generating text data from the audio data of the lecture, dividing the text data into sentences, and linking each sentence to a related object group, and by displaying the object groups and the sentences linked to these object groups as a single unit, students can comprehensively and efficiently review the lecture content without having to watch the lecture video.

[0070] In the above explanation, it was assumed that the server 4's processor 10 performs the generation of object groups from video data, the generation of text data from audio data and the division of text data into sentences, and the linking of sentences to object groups. However, some or all of these processes may be performed by the information terminal 5's processor 20. Furthermore, if the information terminal 5 is equipped with a camera and microphone, the information terminal 5 may acquire the lecture's video and audio data, and the information terminal 5's processor 20 may perform all processing without going through the server 4. [Industrial applicability]

[0071] This invention can, for example, provide information assurance to people with hearing impairments as an alternative to note-taking by a third party, and can also encourage participation in lectures by people with hearing impairments. [Explanation of Symbols]

[0072] 1. Electronic Notebook Creation System 2 cameras 3 Microphone 4 servers 5. Information terminals 6 boards 10 processors 11 memory 12 Storage device 13 Communication Interface 20 processors 21 memory 22 Storage device 23 Display device 24 Communication Interfaces Gr1 First Object Group Gr2 Second Object Group Txt1-Txt14 sentences

Claims

1. It is equipped with a processor that processes video data of a board on which lecture content is written, The aforementioned processor, Still image data is extracted from the aforementioned video data. From the aforementioned still image data, text, figures, and table objects are extracted, and groups of related objects are grouped together as object groups. Text data is generated from the audio data contained in the aforementioned video data. The aforementioned text data is divided into sentences, The sentence derived from the audio data related to the object group is associated with the object group, The object group and the statement associated with the object group are displayed on the display device as a single unit. Electronic note-taking system.

2. An electronic notebook creation system according to claim 1, The aforementioned processor, Recognizing the words and phrases included in the aforementioned object group, At a minimum, sentences containing the phrase from the audio data are associated with the object group as sentences related to the object group. Electronic note-taking system.

3. An electronic notebook creation system according to claim 2, The object group and the sentence derived from the audio data each have time information representing their position on the time axis in the video data, The aforementioned processor, If another object group having second time information that is later in time than the first time information possessed by the aforementioned object group includes the aforementioned phrase, Among the sentences derived from the audio data, sentences containing the phrase and having time information that is earlier in time than the second time information of the other object group are associated with the object group. Among the sentences derived from the audio data, sentences containing the aforementioned phrases and having time information from the second time information onwards in the other object group are linked to the other object group. Electronic note-taking system.

4. The processor that processes the video data of the board on which the lecture content is written performs the following: The steps include extracting still image data from the aforementioned video data, The steps include extracting text, figures, and tables from the still image data and storing related object groups as object groups, The steps include generating text data from audio data contained in the aforementioned video data, The steps include dividing the aforementioned text data into sentences, The steps include: associating the sentence derived from the audio data, which is related to the object group, with the object group; The steps include displaying the object group and the statement associated with the object group as a single unit on a display device, A program that causes the processor to execute an electronic notebook creation method comprising the above-mentioned components.

Images