Information processing device, generation method, and program

Abstract

The present technology relates to an information processing device, a generating method, and a program for making it possible to edit or play back a video containing a lecture in an appropriate manner. The information processing device of the present technology is provided with a generating unit which generates playback-assisting information in accordance with importance determined on the basis of information about a lecture with respect to each of predetermined sections into which data including a video and audio of the lecture is divided. The present technology may be applied to, for example, a lecture capture system used for filming a lecture.

Description

[Technical Field]

[0001] The present technology relates to an information processing device, a generation method, and a program, and more particularly to an information processing device, a generation method, and a program that enable recorded lecture videos to be edited or played back in an appropriate manner. [Background technology]

[0002] In recent years, opportunities to record lectures have increased in the educational field. When recording lecture videos, it is necessary to efficiently record lecture videos by editing the entire lecture video, such as by deleting sections that are not important for learning.

[0003] For example, Patent Document 1 describes a technology that evaluates importance based on the number of comments made in each section of a video segmented based on the time at which a specific person spoke, the number of participants in the discussion, the length of the discussion, volume, gestures, emotions, etc., and edits sections of low importance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-46705 Summary of the Invention [Problem to be solved by the invention]

[0005] When the technology described in Patent Document 1 is applied to editing recorded lecture videos, importance determination is performed based on information linked to the teacher, such as the time of the teacher's speech, volume, gestures, and emotions. If the recorded lecture video is edited based on the importance determined in this way, the section of the video in which the teacher is writing on the blackboard may be determined to be of low importance, and information about the order in which the blackboard writing was performed, which is considered important for learning, may be lost from the recorded lecture video.

[0006] This technology was developed in light of these circumstances, and makes it possible to edit or play back recorded lecture videos in an appropriate manner. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present technology is an information processing device that includes a generation unit that generates playback assistance information for each of predetermined sections that divide data containing video and audio of a lecture, in accordance with the importance determined based on information about the lecture.

[0008] One aspect of the generation method of the present technology is a generation method that generates playback assistance information for each specified section that divides data containing video and audio of a lecture, based on the importance determined based on information about the lecture.

[0009] One aspect of the program of the present technology is a program for causing a computer to execute a process of generating information for playback assistance for each of specified sections divided into data containing video and audio of a lecture, in accordance with the importance determined based on information about the lecture.

[0010] In one aspect of the present technology, playback assistance information is generated for each of specified sections divided into data containing video and audio of a lecture, according to the importance determined based on information about the lecture. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an external appearance of an imaging system according to an embodiment of the present technology. [Figure 2] FIG. 1 is a block diagram illustrating an example of the configuration of an imaging system. [Figure 3] FIG. 2 is a block diagram illustrating an example of a functional configuration of a calculation device. [Figure 4] FIG. 10 is a diagram illustrating an example of an importance determination rule. [Figure 5] FIG. 10 is a diagram illustrating an example of an editing rule. [Figure 6]FIG. 10 is a diagram showing an example of a timeline of recorded lecture video. [Figure 7] FIG. 10 is a diagram showing an example of a timeline of recorded lecture video. [Figure 8] FIG. 10 is a diagram showing an example of a timeline of recorded lecture video. [Figure 9] FIG. 10 is a diagram illustrating an example of an importance determination rule. [Figure 10] FIG. 10 is a diagram illustrating an example of the importance of each piece of analysis information determined for each determination section. [Figure 11] FIG. 10 is a diagram showing an example of a timeline of edited video data. [Figure 12] 10 is a flowchart illustrating processing performed by a computing device. [Figure 13] FIG. 10 is a diagram showing the relationship between the change in the amount of writing on the board over time and the importance level. [Figure 14] FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present technology will be described in the following order. 1. Configuration of an imaging system according to an embodiment of the present technology 2. Example of editing video data 3. Operation of the computing device 4. Variations 5. About Computers

[0013] <1. Configuration of imaging system according to one embodiment of the present technology> Example of a shooting system configuration FIG. 1 is a diagram showing the appearance of an imaging system according to an embodiment of the present technology.

[0014] The imaging system is configured as a lecture capture system and is installed in a classroom or auditorium where a teacher U1 gives a lecture to students U2.

[0015] FIG. 1 shows a classroom (lecture room) in which a teacher (lecturer) U1 is giving a lecture using a whiteboard WB, and a student (audience) U2 is listening to the lecture.

[0016] Teacher U1 is the person who gives the lecture, and during the lecture he explains the lecture while writing on the whiteboard WB.

[0017] Writing is made and erased on the whiteboard WB according to the lecture explanation. The writing is not limited to one color, but multiple colors are used. In Figure 1, characters shown with solid lines on the surface of the whiteboard WB represent characters written with a black pen (a pen with black ink), and characters shown with dotted lines represent characters written with a red pen (a pen with red ink).

[0018] Student U2 is a person attending the lecture, and speaks during the lecture and comes to the front to write on the board. Note that the lecture may be filmed in a dedicated studio or other location where student U2 is not present. Alternatively, the lecture may be filmed in a classroom with multiple students attending the lecture.

[0019] The video capture device 1 is installed in a lecture room and captures images at a field of view that captures the teacher U1 and the whiteboard WB. Video data including video signals and audio signals representing the captured images are output to the calculation device 2.

[0020] The arithmetic unit 2 receives the video data supplied from the video shooting device 1 and determines the importance level based on the video signal and the sound signal. The arithmetic unit 2 edits the video data based on the result of the importance level determination.

[0021] FIG. 2 is a block diagram showing an example of the configuration of the imaging system.

[0022] The imaging system of FIG. 2 is made up of a video imaging device 1, a computing device 2, a recording device 3, and an input / output device 4.

[0023] The video capture device 1 is configured as, for example, a camera that captures images at an angle of view that captures both the teacher U1 and the whiteboard WB at the same time. Video data representing the captured image is output to the calculation device 2. The number of video capture devices 1 is not limited to one, and multiple devices may be provided.

[0024] The arithmetic device 2 is configured as an information processing device that receives video data supplied from the video shooting device 1 and performs importance determination based on the video data. The arithmetic device 2 is connected to the video shooting device 1 via wired or wireless communication. The arithmetic device 2 edits the video data based on the result of the importance determination and outputs the edited video data to the recording device 3 and the input / output device 4.

[0025] The arithmetic unit 2 may be configured with dedicated hardware having each function, or may be configured with a general computer with each function realized by software. Furthermore, the arithmetic unit 2 and the video shooting device 1 may be configured as a single integrated device, rather than as independent devices.

[0026] The recording device 3 records the video data supplied from the arithmetic device 2. The recording device 3 and the arithmetic device 2 may be integrated into one device rather than being configured as independent devices. The recording device 3 may also be connected to the arithmetic device 2 via a network.

[0027] The input / output device 4 is configured by a keyboard and a mouse that accept user operations, a display with a display function, a speaker with a sound output function, etc. The display with a display function may be provided with a touch panel function.

[0028] The input / output device 4 receives instructions based on user operations and outputs a rule signal representing the user's instructions to the arithmetic device 2. For example, the user may instruct an importance determination rule indicating what information is used to determine the importance, and an editing rule indicating what editing is to be performed based on the result of the importance determination.

[0029] Furthermore, the input / output device 4 presents data including the video signal and the audio signal supplied from the arithmetic device 2 to the user.

[0030] The input / output device 4 and the arithmetic device 2 may be configured as a single integrated device rather than as independent devices. The input / output device 4 may also be connected to the arithmetic device 2 via a network.

[0031] Example of functional configuration of arithmetic unit 2 FIG. 3 is a block diagram showing an example of the functional configuration of the arithmetic device 2.

[0032] The arithmetic device 2 in FIG. 3 is made up of a video input unit 101, a video analysis unit 102, a sound analysis unit 103, a control parameter input unit 104, an importance determination unit 105, an automatic editing execution unit 106, and a video output unit 107.

[0033] The video input unit 101 receives at least one piece of video data supplied from the video shooting device 1. As described above, the video data consists of a video signal and an audio signal. The video input unit 101 supplies a video signal representing the video shot by the video shooting device 1 to the video analysis unit 102, and outputs an audio signal representing the audio collected in the lecture room to the audio analysis unit 103.

[0034] The video analysis unit 102 analyzes at least one type of video information (information representing a video related to a lecture) based on the video signal supplied from the video input unit 101. For example, the video analysis unit 102 analyzes information about the teacher's behavior, the students' behavior, the content of the blackboard writing, the amount of increase or decrease in the number of characters written on the blackboard, the color of the characters written on the blackboard, materials attached to the whiteboard, and the like as video information.

[0035] The video analysis unit 102 outputs the analysis result of the video information and the video signal to the importance determination unit 105 .

[0036] The sound analysis unit 103 analyzes at least one type of sound information (information representing sounds related to a lecture) based on the sound signal supplied from the video input unit 101. For example, information about the teacher's voice, the students' voice, and the sound of a chime is analyzed as sound information by the sound analysis unit 103. Note that, hereinafter, when there is no need to distinguish between video information and sound information, the video information and sound information will be collectively referred to as analyzed information.

[0037] The sound analysis unit 103 outputs the analysis result of the sound information and the sound signal to the importance determination unit 105 .

[0038] The control parameter input unit 104 receives a rule signal representing an importance determination rule and a rule signal representing an editing rule, both of which are supplied from the input / output device 4 .

[0039] FIG. 4 is a diagram illustrating an example of the importance determination rule.

[0040] As shown in Figure 4, the user can specify rules for determining the importance of video information, such as "if the teacher is facing forward (towards the back of the classroom), the importance is high," "if the teacher is writing on the blackboard, the importance is low," "if a student is writing on the blackboard, the importance is high," "if a red pen is used to write on the blackboard, the importance is high," and "if the amount of writing on the blackboard decreases, the importance is low."

[0041] In addition, the user may specify rules for determining the importance of sound information, such as "if the teacher is giving an explanation, the importance is high," "if a student is asking a question, the importance is high," or "if the bell rings, the importance is high."

[0042] FIG. 5 is a diagram showing an example of editing rules.

[0043] As shown in Figure 5, the editing rules specified by the user are "delete parts whose importance is lower than the threshold," "compress parts whose importance is lower than the threshold at a high compression rate," and "delete parts in order of decreasing importance so that the length of the lecture recording video is 30 minutes."

[0044] The control parameter input unit 104 in FIG. 3 outputs a rule signal representing the importance determination rule as described above to the importance determination unit 105, and outputs a rule signal representing the editing rule to the automatic editing execution unit .

[0045] The importance determination unit 105 performs importance determination based on the analysis results of the video information supplied from the video analysis unit 102 and the analysis results of the sound information supplied from the sound analysis unit 103 in accordance with the rule signal supplied from the control parameter input unit 104.

[0046] The importance is not determined as a unique value for the entire video data, but as a value for each section into which the video data is divided into short time periods.

[0047] There are various methods for dividing video data. For example, there are a method for dividing video data every predetermined time (e.g., 5 seconds), a method for dividing video data based on the teacher's voice (e.g., sound pressure), a method for recognizing the tip of the pen used for writing on the whiteboard and dividing the video data when the tip of the pen leaves the surface of the whiteboard for a predetermined time, a method for dividing video data based on the increase or decrease in the amount of characters written on the whiteboard, etc. Note that these division methods may be combined to divide the video data.

[0048] The importance determining unit 105 determines the importance of each section into which the video data is divided, not by a binary value such as important or unimportant, but by a value between -1.0 and 1.0, for example.

[0049] The importance of a determination section, which is a section that combines multiple consecutive sections whose importance has been determined, may be further determined. In this case, the importance of the determination section may be determined as the average, maximum, or minimum value of the importance of each section included in the determination section, or a weighted sum according to the duration of each section.

[0050] When the importance is determined based on the analysis results of multiple types of analysis information, the final importance is determined to be one of the average, maximum, minimum, sum, product, or weighted sum according to the weight represented by the rule signal of the importance determined based on the analysis results of each type of analysis information.

[0051] The number of sections to be grouped together as the judgment section may be, for example, a preset number of sections. The judgment section may be a number of sections set based on the teacher's voice, a number of sections set based on the recognition result of the pen tip, or a number of sections set based on the increase or decrease in the amount of characters written on the blackboard.

[0052] The importance determination unit 105 outputs the video data, which is a compilation of the video signal supplied from the video analysis unit 102 and the audio signal supplied from the audio analysis unit 103, as well as the result of the importance determination to the automatic editing execution unit 106.

[0053] The automatic editing execution unit 106 edits the video data based on the result of the importance determination by the importance determination unit 105, in accordance with the rule signal supplied from the control parameter input unit 104. The video data edited by the automatic editing execution unit 106 is output to the video output unit 107.

[0054] The video output unit 107 outputs the video data supplied from the automatic editing execution unit 106 to the recording device 3 and the input / output device 4 .

[0055] <2. Example of editing video data> The following describes an example of editing video data obtained by recording a lecture in the classroom described with reference to Fig. 1. Here, it is assumed that a 120-minute lecture was held in the classroom shown in Fig. 1.

[0056] 6 to 8 are diagrams showing examples of timelines of recorded lecture videos.

[0057] 6 to 8, the video data of the lecture video is divided into 12 judgment intervals, 1 to 12, in chronological order. Each judgment interval is 10 minutes long. Representative screenshots and text representing the sound content are shown for each judgment interval.

[0058] As shown in the upper left of Figure 6, the video of judgment section 1 shows teacher U1 standing in front of a whiteboard WB. No writing is being done on the whiteboard WB. A typical sound recorded in judgment section 1 is the sound of a bell.

[0059] As shown in the upper right of Figure 6, the video of judgment section 2 shows teacher U1 writing on the left side of the whiteboard WB with a black pen. A typical sound recorded in judgment section 2 is the sound of the teacher writing on the board.

[0060] As shown in the lower left of Figure 6, the video of judgment section 3 shows teacher U1 explaining what is written on the whiteboard WB. The representative sound recorded in judgment section 3 is the voice of teacher U1.

[0061] As shown in the lower right of Figure 6, the video of judgment section 4 shows teacher U1 writing on the upper right side of the whiteboard WB with a red pen. A typical sound recorded in judgment section 4 is the sound of writing on the board.

[0062] As shown in the upper left of Figure 7, the video of judgment section 5 shows teacher U1 explaining to student U2 a question. Representative sounds recorded in judgment section 5 include the voice of teacher U1 and the voice of student U2 asking a question.

[0063] As shown in the upper right of Figure 7, the video of judgment section 6 shows teacher U1 explaining while writing on the bottom right side of the whiteboard WB with a black pen. Teacher U1 has written a chemical formula on the whiteboard WB. Typical sounds recorded in judgment section 6 include the sound of writing on the board and the voice of teacher U1.

[0064] As shown in the lower left of Figure 7, the video of determination section 7 shows teacher U1 erasing the writing on the left side of the whiteboard WB. A typical sound recorded in determination section 7 is the sound of erasing the writing.

[0065] As shown in the lower right of Figure 7, the video of the determination section 8 shows the teacher U1 giving a lecture. The representative sound recorded in the determination section 8 is the voice of the teacher U1.

[0066] As shown in the upper left of Figure 8, the video of decision section 9 shows student U2 chatting with teacher U1 and the whiteboard WB. A typical sound recorded in decision section 9 is the chatting voice of student U2.

[0067] As shown in the upper right of Fig. 8, the video of the determination section 10 shows student U2 writing on the lower left side of the whiteboard WB with a black pen. A typical sound recorded in the determination section 10 is the sound of writing on the board.

[0068] As shown in the lower left of Figure 8, the video of the judgment section 11 shows teacher U1 explaining what student U2 wrote on the whiteboard WB. Typical sounds recorded in the judgment section 11 include the voice of teacher U1 and the voice of student U2 chatting.

[0069] As shown in the lower right of Figure 8, the video of the judgment section 12 shows teacher U1 giving a summary of the lecture. Representative sounds recorded in the judgment section 12 are the voice of teacher U1 and the sound of a bell.

[0070] The video analysis unit 102 and the sound analysis unit 103 analyze the video information and sound information for each of the 12 judgment sections. Here, the video information analyzed includes the teacher's movements, the direction of the teacher's face, the students' movements, the color of the blackboard writing, the increase or decrease in the amount of writing on the blackboard, and the content of the writing on the blackboard. Furthermore, the sound information analyzed includes the content of the teacher's voice, the volume of the teacher's voice, the tone of the teacher's voice, the questions students ask, the chatter of students, the chime, the content sound, and the sound of the writing on the blackboard.

[0071] The analysis of video and audio information is carried out using existing methods. For example, it is possible to distinguish between teachers and students using image-based or voiceprint-based personal identification, or it is possible to recognize the content of blackboard writing by combining a blackboard extraction function with OCR (Optical Character Recognition).

[0072] The importance determination unit 105 determines the importance of each of the 12 determination sections based on the analysis results of the video information and the audio information. Specifically, the importance determination unit 105 determines the importance of each piece of analysis information in each section according to the importance determination rules. For example, the video data is divided into sections every 5 seconds.

[0073] Thereafter, the importance determination unit 105 groups together 120 consecutive sections into one determination section, and determines the average value of the importance of each piece of analysis information in each of the 120 sections as the importance of the determination section.

[0074] FIG. 9 is a diagram illustrating an example of the importance determination rule.

[0075] As shown in Figure 9, the importance determination rules for video information are as follows: for the teacher's movements, "if the teacher's movements are greater than a certain size, the importance is 1.0"; and for the direction of the teacher's face, "if the teacher is facing forward, the importance is 1.0."

[0076] Furthermore, as a rule for determining the importance of video information, the importance of student movements is determined according to the rule that "if a student is in the field of view, the importance is 1.0."

[0077] Furthermore, the importance determination rules for video information are as follows: for the color of the writing on the board, "if the color of the writing on the board is red, the importance is 1.0"; for the increase or decrease in the amount of writing on the board, "if the amount of writing on the board is increasing, the importance is 1.0"; and for the content of the writing on the board, "if the writing on the board is a chemical formula, the importance is 1.0."

[0078] The importance determination rules for sound information are as follows: for the volume of the teacher's voice, "if the volume of the teacher's voice is above a certain level, the importance is 1.0," and for the tone of the teacher's voice, "if the tone of the teacher's voice is emotional, the importance is 1.0."

[0079] In addition, the importance determination rules for sound information are as follows: for questions heard from students, the importance is determined as follows: "If a student is asking a question, the importance is 1.0"; for chattering voices heard from students, the importance is determined as follows: "If a student is chatting, the importance is -1.0."

[0080] Furthermore, the importance determination rules for sound information are as follows: for chimes, "if the chime is ringing, the importance is 1.0"; for content, "if the sound of video material (content) is ringing, the importance is 1.0"; for the sound of writing on the blackboard, "if there is the sound of writing on the blackboard, the importance is -0.5"; and for the sound of erasing writing on the blackboard, the importance is -1.0.

[0081] FIG. 10 is a diagram showing an example of the importance of each piece of analysis information determined for each determination section.

[0082] As shown in Figure 10, for each judgment section, the importance of the teacher's movements, the direction of the teacher's face, the student's movements, the color of the blackboard writing, the increase or decrease in the amount of blackboard writing, the content of the blackboard writing, the content of the teacher's voice, the volume of the teacher's voice, the tone of the teacher's voice, the questions asked by the students, the chattering voices of the students, the chime, the content sounds, and the sounds of the blackboard writing are judged.

[0083] For example, for judgment section 1, the importance of the teacher's movements is 0.3, the importance of the teacher's facial direction is 0.9, the importance of student movements is 0, the importance of the color of the blackboard writing is 0, the importance of the increase or decrease in the amount of blackboard writing is 0, the importance of the content of the blackboard writing is 0, the importance of the content of the teacher's voice is 0, the importance of the volume of the teacher's voice is 0, the importance of the tone of the teacher's voice is 0, the importance of questions spoken by students is 0, the importance of casual conversation spoken by students is 0, the importance of the bell is 1.0, the importance of content sounds is 0, and the importance of blackboard sounds is 0.

[0084] Similarly, the importance of each of the judgment sections 2 to 12 is judged.

[0085] As described above, the importance determination unit 105 calculates the sum of the importance determined for each piece of analysis information in each determination section as the final importance.

[0086] 10, the final importance of each of the judgment intervals 1 to 12 is calculated as 2.2, 0.7, 1.9, 2.1, 2.4, 2.5, -0.9, 1.7, 1.6, 2.5, 1.6, and 2.2, as shown in the lower part of Fig. 10. The final importance ranking is as follows: 1st place: judgment interval 6 and judgment interval 10, 3rd place: judgment interval 5, 4th place: judgment interval 1 and judgment interval 12, 6th place: judgment interval 4, 7th place: judgment interval 3, 8th place: judgment interval 8, 9th place: judgment interval 9 and judgment interval 11, 11th place: judgment interval 2, and 12th place: judgment interval 7.

[0087] The automatic editing execution unit 106 performs editing in accordance with the editing rule and the final importance of the judgment sections 1 to 12. Here, it is assumed that the editing rule instructs that "videos of the lecture are deleted in order of decreasing importance so that the length of the recorded video of the lecture is 2 / 3 of the actual length of the lecture."

[0088] In this case, assuming that the importance is determined as shown in Figure 10, the automatic editing execution unit 106 performs editing by deleting four sections from judgment sections 1 to 12 in order of decreasing importance: judgment section 7, judgment section 2, judgment section 9, and judgment section 11.

[0089] FIG. 11 is a diagram showing an example of a timeline of video data after editing.

[0090] As shown in FIG. 11, the edited video data is video data in which determination section 1, determination section 3, determination section 4, determination section 5, determination section 6, determination section 8, determination section 10, and determination section 12 are combined.

[0091] Since the actual lecture duration is 120 minutes, the automatic editing execution unit 106 generates 80 minutes of video data, which is two-thirds of the actual lecture duration.

[0092] The video data obtained by the above editing is output by the video output unit 107 to the recording device 3 or the input / output device 4. The video data obtained by editing is recorded in the recording device 3 or presented to the user by the input / output device 4.

[0093] <3. Operation of the arithmetic unit> Here, the operation of the arithmetic unit 2 having the above configuration will be described.

[0094] The processing performed by the arithmetic unit 2 will be described with reference to the flowchart of FIG.

[0095] 12 starts, for example, when video data is input from the video shooting device 1 to the video input unit 101. Of the video data, a video signal is output to the video analysis unit 102, and a sound signal is output to the sound analysis unit 103.

[0096] In step S1, the video analysis unit 102 analyzes the video information based on the video signal.

[0097] In step S2, the sound analysis unit 103 analyzes the sound information based on the sound signal. Note that the process of step S2 may be performed in parallel with the process of step S1, or may be performed after the process of step S1 has been performed.

[0098] In step S3, the importance determining unit 105 determines the importance of each section into which the video data is divided, based on the analysis result of the video information by the video analysis unit 102 and the analysis result of the sound information by the sound analysis unit 103.

[0099] In step S4, the automatic editing execution unit 106 generates playback assistance information according to the importance determined by the importance determination unit 105. That is, the automatic editing execution unit 106 functions as a generation unit that generates playback assistance information. The playback assistance information is information used to provide the user with recorded lecture video. The automatic editing execution unit 106 generates video data as playback assistance information by performing editing such as deleting video data of low-importance sections or compressing low-importance sections at a higher compression rate than other sections.

[0100] It should be noted that meta information for editing according to the importance level and meta information for playing according to the importance level may be generated as playback assistance information. These meta information will be described later.

[0101] After the playback assistance information is generated, the processing in Fig. 12 ends. The playback assistance information is output by the video output unit 107 to the recording device 3 or the input / output device 4, and is used to provide the recorded lecture video to the user. For example, the input / output device 4 displays the recorded lecture video obtained by playing back the video data as the playback assistance information, and provides it to the user.

[0102] As described above, in the computing device 2, the video data is edited according to the importance determined for each section of the video data based on the analysis information about the lecture information. The lecture information includes information about the teacher and students, blackboard notes, chimes, materials posted on the whiteboard, and information about video materials.

[0103] When the technology described in Patent Document 1 is applied to editing recorded lecture videos, importance is determined based on information linked to people. If the recorded lecture videos are edited according to the importance determined in this way, the section of the video in which the teacher is writing on the blackboard may be determined to be of low importance, and information about the order in which the writing on the blackboard was done may be lost from the recorded lecture videos.

[0104] Furthermore, sections of the video showing blackboard notes written in red pen, which are considered important for learning, may be judged to be of low importance, and the sections of the video showing blackboard notes written in red pen may end up being lost from the recorded lecture video.

[0105] The calculation device 2 edits the video data according to the importance of the analytical information about the lecture, so that the video data can be edited without missing any information that is considered important in recording a lecture, such as information about the order in which the blackboard writing was made and information about the blackboard writing in red pen.

[0106] Therefore, the arithmetic unit 2 can edit the recorded lecture video in an appropriate manner. Furthermore, the arithmetic unit 2 edits the video data by deleting sections that are not important for learning or by compressing the video data at a higher compression rate, so that the video data of the recorded lecture video can be recorded with a reduced amount of data.

[0107] Users who watch recorded lecture videos will be able to learn the lecture content in a shorter time than the actual lecture time, since they will be watching videos from which sections that are not important for learning have been deleted.

[0108] <4. Modifications> -Information about lectures Although an example has been described in which importance is determined based on analytical information about information written on the surface of a whiteboard, importance may also be determined based on analytical information about information about the screen onto which the presentation materials are projected.

[0109] In this case, the importance is determined based on analysis information about slide transitions and animations, for example. In this way, the present technology can be applied to filming lectures that utilize materials other than blackboard writing. Furthermore, the lecture may be filmed with both a whiteboard and a screen simultaneously present within the field of view of the video capture device 1.

[0110] Instead of a whiteboard, the importance may be determined based on analysis information about information written on a blackboard, a green board, or paper such as construction paper.

[0111] Sounds related to the lecture may be collected by a sound collecting device separate from the sound collecting device mounted on the video shooting device 1. For example, the teacher's voice may be collected by a pin microphone worn by the teacher. In this case, the pin microphone is connected to the arithmetic device 2 and outputs a sound signal representing the collected sound to the arithmetic device 2.

[0112] Information about playback assistance Meta information for performing editing according to importance may be generated as playback assistance information by the automatic editing execution unit 106. For example, meta information representing the result of importance determination by the importance determination unit 105 is generated by the automatic editing execution unit 106 as meta information for performing editing according to importance.

[0113] In this case, the video output unit 107 outputs the video data supplied from the video shooting device 1 and the meta information generated by the automatic editing execution unit 106 to the recording device 3 and the input / output device 4.

[0114] For example, if multiple users wish to watch a recorded lecture video at different lengths depending on their level of proficiency, the input / output device 4 edits the video data for each user using the meta information supplied from the computing device 2 and plays back the edited video data. In this way, the video shooting device 1 can provide recorded lecture video of lengths depending on the level of proficiency of each user.

[0115] In addition, editing of video data according to each user's level of proficiency may be performed by the calculation device 2 editing the video data based on meta information recorded in the recording device 3 in accordance with a rule signal representing an editing rule for editing according to each user's level of proficiency.

[0116] Furthermore, meta information for playing back content according to importance may be generated as playback assistance information by the automatic editing execution unit 106. For example, meta information representing the result of importance determination by the importance determination unit 105 is generated by the automatic editing execution unit 106 as meta information for playing back content according to importance.

[0117] In this case, the video output unit 107 outputs the video data supplied from the video shooting device 1 and the meta information generated by the automatic editing execution unit 106 to the recording device 3 and the input / output device 4.

[0118] For example, the input / output device 4 displays the playback position of a section of high importance on a seek bar on the viewing screen of the recorded lecture video. In this way, a user watching the recorded lecture video can easily play back a section of the recorded lecture video that is important for learning by selecting the playback position displayed on the seek bar on the viewing screen. Note that instead of the user selecting the playback position, the input / output device 4 may skip sections of low importance and automatically play back only the playback position displayed on the seek bar as being of high importance.

[0119] Furthermore, together with the playback support information, the automatic editing execution unit 106 may generate thumbnail images representing each section whose importance has been determined.

[0120] For example, the calculation device 2 determines the importance of each frame that constitutes a certain section and sets the frame image of the frame with the highest importance as the thumbnail image. Alternatively, the frame image of the first or last frame in each section may be set as the thumbnail image.

[0121] The video output unit 107 outputs the playback assistance information generated by the automatic editing execution unit 106 and thumbnail images of each section of the recorded lecture video to the recording device 3 and the input / output device 4.

[0122] When thumbnail images are supplied to the input / output device 4 together with meta information for playback according to importance, the input / output device 4 displays the playback position of a section with high importance and a thumbnail image of that section on the seek bar of the viewing screen. This allows the input / output device 4 to present clearer information to the user viewing the recorded lecture video.

[0123] About analysis information The types of analysis information analyzed by the video analysis unit 102 and the sound analysis unit 103 can be set in advance, or can be instructed by the user using a rule signal input via the input / output device 4. For example, if real-time performance is important to the user, the user is instructed to analyze only the necessary and sufficient analysis information.

[0124] -How to determine importance The importance of each element of the analysis information may be determined according to the frequency with which it appears in the video captured by the video capturing device 1.

[0125] For example, if the frequency of blackboard writing written in red pen is high and the frequency of blackboard writing in black pen is low, the importance determination unit 105 determines that the characters written in black pen are characters written for emphasis and determines the importance of the section where the instructor is writing on the black pen to be a high value.

[0126] If most of the writing on the board is done with a red pen, and the importance is determined solely according to an importance determination rule such as "if the writing on the board is done with a red pen, the importance is high," then a large number of sections will be determined to have a high importance value.

[0127] However, if most of the writing on the board is done with a red pen, the importance determination unit 105 can make an importance determination based on the frequency of appearance of writing on the board with a red pen and a black pen, making it possible to make an importance determination that reflects the teacher's intention, such as writing important characters with a black pen.

[0128] Furthermore, for example, when the same formula appears repeatedly on the blackboard, the importance determination unit 105 determines that the repeatedly appearing formula is an important formula for learning, and determines the importance of the section in which the repeatedly appearing formula is written as a high value. It is also possible to determine the importance of the section including the timing when the repeatedly appearing formula was first written as a particularly high value.

[0129] The importance may be determined based on the change over time of each piece of analysis information, for example, the change over time in the amount of writing on the board.

[0130] FIG. 13 is a diagram showing the relationship between the change in the amount of writing on the board over time and the importance level.

[0131] An example of the change in the amount of writing on the board over time is shown in A of Fig. 13. In A of Fig. 13, the horizontal axis represents time, and the vertical axis represents the amount of writing on the board.

[0132] As shown in A of Figure 13, the amount of writing on the board increases (writing is being done) until time t1. At time t1, the increase in the amount of writing on the board stops (writing is completed), and the amount of writing on the board does not change from time t1 to time t2 (explanation continues without writing on the board). After time t2, the amount of writing on the board decreases (writing is erased).

[0133] An example of importance determined according to the change in the amount of writing on the board over time is shown in B of Fig. 13. In B of Fig. 13, the horizontal axis represents time, and the vertical axis represents importance.

[0134] As shown in B of FIG. 13, the importance level decreases during the period up to time t1 when the amount of writing on the board is increasing. At time t1 when the increase in the amount of writing on the board stops, the importance level increases. During the period from time t1 to time t2 when the amount of writing on the board remains unchanged, the importance level gradually decreases from the point when the amount of writing on the board remains unchanged for a certain period of time. At time t2 when the amount of writing on the board begins to decrease, the importance level decreases.

[0135] In this way, the importance determination unit 105 determines the importance of the increase or decrease in the amount of writing on the board as a value as shown in B of FIG. 13, depending on the change over time in the amount of writing on the board as shown in A of FIG.

[0136] As described above, the importance determination unit 105 determines the importance of each section of the video data according to information about the writing on the board based on video and sound. Information about the writing on the board is, for example, information that represents the state of the writing on the board and information that represents the content of the writing on the board. Information that represents the state of the writing on the board includes information that represents the amount of increase or decrease in the writing on the board (change over time), the position of the pen tip, the sound of the writing on the board, the color of the writing on the board, and the frequency of appearance of the color of the writing on the board. Information that represents the content of the writing on the board includes information that represents the characters, equations, and the frequency of appearance of the characters and equations on the board.

[0137] Editing method When determining the final importance ranking of each section, if there are multiple sections with the same final importance value, the ranking of such multiple sections may be determined by random numbers or may be determined according to the order on the timeline.

[0138] Furthermore, if there are multiple sections with the same final importance value, the ranking of each section may be determined based on the importance calculated by referring to the importance of the sections before and after each of the multiple sections.

[0139] 10, the final importance of the judgment section 9 and the judgment section 11 are the same, and the final importance ranking is also the same. For example, when performing editing to delete either the judgment section 9 or the judgment section 11, the automatic editing execution unit 106 compares the sum of the importance of the judgment sections 8 and 10, which are the judgment sections before and after the judgment section 9 (1.7 + 2.5 = 4.2), with the sum of the importance of the judgment sections 10 and 12, which are the judgment sections before and after the judgment section 11 (2.5 + 2.2 = 4.7).

[0140] By comparing the sum of the importance of the judgment sections before and after judgment section 9 with the sum of the importance of the judgment sections before and after judgment section 11, the automatic editing execution unit 106 performs editing by deleting judgment section 9 having the smaller sum of the importance of the judgment sections before and after judgment section 11.

[0141] <5.About Computers> The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

[0142] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0143] A CPU (Central Processing Unit) 301 , a ROM (Read Only Memory) 302 , and a RAM (Random Access Memory) 303 are interconnected by a bus 304 .

[0144] An input / output interface 305 is also connected to the bus 304. An input unit 306 including a keyboard, a mouse, etc., and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305. In addition, a storage unit 308 including a hard disk, a nonvolatile memory, etc., a communication unit 309 including a network interface, etc., and a drive 310 that drives removable media 311 are also connected to the input / output interface 305.

[0145] In the computer configured as above, the CPU 301 loads a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, thereby performing the above-described series of processes.

[0146] The program executed by the CPU 301 is installed in the storage unit 308 by being recorded on a removable medium 311, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting.

[0147] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0148] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0149] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0150] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0151] For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0152] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by multiple devices.

[0153] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0154] <Configuration combination example> The present technology can also be configured as follows.

[0155] (1) A generating unit generates playback support information for each predetermined section of data including video and audio of a lecture, in accordance with the importance determined based on information about the lecture. Information processing device. (2) The information about the lecture is information about the board writing based on the video or the audio. The information processing device according to (1) above. (3) The information about the writing on the board is information that indicates the state of the writing on the board or the content of the writing on the board. The information processing device according to (2) above. (4) The information about the writing on the board is information that indicates at least one of the color of the writing on the board, the increase or decrease of the writing on the board, and a formula included in the writing on the board. The information processing device according to (3) above. (5) The information about the lecture is information that represents the behavior of at least one of the lecturer and the audience of the lecture that are shown in the video. The information processing device according to any one of (1) to (4). (6) The information about the lecture is information representing sounds related to the lecture. The information processing device according to any one of (1) to (5). (7) The generating unit generates edited data as the playback support information by editing the data according to the importance. The information processing device according to any one of (1) to (6). (8) The generating unit generates the edited data by deleting the data in the section with low importance or compressing the data in the section with low importance at a higher compression rate than the data in other sections. The information processing device according to (7) above. (9) The generating unit generates meta information for editing according to the importance as the playback assistance information. The information processing device according to any one of (1) to (6). (10) The generating unit generates meta information for performing playback according to the importance as the playback assistance information. The information processing device according to any one of (1) to (6). (11) a determination unit that determines the importance of each of the predetermined sections based on information about the lecture, The generating unit generates the playback assistance information in accordance with the importance determined by the determining unit. The information processing device according to any one of (1) to (10). (12) the determination unit determines the importance of a determination section that is a group of a plurality of consecutive sections; The generating unit generates the playback assistance information according to the importance determined by the determining unit for each of the determination sections. The information processing device according to (11) above. (13) The determination unit determines the importance of each of the determination sections, each of which is a group of a predetermined number of sections. The information processing device according to (12) above. (14) The determination unit determines the importance of each of the determination sections set based on information about the lecture. The information processing device according to (12) above. (15) The generating unit generates thumbnail images representing each of the sections together with the playback support information. The information processing device according to any one of (1) to (14). (16) The generating unit generates the playback assistance information for the sections having the same importance level in accordance with the importance levels of sections before and after the sections having the same importance level. The information processing device according to any one of (1) to (15). (17) The determination unit determines the importance in accordance with a determination rule instructed by the user via an input device that accepts user operations. The information processing device according to (11) above. (18) The generating unit generates the playback support information in accordance with an editing rule instructed by the user via an input device that accepts user operations. The information processing device according to any one of (1) to (17). (19) Generates playback support information for each of predetermined sections divided into data including video and audio of a lecture, according to the importance determined based on information about the lecture. Generation method. (20) On the computer, Generates playback support information for each of predetermined sections divided into data including video and audio of a lecture, according to the importance determined based on information about the lecture. A program for executing a process. [Explanation of symbols]

[0156] 1 Video shooting device, 2 Calculation device, 3 Recording device, 4 Input / output device, 101 Video input unit, 102 Video analysis unit, 103 Sound analysis unit, 104 Control parameter input unit, 105 Importance determination unit, 106 Automatic editing execution unit, 107 Video output unit

Claims

1. a generating unit that generates playback assistance information for each predetermined section of data including video and audio of a lecture, the predetermined section being divided into predetermined time intervals, in accordance with the importance determined based on information about the lecture; The generating unit generates the playback assistance information for the sections having the same importance level in accordance with the importance levels of sections before and after the sections having the same importance level. Information processing device.

2. The information about the lecture is information about the board writing based on the video or the audio. The information processing device according to claim 1 .

3. The information about the writing on the board is information that indicates the state of the writing on the board or the content of the writing on the board. The information processing device according to claim 2 .

4. The information about the writing on the board is information that indicates at least one of the color of the writing on the board, the increase or decrease of the writing on the board, and a formula included in the writing on the board. The information processing device according to claim 3 .

5. The information about the lecture is information that represents the behavior of at least one of the lecturer and the audience of the lecture that are shown in the video.

5. The information processing device according to claim 1.

6. The information about the lecture is information representing sounds related to the lecture.

6. The information processing device according to claim 1.

7. The generating unit generates edited data as the playback support information by editing the data according to the importance.

7. The information processing device according to claim 1.

8. The generating unit generates the edited data by deleting the data in the section with low importance or compressing the data in the section with low importance at a higher compression rate than the data in other sections. The information processing device according to claim 7 .

9. The generating unit generates meta information for editing according to the importance as the playback assistance information.

7. The information processing device according to claim 1.

10. The generating unit generates meta information for performing playback according to the importance as the playback assistance information.

7. The information processing device according to claim 1.

11. a determination unit that determines the importance of each of the segments based on information about the lecture, The generating unit generates the playback assistance information in accordance with the importance determined by the determining unit. The information processing device according to claim 1 .

12. the determination unit determines the importance of a determination section that is a group of a plurality of consecutive sections; The generating unit generates the playback assistance information according to the importance determined by the determining unit for each of the determination sections. The information processing device according to claim 11.

13. The determination unit determines the importance of each of the determination sections, each of which is a group of a predetermined number of sections. The information processing device according to claim 12.

14. The determination unit determines the importance of each of the determination sections set based on information about the lecture. The information processing device according to claim 12.

15. The generating unit generates thumbnail images representing each of the sections together with the playback support information.

15. The information processing device according to claim 1.

16. The determination unit determines the importance in accordance with a determination rule instructed by the user via an input device that accepts user operations. The information processing device according to claim 11.

17. The generating unit generates the playback support information in accordance with an editing rule instructed by the user via an input device that accepts user operations.

17. The information processing device according to claim 1.

18. The information processing device generating playback assistance information for each of predetermined sections obtained by dividing data including video and audio of a lecture into predetermined time segments in accordance with the importance determined based on information about the lecture; For the sections having the same importance, the playback assistance information is generated according to the importance of sections before and after the sections having the same importance. Generation method.

19. On the computer, generating playback assistance information for each of predetermined sections obtained by dividing data including video and audio of a lecture into predetermined time segments in accordance with the importance determined based on information about the lecture; For the sections having the same importance, the playback assistance information is generated according to the importance of sections before and after the sections having the same importance. A program for executing a process.

Images