Ai question-answering system and method based on time-synchronized multimodal lecture data

Abstract

The present invention relates to an AI question-answer system and method based on time-synchronized multimodal lecture data, configured to generate a Knowledge Chunk by time-synchronizing handwriting data and voice data collected from an electronic whiteboard, and to provide an answer linked to the corresponding lecture point in time to a learner's visual reference question using Retrieval-Augmented Generation (RAG) technology. More specifically, the invention comprises: time-synchronizing the handwriting coordinate stream and voice stream of the electronic whiteboard within a predetermined error range; recognizing visual indicators (stars, underlines, circles, etc.) within the handwriting and assigning semantic tags; generating a multimodal Knowledge Chunk including voice text, handwriting images, timestamps, and indicator tags, and storing it in a vector database; providing an answer containing a timestamp link of the corresponding lecture point in time using Retrieval-Augmented Generation (RAG) technology to a learner's visual reference question ("What is the starred part?"); and simultaneously playing the video and handwriting at that point in time when the link is selected, thereby enabling the learner to effectively review lecture content and receive an accurate answer to the question. This relates to an AI question-answering system and method based on multimodal lecture data.

Description

Technology Field

[0001] The present invention relates to an AI (Artificial Intelligence) question-and-answer system and method based on time-synchronized multimodal lecture data. In particular, it relates to an AI question-and-answer system and method based on time-synchronized multimodal lecture data configured to generate Knowledge Chunks by time-synchronizing handwriting data and voice data collected from an electronic whiteboard, and to provide answers linked to the corresponding lecture point in time to a learner's visual reference questions by utilizing Retrieval-Augmented Generation (RAG) technology. Background Technology

[0002] Recently, digital learning environments, such as electronic whiteboards and online lecture platforms, have been rapidly expanding in the field of education. While lecture videos and handwritten content are being generated in large quantities in this environment, it is currently difficult for learners to receive effective answers when they wish to ask questions about specific parts of the lecture material.

[0003] As a conventional technology, Korean Registered Patent No. 10-1529254 discloses a technology that converts the writing content of an electronic whiteboard into PDF and records audio to provide it in the form of meeting minutes. However, this technology is limited to merely storing and transmitting recorded content and fails to provide a function to intelligently respond to learners' questions.

[0004] Furthermore, while Search Augmented Generative (RAG) technology is being utilized to complement the limitations of Large Language Models (LLM), existing RAG technology is primarily limited to text-based document search, which limits its application to multimodal lecture data that combines whiteboard writing and speech.

[0005] In particular, when a learner asks a question referring to a visual indicator, such as "What is the part the teacher starred?", existing technology made it impossible to grasp the intent of the question and accurately locate the relevant lecture content to provide an answer.

[0006] Accordingly, there is a need for new technology capable of precisely time-synchronizing handwriting and voice data, recognizing visual instructions within the handwriting to perform semantic tagging, and providing accurate answers to learners' visual reference questions linked to the relevant lecture time. Prior art literature

[0007] (0001) Korean Registered Patent No. 10-1529254 The problem to be solved

[0008] The technical problem that the present invention aims to solve is to provide an AI question-and-answer system and method based on time-synchronized multimodal lecture data that collects handwriting coordinate data and voice data collected from an electronic whiteboard in real time and can precisely synchronize time within a predetermined error range.

[0009] Another technical problem that the present invention aims to solve is to provide an AI question-answering system and method based on time-synchronized multimodal lecture data that can automatically recognize visual indicators such as asterisks, underlines, and circles within whiteboard data and assign semantic tags such as 'emphasis', 'important', and 'key' to each indicator.

[0010] Another technical problem that the present invention aims to solve is to provide an AI question-answering system and method based on time-synchronized multimodal lecture data capable of generating a multimodal knowledge chunk that integrates time-synchronized voice text, handwriting images, timestamps, and instruction tags, and storing it in a vector database.

[0011] Another technical problem that the present invention aims to solve is to provide a time-synchronized multimodal lecture data-based AI question-answering system and method capable of generating an accurate answer by first searching the knowledge chunk tagged with the visual indicator when a learner asks a question by referring to a visual indicator, such as "starred part."

[0012] Another technical problem that the present invention aims to solve is to provide an AI question-answer system and method based on time-synchronized multimodal lecture data, configured such that a timestamp link of the corresponding lecture time is included in the generated answer, and when a learner selects the link, the lecture video and whiteboard writing at that time are played simultaneously.

[0013] The technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0014] An AI question-answering method based on time-synchronized multimodal lecture data according to an embodiment of the present invention for achieving the above technical problem is performed through an intelligent question-answering system and comprises: a data collection step of collecting a writing coordinate stream from an electronic whiteboard at predetermined time intervals and collecting a voice stream from a microphone, and assigning a timestamp in milliseconds to each stream; a synchronization step of comparing the timestamps of the writing coordinate stream and the voice stream to perform time synchronization within a predetermined error range; a directive recognition step of recognizing a predefined visual directive pattern in the writing coordinate stream and assigning a semantic tag corresponding to each recognized visual directive; a knowledge creation step of converting the voice stream into text through a speech recognition model and generating a multimodal knowledge chunk including the text, the time-synchronized writing image, the timestamp, and the semantic tag; and an embedding step of vector-embedding the multimodal knowledge chunk and storing it in a vector database. The method is characterized by comprising: a question-answering step of receiving a learner's question, extracting a keyword related to a visual keyword from the question, adjusting the search score or search priority of a knowledge chunk assigned a semantic tag corresponding to the keyword related to the keyword among the knowledge chunks stored in the vector database, performing a hybrid search combining vector similarity search and semantic tag-based search, and generating an answer using a large-scale language model with the searched knowledge chunk as context; and a playback control step of including a link corresponding to the timestamp of the searched knowledge chunk in the answer, and simultaneously playing the lecture video and whiteboard writing at the corresponding point in time in a time-synchronized state when the timestamp link is selected.

[0015] In addition, the above-determined error range is within 500 milliseconds (ms), the above-determined writing coordinate stream is collected at intervals of 50 milliseconds (ms), and each coordinate data is characterized by including an x-coordinate value, a y-coordinate value, pen pressure, pen color, and a timestamp.

[0016] Additionally, the step of performing the time synchronization comprises: detecting a writing start event and a writing end event in a writing stream; matching the start timestamp and end timestamp of each writing segment with the timestamp of a voice chunk; and performing interpolation or reordering when the error exceeds a threshold.

[0017] In addition, in the above-mentioned instruction recognition step, the visual instruction pattern includes at least one of an asterisk, an underline, a circle, an arrow, and parentheses, and is recognized through at least one of a rule-based pattern matching and a machine learning-based classifier that analyzes the geometric characteristics of the coordinate trajectory of the above-mentioned handwriting coordinate stream, and at least one semantic tag among emphasis, key, and test question is assigned to the asterisk, at least one semantic tag among important and memory is assigned to the underline, and at least one semantic tag among definition and concept is assigned to the circle.

[0018] Additionally, the embedding step is characterized by including: a step of generating an integrated text by adding semantic tag information to the text and combining them; and a step of converting the integrated text into a vector embedding through an embedding model and storing it in the vector database.

[0019] In addition, the process of searching for the relevant knowledge chunk in the question-and-answer step comprises: a step of converting the question text into a vector; a step of performing a hybrid search combining vector similarity search and directive tag filtering; and a step of prioritizing the search by assigning weights to the knowledge chunk to which the directive tag is assigned when the question contains directive-related keywords.

[0020] In addition, the process of simultaneously playing the lecture video and the whiteboard in the playback control step comprises: a step of seeking the lecture video player to the timestamp position; a step of loading the whiteboard coordinates corresponding to the timestamp and rendering them sequentially on the whiteboard canvas; and a step of synchronizing the playback of the lecture video with the rendering of the whiteboard so that the learner can check the instructor's whiteboard process in real time.

[0021] Meanwhile, an AI question-answer system based on time-synchronized multimodal lecture data according to one embodiment of the present invention comprises: a data collection unit that collects a handwriting coordinate stream from an electronic whiteboard at predetermined time intervals and collects a voice stream from a microphone, and assigns a timestamp in milliseconds to each stream; a synchronization unit that compares the timestamps of the handwriting coordinate stream and the voice stream to perform time synchronization within a predetermined error range; a directive recognition unit that recognizes a predefined visual directive pattern in the handwriting coordinate stream and assigns a semantic tag corresponding to each recognized visual directive; a knowledge conversion unit that converts the voice stream into text through a speech recognition model and generates a multimodal knowledge chunk including the text, the time-synchronized handwriting image, the timestamp, and the semantic tag; and an embedding unit that vector-embeds the multimodal knowledge chunk and stores it in a vector database. A question-and-answer unit configured to receive a learner's question, extract a keyword related to a visual keyword from the question, perform a hybrid search configured to adjust the search score or search priority of a knowledge chunk assigned a semantic tag corresponding to the keyword related to the keyword among the knowledge chunks stored in the vector database, search for related knowledge chunks by combining vector similarity search and semantic tag-based search, and generate an answer using a large-scale language model with the searched knowledge chunk as context; and a playback control unit configured to include a link corresponding to the timestamp of the searched knowledge chunk in the answer, and to simultaneously play the lecture video and whiteboard writing at that point in time in a time-synchronized state when the timestamp link is selected.

[0022] The above embodiments of the present invention are merely some of the preferred embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by those skilled in the art based on the detailed description of the present invention to be described below. Effects of the invention

[0023] The present invention described above has the following effects.

[0024] First, according to one embodiment of the present invention, by performing a hybrid search that time-synchronizes a whiteboard coordinate stream and a voice stream and adjusts the search score or search priority of a knowledge chunk using semantic tags based on visual instruction patterns, it is possible to provide an accurate answer directly related to the time of the corresponding lecture, along with a timestamp link, to a learner's visual instruction reference question.

[0025] In addition, according to one embodiment of the present invention, by collecting a writing coordinate stream at 50-millisecond intervals and performing time synchronization within an error range of 500 milliseconds, the time discrepancy between the writing and the voice can be minimized, thereby improving the time accuracy of a multimodal knowledge chunk.

[0026] In addition, according to one embodiment of the present invention, precise time synchronization between the writing activity and the utterance content can be reliably achieved by matching the timestamps of the writing segments with the voice chunks based on the writing start and end events and performing interpolation or reordering.

[0027] In addition, according to one embodiment of the present invention, by analyzing the geometric characteristics of the coordinate trajectory of a writing coordinate stream to recognize visual indicator patterns and assigning semantic tags thereto, visual emphasis elements within the writing can be mechanically interpreted and utilized in subsequent search and question-and-answer stages.

[0028] In addition, according to one embodiment of the present invention, by vector embedding an integrated text in which semantic tag information is combined with text, it is possible to construct a vector database that simultaneously reflects the semantic features of knowledge chunks and visual indicator information.

[0029] In addition, according to one embodiment of the present invention, when a question includes keywords related to instructions, knowledge chunks that match the learner's question intent can be efficiently searched by assigning weights to knowledge chunks tagged with the corresponding instructions and searching for them first.

[0030] In addition, according to one embodiment of the present invention, by simultaneously playing a lecture video and sequentially rendering the handwriting coordinates in a time-synchronized state, a learner can review the instructor's handwriting process by verifying it at the same time as the actual lecture.

[0031] In addition, according to one embodiment of the present invention, through a hybrid search configured to adjust the search score or search priority of a knowledge chunk using semantic tags corresponding to visual instructions, a response linked with a time-synchronized lecture video and whiteboard writing can be provided in response to a learner's visual instruction reference question.

[0032] The effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0033] FIG. 1 is a conceptual diagram of a system configuration according to an embodiment of the present invention, FIG. 2 is an exemplary diagram of a multimodal knowledge chunk data structure according to an embodiment of the present invention. FIG. 3 is a flowchart of a question-and-answer processing process according to an embodiment of the present invention, FIG. 4 is a detailed example diagram of a data collection process according to an embodiment of the present invention. FIG. 5 is a flowchart of a time synchronization algorithm according to an embodiment of the present invention, FIG. 6 is an exemplary diagram of an instruction recognition process according to an embodiment of the present invention. FIG. 7 is an example diagram of a timestamp-linked playback process according to an embodiment of the present invention. FIG. 8 is an example diagram of a usage scenario according to an embodiment of the present invention. Specific details for implementing the invention

[0034] Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. It should be noted that in assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the embodiments of the present invention, if it is determined that a detailed description of related known components or functions would hinder understanding of the embodiments of the present invention, such detailed description is omitted.

[0035] In addition, terms such as first, second, A, B, (a), (b), etc., may be used when describing the components of the embodiments of the present invention. These terms are intended merely to distinguish the components from other components, and the essence, order, or sequence of the components is not limited by these terms. Where it is stated that a component is "connected," "combined," or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that components may also be "connected," "combined," or "joined" between each component.

[0036] FIG. 1 is a conceptual diagram of a system configuration according to an embodiment of the present invention, FIG. 2 is an example diagram of a multimodal knowledge chunk data structure according to an embodiment of the present invention, FIG. 3 is a flowchart of a question-and-answer processing process according to an embodiment of the present invention, FIG. 4 is a detailed example diagram of a data collection process according to an embodiment of the present invention, FIG. 5 is a flowchart of a time synchronization algorithm according to an embodiment of the present invention, FIG. 6 is an example diagram of an instruction recognition process according to an embodiment of the present invention, FIG. 7 is an example diagram of a timestamp-linked playback process according to an embodiment of the present invention, and FIG. 8 is an example diagram of a usage scenario according to an embodiment of the present invention.

[0037] Hereinafter, with reference to FIGS. 1 to 8, an AI (Artificial Intelligence) question-answering system and method based on time-synchronized multimodal lecture data according to an embodiment of the present invention will be described. Reference numerals may be referenced together with FIG. 1, and for convenience, the system will be described first.

[0038] First, referring to FIG. 1, a time-synchronized multimodal lecture data-based AI question-and-answer system (hereinafter also referred to as the "system") according to an embodiment of the present invention is performed through an intelligent question-and-answer system and comprises: a data collection unit (10) that collects a writing coordinate stream from an electronic whiteboard at predetermined time intervals and collects a voice stream from a microphone, and assigns a timestamp in milliseconds to each stream; a synchronization unit (20) that compares the timestamps of the writing coordinate stream and the voice stream to perform time synchronization within a predetermined error range; a directive recognition unit (30) that recognizes a predefined visual directive pattern in the writing coordinate stream and assigns a semantic tag corresponding to each recognized visual directive; a knowledge generation unit (40) that converts the voice stream into text through a voice recognition model and generates a multimodal knowledge chunk including the text, the time-synchronized writing image, the timestamp, and the semantic tag; and an embedding unit (60) that vector-embeds the multimodal knowledge chunk and stores it in a vector database (50). The system may be configured to include: a question-answering unit (70) that receives a question from a learner, extracts a keyword related to a visual keyword from the question, performs a hybrid search configured to adjust the search score or search priority of a knowledge chunk assigned a semantic tag corresponding to the keyword related to the keyword among the knowledge chunks stored in a vector database (50), searches for related knowledge chunks by combining vector similarity search and semantic tag-based search, and generates an answer using a large-scale language model with the searched knowledge chunk as context; and a playback control unit (80) configured to include a link corresponding to the timestamp of the searched knowledge chunk in the answer, and to simultaneously play the lecture video and whiteboard writing at that time in a time-synchronized state when the timestamp link is selected.

[0039] An AI question-and-answer system based on time-synchronized multimodal lecture data according to one embodiment of the present invention can be configured to align different types of lecture data generated from an electronic whiteboard and a microphone along a time axis and semantically interpret visual instructions to accurately reproduce the lecture point in time corresponding to a learner's question.

[0040] The system may include a data collection unit (10) configured to collect a writing coordinate stream from an electronic whiteboard at predetermined time intervals, collect a voice stream from a microphone, and assign a timestamp in milliseconds to each stream.

[0041] The data collection unit (10) collects writing coordinate data resulting from the movement of a pen on an electronic whiteboard in the form of a continuous stream, and simultaneously collects the instructor's voice as a voice stream, thereby enabling multimodal lecture data from different sources to be recorded on the same time basis.

[0042] Here, a writing coordinate stream refers to a sequence of coordinate values ​​generated by pen input on an electronic whiteboard, which can mean a data set arranged sequentially over time.

[0043] The system may include a synchronization unit (20) configured to perform time synchronization within a predetermined error range by comparing timestamps assigned to the writing coordinate stream and the voice stream, respectively.

[0044] The synchronization unit (20) can correct the temporal discrepancy between the handwriting coordinate stream and the voice stream generated from different input devices, thereby aligning the handwriting and speech generated at the same time of lecture so that they are logically connected.

[0045] This allows multimodal knowledge chunks generated in subsequent stages to contain temporally consistent information.

[0046] The system may include a directive recognition unit (30) configured to recognize a predefined visual directive pattern in a whiteboard coordinate stream and assign a semantic tag corresponding to each recognized visual directive.

[0047] Visual indicator patterns may refer to forms of visual representation, such as asterisks, underlines, circles, arrows, or parentheses, that an instructor repeatedly uses to emphasize or structure specific concepts during the whiteboard process.

[0048] The instruction recognition unit (30) can automatically identify the visual instruction pattern by analyzing the coordinate trajectory of the whiteboard coordinate stream and assign a semantic tag that reflects the educational meaning of the pattern, thereby adding semantic metadata to the whiteboard information.

[0049] Here, a semantic tag may be information that expresses semantic attributes, such as emphasis, importance, definition, or concept, represented by a visual indicator in a structured form.

[0050] The system may include a knowledge unit (40) configured to convert a voice stream into text through a voice recognition model and to generate a multimodal knowledge chunk including said text, a time-synchronized handwriting image, a timestamp, and a semantic tag.

[0051] A multimodal knowledge chunk refers to a unit of knowledge that combines utterance content at a specific point in time during a lecture with a corresponding whiteboard image, time information, and semantic tags; it can be described as the smallest unit of information structure that can be utilized for subsequent search and Q&A.

[0052] The knowledge unit (40) can generate a knowledge representation that preserves the context of the lecture by combining temporally aligned whiteboard information and semantic tags, going beyond simple voice-to-text conversion.

[0053] The system may include an embedding unit (60) configured to vector embedding multimodal knowledge chunks and storing them in a vector database (50).

[0054] The embedding unit (60) can structure data in a form that allows semantic similarity to be calculated by converting the knowledge chunk into a fixed-dimensional vector based on the text and semantic tag information included in the multimodal knowledge chunk.

[0055] The vector database (50) can be used as a storage for storing vector embedding results and for performing vector similarity search when querying later.

[0056] The system may include a question-answering unit (70) configured to receive a learner's question, extract a keyword related to a visual indicator from the question, perform a hybrid search configured to adjust the search score or search priority of a knowledge chunk with a semantic tag among the knowledge chunks stored in the vector database (50) to search for a related knowledge chunk, and generate an answer using a large-scale language model with the searched knowledge chunk as a context.

[0057] Hybrid search is a search method that combines vector similarity search and semantic tag-based search, and may be a search method that considers both the semantic similarity of a question and the match of semantic tags based on visual indicators.

[0058] When a learner inputs a question that refers to a visual indicator such as "starred part" or "circled definition," the question-and-answer section (70) can increase the search score or search priority of a knowledge chunk that has a semantic tag corresponding to the indicator to produce a more accurate search result.

[0059] The system may include a playback control unit (80) configured to include a link corresponding to a timestamp of a searched knowledge chunk in the generated answer, and to simultaneously play the lecture video and whiteboard at that time in a time-synchronized state when the timestamp link is selected.

[0060] The playback control unit (80) can move the lecture video player to that point in time when the learner selects a timestamp link included in the answer, and sequentially renders the whiteboard coordinates corresponding to the same time information, thereby allowing the learner to visually check the lecture scene directly related to the question.

[0061] Through this, learners can improve their understanding by going beyond text-based answers and simultaneously reproducing whiteboard writing and speech within the actual lecture context.

[0062] Next, referring to FIGS. 2 and 3, an AI question-answering method based on time-synchronized multimodal lecture data according to an embodiment of the present invention is performed through an intelligent question-answering system and comprises: a data collection step of collecting a writing coordinate stream from an electronic whiteboard at predetermined time intervals and collecting a voice stream from a microphone, and assigning a timestamp in milliseconds to each stream; a synchronization step of comparing the timestamps of the writing coordinate stream and the voice stream to perform time synchronization within a predetermined error range; a directive recognition step of recognizing a predefined visual directive pattern in the writing coordinate stream and assigning a semantic tag corresponding to each recognized visual directive; a knowledge creation step of converting the voice stream into text through a voice recognition model and generating a multimodal knowledge chunk including the text, the time-synchronized writing image, the timestamp, and the semantic tag; and an embedding step of vector-embedding the multimodal knowledge chunk and storing it in a vector database (50). The method may be configured to include: a question-answering step of receiving a learner's question, extracting a keyword related to a visual keyword from the question, adjusting the search score or search priority of a knowledge chunk assigned a semantic tag corresponding to the keyword related to the keyword among the knowledge chunks stored in the vector database (50), performing a hybrid search combining vector similarity search and semantic tag-based search, and generating an answer using a large-scale language model with the searched knowledge chunk as a context; and a playback control step of including a link corresponding to the timestamp of the searched knowledge chunk in the answer, and simultaneously playing the lecture video and whiteboard writing at that time in a time-synchronized state when the timestamp link is selected.

[0063] The above intelligent question-and-answer system may include various components capable of performing each of the aforementioned steps, and, for example, a system according to one embodiment of the present invention described above may be applied.

[0064] An AI question-answering method based on time-synchronized multimodal lecture data according to one embodiment of the present invention can proceed through a series of steps to accurately locate and reproduce a lecture point in time that corresponds to a learner's question by chronologically aligning whiteboard writing and utterances occurring during a lecture and reflecting the meaning of visual instructions included in the whiteboard writing.

[0065] First, a data collection step may be performed to collect a writing coordinate stream from an electronic whiteboard at predetermined time intervals, collect a voice stream from a microphone, and assign a timestamp in milliseconds to each stream.

[0066] This step can be performed by the data collection unit (10) of the system, and the writing coordinate stream and the instructor's voice stream resulting from the movement of the pen on the electronic whiteboard can be collected simultaneously and timestamps having the same time standard assigned to each.

[0067] This allows different types of lecture data to be compared temporally in subsequent stages.

[0068] Next, a synchronization step may be performed to compare the timestamps of the whiteboard coordinate stream and the voice stream to perform time synchronization within a predetermined error range.

[0069] This step can be performed by the synchronization unit (20), and can correct the temporal discrepancy that may occur due to the difference in the time of collection of data from the electronic whiteboard and the microphone, thereby aligning the writing and speech that occurred at the same time of the lecture so that they are logically connected.

[0070] Through such time synchronization, the temporal accuracy of knowledge information generated thereafter can be ensured.

[0071] Next, an instruction recognition step may be performed to recognize predefined visual instruction patterns in a whiteboard coordinate stream and assign semantic tags corresponding to each recognized visual instruction.

[0072] This step can be performed by an instruction recognition unit (30), which can automatically identify visual instruction patterns such as asterisks, underlines, circles, arrows, or parentheses by analyzing the coordinate trajectory of the writing coordinate stream.

[0073] Here, a visual indicator pattern may refer to visual expressions repeatedly used by an instructor during a lecture to emphasize or structure specific content, and a semantic tag may be information that structures and expresses the educational meaning of the corresponding visual indicator.

[0074] Subsequently, a knowledge generation step may be performed to convert a voice stream into text through a speech recognition model and generate a multimodal knowledge chunk including the text, a time-synchronized handwriting image, a timestamp, and a semantic tag.

[0075] This step can be performed by the knowledge unit (40), and by combining the speech text corresponding to a specific lecture point in time, the whiteboard image, time information, and semantic tag into a single information unit, a multimodal knowledge chunk in which the lecture context is preserved can be generated.

[0076] Here, a multimodal knowledge chunk may be a minimum unit of lecture knowledge information structured to be utilized in subsequent search and Q&A.

[0077] Next, an embedding step may be performed to vector embedding the generated multimodal knowledge chunk and storing it in a vector database (50).

[0078] This step can be performed by the embedding unit (60), and by converting the knowledge chunk into a vector form based on the text and semantic tag information included in the multimodal knowledge chunk, the semantic similarity between the knowledge can be numerically compared.

[0079] The vector database (50) can be configured to store these vector embedding results so that vector similarity search is possible in a subsequent query step.

[0080] Next, a question-and-answer step may be performed in which a learner's question is received, a keyword related to a directive that references a visual directive is extracted from the question, and the search score or search priority of a knowledge chunk that has been assigned a semantic tag corresponding to the keyword related to the directive is adjusted among the knowledge chunks stored in the vector database (50), thereby performing a hybrid search that combines vector similarity search and semantic tag-based search, and generating an answer using a large-scale language model with the searched knowledge chunk as context.

[0081] This step can be performed by the question-answer unit (70), and when a learner inputs a question that refers to a visual instruction, a knowledge chunk reflecting the corresponding semantic tag is searched first so that a response matching the intent of the question can be provided.

[0082] This enables the generation of accurate answers that reflect the lecture context, compared to simple text similarity-based Q&A.

[0083] Finally, a playback control step may be performed to include a link corresponding to the timestamp of the searched knowledge chunk in the generated answer, and to simultaneously play the lecture video and whiteboard writing at that point in time in a time-synchronized state when the timestamp link is selected.

[0084] This step can be performed by a playback control unit (80), and when a learner selects a timestamp link included in the answer, the lecture video moves to the corresponding lecture point in time, and the whiteboard coordinates corresponding to the same time information are rendered sequentially so that the learner can check the actual lecture scene again.

[0085] Through this, learners can intuitively understand the results of the Q&A along with the lecture video and whiteboard.

[0086] In addition, according to one embodiment of the present invention, the predetermined error range is within 500 milliseconds (ms), the writing coordinate stream is collected at intervals of 50 milliseconds (ms), and each coordinate data may be configured to include an x-coordinate value, a y-coordinate value, pen pressure, pen color, and a timestamp.

[0087] Referring further to Fig. 4, by limiting the collection cycle and time synchronization error range of the whiteboard coordinate stream to specific numerical values, the time accuracy and playback reliability of the multimodal knowledge chunk can be improved.

[0088] That is, the writing coordinate stream can be set to be collected at 50 millisecond intervals, and this can be implemented in such a way that the data collection unit (10) samples the writing input from the electronic whiteboard at regular time intervals to generate coordinate data.

[0089] By collecting the whiteboard coordinate stream at 50-millisecond intervals, sufficient temporal resolution can be secured so that the instructor's whiteboard movements are not oversimplified or omitted, while simultaneously preventing system load caused by the generation of unnecessarily excessive data.

[0090] Such a collection cycle can have critical significance in balancing the continuity and reproduction accuracy of the writing, as well as the quality of the writing rendering in subsequent stages.

[0091] Additionally, when synchronizing time between the writing coordinate stream and the voice stream, a predetermined error range can be limited to within 500 milliseconds, which means that the synchronization unit (20) is configured to determine whether to align based on the maximum allowable time difference by comparing the timestamps of the two streams.

[0092] An error range of within 500 milliseconds can be set as a temporal limit range in which learners do not feel a sense of incongruity when recognizing the whiteboard and speech while watching a lecture video, and if this range is exceeded, there is a possibility that the semantic connectivity between the whiteboard content and the speech content will be reduced.

[0093] Therefore, an error range of within 500 milliseconds can be understood as a critical threshold for maintaining the learner's cognitive immersion when the lecture video and the whiteboard are played back in a time-synchronized state.

[0094] In addition, each writing coordinate data can be configured to include an x-coordinate value, a y-coordinate value, pen pressure, pen color, and a timestamp.

[0095] As such, handwriting coordinate data containing various attributes can reflect the actual expression characteristics of handwriting more precisely than simple position information, and pressure information can be utilized to reproduce emphasis expressions, while pen color information can be used to reproduce visual distinctions used during a lecture.

[0096] Through this, when the writing is played back by the playback control unit (80), the instructor's actual writing style can be faithfully reflected, and more reliable input data can be provided during the process in which the instruction recognition unit (30) recognizes visual instruction patterns.

[0097] Consequently, by specifically defining the collection period of the handwriting coordinate stream, the time synchronization error range, and the components of the handwriting coordinate data, the temporal consistency of the multimodal knowledge chunk and the accuracy of handwriting reproduction can be simultaneously improved.

[0098] Additionally, the step of performing the time synchronization may be configured to include: detecting a writing start event and a writing end event in a writing stream; matching the start timestamp and end timestamp of each writing segment with the timestamp of a voice chunk; and performing interpolation or reordering if the error exceeds a threshold.

[0099] Referring further to FIG. 5, the aforementioned time synchronization step can be further subdivided to precisely perform time alignment with the voice stream based on the start and end of the writing activity.

[0100] First, a step of detecting a writing start event and a writing end event in a writing coordinate stream may be performed, and this may be implemented in such a way that the synchronization unit (20) analyzes whether there is a continuous change in coordinates of the writing coordinate stream and automatically identifies the start and end times of the pen input.

[0101] Here, the whiteboard start event may refer to the point in time when new coordinate data occurs after no coordinate input has occurred for a certain period of time, and the whiteboard end event may refer to the point in time when the state transitions to one where no coordinate input occurs for a certain period of time.

[0102] Such event detection can be used as a criterion for dividing a continuous stream of handwriting coordinates into handwriting segments of meaning.

[0103] Next, a step of matching the start timestamp and end timestamp of each whiteboard segment with the timestamp of a voice chunk generated from a voice stream may be performed.

[0104] At this stage, the synchronization unit (20) can determine which speech content a specific writing action is associated with by identifying the voice chunk that is temporally overlapping with or closest to the time interval in which the writing segment was generated.

[0105] Here, a voice chunk may refer to a segment of voice data obtained by dividing a voice stream into fixed time units or utterance units.

[0106] Next, if the time difference between the writing segment and the voice chunk exceeds a preset threshold, a step of performing interpolation or reordering may be performed.

[0107] Interpolation may be a processing method that supplements time gaps occurring between timestamps of a whiteboard coordinate stream or a voice stream with estimated values, and reordering may be a processing method that rearranges the time order of whiteboard segments or voice chunks to adjust them to align with the logical flow of the lecture.

[0108] This processing is intended to compensate for time deviations that may occur in actual lecture environments where the instructor's writing and speech do not occur completely simultaneously, and can contribute to maintaining the temporal consistency of multimodal knowledge chunks.

[0109] The time synchronization method according to one embodiment of the present invention performs synchronization based on a whiteboard start event and a whiteboard end event, and thus has critical significance in that it can more accurately reflect the actual flow of a lecture compared to a method that simply sorts data at a certain period.

[0110] Through this, the multimodal knowledge chunk generated by the knowledge unit (40) can be configured such that the writing action and the utterance content are semantically connected, and the reliability of the answer and playback result provided by the question-answer unit (70) and the playback control unit (80) can be improved.

[0111] Additionally, in the above-mentioned instruction recognition step, the visual instruction pattern includes at least one of an asterisk, an underline, a circle, an arrow, and parentheses, and is recognized through at least one of a rule-based pattern matching and a machine learning-based classifier that analyzes the geometric characteristics of the coordinate trajectory of the above-mentioned handwriting coordinate stream, and at least one semantic tag among emphasis, key, and test question is assigned to the asterisk, at least one semantic tag among important and memory is assigned to the underline, and at least one semantic tag among definition and concept is assigned to the circle.

[0112] Referring further to Fig. 6, the aforementioned instruction recognition step can be elaborated to recognize visual instruction patterns included in the whiteboard coordinate stream based on geometric characteristics and to assign semantic tags that reflect the educational meaning of each visual instruction.

[0113] Visual indicator patterns may be defined to include at least one of an asterisk, an underline, a circle, an arrow, and parentheses, which may be representative forms of whiteboard expressions that an instructor repeatedly uses during a lecture to emphasize key concepts or to express relationships between concepts.

[0114] Such visual indicator patterns can be the subject of automatic recognition in that they possess specific geometric characteristics that distinguish them from simple drawings or free curves.

[0115] The instruction recognition step can be performed by the instruction recognition unit (30), and can proceed to recognize a visual instruction pattern by analyzing the geometric characteristics of the coordinate trajectory of the writing coordinate stream.

[0116] Geometric characteristics of a coordinate trajectory may refer to mathematical and geometric properties such as changes in the direction of the written coordinates, curvature, whether it is a closed curve, whether it intersects repeatedly, or the ratio of the lengths of the line segments.

[0117] The instruction recognition unit (30) can automatically identify visual instruction patterns within a whiteboard coordinate stream using a rule-based pattern matching method, a machine learning-based classifier, or a combination thereof based on these geometric characteristics.

[0118] A rule-based pattern matching method may be a method that determines a coordinate trajectory satisfying predefined geometric conditions as a specific visual instruction pattern, and a machine learning-based classifier may be configured to classify the type of pattern from learned handwriting coordinate data.

[0119] This complex recognition method can reliably recognize visual instruction patterns despite various handwriting habits or individual differences in whiteboard style.

[0120] For recognized visual indicator patterns, semantic tags can be assigned according to the type of each pattern.

[0121] For example, at least one semantic tag such as emphasis, key, or test question may be assigned to a visual indicator recognized as an asterisk, at least one semantic tag such as important or remember may be assigned to a visual indicator recognized as an underline, and at least one semantic tag such as definition or concept may be assigned to a visual indicator recognized as a circle.

[0122] Such semantic tag assignment is a process of converting the visual representation of the writing into semantic information that can be interpreted by a machine, and can subsequently serve to add semantic metadata to the multimodal knowledge chunk generated in the knowledge unit (40).

[0123] The instruction recognition and semantic tag assignment process expands the whiteboard information into data with a semantic structure rather than simple visual data, thereby enabling the question-answering unit (70) to interpret and respond more accurately to the learner's visual instruction reference questions.

[0124] Through this, learners can engage in Q&A centered on content intentionally emphasized or defined by the instructor during the lecture, and the system can provide responses that reflect the instructor's educational intent.

[0125] Additionally, the embedding step may be configured to include: a step of generating an integrated text by adding the semantic tag information to the text and combining it; and a step of converting the integrated text into a vector embedding through an embedding model and storing it in a vector database (50).

[0126] Referring again to FIG. 2, the aforementioned embedding step can be implemented by combining semantic tag information included in a multimodal knowledge chunk with text, converting it into a vector embedding, and storing it in a vector database (50).

[0127] The embedding step may be a step for integrally reflecting the speech recognition text and the semantic tag information assigned in the instruction recognition step among the multimodal knowledge chunks generated by the knowledge unit (40).

[0128] Here, semantic tags refer to information indicating the educational meaning of visual indicator patterns, which can mean metadata expressed as emphasis, key points, definitions, concepts, etc., corresponding to asterisks, underlines, circles, etc.

[0129] The embedding step can first proceed to the step of generating a combined text by adding semantic tag information to the text.

[0130] In this case, the integrated text can be generated by inserting semantic tags into the original speech recognition text in an identifiable form or by including them together.

[0131] For example, if the speech recognition text of a specific knowledge chunk is "The integral constant C must not be omitted," and an asterisk is recognized on the corresponding writing to assign a semantic tag of emphasis or key, the integrated text can be reconstructed into a sentence structure that reflects the semantic tag information.

[0132] This integrated text generation process allows visually emphasized information during a lecture to be reflected in the embedding result, compared to the case where only spoken text is embedded.

[0133] In other words, even if the text content is identical or similar, knowledge chunks highlighted by visual indicators and knowledge chunks that are not can be distinguished in vector space.

[0134] Subsequently, the embedding step can proceed to converting the generated integrated text into vector embeddings through an embedding model.

[0135] An embedding model can be a model trained to transform natural language sentences into high-dimensional vectors, and can quantify the semantic characteristics of integrated text and map them to a vector space.

[0136] In this case, the generated vector embedding can be used as data representing the meaning of the corresponding multimodal knowledge chunk.

[0137] The converted vector embeddings can be stored in a vector database (50), and the vector database (50) can be configured to efficiently manage vector embeddings for multiple knowledge chunks and perform vector similarity search.

[0138] Through this, the question-and-answer unit (70) can convert the learner's question into a vector and then quickly search for semantically similar knowledge chunks.

[0139] The embedding step according to one embodiment of the present invention performs vector embedding with semantic tag information combined with text, thereby enabling more precise search results for visual indicator-based queries during the subsequent hybrid search process.

[0140] This enables meaning-based question-and-answer that reflects the visually emphasized context during lectures, going beyond simple text similarity-based search.

[0141] Additionally, the process of searching for the relevant knowledge chunk in the above question-and-answer step may be configured to include: a step of converting the question text into a vector; a step of performing a hybrid search combining vector similarity search and directive tag filtering; and a step of prioritizing the search by assigning weights to the knowledge chunk to which the directive tag is assigned when the question contains directive-related keywords.

[0142] Referring again to Fig. 3, the process of searching for relevant knowledge chunks during the aforementioned question-and-answer step can be specified so that vector conversion of the question text, hybrid search execution, and assignment of search weights based on keywords related to instructions can be carried out sequentially.

[0143] The question and answer step can be performed by the question and answer unit (70), and when a learner's question is entered, a step of converting the question text into a vector can be performed first.

[0144] In this case, the question text may be a learner's query sentence entered in the form of natural language, and can be converted into a high-dimensional vector using an embedding model identical or similar to the one used in the embedding step.

[0145] These question vectors are data that numerically represent the meaning of the learner's questions and can subsequently be used as a standard for vector similarity search.

[0146] Next, a hybrid search combining vector similarity search and directive tag filtering can be performed.

[0147] Vector similarity search may be a method of searching for semantically similar knowledge chunks by calculating the distance or similarity between a question vector and a vector embedding of a knowledge chunk stored in a vector database (50).

[0148] Directive tag filtering can be a method of selecting specific types of knowledge chunks or adjusting priorities based on semantic tags assigned to knowledge chunks.

[0149] In particular, if a question includes keywords related to instructions, it can be configured to prioritize searching by assigning weights to knowledge chunks tagged with those instructions.

[0150] Here, keywords related to instructions may refer to expressions in which the learner refers to visual instructions in the question, such as "the starred part," "content said to be important," or "concepts marked with a circle."

[0151] The question and answer unit (70) can extract keywords related to these instructions from the question text and identify semantic tags corresponding to them.

[0152] Weighting can be performed by adjusting the search score or search priority of knowledge chunks to which semantic tags corresponding to directive-related keywords have been assigned.

[0153] For example, if there are multiple knowledge chunks with similar vector similarity scores, the knowledge chunk with matching semantic tags can be sorted with a higher rank.

[0154] This weighted-based priority search can precisely control search results by reflecting that the learner's question intent lies in visual indicators.

[0155] Afterwards, the question-and-answer unit (70) can configure relevant knowledge chunks selected through hybrid search into contexts and provide the contexts to a large-scale language model to generate answers.

[0156] The context may include speech recognition text, semantic tag information, and timestamp information from knowledge chunks, which can support large-scale language models in generating answers that reflect the lecture time and visual emphasis context.

[0157] Unlike search methods that rely on simple text similarity, the related knowledge chunk search process according to one embodiment of the present invention can provide search results that simultaneously consider the learner's query intent and the lecturer's emphasis intent by directly reflecting visual indicator-based semantic information in the search process.

[0158] Through this, it is possible to provide more accurate and explanatory answers to visual reference questions.

[0159] Additionally, the process of simultaneously playing the lecture video and the whiteboard in the playback control step may be configured to include: a step of seeking the lecture video player to the timestamp position; a step of loading the whiteboard coordinates corresponding to the timestamp and rendering them sequentially on the whiteboard canvas; and a step of synchronizing the playback of the lecture video with the rendering of the whiteboard so that the learner can check the instructor's whiteboard process in real time.

[0160] Referring further to FIG. 7, the aforementioned playback control step can be specified so that when a timestamp link included in the answer generated in the question-and-answer step is selected, the lecture video and the whiteboard writing can be played simultaneously in a time-synchronized state.

[0161] The playback control step can be performed by the playback control unit (80), and when a learner selects a link corresponding to a timestamp included in the answer, the playback control unit (80) can first control the lecture video player to seek to the corresponding timestamp location.

[0162] Here, "seek" can refer to the action of immediately moving the playback position of a lecture video to a specified timestamp point.

[0163] At the same time, the playback control unit (80) may be configured to load the writing coordinates corresponding to the selected timestamp.

[0164] The handwriting coordinates may be coordinate data collected from the electronic whiteboard during the data collection phase and stored after a time synchronization process, and only the handwriting coordinates corresponding to the timestamp interval may be selectively loaded.

[0165] Loaded whiteboard coordinates can be controlled to be rendered sequentially on the whiteboard canvas.

[0166] Sequential rendering of a whiteboard canvas can refer to the process in which coordinate data is drawn to be reproduced according to the chronological order in which the instructor actually wrote on the board.

[0167] Through this, learners can check the instructor's writing process step-by-step, rather than just a completed image of the whiteboard.

[0168] The playback control unit (80) can perform synchronization control so that the playback of the lecture video and the rendering of the whiteboard coordinates are time-matched.

[0169] In other words, the playback time of the lecture video and the rendering time of the whiteboard coordinates can be controlled to be synchronized based on the same timestamp.

[0170] Through this synchronization control, the instructor's speech and writing movements can be played back simultaneously without misalignment.

[0171] A playback control step according to one embodiment of the present invention allows a learner to immediately check the lecture content at a specific point in time presented in the question-and-answer results, thereby enabling an intuitive understanding of the context between the lecture video and the whiteboard.

[0172] In particular, by reproducing the actual process of writing on a board that includes visual instructions, along with the instructor's explanation, learners' understanding and engagement can be simultaneously enhanced.

[0173] Furthermore, unlike simple video seek functions or static whiteboard image provision methods, this simultaneous playback method holds significant meaning in that it reproduces time-synchronized multimodal information, thereby naturally extending the Q&A results into the lecture review experience.

[0174] The usage state according to one embodiment of the time-synchronized multimodal lecture data-based AI question-answering system and method of the present invention described above is explained by applying it to a university mathematics lecture as follows (see FIGS. 1 to 8).

[0175] When the lecture begins, the instructor can use the electronic whiteboard to conduct the lecture in the same way as usual.

[0176] At this time, the data collection unit (10) can collect a writing coordinate stream from the electronic whiteboard at predetermined time intervals, and at the same time collect the instructor's voice as a voice stream through a microphone, and assign a timestamp in milliseconds to each stream.

[0177] The instructor can naturally proceed with writing on the board and giving explanations without any separate operation or additional equipment.

[0178] When an instructor marks a star next to a whiteboard or underlines a specific formula while explaining important content during a lecture, the synchronization unit (20) can perform time synchronization within a predetermined error range by comparing the timestamps of the whiteboard coordinate stream and the voice stream.

[0179] In this process, the timing of the writing on the board and the timing of the instructor's speech can be precisely aligned.

[0180] The instruction recognition unit (30) can automatically recognize visual instruction patterns such as stars, underlines, or circles by analyzing a time-synchronized whiteboard coordinate stream.

[0181] For example, if an instructor marks a star next to a formula while saying, "You must remember this part," the instruction recognition unit (30) can analyze the geometric characteristics of the corresponding coordinate trajectory to recognize the star pattern and assign a highlight or key meaning tag to the corresponding visual instruction.

[0182] The knowledge unit (40) can convert a voice stream into text through a voice recognition model and combine the converted text, time-synchronized handwriting image, timestamp and semantic tag to generate a multimodal knowledge chunk.

[0183] The above multimodal knowledge chunk may be data that groups the explanation content and whiteboard state at a specific point in time of a lecture into a single semantic unit.

[0184] The embedding unit (60) can generate an integrated text by combining text and semantic tag information included in a multimodal knowledge chunk, convert it into a vector embedding, and store it in a vector database (50).

[0185] Thus, the entire lecture can be structured and stored in multiple semantic units according to the flow of time.

[0186] After the lecture ends, the learner can enter questions through the Q&A section (70) to check the parts they did not understand during the study.

[0187] For example, when a learner inputs the question "What was the part the teacher starred?", the question-answer unit (70) can extract a keyword related to a visual indicator that refers to the visual indicator from the question.

[0188] The question and answer unit (70) can perform a hybrid search by converting the question text into a vector and adjusting the search score or search priority of the knowledge chunk that has been assigned a semantic tag corresponding to an asterisk among the knowledge chunks stored in the vector database (50).

[0189] As a result, knowledge chunks from a specific point in time during the lecture marked with an asterisk can be searched preferentially.

[0190] The question-and-answer unit (70) can generate an answer using a large-scale language model with the retrieved knowledge chunk as context, and the generated answer may include a link corresponding to the timestamp of the knowledge chunk.

[0191] For example, an answer such as "The content highlighted with an asterisk at 3 minutes 42 seconds is an explanation regarding the integration constant" may be provided.

[0192] When a learner selects a timestamp link included in the answer, the playback control unit (80) can search the lecture video player to the corresponding timestamp location and load the handwriting coordinates at that point in time to sequentially render them on the handwriting canvas.

[0193] At this time, the lecture video and whiteboard writing are played simultaneously in a time-synchronized state, allowing learners to view the instructor's explanation and the whiteboard writing process together.

[0194] Through this, learners can go beyond simply checking answers to questions as text and intuitively review based on actual lecture scenes.

[0195] In addition, regarding the segmentation of semantic tags based on the temporal persistence of a directive, according to one embodiment of the present invention, the directive recognition unit (30) can increase the importance of the semantic tag assigned to the visual directive when the same visual directive is repeated for a certain period of time or is maintained across a plurality of writing segments.

[0196] For example, a distinction can be made between cases where an asterisk is temporarily displayed next to a single formula and cases where it is repeatedly displayed across multiple whiteboard sections, and in the latter case, a key semantic tag can be assigned preferentially.

[0197] Through this, the Q&A section (70) can search for content that is repeatedly emphasized throughout the lecture with a higher priority.

[0198] In addition, regarding the verification of the combination of voice expressions and visual indicators, according to one embodiment of the present invention, the knowledge unit (40) can determine whether the emphasis expression and the visual indicator included in the voice recognition text are detected simultaneously in the same time interval.

[0199] If an instructor marks a star while saying "This is really important," the corresponding multimodal knowledge chunk may be assigned an enhanced semantic tag.

[0200] This can increase the reliability of semantic tags and compensate for simple misrecognition of whiteboard text.

[0201] In addition, regarding the automatic switching of search strategies according to the type of query, according to one embodiment of the present invention, the question-answer unit (70) determines whether the learner's question corresponds to a concept verification type, a procedure explanation type, or a summary request type, and can automatically adjust the weight ratio of the hybrid search according to the type of question.

[0202] For example, for questions like "What is justice?", the search score of knowledge chunks tagged with circle meanings can be relatively higher.

[0203] This allows for the provision of search results tailored to the purpose of the question while using the same database.

[0204] In addition, regarding search correction based on query history per learner, according to one embodiment of the present invention, the question-answer unit (70) analyzes the type of knowledge chunk that the same learner has repeatedly queried, and can give additional weight to the knowledge chunk of the semantic tag type that the learner frequently refers to during subsequent searches.

[0205] This enables customized Q&A that reflects individual learners' weak areas of understanding.

[0206] In addition, according to one embodiment of the present invention regarding the correction of knowledge chunk division based on handwriting density, the knowledge unit (40) can detect a section where the density of handwriting coordinates rapidly increases or decreases, and adjust the timing of the division of the multimodal knowledge chunk.

[0207] This prevents a single concept description from being unnecessarily separated into multiple knowledge chunks and enables the creation of knowledge chunks centered on semantic units.

[0208] In the foregoing, although all components constituting an embodiment of the present invention have been described as being combined or operating in combination, the present invention is not necessarily limited to such embodiments. That is, within the scope of the purpose of the present invention, all components may be selectively combined in one or more ways to operate. Furthermore, terms such as "include," "constitute," or "have" described above, unless specifically stated otherwise, mean that the relevant component may be inherent; thus, they should be interpreted as allowing for the inclusion of additional components rather than excluding other components. All terms, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains, unless otherwise defined. Terms commonly used, such as those defined in advance, should be interpreted in accordance with their meaning in the context of the relevant technology and, unless explicitly defined in the present invention, should not be interpreted in an ideal or overly formal sense.

[0209] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in this invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by such embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0210] 10: Data Collection Unit 20: Synchronization Unit 30: Directive recognition unit 40: Knowledge generation unit 50: Vector Database 60: Embedding Section 70: Q&A section 80: Playback control section

Claims

Claim 1 Performed through an intelligent question-and-answer system, the method comprises: a data collection step of collecting a writing coordinate stream from an electronic whiteboard at predetermined time intervals and a voice stream from a microphone, and assigning a timestamp in milliseconds to each stream; a synchronization step of comparing the timestamps of the writing coordinate stream and the voice stream to perform time synchronization within a predetermined error range; a directive recognition step of recognizing a predefined visual directive pattern in the writing coordinate stream and assigning a semantic tag corresponding to each recognized visual directive; a knowledge creation step of converting the voice stream into text through a speech recognition model and generating a multimodal knowledge chunk including the text, the time-synchronized writing image, the timestamp, and the semantic tag; an embedding step of vector-embedding the multimodal knowledge chunk and storing it in a vector database; receiving a learner's question, extracting directive-related keywords referencing visual directives from the question, and then adjusting the search score or search priority of a knowledge chunk among the knowledge chunks stored in the vector database that has a semantic tag assigned to it corresponding to the directive-related keyword, thereby performing vector similarity search and semantic tag A question-and-answer step of performing a hybrid search combined with a base search and generating an answer using a large-scale language model with the searched knowledge chunk as context; and a playback control step of including a link corresponding to the timestamp of the searched knowledge chunk in the answer, and simultaneously playing a lecture video and whiteboard writing at a corresponding point in time in a time-synchronized state when the timestamp link is selected, wherein the predetermined error range is within 500 milliseconds (ms), the whiteboard coordinate stream is collected at intervals of 50 milliseconds (ms), and each coordinate data includes an x-coordinate value, a y-coordinate value, pen pressure, pen color, and a timestamp, and the step of performing the time synchronization includes detecting a whiteboard start event and a whiteboard end event in the whiteboard stream;The method includes the step of matching the start and end timestamps of each whiteboard segment with the timestamp of the voice chunk; and the step of performing interpolation or reordering if the error exceeds a threshold value; the embedding step includes the step of generating an integrated text by adding the semantic tag information to the text and combining it; and the step of converting the integrated text into a vector embedding through an embedding model and storing it in the vector database; the process of searching for the relevant knowledge chunk in the question-answer step includes the step of converting the question text into a vector; the step of performing a hybrid search combining vector similarity search and directive tag filtering; and the step of prioritizing the search by assigning weights to the knowledge chunk to which the directive tag is assigned when the question contains a directive-related keyword; and the process of simultaneously playing the lecture video and the whiteboard in the playback control step includes the step of seeking the lecture video player to the timestamp position; and the step of loading the whiteboard coordinates corresponding to the timestamp and sequentially rendering them on the whiteboard canvas. A time-synchronized multimodal lecture data-based AI question-answering method characterized by including the step of synchronizing the playback of the lecture video with the rendering of the whiteboard so that the learner can check the instructor's whiteboard process in real time. Claim 2 delete Claim 3 delete Claim 4 The AI ​​question-answering method based on time-synchronized multimodal lecture data according to claim 1, wherein in the instruction recognition step, the visual instruction pattern includes at least one of an asterisk, an underline, a circle, an arrow, and parentheses, and is recognized through at least one of a rule-based pattern matching and a machine learning-based classifier that analyzes the geometric characteristics of the coordinate trajectory of the whiteboard coordinate stream, wherein at least one semantic tag among emphasis, key, and test question is assigned to the asterisk, at least one semantic tag among important and memory is assigned to the underline, and at least one semantic tag among definition and concept is assigned to the circle. Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 A data collection unit that collects a handwriting coordinate stream from an electronic whiteboard at predetermined time intervals and a voice stream from a microphone, and assigns a timestamp in milliseconds to each stream; a synchronization unit that compares the timestamps of the handwriting coordinate stream and the voice stream to perform time synchronization within a predetermined error range; a directive recognition unit that recognizes a predefined visual directive pattern in the handwriting coordinate stream and assigns a semantic tag corresponding to each recognized visual directive; a knowledge generation unit that converts the voice stream into text through a speech recognition model and generates a multimodal knowledge chunk including the text, the time-synchronized handwriting image, the timestamp, and the semantic tag; an embedding unit that vector-embeds the multimodal knowledge chunk and stores it in a vector database; and a hybrid search configured to receive a learner's question, extract directive-related keywords referencing visual directives from the question, and then adjust the search score or search priority of a knowledge chunk among the knowledge chunks stored in the vector database that has a semantic tag assigned to it corresponding to the directive-related keyword, thereby combining vector similarity search and semantic tag-based search. A question-answering unit that searches for related knowledge chunks and generates answers using a large-scale language model with the searched knowledge chunks as context;and includes a playback control unit configured to include a link corresponding to the timestamp of the searched knowledge chunk in the above answer, and to simultaneously play the lecture video and whiteboard handwriting at that point in time in a time-synchronized state when the timestamp link is selected, wherein the predetermined error range is within 500 milliseconds (ms), the whiteboard coordinate stream is collected at intervals of 50 milliseconds (ms), and each coordinate data includes an x-coordinate value, a y-coordinate value, pen pressure, pen color, and a timestamp; the synchronization unit is configured to detect a whiteboard start event and a whiteboard end event in the whiteboard stream, match the start timestamp and end timestamp of each whiteboard segment with the timestamp of the voice chunk, and perform interpolation or reordering if the error exceeds a threshold; the embedding unit is configured to generate an integrated text by adding the semantic tag information to the text and combining it, and to convert the integrated text into a vector embedding through an embedding model and store it in the vector database; the question-answer unit is configured to convert the question text into a vector and vector A time-synchronized multimodal lecture data-based AI question-and-answer system characterized by performing a hybrid search combining similarity search and directive tag filtering, and, if the question contains directive-related keywords, assigning weights to knowledge chunks to which the corresponding directive tags are assigned to prioritize the search, and the playback control unit seeking the lecture video player to the timestamp location, loading the whiteboard coordinates corresponding to the timestamp and rendering them sequentially on the whiteboard canvas, and synchronizing the playback of the lecture video with the rendering of the whiteboard so that the learner can check the instructor's whiteboard process in real time.

Images