An AI-powered question-answering method and system based on the correlation between video keyframes and progress.

Abstract

This application provides an AI-powered Q&A method and system based on the correlation between video keyframes and progress. Specifically, this application captures the progress and keyframe coordinates of knowledge points when the user pauses the video. It generates a touch heatmap of knowledge points and attention markers based on touch trajectories. The application then integrates the heatmap, markers, coordinate topology, and pre-pause speech semantics to generate a question confidence level. Based on this confidence level, a decision is made: high confidence results in a direct answer; low confidence activates a dynamic follow-up question engine, analyzing the knowledge point type and matching it to a subject-specific strategy to generate follow-up question options. After the user selects an option, the application outputs Q&A content containing core answers and knowledge extensions. This application accurately identifies user questions by integrating multimodal information such as video keyframes, touch heatmaps, and speech semantics, and intelligently makes decisions based on confidence levels to achieve instant and accurate Q&A or guided follow-up questions, ultimately generating personalized Q&A content containing core answers and extended knowledge.

Description

Technical Field

[0001] This application relates to the field of intelligent educational video interaction technology, and in particular to an AI Q&A method and system based on the correlation between video keyframes and progress. Background Technology

[0002] In intelligent education video platforms, users often pause videos because they don't understand specific knowledge points. To provide efficient and accurate Q&A services, there is an urgent need for a technology that can automatically understand a user's questions and respond intelligently at the moment of pause, based on the current video progress, keyframe visual content, and the user's possible interactive intent. The core requirements of this technology are: accurately capturing the focus of knowledge points at the moment of pause, understanding the context of the user's potential questions, and proactively initiating multi-round follow-up questions related to the context when the AI ​​itself cannot accurately determine the details of the question, such as "Do you have a question about the data formula in area A of the diagram, or do you not understand the operation steps in area B?", to ultimately pinpoint the problem and provide layered solutions, thereby significantly improving the accuracy of Q&A and the user experience.

[0003] Currently, one targeted technical solution involves extracting keywords from video subtitles or Automatic Speech Recognition (ASR) text near the pause point and matching them with a pre-defined question-and-answer knowledge base. This solution analyzes the text content before and after the pause to identify possible keywords or phrases, compares them with question-and-answer entries organized by knowledge points, and directly returns the most matching entry as the answer to the user. Some implementations combine simple progress markers to associate them with general knowledge points.

[0004] However, this existing solution has significant limitations: it relies heavily on textual information, completely ignoring the rich visual elements and their spatial relationships in keyframes, making it impossible to locate the actual focus of the question in the image; at the same time, the solution is essentially a passive response mechanism, lacking the ability to assess the confidence level of user questions, and when the matching of text keywords is ambiguous or unclear, the system cannot proactively initiate targeted multi-round follow-up questions based on the current image content to clarify details; in addition, its understanding of the coherence of the speech semantics before the pause and the topological relationships of image elements is weak, making it difficult to support intelligent decision-making combined with multimodal context, ultimately leading to inaccurate or irrelevant answers to complex questions. Summary of the Invention

[0005] This application provides an AI question-answering method and system based on the correlation between video keyframes and progress, in order to solve the problem in the prior art that the reliance on single text matching ignores visual focus and user interaction intent, resulting in the inability to accurately locate question points on the screen and the lack of an active questioning mechanism.

[0006] Firstly, this application provides an AI-based question-answering method based on the correlation between video keyframes and progress, including:

[0007] When the user performs a video pause operation, capture the progress indicator and the corresponding key frame, and extract the spatial coordinate information of the target area representing the knowledge point in the key frame;

[0008] The touch trajectory data that meets the pressure triggering conditions is captured by the touch screen, and a spatial association map between the touch hot zone and the knowledge point is generated by combining the progress indicator and the spatial coordinate information. The attention area marker is generated based on the dwell characteristics of the touch trajectory data.

[0009] By integrating the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest markers, the topological relationships of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation, a question confidence score is generated.

[0010] The decision routing is based on the quantification level of the question confidence. When the direct response threshold is reached, instant answer content is generated. When the direct response threshold is not reached, the dynamic follow-up question engine is activated.

[0011] The dynamic questioning engine parses the semantic types of knowledge points in the spatial association graph to match the subject-specific questioning strategy library and generate a set of questioning options.

[0012] Based on the user's selection of options in the set of follow-up questions, answer content containing a core answer layer and an associated knowledge extension layer is generated.

[0013] Optionally, the step of parsing the semantic types of knowledge points in the spatial association graph through the dynamic questioning engine to generate a set of questioning options by matching them with a subject-specific questioning strategy library includes:

[0014] Extract the target regions of knowledge points corresponding to strongly correlated markers in the spatial correlation map, and classify the target regions into semantic types such as formula, concept, or chart based on their content;

[0015] Based on the semantic type matching subject-specific follow-up questioning strategy library, call the step derivation option set corresponding to formula class, the definition comparison option set corresponding to concept class, or the data interpretation option set corresponding to chart class as the basic follow-up questioning option set;

[0016] Based on the average duration of all touch points in the touch trajectory data, the priority order of the options in the basic follow-up question option set is adjusted, and the high-frequency question options are arranged at the top to generate the final follow-up question option set.

[0017] Optionally, adjusting the priority order of options in the basic follow-up question option set based on the average duration of all touch points in the touch trajectory data, and arranging the high-frequency question options at the top to generate the final follow-up question option set, includes:

[0018] Calculate the arithmetic mean of the duration of all touch points in the touch trajectory data as the average duration, and count the number of times each option in the basic follow-up question option set has been selected from the historical Q&A database.

[0019] The options in the basic follow-up question set are rearranged in descending order according to the historical selection frequency to generate an adjusted set with the most frequently asked question options at the top.

[0020] When the average duration exceeds the interaction depth threshold, deep parsing options are added to the rearranged basic follow-up question option set to form the final follow-up question option set;

[0021] When the average duration does not exceed the interaction depth threshold, the rearranged basic set of follow-up questions will be directly used as the final set of follow-up questions.

[0022] Optionally, the step of generating answer content containing a core answer layer and a related knowledge extension layer based on the user's selection operation of options in the set of follow-up questions includes:

[0023] In response to a user clicking on a specific option in the set of follow-up questions, obtain the identification information of that specific option;

[0024] Based on the identification information, standard analysis content corresponding to the knowledge points is extracted from the pre-stored parsed text library as the core solution layer;

[0025] Based on the progress markers and marker information, the video metadata database is retrieved to obtain the background knowledge module and advanced knowledge module associated with the current knowledge point, and they are combined to form an associated knowledge extension layer.

[0026] The core answer layer is placed at the beginning of the Q&A content, and the related knowledge extension layer is attached to the end of the core answer layer in an expandable hierarchical structure to generate complete Q&A content.

[0027] Optionally, the step of capturing touch trajectory data that meets the pressure triggering conditions through the touch screen, generating a spatial association map between touch hotspots and knowledge points by combining the progress indicator and the spatial coordinate information, and generating a region of interest marker based on the dwell characteristics of the touch trajectory data includes:

[0028] Monitor the pressure sensor data of the touch screen. When the pressure value continuously exceeds the preset pressure threshold, record the touch point coordinate sequence and the duration of each touch point to form touch trajectory data.

[0029] A closed polygon formed by continuous touch points is defined as a touch hot zone. The spatial coordinate information of the touch hot zone and the target area of ​​each knowledge point is compared. Based on the comparison results, a spatial association map between the touch hot zone and the knowledge point is generated.

[0030] Touch points whose duration exceeds the dwell threshold in the touch trajectory data are extracted as dwell feature points. A circular marker with a radius proportional to the duration is generated as a region of interest marker, with the coordinates of each dwell feature point as the center.

[0031] Optionally, the step of comparing the spatial coordinates of the touch hotspot with the target areas of each knowledge point, and generating a spatial association map between the touch hotspot and the knowledge points based on the comparison results, includes:

[0032] When the closed polygon formed by the touch hot area completely covers the spatial coordinate information of a single knowledge point target area, a strong association mark between the touch hot area and the corresponding knowledge point is established.

[0033] When the closed polygon formed by the touch hot area simultaneously covers the spatial coordinate information of multiple knowledge point target areas, a weak association marker is established between the touch hot area and the corresponding multiple knowledge points.

[0034] By aggregating the strong and weak association tags, a spatial association map is generated that contains a set of tags relating touch hotspots to knowledge points.

[0035] Optionally, the step of fusing the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest markers, the topological relationships of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation to generate a question confidence score includes:

[0036] The ratio of the number of strongly correlated markers to the total number of markers in the spatial correlation map is used as a spatial distribution characteristic.

[0037] The straight-line distance between the center coordinates of each region of interest marker and the center point coordinates of the nearest knowledge point target region is calculated as a geometric attribute.

[0038] Traverse the spatial coordinate information of all knowledge point target regions. When the distance between the bounding boxes of any two regions is less than the distance threshold, establish a connection relationship. Count the total number of such connections as the topological relationship.

[0039] The audio waveform within a set time before the pause operation is extracted and converted into text. The frequency of occurrence of predefined question keywords is counted as semantic features.

[0040] The spatial distribution characteristics, geometric attributes, topological relationships, and semantic features are input into a weighted calculation model to perform a comprehensive calculation to generate the question confidence level.

[0041] Optionally, the decision routing based on the quantification level of the question confidence, generating immediate answer content when the direct response threshold is reached, and activating the dynamic follow-up question engine when the direct response threshold is not reached, includes:

[0042] The confidence level of the question is compared numerically with the preset direct response threshold;

[0043] When the confidence level of the question is greater than or equal to the direct response threshold, the pre-stored parsed text corresponding to the target area of ​​the current knowledge point is retrieved to generate instant Q&A content.

[0044] When the confidence level of a question is less than the direct response threshold, the dynamic follow-up question engine is activated and follow-up question option generation is performed, while the generation of instant answer content is prohibited.

[0045] Optionally, the step of capturing the progress indicator and corresponding keyframe when the user performs a video pause operation, and extracting the spatial coordinate information of the target area representing the knowledge point in the keyframe, includes:

[0046] When the video playback engine receives a pause command, it reads the current time value on the video timeline as a progress indicator.

[0047] The frame buffer module of the video decoder is invoked to output a static image that is precisely aligned with the progress indicator as a keyframe.

[0048] Identify independent blocks in the keyframe that conform to the characteristics of text clustering areas, formula clustering areas, or graphic clustering areas as target areas of knowledge points, and simultaneously generate spatial coordinate information including the bounding box positioning coordinates, bounding box size, and center point coordinates of each target area.

[0049] Secondly, this application provides an AI question-answering system based on the correlation between video keyframes and progress, including:

[0050] The acquisition module is used to capture the progress indicator and the corresponding key frame when the user performs a video pause operation, and extract the spatial coordinate information of the target area representing the knowledge point in the key frame;

[0051] The generation module is used to capture touch trajectory data that meets the pressure triggering conditions through the touch screen, combine the progress indicator and the spatial coordinate information to generate a spatial association map between the touch hot zone and the knowledge point, and generate a region of interest marker based on the dwell characteristics of the touch trajectory data.

[0052] The fusion module is used to fuse the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest markers, the topological relationship of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation to generate a question confidence score.

[0053] The decision module is used to make decision routing based on the quantitative level of the question confidence. When the direct response threshold is reached, instant answer content is generated. When the direct response threshold is not reached, the dynamic follow-up question engine is activated.

[0054] The parsing module is used to parse the semantic types of knowledge points in the spatial association graph through the dynamic questioning engine, so as to match the subject-specific questioning strategy library to generate a set of questioning options;

[0055] The output module is used to generate answer content, which includes a core answer layer and a related knowledge extension layer, based on the user's selection operation of the options in the set of follow-up questions.

[0056] In this application example, when a user performs a video pause operation, a progress indicator and corresponding keyframe image are captured, and spatial coordinate information representing the target area of ​​the knowledge point in the keyframe image is extracted; touch trajectory data that meets the pressure trigger condition is captured through the touch screen, and a spatial association map between the touch hot zone and the knowledge point is generated by combining the progress indicator and the spatial coordinate information, and an attention area marker is generated based on the dwell characteristics of the touch trajectory data; the spatial distribution characteristics of the spatial association map, the geometric attributes of the attention area marker, the topological relationship of the spatial coordinate information, and the semantic features of the associated voice segment before the pause operation are integrated to generate a question confidence level; decision routing is performed based on the quantification level of the question confidence level, and instant Q&A content is generated when the direct response threshold is reached, and a dynamic follow-up question engine is activated when the direct response threshold is not reached; the semantic type of the knowledge point in the spatial association map is parsed by the dynamic follow-up question engine to match the subject-specific follow-up question strategy library to generate a follow-up question option set; based on the user's selection operation of the options in the follow-up question option set, Q&A content containing a core answer layer and an associated knowledge extension layer is generated.

[0057] The technical solution of this application has the following beneficial effects:

[0058] This application captures progress markers and keyframe knowledge point coordinates when the video is paused, and combines these with touch trajectories to generate a spatial correlation map of touch hotspots and knowledge points, as well as markers for areas of interest. It then dynamically generates question confidence levels by integrating the spatial distribution of the map, the geometric attributes of the markers, the topological relationships of the coordinates, and the semantic features of the speech before the pause. Based on the confidence level, intelligent decision-making routing is implemented. When a threshold is reached, the answer is directly output; otherwise, a dynamic follow-up question engine is activated, generating follow-up question options by parsing the semantic type of the knowledge points and matching them with a subject strategy library. Finally, based on the user's selection, layered answer content containing a core answer layer and a related knowledge extension layer is output. This achieves precise question positioning, adaptive interactive follow-up questioning, and structured knowledge-based answers under multimodal information fusion, significantly improving the accuracy of Q&A in video scenarios and enhancing the user experience.

[0059] The further dynamic follow-up question engine extracts knowledge point regions corresponding to strongly correlated markers in the spatial correlation graph, classifying their content into semantic types such as formulas, concepts, or charts. It then matches these with a subject-specific follow-up question strategy library, calling upon formula derivation steps, concept definition comparisons, or chart data interpretation options as basic follow-up question options. Finally, based on the average duration of touch points, the priority of each option in the basic options is dynamically adjusted to generate a final, optimized set of follow-up question options. This solution achieves precise customization of subject-related follow-up question strategies based on the semantic type of knowledge points and intelligently adjusts option priorities by analyzing user touch behavior. This generates a highly contextualized set of follow-up question options that aligns with users' potential question tendencies, significantly improving the targeting of follow-up questions and user selection efficiency, effectively guiding users to quickly clarify core questions.

[0060] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart of an AI question-answering method based on the correlation between video keyframes and progress, provided in this application, is shown.

[0063] Figure 2 The illustration shows a scenario diagram of an AI question-answering method based on the correlation between video keyframes and progress, as provided in this application.

[0064] Figure 3 The diagram shows a structural schematic of an AI Q&A system based on the correlation between video keyframes and progress, as provided in this application. Detailed Implementation

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0066] In some of the processes described in the specification, claims, and accompanying drawings of this application, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as 101, 102, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a chronological order, nor do they limit "first" and "second" to different types.

[0067] Research indicates that current AI-powered Q&A technologies in intelligent educational video platforms generally rely on keyword matching mechanisms for subtitles and text near pause points. This approach suffers from fundamental flaws: it severely neglects core visual information in keyframes of the video and the explicit intentions expressed by users through touch interaction, resulting in an inability to accurately locate the actual object of the question on the screen; furthermore, this approach is essentially a passive response mode, lacking the ability to dynamically assess the confidence level of user questions; additionally, the understanding of the semantic coherence of speech before pauses and the spatial topological relationships of screen elements is fragmented, making it difficult to support multimodal association analysis of complex knowledge points. These flaws collectively lead to Q&A responses that often deviate from the user's true focus, providing broad or incorrect answers, and resulting in a clunky interactive experience.

[0068] To address the aforementioned issues, this application proposes an AI-powered question-answering method based on the correlation between video keyframes and progress. Its core lies in: simultaneously capturing progress markers, spatial coordinates of keyframe knowledge points, and touch trajectory data meeting pressure conditions at the moment the user pauses the video; fusing these data to generate a spatial correlation map of touch hotspots and knowledge points, along with markers for areas of interest; then, combining the spatial distribution characteristics of the map, the geometric attributes of the markers, the topological relationships of the coordinates, and the semantic features of the pre-pause audio segment, dynamically generating a multimodal fusion question confidence level. Intelligent decision-making routing is performed based on the confidence level quantification. When a threshold is reached, a precise answer is directly generated; otherwise, a dynamic follow-up question engine is activated. This engine parses the semantic type of knowledge points in the map, matches them with a subject-specific strategy library to generate a customized set of follow-up question options, and outputs layered content containing a core answer layer and a related knowledge extension layer after the user selects an option. This method fundamentally solves three major pain points of existing technologies: First, by fusing spatial coordinates and touch trajectories, it accurately captures the visual focus and interaction intent in the screen, eliminating "visual blind spots"; Second, based on confidence routing and dynamic follow-up questioning mechanisms, it achieves a leap from passive response to proactive guided interaction, effectively clarifying ambiguous questions; Third, by integrating multimodal context, it supports the associative understanding and structured answers to complex knowledge points, significantly improving the accuracy of answering questions and user experience.

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0070] Figure 1 This application provides a flowchart of an AI question-answering method based on the correlation between video keyframes and progress, as shown in the embodiments. Figure 1 As shown, the method includes:

[0071] 101. When the user performs a video pause operation, capture the progress indicator and the corresponding key frame, and extract the spatial coordinate information of the target area representing the knowledge point in the key frame;

[0072] Optionally, step 101 may specifically include the following steps:

[0073] 1011. When the video playback engine receives a pause command, it reads the current time value on the video timeline as a progress indicator;

[0074] 1012. Call the frame buffer module of the video decoder and output a static image that is precisely aligned with the progress indicator as a keyframe image;

[0075] 1013. Identify independent blocks in the keyframe that conform to the characteristics of text clustering area, formula clustering area, or graphic clustering area as target areas of knowledge points, and simultaneously generate spatial coordinate information including the bounding box positioning coordinates, bounding box size, and center point coordinates of each target area.

[0076] In the above scheme, the progress marker refers to the precise timestamp captured by the system from the video timeline when the user triggers a pause operation during video playback. It uniquely identifies the moment the pause occurred and serves as the core index for associating the corresponding video content segment. The keyframe refers to the static image frame precisely corresponding to the moment the user pauses the video, located and extracted by the system based on the progress marker. It completely preserves the visual scene seen by the user at the moment of pause. The knowledge point target region refers to the independent visual block carrying core teaching information identified by the system through analysis within the keyframe. These regions are the physical carriers of potential user questions. Spatial coordinate information refers to the data set used to quantitatively describe the specific location and range of the knowledge point target region within the keyframe. It accurately characterizes the geometric attributes of the target region, providing a foundation for subsequent analysis of the spatial relationship between user interaction focus and knowledge points.

[0077] In this embodiment, firstly, in step 1011, when the user triggers a pause operation through the video playback interface, the system monitors and captures this pause event. Then, it calls the video playback engine's application programming interface (API) to read the precise position of the current video playback head on the timeline. This value, typically in milliseconds (ms), is recorded by the system and defined as a progress indicator. For example, if the user clicks the pause button or presses the spacebar through the app's built-in player, the system captures this pause event and captures the precise position value 00:02:05.670 on the timeline as the progress indicator. This progress indicator is the core basis for locating the video content in subsequent steps.

[0078] Next, after obtaining the progress indicator generated in step 1011 through step 1012, the system calls the frame positioning function provided by the underlying video decoder, passing the progress indicator as an input parameter to the decoder. The decoder then searches for the nearest keyframe before or after that point in time, based on the video's encoding structure. Once the corresponding keyframe is found, the decoder extracts the complete static image data corresponding to that keyframe from the keyframe buffer. This static image data is the keyframe image that strictly corresponds to the content at the moment the user pauses the video. For example, the progress indicator 00:02:05.670 might actually correspond to frame 3012 of the video, and the decoder would output the image of frame 3012.

[0079] Finally, after obtaining the keyframe image output in step 1012 via step 1013, the pre-trained object detection model is used to analyze the image. This object detection model is designed to identify specific visual blocks in an image that carry knowledge content. The object detection model scans the entire image and outputs all detected candidate blocks, their category confidence scores, and bounding box coordinates. Next, the system applies a rule engine to filter and select all blocks detected by the model. Blocks containing dense, line-arranged text content, typically with a text width significantly greater than its height, are selected as text clusters; blocks containing numerous mathematical symbols, operators, and variables are selected as formula clusters; and blocks containing clear geometric shapes, data charts, or diagrams are selected as graphic clusters. Blocks identified by the model and conforming to at least one of the above feature rules are confirmed as target regions of knowledge points. For each target region of a knowledge point, its bounding box information is calculated: the coordinates of the top-left pixel of the target region's bounding box are obtained. and the coordinates of the bottom right pixel Used as bounding box positioning coordinates; based on the bounding box positioning coordinate information, calculate the width including the bounding box. and height Bounding box dimensions and center point coordinates The bounding box positioning coordinates, bounding box dimensions, and center point coordinates are packaged together to form spatial coordinate information describing the location and extent of the target area, and a set of spatial coordinate information for all target areas is output.

[0080] In practical applications, suppose a user is watching a physics video lesson on the online learning platform "EduLearn" explaining "Newton's Second Law F=ma". When the video reaches the point where it's explaining the derivation of the formula and showing a slide containing the formula F=ma and a force diagram, the user becomes confused and presses the pause button at time 00:08:23.105. The system receives the pause command and immediately records this precise timestamp. Next, the system calls the platform's video decoding module, and based on the timestamp 00:08:23.105, locates the keyframe corresponding to that moment as frame 150. The decoder successfully extracts the static image of this frame, which contains the slide containing the formula and force diagram. Then, the system inputs this slide image into the object detection model. The model identifies several candidate areas in the image: the title bar "Newton's Second Law" at the top of the page, the formula block in the middle, the force diagram block on the right, and the page number at the bottom of the page. The rules engine begins to work: While the title bar contains text, it's generally not considered a core knowledge point area; page number information is irrelevant and is filtered out; the formula block contains numerous mathematical symbols and is identified as a formula cluster area; the force analysis diagram contains arrows representing force vectors and squares representing objects and is identified as a graphic cluster area. These two areas are confirmed as the target areas for the knowledge points. Their spatial coordinates are then calculated: the formula block's bounding box has its top-left corner at (120, 80) and bottom-right corner at (400, 300), so its width is 280 pixels, its height is 220 pixels, and its center point is at (260, 190); the force diagram block's bounding box is (450, 150, 800, 400), its width is 350 pixels, its height is 250 pixels, and its center point is (625, 275).

[0081] The above-mentioned 101 overall solution achieves automated and accurate localization of knowledge point areas during pauses: by strictly aligning timestamps and keyframes, it ensures that the captured image matches the user's question scenario; by combining object detection and rule filtering, it intelligently identifies core knowledge blocks and quantifies their spatial location, laying a data foundation for subsequent integration of touch interaction and semantic analysis, and avoiding the localization deviation problem caused by relying on manual annotation or text matching in traditional solutions.

[0082] 102. Capture touch trajectory data that meets the pressure triggering conditions through the touch screen, combine the progress indicator and the spatial coordinate information to generate a spatial association map between the touch hot zone and the knowledge point, and generate a region of interest marker based on the dwell characteristics of the touch trajectory data;

[0083] Optionally, step 102 may specifically include the following steps:

[0084] 1021. Monitor the pressure sensor data of the touch screen. When the pressure value continuously exceeds the preset pressure threshold, record the touch point coordinate sequence and the duration of each touch point to form touch trajectory data.

[0085] 1022. Define the closed polygon formed by continuous touch points as the touch hot zone, compare the position of the touch hot zone with the spatial coordinate information of each knowledge point target area, and generate a spatial association map between the touch hot zone and the knowledge point based on the comparison results.

[0086] Specifically, step 1022 may include the following process: when the closed polygon formed by the touch hot area completely covers the spatial coordinate information of a single knowledge point target area, a strong association marker between the touch hot area and the corresponding knowledge point is established; when the closed polygon formed by the touch hot area simultaneously covers the spatial coordinate information of multiple knowledge point target areas, a weak association marker between the touch hot area and the corresponding multiple knowledge points is established; the strong association markers and weak association markers are aggregated to generate a spatial association map containing a set of association markers between the touch hot area and the knowledge points.

[0087] 1023. Extract touch points in the touch trajectory data whose duration exceeds the dwell threshold as dwell feature points, and generate circular markers with a radius proportional to the duration as interest area markers, centered on the coordinates of each dwell feature point.

[0088] In the above scheme, the pressure trigger condition refers to the lower limit of the touchscreen pressure sensor value preset by the system. When the pressure value generated by the user's touch operation continuously exceeds this lower limit, the system determines that the touch is a valid interaction rather than an unintentional touch, thus filtering out interference signals. Touch trajectory data refers to the dynamic recording information generated by the user's touch behavior that meets the pressure trigger condition. It includes the continuous coordinate sequence of the touch points on the screen and the duration of continuous pressing of each touch point, completely depicting the screen path actively marked or focused on by the user. The touch hot zone refers to the closed polygonal area formed by connecting the beginning and end of the continuous touch point coordinate sequence. This area intuitively represents the range of visual content that the user intends to highlight in screen space. The spatial association map is a structured data model that establishes strong or weak association markers between the touch hot zone and the knowledge point target area by calculating the inclusion relationship between the spatial coordinate information of the touch hot zone and the knowledge point, forming a mapping map that reflects the correspondence between the user's question focus and the location of the knowledge content. Dwell time characteristics refer to specific touch point behavior patterns identified from touch trajectory data. These patterns are characterized by a sustained press duration at a single touch point exceeding a preset dwell time threshold, reflecting the user's deep attention to the corresponding location on the screen. Attention area markers are circular visual identifiers generated based on the dwell time characteristics, used to spatially quantify the intensity and range of the user's attention to a specific location.

[0089] In this embodiment, step 1021 firstly involves continuously monitoring the user's pressure value using a pressure sensor built into the touchscreen. When the pressure value of a touch point continuously exceeds a preset threshold, the system begins recording the coordinates (x, y) of that touch point and its duration from the start to the end of the press. As the user moves on the screen, the system continuously captures multiple touch points that meet the pressure conditions, forming a touch trajectory dataset containing coordinate sequences and durations. For example, when the user draws a circle on the screen with their finger, the system captures five consecutive points: starting at (100, 100) and pressing for 1.0 second, moving to (100, 200) and pressing for 1.1 seconds, moving to (200, 200) and pressing for 0.9 seconds, moving to (200, 100) and pressing for 1.2 seconds, returning to the starting point (100, 100) and pressing for 1.0 second, ultimately forming complete trajectory data containing five sets of coordinates and durations.

[0090] Next, in step 1022, the continuous coordinate points in the touch trajectory data output in step 1021 are connected sequentially, and a closed polygon region is constructed using the convex hull algorithm. Then, the spatial coordinate information of the polygon and the target region of the knowledge point is compared, and the coverage state of the polygon on the bounding box of each knowledge point is calculated using the ray method. When all vertices of a knowledge point bounding box are located inside the polygon, a strong association mark between the touch hotspot and the corresponding knowledge point is generated. When the polygon only covers part of the bounding box of multiple knowledge points, a weak association mark between the touch hotspot and the set of related knowledge points is generated. Finally, all marks are integrated to form a structured spatial association map, which clearly records the mapping relationship between the user interaction area and the knowledge content. For example, if the user selects a pentagon, the spatial coordinates of the pentagon are compared with those of the target knowledge point area. If the selected area completely encloses the four corners of the formula area (50, 100, 300, 300), a strong association marker is generated between the touch area and the corresponding knowledge point. If the selected area includes both the upper left corner of the formula area and the lower right corner of the chart area, a weak association marker is generated between the touch area and the set of related knowledge points, marking the touch area 1 as associated with [formula area + chart area].

[0091] Finally, using the touch trajectory data scanned in step 1023, touch points whose duration exceeds a preset dwell threshold are selected as dwell feature points; and for each feature point, with its coordinates (x, y) as the center, according to the formula: The system dynamically calculates the range of a circular region. Ultimately, it generates a circular attention region marker centered on a feature point, with a radius proportional to the attention duration, to quantify the user's deep attention intensity to a specific location. For example, if a user continuously presses on coordinates (500, 200) for 1.2 seconds, exceeding the threshold of 0.8 seconds, a circular region is calculated with (500, 200) as the center and a scaling factor k = 20 pixels / second, resulting in a red semi-transparent circle with a center of (500, 200) and a radius of 24 pixels.

[0092] In practical application, when a user pauses a physics teaching video on an online learning platform, two core knowledge areas are displayed on the screen: one is a formula derivation area located between the top left corner (coordinates 50 x 100 x 1) and the bottom right corner (coordinates 300 x 300 x 1) and an experimental chart area located between 350 x 150 x 1) and 700 x 400 x 1) and an experimental chart area. The user applies approximately 400 millipascals of pressure to the screen with their finger, moving their finger along the boundary of the formula derivation area, forming a roughly closed polygonal trajectory that encloses the area, and then holds a point in the experimental chart area for approximately 1.2 seconds. The system first captures the sequence of continuous touch points in the formula area and their respective press durations, while also recording the coordinates and duration of the long-press point in the chart area. Next, the system analyzes the polygonal area formed by the selection, confirming that it completely encompasses the entire boundary of the formula derivation area, thus establishing a strong correlation between this touch area and the formula derivation knowledge point. Simultaneously, the system detected that the duration of a long press on the chart area exceeded the 0.8-second dwell threshold. Therefore, it generated a circular marker with a radius of approximately 24 pixels, centered on the long press coordinates and proportionally based on the 1.2-second duration, to mark the area of ​​deep user attention at that location. Finally, the system integrated this information to generate a spatial correlation map indicating a strong correlation between the touch hotspot and the formula derivation area, as well as a circular attention area marker located in the experimental chart area.

[0093] The overall solution described above (102) captures user interaction intent through pressure-sensitive touch trajectories, generates a spatial correlation map between touch hotspots and knowledge points, and dynamically constructs attention area markers based on dwell time. This enables precise spatial positioning of question focus, visual mapping of user intent, and quantitative expression of attention intensity, providing structured spatial relationship data support for subsequent multimodal fusion analysis and significantly improving the accuracy of question understanding and the efficiency of interaction guidance.

[0094] 103. By integrating the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest markers, the topological relationships of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation, a question confidence score is generated.

[0095] Optionally, step 103 may specifically include the following steps:

[0096] 1031. The ratio of the number of strongly correlated markers to the total number of markers in the spatial correlation map is used as a spatial distribution characteristic;

[0097] 1032. Calculate the straight-line distance between the center coordinates of each interest region marker and the center point coordinates of the nearest knowledge point target region as a geometric attribute;

[0098] 1033. Traverse the spatial coordinate information of all knowledge point target regions. When the distance between the bounding boxes of any two regions is less than the distance threshold, establish a connection relationship. Count the total number of such connections as the topological relationship.

[0099] 1034. Extract the voice waveform within a set time before the pause operation, convert it into text, and count the frequency of predefined question keywords as semantic features;

[0100] 1035. Input the spatial distribution characteristics, geometric attributes, topological relationships and semantic features into the weighted calculation model to perform comprehensive calculation to generate the question confidence level.

[0101] In the above scheme, spatial distribution characteristics refer to a quantitative indicator reflecting the concentration of user interaction focus, characterizing the user's level of focus on core knowledge points, and can be used to assess the clarity of the question intent. Geometric attributes refer to a physical quantity that measures the degree of deviation between the user's deep focus and the core location of the knowledge point; the smaller the distance value, the more closely the focus matches the core knowledge. Topological relationships refer to the connection network characteristics describing the spatial proximity between target areas of knowledge points; the larger the value, the closer the knowledge points are related. Semantic features refer to the question tendency clues extracted from the associated speech segments before the pause; the higher the frequency, the more significant the question intent in the speech. Question confidence refers to a probabilistic score generated by combining the above four types of features to assess the clarity of the user's question; the larger the value, the more confident the system is in judging the user's question.

[0102] In this embodiment of the application, firstly, the spatial correlation map generated in step 102 is read through step 1031, all marker types are extracted from it, and the number of strongly correlated markers is counted and recorded as follows. Meanwhile, the number of weak association markers is counted and denoted as . Next, the spatial distribution characteristic index is calculated using the following formula: For example, if the graph contains two strongly correlated markers and one weakly correlated marker, then... This value reflects the user's focus and concentration on core knowledge points.

[0103] Next, in step 1032, for each region of interest marker generated in step 102, the coordinates of its center are obtained. Then, the spatial coordinates of the target regions of all knowledge points are traversed, and the center point coordinates of the bounding box of each knowledge point are extracted. Calculate the Euclidean distance from the center of the circle to the center of each knowledge point. And select the minimum value as the geometric attribute value. For example, the distance from the center of the circle (200, 150) to the center of the nearest knowledge point (180, 160). The smaller the pixel value, the closer the focus is to the core knowledge.

[0104] Then, load the bounding box coordinates of all knowledge point target regions, and calculate the minimum bounding box spacing by combining any two regions: first, calculate the horizontal and vertical spacing components, and take the maximum value of the two as the actual spacing. Negative values ​​are considered as 0; when Connections are established sequentially, and the total number of connections that meet the conditions is ultimately used as the topology relationship value. For example, in regions A (50, 100, 300, 300) and B (350, 150, 700, 400), the horizontal distance between the right boundary 300 of region A and the left boundary 350 of region B is 350-300=50 pixels, which is equal to the threshold and counted as one connection.

[0105] The raw audio waveform data for a fixed duration before the pause operation is extracted and converted into text using a speech recognition engine. Then, it is matched against a predefined interrogative keyword library Q={"why","how","whether",...}, and the frequency of these words in the text is used as semantic feature values. For example, if the text is identified as "Why does this formula need to be transformed?", and the keyword "why" appears once, then... .

[0106] Finally, step 1035 converts the four-dimensional features generated in the previous steps into spatial distribution characteristics. Geometric properties Topological relationships With semantic features The pre-trained weighted model is input for fusion calculation, and the calculation formula is as follows: The weight The attenuation coefficient k=0.02. The final output is the confidence level of the question. Used for subsequent routing decisions. For example, spatial distribution characteristics. 0.67, geometric properties 22.36 pixels, topological relationship 1 and semantic features The value is 1. Substituting this into the formula and performing the fusion calculation yields the confidence level of the question as 1. .

[0107] In practical applications, when a user pauses a physics video on the online learning platform, the system detects two strongly correlated markers in the spatial correlation graph, completely covering the formula derivation area and experimental data area on the screen, respectively. There is also a weakly correlated marker in the text description area. The calculated proportion of the strongly correlated markers is approximately two-thirds, or 0.67. Simultaneously, a user-generated circular area of ​​interest is identified, with its center located at 200 pixels horizontally and 150 pixels vertically on the screen. The straight-line distance from the nearest formula area center point (180 pixels horizontally and 160 pixels vertically) is approximately 22.36 pixels. Within the knowledge point area, the formula area boundary extends from the top left corner (50 pixels horizontally and 100 pixels vertically) to the bottom right corner (300 pixels horizontally and 300 pixels vertically), while the chart area boundary extends from 350 pixels horizontally and 150 pixels vertically to 700 pixels horizontally and 400 pixels vertically. The minimum horizontal distance between the two areas is 50 pixels, equal to a preset threshold; therefore, a topological connection is established, and the total number of connections is recorded as 1. The audio segment 5 seconds before the pause was extracted and converted into text "Why does this formula need to be converted?" The question word "why" triggered a predefined keyword library with a frequency of 1. The question confidence was calculated using a weighted fusion model: the spatial distribution characteristic value of 0.67, the geometric attribute distance value of 22.36 pixels was reduced by the attenuation function e^(-0.02×22.36)≈0.64, the topological relationship value of 1 was reduced by the hyperbolic tangent function tanh(1)≈0.76, and the semantic feature value of 1 was linearly superimposed with weight coefficients of 0.3, 0.3, 0.2, and 0.2: 0.3×0.67+0.3×0.64+0.2×0.76+0.2×1=0.72. Finally, a question confidence of 0.72 was generated, triggering an immediate answer response.

[0108] The above-mentioned overall solution innovatively integrates four-dimensional information: spatial interaction intensity, visual attention accuracy, knowledge point association density, and voice question clues. It generates a comprehensive question confidence score through a weighted model, realizing the quantitative evaluation and unified expression of multimodal intent, providing an objective basis for subsequent decision-making and routing, and significantly improving the comprehensiveness and reliability of question recognition.

[0109] 104. Decision routing is performed based on the quantification level of the question confidence level. When the direct response threshold is reached, instant answer content is generated. When the direct response threshold is not reached, the dynamic follow-up question engine is activated.

[0110] Optionally, step 104 may specifically include the following steps:

[0111] 1041. Compare the question confidence level with the preset direct response threshold.

[0112] 1042. When the confidence level of the question is greater than or equal to the direct response threshold, retrieve the pre-stored parsed text corresponding to the target area of ​​the current knowledge point to generate instant Q&A content.

[0113] 1043. When the confidence level of the question is less than the direct response threshold, activate the dynamic follow-up question engine and execute the follow-up question option generation operation, while prohibiting the generation of instant answer content.

[0114] In the above scheme, the direct response threshold refers to a preset threshold for determining the clarity of the question. When the confidence level of the question calculated by the system reaches or exceeds this value, it indicates that the user's question is highly clear, triggering the direct answering mechanism. Instant answer content refers to the pre-stored structured answer text retrieved by the system based on the target area identifier of the current knowledge point when the question confidence level meets the threshold condition, used to directly respond to the user's question. The dynamic follow-up question engine is an interactive decision module activated when the question confidence level is below the threshold. It generates a set of targeted follow-up question options by parsing the context of the user's potential questions, guiding the user to clarify ambiguous intentions. The prohibition generation mechanism is a control command triggered synchronously when the dynamic follow-up question engine is activated. It forcibly closes the direct answer content output channel to ensure that the system maintains only a single interaction path in scenarios with ambiguous intentions, avoiding information conflicts.

[0115] In this embodiment, step 1041 obtains the question confidence level value (e.g., 0.72) and compares it with a preset direct response threshold. This process is implemented using a numerical comparator: if the confidence level is greater than or equal to the threshold, a direct response flag is triggered; if the confidence level is less than the threshold, a follow-up question flag is triggered. For example, in a physics teaching scenario, if a user's question confidence level of 0.75 regarding the electromagnetic formula exceeds the threshold of 0.6, the system generates a direct response flag.

[0116] When the confidence level of the question is greater than or equal to the threshold triggering the direct response flag, the system performs the following chain operation in step 1042: Extract the current main region identifier from the knowledge point target region information; then, using this identifier as the index key, query the pre-stored parsed text database; finally, encapsulate the retrieved structured text into instant Q&A content. For example, the retrieved pre-stored text is: "Faraday's law describes the relationship between induced electromotive force and the rate of change of magnetic flux: , commonly found in the working principle of generators...", directly output to the user interface.

[0117] When the confidence level of a question is less than the threshold triggering a follow-up question flag, the system performs a dual operation in step 1043: on the one hand, it activates the dynamic follow-up question engine, initializes the engine core processor, and calls the follow-up question option generation algorithm; on the other hand, it activates the generation prohibition mechanism, forcibly locking the data transmission channel of the direct answer interface. For example, in a math video scenario, if the user's confidence level for the integral application is only 0.52 < 0.6, the system activates the follow-up question engine to generate options ["Do you need an example demonstration?", "Or concept explanation?"], while simultaneously blocking access to the pre-stored text library to ensure that the interface only displays follow-up question options.

[0118] In practical applications, when a user pauses a circuit analysis video, the system calculates the confidence level of the question. If the confidence level is 0.68, which is greater than the threshold of 0.6, a direct response flag is triggered. The system then retrieves the pre-stored parsing text based on the current knowledge point ID "ohm_law," stating that "in the Ohm's Law formula V=IR, V represents voltage..." and outputs it. If the confidence level is 0.55, which is less than the threshold of 0.6, a follow-up question flag is triggered; the follow-up question engine generates options such as "Is the question about the formula symbol?" or "Or the experimental procedure?", while the direct answer function is disabled.

[0119] The above-mentioned overall solution of 104 achieves accurate decision-making and routing through intelligent comparison of confidence level and threshold: when the confidence level is high, structured Q&A content is directly output to ensure efficient response; when the confidence level is low, the follow-up question engine is activated to guide users to clarify their questions, while prohibiting the output of conflicting information, which significantly improves the rigor of the interaction logic and the smoothness of the user experience.

[0120] 105. The dynamic questioning engine is used to parse the semantic types of knowledge points in the spatial association graph to match the subject-specific questioning strategy library and generate a set of questioning options;

[0121] Optionally, step 105 may specifically include the following steps:

[0122] 1051. Extract the target regions of knowledge points corresponding to strongly correlated markers in the spatial correlation map, and classify the target regions into semantic types such as formula, concept, or chart based on their content;

[0123] 1052. Match the subject-specific follow-up questioning strategy library according to the semantic type, and call the step derivation option set corresponding to the formula class, the definition comparison option set corresponding to the concept class, or the data interpretation option set corresponding to the chart class as the basic follow-up questioning option set;

[0124] 1053. Based on the average duration of all touch points in the touch trajectory data, adjust the priority order of the options in the basic follow-up question option set, and arrange the high-frequency question options at the top to generate the final follow-up question option set.

[0125] Specifically, step 1053 may include the following processes: calculating the arithmetic mean of the duration of all touch points in the touch trajectory data as the average duration, and counting the historical selection count of each option in the basic follow-up question option set from the historical question database; rearranging the option order of the basic follow-up question option set in descending order according to the historical selection count to generate an adjusted set with high-frequency question options at the top; when the average duration exceeds the interaction depth threshold, adding depth analysis options to the rearranged basic follow-up question option set to form the final follow-up question option set; when the average duration does not exceed the interaction depth threshold, directly using the rearranged basic follow-up question option set as the final follow-up question option set.

[0126] In the above scheme, the semantic type of knowledge points refers to a category system based on the visual content of the target area, including formulas, concepts, and charts. This classification is used to accurately match subject-specific follow-up question strategies. The subject-specific follow-up question strategy library refers to a set of pre-built follow-up question option templates based on subject area and semantic type, used to generate scenario-based follow-up question content. Average duration refers to the arithmetic mean of the press duration of all touch points in the touch trajectory, reflecting the overall depth of user interaction. The interaction depth threshold refers to the critical duration value that triggers the addition of deep analysis options. When the average duration exceeds this value, advanced items are added to the follow-up question options. Deep analysis options are advanced follow-up questions added to the end of the basic option set to meet the needs of deep interaction. High-frequency question options are prioritized by rearranging the basic options in descending order of the number of times each option has been selected based on historical database statistics, to improve user selection efficiency.

[0127] In this embodiment, step 1051 first extracts all knowledge point target regions with strong correlation markers from the spatial association map. Then, each region is semantically classified: an OCR engine is called to recognize the text content within the region, while a symbol detection algorithm scans mathematical symbols and graphic elements. If the density of detected mathematical symbols exceeds a threshold, it is marked as a formula; if keywords such as "definition" or "theorem" are detected without dense symbols, it is marked as a concept; if the proportion of visual elements exceeds 50%, it is marked as a chart. For example, a region containing the expression " It is classified as a formula.

[0128] Next, based on the semantic type output by 1051, the system queries the subject-specific follow-up question strategy library to match the corresponding option templates. First, the video's subject tag is determined, then the corresponding option template is called according to the semantic type: formula-related templates are activated ["derived steps", "symbol meaning", "application scenarios"], concept-related templates are activated ["definition comparison", "example analysis", "common misconceptions"], and chart-related templates are activated ["data interpretation", "graphing principles", "trend analysis"]. For example, mathematics with formula-related semantics outputs a basic set of follow-up question options. ={"1. Derivation steps","2. Meaning of symbols","3. Application scenarios"}.

[0129] Finally, based on the touch trajectory data, a third-order optimization is performed on the basic set of follow-up questions: the arithmetic mean of the duration of all touch points in the touch trajectory data is calculated using the following formula: , The duration of the touch point is denoted by n, and the number of touch points is denoted by n. The sum of the durations of n touch points; then, the historical database is queried to obtain... The options are reordered in descending order of their historical selection counts. For example, "Derivation steps": 120 times, "Symbol meaning": 80 times. ={"1. Derivation Steps","2. Symbol Meaning","3. Application Scenarios"}; Finally, determine if the average duration exceeds the interaction depth threshold. If the average duration exceeds the interaction depth threshold, add options; if the average duration does not exceed the interaction depth threshold, output directly. For example, average duration The final output is a set of follow-up question options. ={"Derivation Steps","Symbol Meaning","Application Scenarios"}; If the average duration Then add to ={"Derivation steps","Symbol meaning","Application scenarios","Step-by-step explanation"}.

[0130] In practical applications, when users watch calculus videos, they circle the integral formula area to form a strong association marker. The system uses OCR to recognize the formula symbols and classify them into formula semantic types. It then matches the mathematical subject strategy library to generate a basic option set ["derive steps", "symbol meaning", "application scenario"]. Based on the touch trajectory, the average duration is calculated to be 1.4 seconds, calculated from the durations of 1.4 seconds, 1.0 seconds, and 1.8 seconds at three points. Historical data shows that "derive steps" was selected 105 times and "symbol meaning" was selected 62 times. The options are rearranged to ["derive steps", "symbol meaning", "application scenario"]. Since the average duration of 1.4 seconds does not exceed the 1.5-second threshold, the final output set of follow-up questions is ["1. Derivation steps", "2. Symbol meaning", "3. Application scenario"]. If the duration becomes 2.0 seconds, 1.5 seconds, and 1.7 seconds, the average duration of 1.73 seconds exceeds the threshold, so the option "4. Step-by-step explanation" is added.

[0131] The above-mentioned 105 overall solution generates subject-specific customized follow-up questions through semantic type matching, and dynamically adjusts the priority and depth of the options based on the average touch duration: high-frequency question items are placed at the top based on historical selection data to improve selection efficiency, and in-depth analysis options are added when the interaction duration exceeds the threshold, so as to achieve accurate matching between the follow-up questions and user behavior characteristics, and significantly improve the pertinence and interaction depth of question clarification.

[0132] 106. Based on the user's selection of options in the set of follow-up questions, generate answer content that includes a core answer layer and a related knowledge extension layer.

[0133] Optionally, step 106 may specifically include the following steps:

[0134] 1061. Responding to the user's action of clicking on a specific option in the set of follow-up questions, obtain the identification information of the specific option;

[0135] 1062. Based on the identification information, extract the standard analysis content of the corresponding knowledge points from the pre-stored parsing text library as the core solution layer;

[0136] 1063. Based on the progress markers and marker information, retrieve the video metadata database to obtain the background knowledge module and advanced knowledge module associated with the current knowledge point, and combine them to form an associated knowledge extension layer;

[0137] 1064. Place the core answer layer at the beginning of the Q&A content, and attach the related knowledge extension layer at the end of the core answer layer in an expandable hierarchical structure to generate complete Q&A content.

[0138] In the above scheme, the specific option identifier information refers to the unique code captured by the system when the user clicks on the follow-up question option. This code includes the index key-value pair of that option in the strategy library, used to accurately locate pre-stored parsed content. The core answer layer refers to the standard answer content extracted from the pre-stored parsed text library based on the identifier information. It includes direct parsed text addressing the user's selected question points, used to instantly answer core questions. The related knowledge extension layer refers to supplementary knowledge modules retrieved from the video metadata database based on progress and option identifiers. It includes background knowledge modules and advanced knowledge modules, used to provide contextual extension and in-depth expansion. The expandable hierarchical structure refers to the front-end interaction design scheme, which encapsulates the related knowledge extension layer as a default collapsed control, used to balance information density and user cognitive load.

[0139] In this embodiment, firstly, in step 1061, when the user clicks a specific option on the follow-up option set interface, the system captures this interaction operation through the front-end event monitoring module and parses the unique identifier information of the clicked control. This identifier information serves as the core index key value for subsequent content retrieval; for example, when the user selects the "symbol meaning" option, the system captures ID="symbol_meaning".

[0140] Next, using the specific option identifier information obtained in step 1061 as the query key, the pre-stored parsed text library structured SQL database is retrieved via step 1062: the SQL statement `SELECT content FROM knowledge_responses WHERE option_id='symbol_meaning'` is executed to extract the standard parsed text of the matching field. This text, after being formatted, serves as the core solution layer. For example, a 500-word explanation of the integral symbol "∫" is retrieved: "The integral symbol originates from the Latin word summa...representing an infinite accumulation process."

[0141] Then, by fusing the progress marker and option identifier information in step 1063, a joint query is initiated to the video metadata database: first, the background knowledge module is retrieved to obtain historical background and application scenarios; then, the advanced knowledge module is retrieved to obtain extended content, such as the progress marker 00:15:30.500 and the identifier information "derivation_steps", and the search is executed. ANDoption_id='derivation_steps' returns 150 words of text titled "The Creation Process of the Newton-Leibniz Formula". Then, the advanced knowledge module SELECT advanced FROM metadata... returns 200 words of text including the formula, titled "The Relationship Between Double Integrals and Iterated Integrals", to obtain extended content. Finally, the obtained historical background and application scenarios are combined with the obtained extended content to form a related knowledge extension layer.

[0142] Finally, the core answer layer generated in step 1062 is placed at the beginning of the Q&A content, and the related knowledge extension layer output in step 1063 is encapsulated as an expandable control: using HTML5. <details>The label wraps the background module and advanced module text, and sets... <summary> ▼Related Knowledge Expansion< / summary> As a clickable title, and initialized with a default collapsed state, it forms a complete Q&A content structure of

Core Answer

▼ Expandable Expansion Layer

[0143] In practical applications, after the user selects the "Derivation Steps" option ID="derivation_steps", the system queries the parsing library based on the progress marker 00:15:30.500 to obtain a 300-word core solution layer, and jointly queries the metadata database to obtain a 150-word background module about the history of the Newton-Leibniz formula and a module containing the double integral formula. The 200-word advanced module is ultimately assembled into a layered Q&A content consisting of a core text at the beginning and an expandable control at the end.

[0144] The above-mentioned overall solution of 106 achieves layered knowledge delivery through structured content assembly. The core answer layer accurately responds to users' immediate questions, while the related extension layer provides background and advanced knowledge that can be explored. The expandable design balances information density and cognitive load, forming an adaptive Q&A paradigm of "focusing on the core and expanding on demand", which significantly improves the efficiency of knowledge transfer and support for deep learning.

[0145] The following is a complete example for steps 101-106, such as Figure 2 As shown, when user Xiao Wang was watching a physics course video on an online learning platform, he pressed the pause button at the timestamp 00:12:30.500 during the "Derivation of Newton's Second Law" segment. The system captured this progress marker, and the decoder output the current keyframe: the left side of the slide displays the formula F=ma, and the right side shows a force analysis diagram of an inclined plane. The target detection model identified two core knowledge point areas: the formula area's bounding box coordinates were 50 (horizontal 100, vertical to bottom right 300, horizontal 300, vertical); the diagram area's bounding box coordinates were 350 (horizontal 150, vertical to bottom right 700, horizontal 400, vertical).

[0146] Xiao Wang used a stylus to circle the formula area, applying a pressure of 480 mPa, exceeding the preset threshold of 300 mPa. The system recorded the closed trajectory point, forming a hot zone that completely covered the boundary of the formula area, generating a strongly correlated marker. Simultaneously, by pressing and holding for 1.8 seconds at the force diagram coordinate point 600 (horizontal) 250 (vertical), a dwell feature was triggered, generating a circular attention marker with a radius of 36 pixels. The system fused multimodal features to calculate the interrogative confidence score: spatial distribution characteristics scored 1.0 points due to the unique strongly correlated marker; the Euclidean distance from the center of the attention marker to the center of the illustrated area was approximately 76 pixels; the horizontal distance between the boundary of the formula area and the illustrated area was 50 pixels, reaching the connection threshold, and the topological relationship scored 1 point; the voice prompt "Why is mass inversely proportional to acceleration?" 5 seconds before the pause, recognizing the interrogative word "why," scored 1 point for semantic features. The weighted model output confidence score was 0.92, exceeding the direct response threshold of 0.6.

[0147] After determining a high level of confidence, the system skips the follow-up questioning process and directly generates layered answer content: First, based on the formula area identifier, it retrieves the core answer layer "mass m represents the inertial property of an object, acceleration a is proportional to the force F..."; then, combined with the progress marker 00:12:30.500, it queries the metadata database to obtain the background module "Newton first proposed this law in 1687 in *Mathematical Principles of Natural Philosophy*" and the advanced module "Relativistic Correction Formula". The derivation explanation is then assembled into a front-end interface: the core analytical text is displayed at the top, and an expandable control "▼ Related Knowledge Extension" is attached at the bottom, which expands the historical background and extended content on relativity.

[0148] After quickly grasping the essential properties of F=ma through the core layer, Xiao Wang clicks the expand icon to explore the knowledge evolution from classical mechanics to relativity in depth. The system achieves a closed-loop learning experience of "precisely responding to core questions - expanding knowledge boundaries as needed" through an adaptive mechanism of multimodal perception, confidence decision-making, and hierarchical output.

[0149] Figure 3 This application provides a schematic diagram of the structure of an AI question-answering system based on the correlation between video keyframes and progress, as shown in the embodiment of the present application. Figure 3 As shown, the system includes:

[0150] The acquisition module 31 is used to capture the progress indicator and the corresponding key frame when the user performs a video pause operation, and extract the spatial coordinate information of the target area representing the knowledge point in the key frame;

[0151] The generation module 32 is used to capture touch trajectory data that meets the pressure triggering conditions through the touch screen, combine the progress indicator and the spatial coordinate information to generate a spatial association map between the touch hot zone and the knowledge point, and generate a region of interest marker based on the dwell characteristics of the touch trajectory data.

[0152] The fusion module 33 is used to fuse the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest marker, the topological relationship of the spatial coordinate information, and the semantic features of the associated speech segment before the pause operation to generate a question confidence score.

[0153] Decision module 34 is used to make decision routing based on the quantitative level of the question confidence, generate instant answer content when the direct response threshold is reached, and activate the dynamic follow-up question engine when the direct response threshold is not reached.

[0154] The parsing module 35 is used to parse the semantic types of knowledge points in the spatial association graph through the dynamic questioning engine, so as to match the subject-specific questioning strategy library to generate a set of questioning options;

[0155] The output module 36 is used to generate answer content containing a core answer layer and an associated knowledge extension layer based on the user's selection operation of the options in the set of follow-up questions.

[0156] Figure 3 The aforementioned AI question-answering system based on the correlation between video keyframes and progress can execute... Figure 1 The implementation principle and technical effects of the AI ​​Q&A method based on the correlation between video keyframes and progress described in the illustrated embodiment will not be repeated here. The specific methods by which each module and unit performs operations in the AI ​​Q&A system based on the correlation between video keyframes and progress in the above embodiments have been described in detail in the embodiments related to this method, and will not be elaborated upon here.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / details>

Claims

1. An AI-based question-answering method based on the correlation between video keyframes and progress, characterized in that, include: When the user performs a video pause operation, capture the progress indicator and the corresponding key frame, and extract the spatial coordinate information of the target area representing the knowledge point in the key frame; The touch trajectory data that meets the pressure triggering conditions is captured by the touch screen, and a spatial association map between the touch hot zone and the knowledge point is generated by combining the progress indicator and the spatial coordinate information. The attention area marker is generated based on the dwell characteristics of the touch trajectory data. By integrating the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest markers, the topological relationships of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation, a question confidence score is generated. The decision routing is based on the quantification level of the question confidence. When the direct response threshold is reached, instant answer content is generated. When the direct response threshold is not reached, the dynamic follow-up question engine is activated. The dynamic questioning engine parses the semantic types of knowledge points in the spatial association graph to match the subject-specific questioning strategy library and generate a set of questioning options. Based on the user's selection of options in the set of follow-up questions, answer content is generated that includes a core answer layer and a related knowledge extension layer. The process of fusing the spatial distribution characteristics of the spatial correlation map, the geometric attributes of the region of interest markers, the topological relationships of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation to generate a question confidence score includes: The ratio of the number of strongly correlated markers to the total number of markers in the spatial correlation map is used as a spatial distribution characteristic. The straight-line distance between the center coordinates of each region of interest marker and the center point coordinates of the nearest knowledge point target region is calculated as a geometric attribute. Traverse the spatial coordinate information of all knowledge point target regions. When the distance between the bounding boxes of any two regions is less than the distance threshold, establish a connection relationship. Count the total number of such connections as the topological relationship. The voice waveform within a set time before the pause operation is extracted and converted into text, and the frequency of occurrence of predefined question keywords is counted as semantic features. The spatial distribution characteristics, geometric attributes, topological relationships, and semantic features are input into a weighted calculation model to perform a comprehensive calculation to generate the question confidence level.

2. The method according to claim 1, characterized in that, The process involves parsing the semantic types of knowledge points in the spatial correlation graph using the dynamic questioning engine to generate a set of questioning options by matching them with a subject-specific questioning strategy library, including: Extract the target regions of knowledge points corresponding to strongly correlated markers in the spatial correlation map, and classify the target regions into semantic types such as formula, concept, or chart based on their content; Based on the semantic type matching subject-specific follow-up questioning strategy library, call the step derivation option set corresponding to formula class, the definition comparison option set corresponding to concept class, or the data interpretation option set corresponding to chart class as the basic follow-up questioning option set; Based on the average duration of all touch points in the touch trajectory data, the priority order of the options in the basic follow-up question option set is adjusted, and the high-frequency question options are arranged at the top to generate the final follow-up question option set.

3. The method according to claim 2, characterized in that, The step involves adjusting the priority order of options in the basic follow-up question option set based on the average duration of all touch points in the touch trajectory data, and placing the most frequently asked question options at the top to generate the final follow-up question option set, including: Calculate the arithmetic mean of the duration of all touch points in the touch trajectory data as the average duration, and count the number of times each option in the basic follow-up question option set has been selected from the historical Q&A database. The options in the basic follow-up question set are rearranged in descending order according to the historical selection frequency to generate an adjusted set with the most frequently asked question options at the top. When the average duration exceeds the interaction depth threshold, deep parsing options are added to the rearranged basic follow-up question option set to form the final follow-up question option set; When the average duration does not exceed the interaction depth threshold, the rearranged basic set of follow-up questions will be directly used as the final set of follow-up questions.

4. The method according to claim 1, characterized in that, The process of generating answer content that includes a core answer layer and a related knowledge extension layer based on the user's selection of options from the set of follow-up questions includes: In response to a user clicking on a specific option in the set of follow-up questions, obtain the identification information of that specific option; Based on the identification information, standard analysis content corresponding to the knowledge points is extracted from the pre-stored parsed text library as the core solution layer; Based on the progress markers and marker information, the video metadata database is retrieved to obtain the background knowledge module and advanced knowledge module associated with the current knowledge point, and they are combined to form an associated knowledge extension layer. The core answer layer is placed at the beginning of the Q&A content, and the related knowledge extension layer is attached to the end of the core answer layer in an expandable hierarchical structure to generate complete Q&A content.

5. The method according to claim 1, characterized in that, The process involves capturing touch trajectory data that meets pressure triggering conditions via a touchscreen, combining the progress indicator and spatial coordinate information to generate a spatial association map between touch hotspots and knowledge points, and generating attention area markers based on the dwell characteristics of the touch trajectory data, including: Monitor the pressure sensor data of the touch screen. When the pressure value continuously exceeds the preset pressure threshold, record the touch point coordinate sequence and the duration of each touch point to form touch trajectory data. A closed polygon formed by continuous touch points is defined as a touch hot zone. The spatial coordinate information of the touch hot zone and the target area of ​​each knowledge point is compared. Based on the comparison results, a spatial association map between the touch hot zone and the knowledge point is generated. Touch points whose duration exceeds the dwell threshold in the touch trajectory data are extracted as dwell feature points. A circular marker with a radius proportional to the duration is generated as a region of interest marker, with the coordinates of each dwell feature point as the center.

6. The method according to claim 5, characterized in that, The step of comparing the spatial coordinates of the touch hotspot with the target areas of each knowledge point, and generating a spatial association map between the touch hotspot and the knowledge points based on the comparison results, includes: When the closed polygon formed by the touch hot area completely covers the spatial coordinate information of a single knowledge point target area, a strong association mark between the touch hot area and the corresponding knowledge point is established. When the closed polygon formed by the touch hot area simultaneously covers the spatial coordinate information of multiple knowledge point target areas, a weak association marker is established between the touch hot area and the corresponding multiple knowledge points. By aggregating the strong and weak association tags, a spatial association map is generated that contains a set of tags relating touch hotspots to knowledge points.

7. The method according to claim 1, characterized in that, The decision-making routing based on the quantification level of the question confidence, generating immediate Q&A content when the direct response threshold is reached, and activating the dynamic follow-up question engine when the direct response threshold is not reached, includes: The confidence level of the question is compared numerically with the preset direct response threshold; When the confidence level of the question is greater than or equal to the direct response threshold, the pre-stored parsed text corresponding to the target area of ​​the current knowledge point is retrieved to generate instant Q&A content. When the confidence level of a question is less than the direct response threshold, the dynamic follow-up question engine is activated and follow-up question option generation is performed, while the generation of instant answer content is prohibited.

8. The method according to claim 1, characterized in that, The step of capturing the progress indicator and corresponding keyframe when the user pauses the video, and extracting the spatial coordinate information of the target area representing the knowledge point in the keyframe, includes: When the video playback engine receives a pause command, it reads the current time value on the video timeline as a progress indicator. The frame buffer module of the video decoder is invoked to output a static image that is precisely aligned with the progress indicator as a keyframe. Identify independent blocks in the keyframe that conform to the characteristics of text clustering areas, formula clustering areas, or graphic clustering areas as target areas of knowledge points, and simultaneously generate spatial coordinate information including the bounding box positioning coordinates, bounding box size, and center point coordinates of each target area.

9. An AI-powered question-answering system based on the correlation between video keyframes and progress, characterized in that: include: The acquisition module is used to capture the progress indicator and the corresponding key frame when the user performs a video pause operation, and extract the spatial coordinate information of the target area representing the knowledge point in the key frame; The generation module is used to capture touch trajectory data that meets the pressure triggering conditions through the touch screen, combine the progress indicator and the spatial coordinate information to generate a spatial association map between the touch hot zone and the knowledge point, and generate a region of interest marker based on the dwell characteristics of the touch trajectory data. The fusion module is used to fuse the spatial distribution characteristics of the spatial association map, the geometric attributes of the region of interest markers, the topological relationship of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation to generate a question confidence score. The decision module is used to make decision routing based on the quantitative level of the question confidence. When the direct response threshold is reached, instant answer content is generated. When the direct response threshold is not reached, the dynamic follow-up question engine is activated. The parsing module is used to parse the semantic types of knowledge points in the spatial association graph through the dynamic questioning engine, so as to match the subject-specific questioning strategy library to generate a set of questioning options; The output module is used to generate Q&A content that includes a core answer layer and an associated knowledge extension layer based on the user's selection operation of the options in the set of follow-up questions; The process of fusing the spatial distribution characteristics of the spatial correlation map, the geometric attributes of the region of interest markers, the topological relationships of the spatial coordinate information, and the semantic features of the associated speech segments before the pause operation to generate a question confidence score includes: The ratio of the number of strongly correlated markers to the total number of markers in the spatial correlation map is used as a spatial distribution characteristic. The straight-line distance between the center coordinates of each region of interest marker and the center point coordinates of the nearest knowledge point target region is calculated as a geometric attribute. Traverse the spatial coordinate information of all knowledge point target regions. When the distance between the bounding boxes of any two regions is less than the distance threshold, establish a connection relationship. Count the total number of such connections as the topological relationship. The voice waveform within a set time before the pause operation is extracted and converted into text, and the frequency of occurrence of predefined question keywords is counted as semantic features. The spatial distribution characteristics, geometric attributes, topological relationships, and semantic features are input into a weighted calculation model to perform a comprehensive calculation to generate the question confidence level.

Images