Real-time screenshot splicing-based training room desktop operation key step playback method

Abstract

The application discloses a real-time screenshot splicing-based operation key step playback method in a practical training room, relates to the technical field of data processing, and balances integrity and storage economy by aligning desktop screenshots with multi-dimensional event timestamps and combining operation strength adaptive sampling, which not only completely captures operation details, but also avoids invalid screenshot waste; relying on screenshot splicing and incremental storage technology, a globally consistent operation context is constructed to solve the information loss problem under window switching and complex operation; through intelligent screening and atlas indexing of key steps, accurate positioning of operation steps is realized, and playback step-level time compression and event superposition presentation are matched to greatly improve the backtracking efficiency; the playback consistency verification and rollback correction mechanism guarantee the accuracy of reproduction, and finally, the operation review effect in practical training teaching is optimized, the students are helped to quickly master core skills, and the training quality is significantly improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method for replaying key steps of desktop operation in a training room based on real-time screenshot stitching. Background Technology

[0002] With the rapid development of information technology, modern training environments have gradually incorporated virtual desktops, remote teaching, and self-study. By simulating real-world operating environments, trainees can conduct simulated training, improving their practical mastery of various technologies and operations. However, as the scale of training expands, efficiently and accurately recording and replaying trainees' operational processes becomes crucial for improving training quality. Existing technologies mostly use video recording or screenshots to record trainees' desktop operations, but these methods have limitations in playback, failing to provide a globally consistent context of the operational steps, thus affecting operational understanding and teaching effectiveness.

[0003] Currently, mainstream technologies for recording and replaying desktop operations in training labs mainly include screen recording video, timed screenshots, and event-driven screenshots. Screen recording video: By recording desktop operations in real time and generating video files for playback, it can intuitively demonstrate the student's operation process. However, the video files are large, making it difficult to accurately locate key operation steps during playback, and storage and transmission costs are high. Timed screenshots: The system captures desktop images at fixed intervals and saves them, displaying them sequentially during playback. However, this method cannot fully reflect the operation details, and in cases of frequent window switching or complex operations, the missing information caused by the screenshot intervals makes the operation reproduction incomplete.

[0004] Event-driven screenshots capture screenshots only when an interaction event occurs. Theoretically, this can reduce storage overhead, but it still lacks a comprehensive record of continuous operations and cannot provide continuous context between operation steps, affecting the user's understanding and recall of the operation process.

[0005] Therefore, how to improve the reproduction of operations and the effectiveness of teaching by enabling rapid playback while ensuring that the operation steps are clear and completely recorded is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] In order to solve the above-mentioned technical problems, this application proposes the following technical solution:

[0007] In a first aspect, embodiments of this application provide a method for replaying key steps of a training lab desktop operation based on real-time screenshot stitching, including:

[0008] Capture a sequence of desktop screenshots and simultaneously capture mouse, keyboard, window focus, and window geometry change events, and align the timestamps.

[0009] The screenshot sampling frequency is adaptively adjusted based on the event intensity and the visual changes of adjacent screenshots.

[0010] The screenshot is used to generate fingerprints and local features and match them. The geometric mapping is estimated and the splicing confidence is calculated. If the confidence meets the threshold, the mapped screenshot is merged and updated to the canvas and stored incrementally with differential patches. Otherwise, a new canvas fragment is created.

[0011] By integrating visual changes, event intensity, splicing confidence, and window layer changes, step boundary detection is performed and importance is calculated to filter key steps;

[0012] Construct a step graph and index that includes canvas identifiers, keyframe identifiers, and event summaries. On the playback end, prioritize key steps by step-level time compression and overlay cursors and event markers. Perform consistency checks and trigger rollback corrections if the consistency is not met.

[0013] In one possible implementation, the step of capturing a sequence of desktop screenshots and simultaneously capturing mouse, keyboard, window focus, and window geometry change events, and aligning the timestamps, includes:

[0014] On the training equipment terminal, obtain the current desktop frame buffer and generate a screenshot sequence. ;

[0015] Synchronously collect event streams consisting of mouse movement or clicks, keyboard key presses, window focus changes, window geometric changes in position or size, and application switching information. ;

[0016] The screenshot sequence and the event stream are aligned using a unified timestamp to obtain an aligned dataset. in For timestamp The corresponding event set includes all user interaction events that occurred within the event window before and after the timestamp.

[0017] In one possible implementation, the adaptive adjustment of the screenshot sampling frequency based on event intensity and visual changes in adjacent screenshots includes:

[0018] Based on the event set Calculate event intensity And calculate the intensity of visible change based on adjacent screenshots. The The ,in: This indicates the weighting coefficient for each event category, representing the degree to which different events contribute to the overall event intensity. Indicates event type q in Frequency or count of occurrences within the time window Indicates adjacent screenshots Structural similarity;

[0019] According to the above With the Adaptive determination of the next screenshot sampling interval To control the sampling frequency of the screenshot sequence:

[0020]

[0021] in: For a moment The next screenshot interval, Minimum screenshot interval, Maximum screenshot interval Based on the sampling scale constant, This is the event intensity weighting coefficient. The weighting coefficient is the intensity of visible change. Indicates will Cut off to interval .

[0022] In one possible implementation, the step of generating a fingerprint and matching local features from the screenshot, estimating the geometric mapping, and calculating the splicing confidence score, if the confidence score meets a threshold, then fusing and updating the mapped screenshot to the canvas and storing it incrementally with differential patches; otherwise, creating a new canvas fragment, including:

[0023] The fingerprint of the current screenshot is calculated to retrieve candidate overlapping frames and local features are extracted. A set of matching pairs is obtained based on feature matching.

[0024] The geometric mapping of the current screenshot to the target canvas coordinate system is estimated based on the set of matching pairs, and the stitching confidence is calculated based on the reprojection error and the inlier ratio.

[0025] When the splicing confidence meets the preset threshold condition, the current screenshot is fused and updated to the canvas after geometric mapping remapping, and the canvas update process is incrementally fused and updated in the form of differential patching.

[0026] When the splicing confidence does not meet the preset threshold, a new canvas fragment is created and subsequent splicing and updates are performed on it.

[0027] In one possible implementation, the step of calculating the fingerprint of the current screenshot for retrieving candidate overlapping frames and extracting local features, and obtaining a set of matching pairs based on feature matching, includes:

[0028] Calculate the global fingerprint for each screenshot , used for quickly coarsely screening historical candidate frames or canvas;

[0029] Calculate the local feature set ,in: For the coordinates of the key points, For descriptors;

[0030] Feature matching is performed on the candidate set to obtain a set of matching pairs. .

[0031] In one possible implementation, the step of estimating the geometric mapping of the current screenshot to the target canvas coordinate system based on the set of matching pairs, and calculating the stitching confidence based on the reprojection error and the inlier ratio, includes:

[0032] Robust estimation is used to obtain the mapping matrix from the current screenshot to the target canvas coordinate system. And calculate the error:

[0033]

[0034] in: The average reprojection error is used to measure the reliability of the stitching. The coordinates of the corresponding point in the target canvas or reference frame. The coordinates of the matching point in the current screenshot. For homogeneous normalized projection operators, The square of the L2 norm;

[0035] The stitching confidence level has been defined based on the average reprojection error:

[0036]

[0037] in: To construct the confidence level, a higher confidence level indicates greater reliability. For the Sigmoid function, To match the quantity indicators, For interior point proportions, , and These are the weighting coefficients.

[0038] In one possible implementation, when the splicing confidence level meets a preset threshold condition, the current screenshot is geometrically remapped and then fused and updated onto the canvas. The canvas update process is performed as an incremental fusion update in the form of differential patches, including:

[0039] Based on the mapping matrix, the current screenshot Mapping to the canvas coordinate system yields the alignment map. ;

[0040] According to the above Perform incremental blending updates on canvas C:

[0041]

[0042] in: For the current canvas, To align the graph and fuse weights;

[0043] The storage employs a critical canvas combined with a patching mechanism, recording differential patch sets and storing only the changed areas.

[0044] In one possible implementation, the step boundary detection and importance calculation for key steps, which integrates visual changes, event intensity, splicing confidence, and window layer changes, includes:

[0045] The visible change intensity, event intensity, splicing confidence, and window layer change index are fused on the timeline to calculate the step boundary score. The step boundary points are determined based on the step boundary score and the preset boundary judgment rules, thereby dividing the aligned data unit into multiple step segments.

[0046] Calculate an importance score for each step segment and determine the set of key steps from the step segments based on the importance scores.

[0047] In one possible implementation, the visible change intensity, event intensity, splicing confidence, and window layer change index are fused on the timeline to calculate a step boundary score. Step boundary points are then determined based on the step boundary score and a preset boundary determination rule, thereby dividing the aligned data unit into multiple step segments, including:

[0048] Calculate boundary scores:

[0049]

[0050] in: For boundary scores, a larger score indicates a higher likelihood of step switching. For window layer change indicators, , , and These are the fusion weights for visual change intensity, event intensity, splicing confidence, and window layer change index, respectively.

[0051] when When the threshold is exceeded and a local maximum is reached, the boundary points of the critical steps are determined, and step segments are formed between the boundaries. .

[0052] In one possible implementation, calculating an importance score for each step segment and determining a set of key steps from the step segments based on the importance scores includes:

[0053] For each step segment Calculate importance score :

[0054]

[0055] in: For the j-th step segment Importance score, The average visible change intensity within the step. The average event intensity within the step. for, This refers to the number of window switches within the step. The duration of the step. , , and These are the weighting coefficients;

[0056] Based on the importance score Sort the steps from largest to smallest, and select the steps with the highest preset number of steps as key steps to form the key step set.

[0057] In one possible implementation, the construction includes a step graph and index comprising canvas identifiers, keyframe identifiers, and event summaries. At the playback end, key steps are presented first with step-level time compression and overlaid with cursors and event markers. A consistency check is performed; if the consistency is not met, a rollback correction is triggered, including:

[0058] Construct a step graph and index structure for the step segments and key steps, including canvas identifiers, keyframe identifiers, event summaries, and searchable fields;

[0059] At the playback end, the canvas and differential patch are decoded and reconstructed according to the step-level time mapping function, and the real time axis is mapped to the playback time axis according to the step-level time mapping function.

[0060] The key steps are given priority for playback, while the cursor trajectory and event visualization markers are overlaid and rendered. A consistency check is performed on the playback reconstruction results. When the consistency check does not meet the preset conditions, a rollback strategy is triggered to correct the playback reconstruction results.

[0061] In one possible implementation, at the playback end, the canvas and differential patches are decoded and reconstructed based on the step-level graph, and the real timeline is mapped to the playback timeline based on the step-level time mapping function, including:

[0062] Define from real time Replay time Mapping:

[0063]

[0064] in: To replay the timeline position, This is the step time scaling factor, with a larger value for critical steps and a smaller value for ordinary steps; Let j be the actual duration of step j. This is the scaling factor for the step time. For inclusion The step numbers.

[0065] In one possible implementation, the key steps are prioritized for playback, while simultaneously overlaying and rendering cursor trajectories and event visualization markers. A consistency check is performed on the playback reconstruction results. When the consistency check fails to meet preset conditions, a rollback strategy is triggered to correct the playback reconstruction results, including:

[0066] Load the corresponding canvas according to the map. Desktop state stream within the patch sequence restoration step;

[0067] Overlay the cursor trajectory and click ripples, and display them aligned with the event timestamps.

[0068] If the difference between the reconstructed frame and the step keyframe exceeds a threshold, a rollback strategy is triggered, either loading a denser patch or switching to an alternate canvas node.

[0069] Secondly, embodiments of this application provide a system for replaying key steps of desktop operations in a training lab based on real-time screenshot stitching, including:

[0070] The acquisition module is used to acquire a sequence of desktop screenshots and simultaneously acquire mouse, keyboard, window focus, and window geometry change events, and align the timestamps.

[0071] The adaptive control module is used to adaptively adjust the screenshot sampling frequency based on the event intensity and the visual changes of adjacent screenshots;

[0072] The stitching and reconstruction module is used to generate fingerprints and local features from the screenshot and match them, estimate the geometric mapping and calculate the stitching confidence. If the confidence meets the threshold, the mapped screenshot is merged and updated to the canvas and stored incrementally with differential patches; otherwise, a new canvas fragment is created.

[0073] The step filtering module is used to integrate visual changes, event intensity, splicing confidence, and window layer changes to perform step boundary detection and calculate importance to filter key steps.

[0074] The indexing and playback module is used to build a step graph and index that includes canvas identifiers, keyframe identifiers and event summaries. On the playback end, key steps are presented first by step-level time compression and cursors and event markers are superimposed. Consistency checks are performed, and if the consistency is not met, a rollback correction is triggered.

[0075] In this embodiment, desktop screenshots are aligned with multi-dimensional event timestamps, and adaptive sampling based on operation intensity is used to capture operation details completely while avoiding wasted invalid screenshots, balancing integrity and storage economy. By relying on screenshot stitching and incremental storage technology, a globally consistent operation context is constructed, solving the problem of information loss during window switching and complex operations. Intelligent filtering of key steps and graph indexing enable precise positioning of operation steps, and combined with step-level time compression and event overlay presentation at the playback end, significantly improving backtracking efficiency. Playback consistency verification and rollback correction mechanisms ensure accurate reproduction. Ultimately, with low storage and transmission costs, high-completeness operation reproduction, and efficient backtracking experience, the effectiveness of operation review in practical training is optimized, helping trainees quickly master core skills and significantly improving training quality. Attached Figure Description

[0076] Figure 1 A flowchart illustrating a method for replaying key steps of a training room desktop operation based on real-time screenshot stitching, provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram of canvas fragment splicing and reconstruction provided in an embodiment of this application;

[0078] Figure 3 This is a schematic diagram of step-level time mapping provided for embodiments of this application;

[0079] Figure 4 This is a schematic diagram of a training room desktop operation key step playback system based on real-time screenshot stitching, provided as an embodiment of this application. Detailed Implementation

[0080] The present solution will now be described in conjunction with the accompanying drawings and specific embodiments.

[0081] See Figure 1 The method for replaying key steps of desktop operation in a training room based on real-time screenshot stitching provided in this embodiment includes:

[0082] S101 captures a sequence of desktop screenshots and simultaneously captures mouse, keyboard, window focus, and window geometry change events, and aligns them with timestamps.

[0083] In this embodiment, the current desktop frame buffer is obtained on the training equipment terminal, and a screenshot sequence is generated. The event stream is composed of synchronously collected data on mouse movements or clicks, keyboard presses, window focus changes, window geometric changes (position or size), and application switching information. The screenshot sequence and the event stream are aligned using a unified timestamp to obtain an aligned dataset. in For timestamp The corresponding event set includes all user interaction events that occurred within the event window before and after the timestamp.

[0084] S102 adaptively adjusts the screenshot sampling frequency based on the event intensity and the visual changes of adjacent screenshots.

[0085] Based on the event set Calculate event intensity , This includes a weighted sum of clicks, keystrokes, and window switches per unit time. It also calculates the intensity of visual change based on adjacent screenshots. The The ,in: This indicates the weighting coefficient for each event category, representing the degree to which different events contribute to the overall event intensity. Indicates event type q in Frequency or count of occurrences within the time window Indicates adjacent screenshots Structural similarity.

[0086] According to the above With the Adaptive determination of the next screenshot sampling interval To control the sampling frequency of the screenshot sequence:

[0087]

[0088] in: For a moment The next screenshot interval, Minimum screenshot interval, Maximum screenshot interval Based on the sampling scale constant, This is the event intensity weighting coefficient. The weighting coefficient is the intensity of visible change. Indicates will Cut off to interval .

[0089] S103: Generate fingerprints and local features from the screenshot and match them, estimate the geometric mapping and calculate the splicing confidence. If the confidence meets the threshold, the mapped screenshot is merged and updated to the canvas and stored incrementally with differential patches; otherwise, a new canvas fragment is created.

[0090] See Figure 2 This embodiment first calculates the fingerprint for retrieving candidate overlapping frames from the current screenshot and extracts local features, then obtains a set of matching pairs based on feature matching. Specifically, a global fingerprint is calculated for each screenshot. It is used for quickly and coarsely screening historical candidate frames or canvases; calculating local feature sets. ,in: For the coordinates of the key points, For descriptors; perform feature matching in the candidate set to obtain a set of matching pairs. .

[0091] Then, based on the matching pair set, the geometric mapping from the current screenshot to the target canvas coordinate system is estimated, and the stitching confidence is calculated based on the reprojection error and the inlier ratio. Specifically, robust estimation is used to obtain the mapping matrix from the current screenshot to the target canvas coordinate system. And calculate the error:

[0092]

[0093] in: The average reprojection error is used to measure the reliability of the stitching. The coordinates of the corresponding point in the target canvas or reference frame. The coordinates of the matching point in the current screenshot. For homogeneous normalized projection operators, It is the square of the L2 norm.

[0094] The stitching confidence level has been defined based on the average reprojection error:

[0095]

[0096] in: To construct the confidence level, a higher confidence level indicates greater reliability. For the Sigmoid function, To match the quantity indicators, For interior point proportions, , and These are the weighting coefficients.

[0097] When the splicing confidence level meets a preset threshold, the current screenshot is geometrically remapped and then fused to the canvas. The canvas update process is performed incrementally using differential patches. Specifically, the current screenshot is fused to the canvas based on the mapping matrix. Mapping to the canvas coordinate system yields the alignment map. According to the above Perform incremental blending updates on canvas C:

[0098]

[0099] in: For the current canvas, To align the graph and fuse weights;

[0100] A key canvas combined with a patching mechanism is used for storage, recording the differential patch set and storing only the changed areas. When the splicing confidence does not meet the preset threshold, a new canvas segment is created and subsequent splicing updates are performed on it.

[0101] S104 integrates visual changes, event intensity, splicing confidence, and window layer changes to perform step boundary detection and calculate importance to screen key steps.

[0102] In this implementation, to achieve the screening of key steps, the visible change intensity, the event intensity, the splicing confidence, and the window layer change index are first integrated on the time axis to calculate the step boundary score. Then, the step boundary point is determined based on the step boundary score and the preset boundary judgment rule, thereby dividing the aligned data unit into multiple step segments.

[0103] Specifically, calculate the boundary score:

[0104]

[0105] in: For boundary scores, a larger score indicates a higher likelihood of step switching. For window layer change indicators, , , and These are the fusion weights for visual change intensity, event intensity, splicing confidence, and window layer change index, respectively.

[0106] when When the threshold is exceeded and a local maximum is reached, the boundary points of the critical steps are determined, and step segments are formed between the boundaries. .

[0107] An importance score is calculated for each step segment, and a set of key steps is determined from the step segments based on the importance score. Specifically, for each step segment... Calculate importance score :

[0108]

[0109] in: For the j-th step segment Importance score, The average visible change intensity within the step. The average event intensity within the step. for, This refers to the number of window switches within the step. The duration of the step. , , and These are the weighting coefficients.

[0110] Based on the importance score Sort the steps from largest to smallest, and select the steps with the highest preset number of steps as key steps to form the key step set.

[0111] S105: Construct a step graph and index containing canvas identifiers, keyframe identifiers, and event summaries. On the playback end, prioritize key steps by step-level time compression and overlay cursors and event markers. Perform consistency checks and trigger rollback corrections if the consistency is not met.

[0112] In this embodiment, a step graph and index structure, including canvas identifiers, keyframe identifiers, event summaries, and searchable fields, are first constructed for the step fragments and key steps. The step graph constructed in this embodiment is... , For each step node, it contains canvas ID, keyframe ID, event summary, and text label, which can be extracted from window title / process name / control text. For step transition edges, record the time span, trigger event type, and concatenation confidence statistics. Simultaneously, build an inverted index: supporting retrieval by application name, window title, key operation (such as "Save / Submit"), time period, student ID, etc.

[0113] Secondly, at the playback end, the canvas and differential patches are decoded and reconstructed according to the aforementioned step-level graph, and the real timeline is mapped to the playback timeline according to the step-level time mapping function. In this embodiment, a step-level time mapping function is defined from the real time... Replay time Mapping:

[0114]

[0115] in: To replay the timeline position, This is the step time scaling factor, with a larger value for critical steps and a smaller value for ordinary steps; Let j be the actual duration of step j. This is the scaling factor for the step time. For inclusion The step numbers.

[0116] See Figure 3 In this embodiment, the key steps are presented and replayed first, while the cursor trajectory and event visualization markers are overlaid and rendered. A consistency check is performed on the replay reconstruction results. When the consistency check does not meet the preset conditions, a rollback strategy is triggered to correct the replay reconstruction results.

[0117] Specifically, in this embodiment, the corresponding canvas is loaded according to the map. Restore the desktop state stream within the patch sequence step; overlay and render the cursor trajectory and click ripples, aligning them according to the event timestamp; if the difference between the reconstructed frame and the step keyframe exceeds a threshold, trigger a rollback strategy, load a denser patch, or switch to an alternate canvas node.

[0118] Corresponding to the above embodiment of the method for replaying key steps of desktop operation in a training room based on real-time screenshot stitching, this application also provides an embodiment of a system for replaying key steps of desktop operation in a training room based on real-time screenshot stitching.

[0119] See Figure 4 A training lab desktop operation key step playback system 20 based on real-time screenshot stitching includes:

[0120] The acquisition module 201 is used to acquire a sequence of desktop screenshots and simultaneously acquire mouse, keyboard, window focus, and window geometric change events, and perform timestamp alignment.

[0121] The adaptive control module 202 is used to adaptively adjust the screenshot sampling frequency based on the event intensity and the visual changes of adjacent screenshots.

[0122] The stitching and reconstruction module 203 is used to generate fingerprints and local features from the screenshot and match them, estimate the geometric mapping and calculate the stitching confidence. If the confidence meets the threshold, the mapped screenshot is merged and updated to the canvas and stored incrementally with differential patches; otherwise, a new canvas fragment is created.

[0123] The step filtering module 204 is used to integrate visual changes, event intensity, splicing confidence and window layer changes to perform step boundary detection and calculate importance to filter key steps.

[0124] The indexing and playback module 205 is used to construct a step graph and index containing canvas identifiers, keyframe identifiers and event summaries. On the playback end, key steps are presented first by step-level time compression and cursors and event markers are superimposed. Consistency checks are performed, and if the consistency is not met, a rollback correction is triggered.

[0125] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0126] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A real-time screenshot-based splicing real training room desktop operation key step playback method, characterized in that, The method comprises the following steps: Collecting a sequence of desktop screenshots and synchronously collecting mouse, keyboard, window focus and window geometry change events, and performing timestamp alignment; Adaptively adjusting the screenshot sampling frequency according to the event intensity and the visual change of adjacent screenshots; Generating fingerprints and local features for the screenshots and performing matching, estimating the geometric mapping and calculating the stitching confidence, and if the confidence meets the threshold, updating the screenshots after mapping to the canvas and storing the differential patches, otherwise creating a new canvas segment, including: Calculating the fingerprints of the current screenshot for retrieving candidate overlapping frames and extracting local features, and obtaining a matching pair set based on feature matching; Estimating the geometric mapping of the current screenshot to the target canvas coordinate system based on the matching pair set, and calculating the stitching confidence based on the re-projection error and the inlier ratio; When the stitching confidence meets the preset threshold condition, updating the current screenshot after geometric mapping and remapping to the canvas, and storing the canvas update process in the form of differential patches; When the stitching confidence does not meet the preset threshold condition, a new canvas segment is established and subsequent stitching update is performed thereon; The method comprises the following steps: Compute global fingerprints for each screenshot for fast coarse screening of historical candidate frames or canvases; Computing a local feature set wherein: is a keypoint coordinate, is a descriptor; performing feature matching in the candidate set to obtain a matching pair set ; The method comprises the following steps: robust estimation to obtain a mapping matrix from the current screenshot to the target canvas coordinate system and compute the error: wherein: is the average re-projection error, used to measure the reliability of the stitching, is the corresponding point coordinate in the target canvas or reference frame, is the matching point coordinate in the current screenshot, is the homogeneous normalized projection operator, is the square of the L2 norm; According to the average re-projection error, the stitching confidence is defined as: wherein: is a stitching confidence, the greater the more reliable, is a Sigmoid function, is a matching number indicator, is an inlier ratio, , and is a weight coefficient; Fusion visual change, event intensity, stitching confidence and window layer change for step boundary detection and calculation of important degree screening key steps, including: Fusing the visual change intensity, the event intensity, the stitching confidence and the window layer change indicator on the time axis, calculating the step boundary score, and determining the step boundary point according to the step boundary score and the preset boundary determination rule, so as to divide the aligned data unit into multiple step segments; For each step segment, calculate the importance score and determine the key step set from the step segment according to the importance score; The method comprises the following steps: The method comprises the following steps: wherein: is a boundary score, the larger the more likely a step switch occurs, is a window layer change indicator, , , and are the fusion weights of visual change intensity, event intensity, stitching confidence and window layer change indicator, respectively. When When exceeding the threshold and being a local maximum, determine a key step boundary point, the step fragment is formed between the boundaries ; For each step segment, calculate the importance score and determine the key step set from the step segment according to the importance score; For each step segment Computing importance score : wherein: is the importance score of the jth step segment , is the average visual change intensity within a step, is the average event intensity within a step, is, is the number of window switches within a step, is the step duration, , , and are weight coefficients; According to the importance scores From large to small, select the top preset number of step fragments as the key step set Building a step atlas and index containing canvas identifier, key frame identifier and event summary, and preferentially presenting key steps and superimposing cursors and event markers according to step-level time compression on the playback end, performing consistency verification, and triggering rollback correction when not meeting the condition.

2. The real-time screenshot-based splicing real training room desktop operation key step playback method according to claim 1, characterized in that, The method comprises the following steps: Obtain the current desktop frame buffer on the training device terminal to generate a screenshot sequence ; Synchronously collect mouse movement or click, keyboard key, window focus change, window geometry change position or size, application switching information group event stream ; unifying timestamp alignment of the screenshot sequence and the event stream, resulting in an aligned dataset wherein is a timestamp corresponding event collection, containing the timestamp and all user interaction events occurring within a window of events before and after the timestamp.

3. The real-time screenshot-based splicing real training room desktop operation key step playback method according to claim 2, characterized in that, The method comprises the following steps: Based on the event set Calculate event intensity And calculate the intensity of visible change based on adjacent screenshots. The The ,in: This indicates the weighting coefficient for each event category, representing the degree to which different events contribute to the overall event intensity. Indicates event type q in Frequency or count of occurrences within the time window Indicates adjacent screenshots Structural similarity; According to the described With the described Adaptive determination of next screenshot sampling interval To control the sampling frequency of the screenshot sequence: wherein: is the time of day is the next screenshot interval, is the minimum screenshot interval, is the maximum screenshot interval, is the base sampling scale constant, is the event intensity weight coefficient, is the visual change intensity weight coefficient, denotes truncating to the interval .

4. The real-time screenshot-based splicing real training room desktop operation key step playback method according to claim 1, characterized in that, The construction comprises a step atlas and index containing canvas identification, key frame identification and event summary, and in the playback end, key steps are presented in time compression priority, a cursor and event mark are superimposed, consistency verification is performed, and when the consistency verification does not meet the requirements, a rollback modification is triggered, including: A step atlas and index structure containing canvas identification, key frame identification, event summary and searchable fields are constructed for the step segment and the key step; In the playback end, the canvas and differential patch are decoded and reconstructed according to the step atlas, and the real time axis is mapped into the playback time axis according to the step-level time mapping function; The key step is presented in priority playback, while the cursor trajectory and event visualization mark are superimposed and rendered, and the consistency of the playback reconstruction result is verified, and when the consistency verification does not meet the preset condition, a rollback strategy is triggered to modify the playback reconstruction result.

Images