Real-time interactive image generation system based on multi-point touch canvas

Abstract

The invention relates to the technical field of computer graphic interactive processing, in particular to a real-time interactive image generation system based on a multi-point touch canvas. The input acquisition unit is used for acquiring original touch data from an operating system and preprocessing the original touch data; the gesture recognition and analysis unit is used for performing high-level semantic behavior analysis on the contact data sequence processed by the preprocessing module; the interaction control and parameter mapping unit is used for receiving the semantic event output by the gesture recognition and analysis unit, analyzing the semantic event into an image control command and generating an executable command sequence; and the image generation unit generates interactive image content dynamically responded in real time based on an internal graph state management mechanism. By introducing the adaptive Kalman filtering and trajectory prediction auxiliary mechanism, the filtering intensity of the contact data can be dynamically adjusted, the efficient suppression of finger jitter and the consistent reconstruction of the contact ID are realized, and the input stability and data continuity under the multi-point touch operation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer graphics interactive processing, and in particular to a real-time interactive image generation system based on a multi-touch canvas. Background Art

[0002] With the widespread application of multi-touch technology, interactive image generation systems have developed rapidly in fields such as education, design, demonstration, visual programming, and human-computer collaboration. Especially in devices such as touch-screen tablets, smart whiteboards, graphic editing software, and digital drawing terminals, users expect to perform natural and smooth operations directly on the touch canvas with their fingers or stylus to achieve interactive tasks such as free drawing, graphic transformation, and gesture control.

[0003] However, when sketching or drawing multi-layers on a tablet, users may experience discontinuous touch tracks and unstable recognition results due to finger physiological jitter, operating system contact ID reallocation, and other issues when carefully dragging, drawing accurately, or sliding quickly, which seriously affects the drawing quality and subsequent recognition accuracy. In addition, some systems use a global redrawing method to respond to interactive changes, without effectively dividing and caching the layer status and change areas, resulting in a waste of computing resources and a reduction in frame rate, making it difficult to support real-time rendering requirements in high-frequency multi-finger interaction scenarios. Therefore, a real-time interactive image generation system based on a multi-touch canvas is designed. Summary of the Invention

[0004] The purpose of the present invention is to provide a real-time interactive image generation system based on a multi-touch canvas to solve the problems raised in the above-mentioned background technology, such as the physiological jitter of the finger and the reallocation of the contact ID of the operating system when the user is dragging carefully, drawing accurately or sliding quickly, resulting in discontinuous touch tracks and unstable recognition results, which seriously affect the drawing quality and subsequent recognition accuracy. In addition, some systems use a global redrawing method to respond to interactive changes, and do not effectively divide and cache the layer status and change area, resulting in a waste of computing resources and a reduction in frame rate, making it difficult to support the real-time rendering requirements in high-frequency multi-finger interaction scenarios.

[0005] To achieve the above-mentioned object, the present invention aims to provide a real-time interactive image generation system based on a multi-touch canvas, comprising an input acquisition unit, the input acquisition unit being configured to obtain raw touch data from an operating system and pre-process the raw touch data; A gesture recognition and analysis unit, which is used to perform high-level semantic behavior analysis on the contact data sequence processed by the preprocessing module, automatically identify and classify user operation behaviors in a structured manner by combining a preset gesture rule library and a supervised learning model, and generate corresponding semantic events; An interactive control and parameter mapping unit, configured to receive the semantic events output by the gesture recognition and parsing unit, parse them into image control commands, and generate executable command sequences through state smoothing and an abnormal fallback mechanism; An image generation unit is configured to receive an executable command sequence output by the interactive control and parameter mapping unit, and to perform image state updates, path drawing, transformation operations, and layer output based on an internal graphics state management mechanism and a rendering engine, thereby generating interactive image content with real-time dynamic response.

[0006] As a further improvement of the present technical solution, the input acquisition unit includes an input module and a preprocessing module; The input module is used to collect raw touch data, which includes touch point coordinates, quantity, touch point ID, pressure, size, direction and time series; The preprocessing module preprocesses the original contact data based on an adaptive Kalman filter algorithm and a trajectory prediction auxiliary mechanism.

[0007] As a further improvement of this technical solution, the specific steps involved in preprocessing the original contact data are: Adaptive Kalman filter algorithm is used to perform dynamic smoothing on the touch point coordinates: For each touch point, a state vector is constructed on the time series. The state vector contains the two-dimensional position and velocity information of the touch point, and the velocity estimation value between consecutive frames is obtained through the time series buffer. Dynamically adjust the process noise covariance matrix of the filter according to the speed to achieve speed adaptive regulation of the filter strength; The steps involved in speed adaptive adjustment are as follows: When the contact is detected to be at low speed or stationary, the adaptive Kalman filter is used to enhance the filtering strength to smooth out subtle jitters; When the contact is detected to be in a fast-moving state, the adaptive Kalman filter is used to reduce the filtering intensity to maintain responsiveness and trajectory continuity; This approach allows the touch point trajectory to maintain high smoothness and continuity during dynamic operations without sacrificing the real-time nature of interactive responses, effectively improving the input quality and user experience in fine-grained interactive tasks such as high-precision drawing and gesture control. To further improve the consistency of touch point IDs in multi-touch, a trajectory prediction-assisted mechanism is introduced based on the adaptive Kalman filter to correct touch point ID consistency. This is used to address the problem of touch point ID jumps caused by occlusion and system misidentification. The specific steps are as follows: By analyzing the historical movement trajectory and current velocity of each touch point, the expected touch point position in the next frame is predicted based on the linear motion model; For all newly identified touch points in the current frame, calculate the Euclidean distance and direction angle between them and each predicted position; If the spatial distance between the touch point and the predicted point is less than the preset threshold and the movement direction of a new touch point in the current frame is consistent with the corresponding predicted touch point in the previous frame, the system determines that it is a continuation of the same physical touch point and inherits the original touch point ID; In the multi-point matching scenario, the optimal matching strategy based on the Hungarian algorithm is adopted to achieve the optimal solution for global contact ID allocation.

[0008] As a further improvement of the present technical solution, the gesture recognition and parsing unit includes a contact event sequence generation module, a behavior pattern recognition module and a high-level semantic event module; The contact event sequence generation module extracts the lifecycle events of each contact based on the structured contact data output by the preprocessing module, and constructs a time series sliding window based on the time window buffer mechanism. For each moment, the corresponding contact set state snapshot is generated. The corresponding contact set state snapshot includes the number, position, speed and trajectory direction; The behavior pattern recognition module is used to identify typical gesture operations contained in user operations; And through the high-level semantic event module, each recognized gesture operation is encapsulated as a structured semantic event object.

[0009] As a further improvement of the present technical solution, the specific steps involved in the behavior pattern recognition module recognizing the typical gesture behavior included in the user operation are: Within the time sliding window, for each touch point and its trajectory, spatial features, temporal features, and behavioral common features are extracted, and these features are combined to form an input vector sequence; Build a gesture recognition rule library including a set of spatial / temporal constraint rules for typical touch gestures; A supervised learning-based gesture classification model is deployed in parallel with the gesture recognition rule library to identify behavior judgments for fuzzy rules and new behavior recognition that is not included. The supervised learning-based gesture classification model uses a lightweight sequence classification network. Its input is the extracted input vector sequence, and its output is the gesture category probability distribution, which is used to determine which gesture category the current operation is more likely to belong to. When only a single rule in the gesture recognition rule library is matched and uniquely corresponds to a certain gesture category, the rule judgment result is directly used; When there are multiple possible options for the rule matching result, the gesture classification model is called to assist in the judgment and the one with the highest probability is adopted; A time series stability factor is introduced in the output stage of the behavior pattern recognition module to smooth the output, and the output category is used as the final recognition result to suppress behavior misjudgment caused by single-frame recognition errors.

[0010] As a further improvement of the present technical solution, the interactive control and parameter mapping unit includes a semantic event parsing and command translation module, a state smoothing module and an abnormal fallback module; The semantic event parsing and command translation module is used to receive the semantic event object output by the gesture recognition parsing unit, and map it to the corresponding image generation control command and its original control parameters for different event types, and output the initial control command set CMD raw ; The state smoothing module optimizes the image generation control command based on the state buffer and interpolation mechanism to obtain the control command sequence CMD after smooth interpolation. smooth ; The abnormal fallback module is used to receive the interpolated control command sequence CMD smooth , and based on the parameter validity verification mechanism, the interpolated control command sequence CMD smooth Perform a validity check and finally output the executable command sequence CMD exec .

[0011] As a further improvement of this technical solution, the state buffering and interpolation mechanism is specifically as follows: Establish a buffer pool of multi-frame continuous image control states to store the image state parameters of the latest N frames as a dynamic reference benchmark for subsequent interpolation calculations; Initial control command set CMD raw The continuous control parameters involved in the process are interpolated using a linear interpolation algorithm to generate intermediate frame control parameters in the time dimension t. For instructions with sudden changes between frames, enforce step-by-step transitions and insert intermediate frame control parameters to construct a smooth interpolated control command sequence CMD. smooth .

[0012] As a further improvement of the present technical solution, the image generation unit includes an image state management module, a graphics rendering and synthesis module, and an image output module; The image state management module is used to store and maintain the data structure set of the core control parameters during the image interaction process, and the core control parameters are output by the interactive control and parameter mapping unit through the executable command sequence CMD exec Two-way binding ensures that every image state change caused by gesture interaction is dynamically updated and recorded in the image state management module; The graphics rendering and synthesis module takes the status data provided by the image state management module as input, performs graphics rendering and layer synthesis on the drawing content generated by the user's current interaction, and passes the rendering result as the final frame image to the image output module for caching and exporting; The image output module is used to receive the final frame image generated by the graphics rendering and synthesis module, and temporarily store the bitmap data of the current frame by establishing a real-time image frame buffer; When the system detects that the user triggers the export behavior, the image output module completes the generation and format packaging export of the final output data based on the current frame bitmap data stored in the real-time image frame buffer, combined with the drawing path set and layer structure provided by the image state management module.

[0013] As a further improvement of this technical solution, the specific steps involved in the graphics rendering and synthesis module generating the final frame image are: Parse the executable command sequence CMD output by the interactive control and parameter mapping unit exec , and parse each frame control instruction in the command sequence, and complete the scheduling and change of the state object in the image state management module according to the command type; Traverse the layer stack structure in the image state management module frame by frame, and perform layer synthesis rendering in sequence according to the Z order of the layers to construct a composite image structure in a multi-layer overlay scene; During layer rendering, the system obtains a path set from the image state management module, renders each path in the path set as an image primitive, and introduces a Bezier curve interpolation algorithm to achieve smooth trajectory reconstruction. It also combines pixel coverage-based anti-aliasing rendering technology to improve path edge quality and overall visual smoothness. The global view control parameters are encapsulated into a unified affine transformation matrix, which is applied to all graphic elements in the current frame. The coordinate transformation of all drawn objects is then processed to synchronize the displacement and scaling of graphic elements in the current canvas view. Furthermore, a lazy transformation refresh strategy is introduced to skip redundant calculations if the transformation remains unchanged. Matrix updates and screen redrawing are triggered only when state parameters change. Executable command sequence CMD based on the current command frame exec The image is compared with the previous frame state, the changed area is dynamically detected, the dirty region caching strategy is used to identify the minimum boundary of the difference area, and only the local image is redrawn in this area.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This real-time interactive image generation system based on a multi-touch canvas introduces an adaptive Kalman filter and trajectory prediction auxiliary mechanism during the input acquisition phase. This allows for dynamic adjustment of the filtering intensity of the touch data, effectively suppressing finger jitter and reconstructing consistent touch IDs. This significantly improves input stability and data continuity under multi-touch operations, providing a high-quality input foundation for subsequent gesture recognition and image control.

[0015] 2. In this real-time interactive image generation system based on a multi-touch canvas, the interactive control and parameter mapping module supports high-precision mapping of semantic events to control commands. Combined with a smooth state interpolation mechanism and an abnormal fallback strategy, this system avoids command jumps and control conflicts, improving interactive coherence and user experience consistency. At the same time, by building a behavior recognition module that integrates the rule library and the supervised learning model, and combining the spatial characteristics, temporal characteristics and common behavioral characteristics of the contact trajectory, it can effectively identify a variety of typical touch behaviors including drawing, zooming, rotating, long pressing, etc., and at the same time support the recognition and expansion of new or blurred-boundary gestures.

[0016] 3. This real-time interactive image generation system based on multi-touch canvas introduces an adaptive Kalman filter and trajectory prediction auxiliary mechanism during the input acquisition phase. This allows for dynamic adjustment of the filtering intensity of the contact data, enabling efficient suppression of finger jitter and consistent reconstruction of contact IDs. This significantly improves input stability and data continuity under multi-touch operations, providing a high-quality input foundation for subsequent gesture recognition and image control. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the overall process of the present invention.

[0018] The meaning of each number in the figure is: 1. Input acquisition unit; 2. Gesture recognition and analysis unit; 3. Interaction control and parameter mapping unit; 4. Image generation unit. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0020] Example: See Figure 1 As shown, a real-time interactive image generation system based on a multi-touch canvas is provided, comprising an input acquisition unit 1, the input acquisition unit 1 being used to obtain raw touch data from an operating system and pre-process the raw touch data; In this embodiment, the input acquisition unit 1 includes an input module and a preprocessing module; The input module is used to collect raw touch data, which includes touch point coordinates, quantity, touch point ID, pressure, size, direction and time series; The preprocessing module preprocesses the original contact data based on an adaptive Kalman filter algorithm and a trajectory prediction auxiliary mechanism.

[0021] Furthermore, users' fingers inevitably experience physiological jitter during natural operations, especially during fine dragging or drawing operations. This high-frequency noise will significantly affect the stability of the touch point trajectory. At the same time, during multi-point concurrent touch, factors such as palm occlusion and rapid sliding can easily cause the contact ID to jump or be reallocated at the operating system level, affecting subsequent gesture recognition and interaction stability. Therefore, adaptive Kalman filtering and trajectory prediction assisted tracking are used to pre-process the original contact data. The specific steps involved in the pre-processing of the original contact data are: Adaptive Kalman filter algorithm is used to perform dynamic smoothing on the touch point coordinates: For each touch point, a state vector is constructed on the time series. The state vector contains the two-dimensional position and velocity information of the touch point, and the velocity estimation value between consecutive frames is obtained through the time series buffer. Dynamically adjust the process noise covariance matrix of the filter according to the speed to achieve speed adaptive regulation of the filter strength; The steps involved in speed adaptive adjustment are as follows: When the contact is detected to be at low speed or stationary, the adaptive Kalman filter is used to enhance the filtering strength to smooth out subtle jitters; When the contact is detected to be in a fast-moving state, the adaptive Kalman filter is used to reduce the filtering intensity to maintain responsiveness and trajectory continuity; This approach allows the touch point trajectory to maintain high smoothness and continuity during dynamic operations without sacrificing the real-time nature of interactive responses, effectively improving the input quality and user experience in fine-grained interactive tasks such as high-precision drawing and gesture control. To further improve the consistency of touch point IDs in multi-touch, a trajectory prediction auxiliary mechanism is introduced based on the adaptive Kalman filter to correct the consistency of touch point IDs. This is used to solve the problem of touch point ID jumps caused by occlusion and system misidentification): By analyzing the historical movement trajectory and current velocity of each touch point, the expected touch point position in the next frame is predicted based on the linear motion model; In this embodiment, the specific steps involved in predicting the expected touch point position of the next frame using the linear motion model are: For a contact i at time t, its state vector X i (t) contains its two-dimensional position and 2D velocity : , Where, X i (t) represents the state vector of the i-th contact at frame t; Indicates the x-coordinate of contact i in frame t; Indicates that contact i is in frame t y coordinate; represents the velocity of contact i in the x direction at frame t; Indicates that contact i is in frame t y Speed ​​in direction; T express; The inter-frame time interval is , the predicted state of the next frame t+1 Calculated by the following state transfer equation: , Among them, the state transfer matrix F is: , Where F represents the state transfer matrix; W t represents the process noise vector; Indicates the time interval between frames; The predicted contact position of the next frame for: , Where, Indicates the x-coordinate of the t+1 frame prediction; Represents the y coordinate of the t+1 frame prediction; where the two-dimensional velocity Velocity estimates between consecutive frames obtained through the state estimate of the Kalman filter or the time series buffer; For all newly identified touch points in the current frame, calculate the Euclidean distance and direction angle between them and each predicted position; If the spatial distance between the touch point and the predicted point is less than the preset threshold and the movement direction of a new touch point in the current frame is consistent with the corresponding predicted touch point in the previous frame (direction consistency, specifically judged as: the angle between the movement direction vector of the touch point in the previous frame and the movement direction vector of the touch point in the current frame is within the set range (such as less than 25 degrees), that is, the cosine value of the angle between the two meets ,in, is the direction consistency threshold (e.g. =0.9), used to determine whether the current touch point is a continuation of the previous touch point), the system determines that it is a continuation of the same physical touch point, thus inheriting the original touch point ID; In multi-point matching scenarios, an optimal matching strategy based on the Hungarian algorithm is used to achieve the optimal solution for global contact ID assignment. Specifically, a one-to-one match is performed between the predicted contact points (the predicted positions of known contacts in the previous frame) and the new contacts actually observed in the current frame, with the goal of minimizing the total matching cost (usually spatial distance) globally to obtain the optimal ID inheritance relationship. In multi-touch operations, users slide, rotate, and click on the screen with multiple fingers at the same time. The system needs to continuously identify each touch point and ensure that each touch point ID does not jump during its life cycle. Due to problems such as occlusion, finger leaving and returning, and ID reallocation caused by system misidentification during the operation, the contacts cannot be matched simply in sequence. Instead, global optimization matching must be performed based on motion trajectory prediction and current observation data.

[0022] This trajectory prediction assistance mechanism maintains the continuity and consistency of touch trajectory IDs at the logical level, even if touch point reallocation or ID confusion occurs at the system level. This significantly improves the stability and robustness of downstream interaction recognition modules. Based on the above-mentioned adaptive Kalman filtering algorithm and ID reconstruction, the system further automatically determines the state of each contact as Down (pressed), Move (sliding), or Up (lifted) based on the life cycle of the contact and the events of the first appearance, continuous movement, and disappearance of each contact in the time series. The filtering results are combined with the predicted trajectory to form a structured standard output for subsequent gesture recognition and image generation modules to call.

[0023] The real-time interactive image generation system based on the multi-touch canvas also includes a gesture recognition and analysis unit 2, which is used to perform high-level semantic behavior analysis on the contact data sequence processed by the pre-processing module, automatically identify and classify the user's operation behavior in a structured manner by combining a preset gesture rule library and a supervised learning model, and generate corresponding semantic events; In this embodiment, the gesture recognition and analysis unit 2 includes a contact event sequence generation module, a behavior pattern recognition module and a high-level semantic event module; The contact event sequence generation module extracts the lifecycle events of each contact: Down (press), Move (slide), and Up (lift) based on the structured contact data output by the preprocessing module. Based on the time window buffering mechanism, it constructs a time series sliding window to facilitate subsequent behavior analysis within a local time period. At each moment, a corresponding contact set state snapshot is generated, which includes dynamic attributes such as quantity, position, speed, and trajectory direction. The behavior pattern recognition module is used to identify typical gesture operations contained in user operations (including single-finger drawing, two-finger zooming, two-finger rotation, long press, and erase); Each recognized gesture operation is encapsulated into a structured semantic event object (including event type, target contact ID, duration, and related parameters (such as rotation angle, zoom factor, etc.)) through the high-level semantic event module. The semantic event serves as the input of the subsequent interactive control and parameter mapping unit 3 to achieve accurate mapping and control of image generation instructions.

[0024] Furthermore, to address the poor adaptability and scalability of traditional gesture recognition methods based on fixed thresholds or hard-coded rules, and considering the overlapping characteristics of temporal dynamics and spatial interactivity of gesture behavior, the behavior pattern recognition module identifies typical gesture behaviors contained in user operations. The specific steps involved are as follows: Within the time sliding window, for each touch point and its trajectory, spatial features (trajectory length, rate of change of distance between touch points, trajectory curvature, and angle change), temporal features (contact event duration, velocity change rate, and dwell time), and behavioral common features (touch point quantity change pattern, synchronization, interleaving, and relative movement direction) are extracted. These features (spatial features, temporal features, and behavioral common features) are combined to form an input vector sequence for joint use by rule matching and model inference. Build a gesture recognition rule library including a set of spatial / temporal constraint rules for typical touch gestures; The gesture recognition rule library includes but is not limited to the following rules If the number of contacts in the current frame is equal to 1 and the trajectory moves continuously and the moving direction changes smoothly, then the matching rule R 绘图 Marked as drawing gesture; If the number of contacts in the current frame is equal to 2 and the Euclidean distance between contacts continues to increase and the direction angle remains basically unchanged in multiple time frames, then the matching rule R 缩放 Mark as magnification gesture; If the number of contacts in the current frame is equal to 2 and the trajectory of each contact rotates clockwise / counterclockwise around the center point, the trajectory direction angle changes continuously, and the angle increment exceeds the set rotation angle threshold, then the matching rule R 旋转 Marked as rotation gesture; If a contact maintains a low speed or static state in the same spatial area and the duration exceeds the set long press time threshold, and the pressure value maintains a stable or rising trend, then the matching rule R 长按 Mark as long press gesture; If the number of contacts in the current frame is greater than 2, and multiple contacts slide horizontally at high speed in similar directions, the trajectories are compactly distributed in the horizontal direction, and the overall speed average is higher than the speed setting threshold, then the matching rule R 擦除 Mark as erase gesture; To adapt to individual operational differences and complex contextual changes, a supervised learning-based gesture classification model is deployed in parallel with the gesture recognition rule library. (The supervised learning model serves as a supplementary recognition path for fuzzy judgment, uncertain scenarios, and fault-tolerant fallback.) This model is used for behavior judgment in areas with fuzzy recognition rules and unrecognized new behavior recognition (i.e., for multi-category gesture discrimination based on the extracted touch spatial-temporal feature sequences). Among them, the cases where the rules are ambiguous include mixing scaling and rotation behaviors, and unclear boundaries between drawing and dragging. New behavior recognition scenarios not included include rapid zoom + rotation combination actions; The supervised learning-based gesture classification model uses a lightweight sequence classification network (such as LSTM or TCN). Its input is an extracted input vector sequence (such as contact spacing changes, velocity curves, duration, relative direction, etc.), and its output is a probability distribution of gesture categories (such as drawing, zooming, rotating, long press, erasing, etc.). It is used to determine which gesture category the current operation is most likely to belong to. By introducing different user datasets during the training phase, the model can gradually learn the operation patterns of different users, thereby achieving adaptive generalization of gesture recognition. Specifically, to enable the supervised learning-based gesture classification model to effectively recognize diverse touch gestures, the training sample structure is designed to cover spatial features, temporal features, topological relationships between fingers, and contextual features. These include: the number of touch points involved in the current gesture (e.g., 1 finger, 2 fingers, 3 fingers, etc.), the trajectory of each touch point, the trajectory velocity vector, the contact spacing change curve (used to determine gestures such as zooming in / out), the angle change rate (used to identify rotation movements), the trajectory duration (used to identify rotation movements), the average pressure value (used to determine long press, pressure-sensitive drawing, etc.), and the variance / energy index (characterizing trajectory stability and used to determine whether there is jitter). All sequence features can use a sliding window structure with a unified length to facilitate network processing. Furthermore, for the multi-category gesture classification task, the supervised learning model uses the cross-entropy loss function to measure the difference between the predicted probability distribution and the true label; And to support multi-category and multi-gesture recognition, the labels adopt a one-hot encoding structure; When only a single rule in the gesture recognition rule library is matched and uniquely corresponds to a certain gesture category, the rule judgment result is directly used; When there are multiple possible options or low confidence in the rule matching results, the gesture classification model is called to assist in the judgment and the one with the highest probability is adopted; In the output stage of the behavior pattern recognition module, a time series stability factor is introduced (for example, if the number of continuous recognition frames is greater than N, a minimum continuous recognition frame number threshold is set, where N represents the minimum consistent frame number threshold in a time sliding window, generally between 3 and 5 frames, determined based on the gesture recognition sampling rate (for example, 30 frames per second) and the target response time (less than 0.5s)). Output smoothing is performed and the category is output as the final recognition result (output is only made when the recognition result of a candidate gesture category remains consistent within N consecutive frames (or the category with the highest probability output by the classification model remains constant)). This is used to suppress behavior misjudgments caused by single-frame recognition errors, improving the overall robustness of the system and the consistency of user experience. In this embodiment, the specific steps involved in introducing the time series stability factor to perform output smoothing are: Set a time sliding window of length N, where N is the minimum continuous recognition frame number threshold, generally 3 to 5 frames, determined based on the sampling rate (e.g., 30 frames / second) and system response requirements. The window stores gesture recognition results and their corresponding classification probabilities for N consecutive frames. When each new recognition frame arrives, the number of occurrences and probability distribution of each gesture category in the sliding window are counted. If the number of recognition results of a certain category in the window reaches or exceeds a preset threshold (such as N), and the average classification probability of this category is the highest, then the current output category is determined to be this gesture category; If the current recognition category is the same as the output category at the previous moment, then continue to output that category directly; If the category changes, the delayed confirmation mechanism waits for N consecutive frames within the window to show consistent results before switching outputs, avoiding result jitter caused by a single frame misjudgment. At the same time, the sliding window length N and the category occurrence threshold are dynamically adjusted according to the specific application scenario to achieve a balance between recognition speed and stability.

[0025] The real-time interactive image generation system based on the multi-touch canvas further includes an interactive control and parameter mapping unit 3, which is used to receive the semantic events output by the gesture recognition and parsing unit, parse them into image control commands, and generate an executable command sequence through state smoothing and an abnormal fallback mechanism; In this embodiment, the interactive control and parameter mapping unit 3 includes a semantic event parsing and command translation module, a state smoothing module and an abnormal fallback module; The semantic event parsing and command translation module is used to receive the semantic event object (including event type, contact ID, parameter value, etc.) output by the gesture recognition parsing unit, and for different event types (drawing, scaling, rotation, dragging, erasing, etc.), map it to the corresponding image generation control command and its original control parameters, and output the initial control command set CMD raw ; In this embodiment, the initial control command set CMD raw It is a sequence of instructions extracted from the semantic event object output by gesture recognition; The following are examples of mappings for various events: Scaling class: translates the scaling factor (such as 1.2) in the scaling semantic event into the instruction setScale(currentScale*1.2); Drag class: Extract the track offset (dx, dy) and translate it into: translateCanvas(dx, dy); Drawing class: inject the trajectory sequence into the drawing path array in order to generate drawing line segments; Eraser class: Finds graphic elements in the intersection area based on the sliding trajectory and executes deletePath(*); Long press class: Construct a right-click menu call instruction, such as: triggerContextMenu(x,y); The state smoothing module optimizes the image generation control command based on the state buffer and interpolation mechanism to obtain the control command sequence CMD after smooth interpolation. smooth ; The abnormal fallback module is used to receive the interpolated control command sequence CMD smooth , and based on the parameter validity verification mechanism, the interpolated control command sequence CMD smooth Perform legality check, legality verification, state boundary constraints, conflict detection and exception handling, and decide whether to execute, modify or rollback, and finally output the executable command sequence CMD exec In addition to outputting the executable command CMD, the exception fallback module exec In addition, the state limit information is returned to the state smoothing module to adjust the subsequent interpolation strategy (such as dynamically shrinking the interpolation step).

[0026] The parameter validity verification mechanism is as follows: The boundary conditions of the parameters of each frame control command are checked for legitimacy, such as the lower limit of the scaling factor is not less than 0.1 and the rotation angle does not exceed ±360 degrees; Check whether there are any command conflicts or interaction logic anomalies in the command sequence (such as the same touch point triggering multiple events), and perform state rollback or request confirmation (such as the hover confirmation mechanism); If a control frame is found to contain illegal parameters or abnormal conflicts, then: Abort the execution of this frame and its subsequent frames; Roll back to the most recent valid state or query the state stack; Trigger the abnormal feedback mechanism.

[0027] Furthermore, the state buffering and interpolation mechanism is the core of the control parameter interpolation and state smoothing module, which is used to optimize the original control commands output by the semantic event parsing module to generate a smooth control command sequence with stronger temporal continuity and operational stability. The state buffering and interpolation mechanism is specifically as follows: Establish a buffer pool of multi-frame continuous image control states to store the image state parameters (such as position, scale, and rotation angle) of the latest N frames as a dynamic reference benchmark for subsequent interpolation calculations; Initial control command set CMD raw The continuous control parameters involved (such as targetScale, rotationAngle, offsetX / Y) use linear interpolation algorithm to generate intermediate frame control parameters in the temporal dimension t; The linear interpolation algorithm is used to perform linear transitions on the scaling, rotation, and translation parameters at equal intervals along the time dimension between two consecutive state frames to generate a sequence of interpolated control parameters; Among them, the interpolation process satisfies the following constraints; Time resolution: (33ms per frame); Maximum step length limit: ; For commands with sudden changes between frames (such as changing the zoom from 1.0 to 2.5 instantly), a step-by-step transition is enforced and intermediate frame control parameters are inserted to prevent the sudden change from causing an abrupt interactive experience, thereby constructing a smoothly interpolated control command sequence. , where each c i It is the control frame obtained by smooth interpolation at a reasonable step.

[0028] The real-time interactive image generation system based on the multi-touch canvas also includes an image generation unit 4, which is used to receive the executable command sequence output by the interactive control and parameter mapping unit 3, and perform image state updates, path drawing, transformation operations and layer output based on the internal graphics state management mechanism and rendering engine to generate real-time dynamic response interactive image content.

[0029] In this embodiment, the image generation unit 4 includes an image state management module, a graphics rendering and synthesis module, and an image output module; The image state management module is used to store and maintain the data structure set of the core control parameters in the image interaction process, and the core control parameters are output through the executable command sequence CMD output by the interactive control and parameter mapping unit 3. execTwo-way binding (dynamically manages the current canvas's zoom ratio, rotation angle, offset position, path collection, layer structure, and drawing style parameters through an event-driven state synchronization mechanism (the event-driven state synchronization mechanism is a responsive state management paradigm commonly used in front-end / interactive systems, indicating that the system triggers state changes based on events (such as drawing and zooming)). The zoom ratio, rotation angle, and canvas offset constitute the global view control parameters, and the drawing path collection stores the trajectory data (including coordinate sequence, color, line width, brush type, etc.) generated by each user drawing operation. The layer structure adopts a stacked layer management method. LayerStack supports path grouping, layer switching, and visibility control. The state snapshot stack (StateStack) records the state snapshot of the canvas after each operation, supporting timing control operations such as undo, redo, and state rollback). This ensures that the image state changes generated by each gesture interaction are dynamically updated and recorded in the image state management module. In this embodiment, the core control parameters include: Global view control parameters: such as zoom ratio, rotation angle, and canvas offset; Drawing path parameters: such as the coordinate points of the path sequence, color value, line width, brush type, etc.; Layer control parameters: such as layer visibility flag, Z order, transparency, blending mode, lock status, etc. Style control parameters: such as brush mode, dynamic line width rules, trajectory smoothness parameters, etc. The above core control parameters serve as the state basis for each frame of image generation in this system. They are updated in real time under the drive of interactive instructions and mapped to the graphics rendering process. The layer stack structure (LayerStack) corresponding to the layer control parameters uses a stack-like organization method to arrange layers in Z order. It is used to record the state of each layer to support rendering traversal and image synthesis operations, ensuring that the system can achieve orderly state management and consistency in graphics structure construction in a multi-layer environment. The data structure set refers to a structured organizational form used to carry the above core control parameters and their timing change records, specifically including: Global view state object (used to record the zoom, rotation and translation status of the current canvas); Path collection (used to store all user drawing tracks, each path is an object unit containing coordinates, color, and style); Layer stack structure (used to manage the hierarchical relationship of image content, stacking and organizing independent drawing layers in Z-axis order. Each layer can have its own graphic elements, transformation properties and visibility state, and supports compositing and overlaying between layers; for example, background layer, drawing layer, special effects layer, etc. Each layer may contain multiple paths, shapes or other graphic elements. The stack structure means that there is a top-down relationship (Z order) between layers, which determines their rendering order and overlay relationship); Multiple structured carriers, such as the state snapshot stack (used to record the complete image state snapshot after each interaction operation, enabling state backtracking, undo and redo), are used to maintain the consistency, traceability and dynamic responsiveness of the image state during the interaction process; Among them, the global view control parameters are output by monitoring the executable command sequence CMD output by the interactive control and parameter mapping unit (3). exec Real-time state changes: After receiving each control command, the system compares its parameters with the current canvas state and updates the state parameters through mapping strategies (such as incremental overlay, direct overlay, or buffered interpolation), ensuring that the image state is highly synchronized and consistent during multi-finger interaction, complex gestures, and continuous drawing. The graphics rendering and synthesis module takes the status data provided by the image state management module as input, performs graphics rendering and layer synthesis on the drawing content generated by the user's current interaction, and passes the rendering result as the final frame image to the image output module for caching and exporting; The image output module is used to receive the final frame image generated by the graphics rendering and synthesis module, and temporarily store the bitmap data of the current frame by establishing a real-time image frame buffer (FrameBuffer) to support real-time preview, image snapshot and subsequent export operations; When the system detects that the user triggers an export action (such as executing a save or export command), the image output module generates the final output data and exports it in a format package based on the current frame bitmap data stored in the real-time image frame buffer, combined with the drawing path set and layer structure provided by the image state management module.

[0030] When the system detects that the user executes an explicit output instruction such as save or export, the image output module, based on the global view control parameters, layer structure and path set stored in the current image state management module, combines the bitmap frame image output by the graphics rendering synthesis module to complete the packaging and export of the final output image data; The exported content includes but is not limited to: (1) bitmap data of the current frame image (the image output is saved in common image formats such as PNG and JPG for image storage and external display); (2) a complete path set and layer structure (all layers, path tracks and their style parameters are serialized into structured data formats such as JSON and SVG for image storage and external display) to support subsequent retrospective editing, cross-device migration or project-level archiving management.

[0031] Specifically, the steps involved in the graphics rendering and synthesis module generating the final frame image are: Parse the executable command sequence CMD output by the interactive control and parameter mapping unit exec , and parse each frame control instruction in the command sequence, and complete the scheduling and change of the state objects (related layers, paths, styles, etc.) in the image state management module according to the command type (such as setScale, translateCanvas, drawPath, deletePath); Traverse the layer stack structure (LayerStack) in the image state management module frame by frame, and perform layer synthesis rendering in sequence according to the Z order (stack order) of the layers. During the layer traversal process, support transparency adjustment, blending mode control and editable status identification of each layer, thereby realizing the construction of a composite image structure in a multi-layer overlay scene; During layer rendering, the system obtains a path collection from the image state management module and renders each path in the path collection (based on its trajectory coordinates, color, line width, brush type, etc.) as an image primitive. It supports pressure-driven stroke width changes and speed-driven brush deformation effects, introduces a Bezier curve interpolation algorithm to achieve smooth trajectory reconstruction, and combines pixel coverage-based anti-aliasing rendering technology (such as MSAA or FXAA) to improve path edge quality and overall visual smoothness. In this embodiment, when performing path rendering, the graphics rendering and synthesis module uses a Bezier curve interpolation algorithm to smoothly reconstruct the trajectory of the original coordinate point sequence in the user's drawing path. The specific steps involved are: Using the continuous trajectory points input by the user as control points, construct quadratic or cubic Bezier curve segments to replace the original broken line trajectory for path drawing, thereby improving the smoothness of the curve while maintaining the original gesture trend and avoiding jagged and discontinuous phenomena caused by sparse sampling points or gesture jitter. At the same time, during the graphics rasterization rendering stage, an anti-aliasing rendering mechanism based on pixel coverage is introduced. By weightedly blending the pixel grayscale of the edge areas of image primitives (such as using MSAA or FXAA methods), the jaggedness of path boundaries is reduced, improving overall visual quality. The above path interpolation and anti-aliasing mechanisms work together in the path drawing process, effectively improving the smoothness and aesthetics of handwritten paths. This is particularly suitable for image generation requirements in complex gesture scenarios such as high-frequency curve drawing and multi-finger synchronous writing. Specifically, the anti-aliasing rendering mechanism based on pixel coverage determines the partial coverage of the edge pixels of the graphics during the rasterization process, and dynamically adjusts the grayscale value or color weight of the pixel to achieve a smooth transition in the edge area. Specifically, when the edge of a path falls between multiple pixel units, the system calculates its output color value based on the area ratio (coverage) of the pixel covered by the path primitive, achieving grayscale blending or sub-pixel filling to avoid jagged discontinuous stripes on the edge of the path. In order to further improve rendering performance and image quality, the system chooses to use multi-sampling anti-aliasing technology or fast approximate anti-aliasing technology; among them, multi-sampling anti-aliasing technology generates a more refined edge blending effect by performing coverage statistics on multiple sampling points of each pixel, and fast approximate anti-aliasing technology performs post-processing on the image after rendering is completed, and quickly blurs and smoothes the jagged area based on the edge detection algorithm, which has the advantages of low performance overhead and strong compatibility. Global view control parameters (scaling, rotation, and translation, etc.) are encapsulated as a unified affine transformation matrix, applied to all graphic elements in the current frame, and coordinate transformation processing is performed on all drawn objects. This is used to synchronize the displacement and scaling of graphic elements in the current canvas view. As long as the transformation remains unchanged, a transformation lazy refresh strategy is introduced to skip redundant calculations, triggering matrix updates and screen redrawing only when state parameters change or new commands are received. Executable command sequence CMD based on the current command frame exec By comparing the image state with the previous frame, the changed area is dynamically detected, and a dirty region caching strategy is used to identify the minimum boundary of the different area. Only the local image redrawing is performed on this area, which significantly reduces the invalid graphics calculation overhead and improves the frame rate and response speed of the graphics system. In this embodiment, the graphics rendering and synthesis module adopts a local refresh strategy based on the dirty region caching strategy to dynamically identify the difference area between the current frame and the previous frame image state and redraw the local image, which specifically includes the following steps: In each rendering cycle, the system receives an executable command sequence CMD from the interactive control and parameter mapping unit exec, and compare the control parameters in the current command (such as zoom ratio, translation offset, path trajectory, layer state, etc.) with the state parameters of the previous frame cached in the image state management module, and extract the image elements that have changed and their affected range; For any graphics objects that have changed (such as adding a new path, changing the path style, or offsetting a layer), the minimum enclosing rectangular area in the canvas coordinate system is calculated based on its spatial bounding box and added to the dirtyregion set. If the global view transformation (such as scaling or rotation) occurs, the entire canvas is marked as dirty. Perform spatial overlap detection and merge processing on multiple dirty areas collected in the current cycle to avoid repeated drawing, forming a set of minimized and non-overlapping dirty areas as the target area set for this local redrawing; When executing a drawing task, the rendering engine only traverses and redraws the target area content in the dirty area set, keeping the unchanged area image cached in the frame buffer, thereby reducing GPU drawing instructions and computing overhead; After the redraw is completed, the newly generated image frame and the dirty area are written into the real-time image frame buffer (FrameBuffer) in the image output module to ensure that the next rendering cycle can continue the difference recognition based on the latest image state and build a continuous area refresh chain; By introducing a dirty region caching strategy, this system can significantly reduce full-canvas redraw operations in scenarios with high-frequency multi-finger interaction, complex path drawing, and frequent view changes, thereby lowering GPU resource consumption, improving rendering efficiency, and enhancing system frame rate and user operation smoothness.

[0032] The basic principles, main features, and advantages of the present invention are shown and described above. It should be understood by those skilled in the art that the present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention, and such changes and modifications fall within the scope of the invention claimed.

Claims

1. A real-time interactive image generation system based on a multi-touch canvas, characterized by: include An input acquisition unit (1), the input acquisition unit (1) being used to acquire original touch data from an operating system and to pre-process the original touch point data; A gesture recognition and analysis unit (2), the gesture recognition and analysis unit (2) is used to perform high-level semantic behavior analysis on the contact data sequence processed by the pre-processing module, automatically recognize and structure the user operation behavior in combination with a preset gesture rule library and a supervised learning model, and generate corresponding semantic events; An interactive control and parameter mapping unit (3), the interactive control and parameter mapping unit (3) is used to receive the semantic events output by the gesture recognition and parsing unit, parse them into image control commands, and generate an executable command sequence through state smoothing and an abnormal fallback mechanism; An image generation unit (4) is used to receive the executable command sequence output by the interactive control and parameter mapping unit (3), and perform image state update, path drawing, transformation operation and layer output based on the internal graphics state management mechanism and the rendering engine to generate interactive image content with real-time dynamic response.

2. The real-time interactive image generation system based on multi-touch canvas according to claim 1, characterized in that: The input acquisition unit (1) comprises an input module and a preprocessing module; The input module is used to collect raw touch data, which includes touch point coordinates, quantity, touch point ID, pressure, size, direction and time series; The preprocessing module preprocesses the original contact data based on an adaptive Kalman filter algorithm and a trajectory prediction auxiliary mechanism.

3. The real-time interactive image generation system based on multi-touch canvas according to claim 2, characterized in that: The specific steps involved in preprocessing the original contact data are: Adaptive Kalman filter algorithm is used to perform dynamic smoothing on the touch point coordinates: For each touch point, a state vector is constructed on the time series. The state vector contains the two-dimensional position and velocity information of the touch point, and the velocity estimation value between consecutive frames is obtained through the time series buffer. Dynamically adjust the process noise covariance matrix of the filter according to the speed to achieve speed adaptive regulation of the filter strength; Based on the adaptive Kalman filter, a trajectory prediction auxiliary mechanism is introduced to correct the contact ID consistency. The specific steps are as follows: By analyzing the historical movement trajectory and current velocity of each touch point, the expected touch point position in the next frame is predicted based on the linear motion model; For all newly identified touch points in the current frame, calculate the Euclidean distance and direction angle between them and each predicted position; If the spatial distance between the touch point and the predicted point is less than the preset threshold and the movement direction of a new touch point in the current frame is consistent with the corresponding predicted touch point in the previous frame, the system determines that it is a continuation of the same physical touch point and inherits the original touch point ID; In the multi-point matching scenario, the optimal matching strategy based on the Hungarian algorithm is adopted to achieve the optimal solution for global contact ID allocation.

4. The real-time interactive image generation system based on multi-touch canvas according to claim 1, characterized in that: The gesture recognition and parsing unit (2) includes a contact event sequence generation module, a behavior pattern recognition module and a high-level semantic event module; The contact event sequence generation module extracts the lifecycle events of each contact based on the structured contact data output by the preprocessing module, and constructs a time series sliding window based on the time window buffer mechanism. For each moment, the corresponding contact set state snapshot is generated. The corresponding contact set state snapshot includes the number, position, speed and trajectory direction; The behavior pattern recognition module is used to identify typical gesture operations contained in user operations; And through the high-level semantic event module, each recognized gesture operation is encapsulated as a structured semantic event object.

5. The real-time interactive image generation system based on multi-touch canvas according to claim 4, characterized in that: The specific steps involved in the behavior pattern recognition module recognizing the typical gesture behavior included in the user operation are: Within the time sliding window, for each touch point and its trajectory, spatial features, temporal features, and behavioral common features are extracted, and these features are combined to form an input vector sequence; Build a gesture recognition rule library including a set of spatial / temporal constraint rules for typical touch gestures; A supervised learning-based gesture classification model is deployed in parallel with the gesture recognition rule library to identify behavior judgments for fuzzy rules and new behavior recognition that is not included. The supervised learning-based gesture classification model uses a lightweight sequence classification network. Its input is the extracted input vector sequence, and its output is the gesture category probability distribution, which is used to determine which gesture category the current operation is more likely to belong to. When only a single rule in the gesture recognition rule library is matched and uniquely corresponds to a certain gesture category, the rule judgment result is directly used; When there are multiple possible options for the rule matching result, the gesture classification model is called to assist in the judgment and the one with the highest probability is adopted; A time series stability factor is introduced in the output stage of the behavior pattern recognition module to smooth the output, and the output category is used as the final recognition result to suppress behavior misjudgment caused by single-frame recognition errors.

6. The real-time interactive image generation system based on multi-touch canvas according to claim 1, characterized in that: The interactive control and parameter mapping unit (3) includes a semantic event parsing and command translation module, a state smoothing module and an abnormal fallback module; The semantic event parsing and command translation module is used to receive the semantic event object output by the gesture recognition parsing unit, and map it to the corresponding image generation control command and its original control parameters for different event types, and output the initial control command set CMD raw ; The state smoothing module optimizes the image generation control command based on the state buffer and interpolation mechanism to obtain the control command sequence CMD after smooth interpolation. smooth ; The abnormal fallback module is used to receive the interpolated control command sequence CMD smooth , and based on the parameter validity verification mechanism, the interpolated control command sequence CMD smooth Perform a validity check and finally output the executable command sequence CMD exec .

7. The real-time interactive image generation system based on multi-touch canvas according to claim 6, characterized in that: The state buffering and interpolation mechanism is specifically as follows: Establish a buffer pool of multi-frame continuous image control states to store the image state parameters of the latest N frames as a dynamic reference benchmark for subsequent interpolation calculations; Initial control command set CMD raw The continuous control parameters involved in the process are interpolated using a linear interpolation algorithm to generate intermediate frame control parameters in the time dimension t. For instructions with sudden changes between frames, enforce step-by-step transitions and insert intermediate frame control parameters to construct a smooth interpolated control command sequence CMD. smooth .

8. The real-time interactive image generation system based on multi-touch canvas according to claim 1, characterized in that: The image generation unit (4) includes an image state management module, a graphics rendering and synthesis module, and an image output module; The image state management module is used to store and maintain the data structure set of the core control parameters in the image interaction process, and the core control parameters are output through the executable command sequence CMD output by the interactive control and parameter mapping unit (3). exec Two-way binding ensures that every image state change caused by gesture interaction is dynamically updated and recorded in the image state management module; The graphics rendering and synthesis module takes the status data provided by the image state management module as input, performs graphics rendering and layer synthesis on the drawing content generated by the user's current interaction, and passes the rendering result as the final frame image to the image output module for caching and exporting; The image output module is used to receive the final frame image generated by the graphics rendering and synthesis module, and temporarily store the bitmap data of the current frame by establishing a real-time image frame buffer; When the system detects that the user triggers the export behavior, the image output module completes the generation and format packaging export of the final output data based on the current frame bitmap data stored in the real-time image frame buffer, combined with the drawing path set and layer structure provided by the image state management module.

9. The real-time interactive image generation system based on multi-touch canvas according to claim 8, characterized in that: The specific steps involved in the graphics rendering and synthesis module generating the final frame image are: Parse the executable command sequence CMD output by the interactive control and parameter mapping unit exec , and parse each frame control instruction in the command sequence, and complete the scheduling and change of the state object in the image state management module according to the command type; Traverse the layer stack structure in the image state management module frame by frame, and perform layer synthesis rendering in sequence according to the Z order of the layers to construct a composite image structure in a multi-layer overlay scene; During layer rendering, the system obtains a path set from the image state management module, renders each path in the path set as an image primitive, and introduces a Bezier curve interpolation algorithm to achieve smooth trajectory reconstruction. It also combines pixel coverage-based anti-aliasing rendering technology to improve path edge quality and overall visual smoothness. The global view control parameters are encapsulated into a unified affine transformation matrix, which is applied to all graphic elements in the current frame. The coordinate transformation of all drawn objects is then processed to synchronize the displacement and scaling of graphic elements in the current canvas view. Furthermore, a lazy transformation refresh strategy is introduced to skip redundant calculations if the transformation remains unchanged. Matrix updates and screen redrawing are triggered only when state parameters change. Executable command sequence CMD based on the current command frame exec The image is compared with the previous frame state, the changed area is dynamically detected, the dirty region caching strategy is used to identify the minimum boundary of the difference area, and only the local image is redrawn in this area.

Images