Real-time interactive image generation system based on multi-touch canvas

Abstract

The present application relates to the technical fields of computer graphics interactive processing, in particular to a real-time interactive image generation system based on multi-point touch canvas. It comprises an input acquisition unit for obtaining original touch data from an operating system and pre-processing the original touch data; a gesture recognition and analysis unit for high-level semantic behavior analysis on the touch data sequence processed by the pre-processing module; an interactive control and parameter mapping unit for receiving semantic events output by the gesture recognition and analysis unit, analyzing them into image control commands, and generating executable command sequences; and an image generation unit for generating real-time dynamic response interactive image content based on an internal graphics state management mechanism. By introducing an adaptive Kalman filtering and trajectory prediction auxiliary mechanism, the filtering strength of the touch data can be dynamically adjusted, the finger jitter can be efficiently suppressed, and the touch ID consistency can be reconstructed, thereby significantly improving the input stability and data continuity under multi-point touch operation.

Description

Technical Field

[0001] This invention relates to the field of computer graphics interactive processing technology, and more specifically, to a real-time interactive image generation system based on a multi-touch canvas. Background Technology

[0002] With the widespread application of multi-touch technology, interactive image generation systems have developed rapidly in education, design, presentation, visual programming, human-computer collaboration and other fields. Especially in devices such as touch screen tablets, smart whiteboards, graphic editing software and digital drawing terminals, users expect to perform natural and smooth operations directly on the touch canvas with their fingers or styluses to achieve interactive tasks such as free drawing, graphic transformation and gesture control.

[0003] However, when sketching or drawing on multiple layers on a tablet, users experience discontinuous touch trajectories and unstable recognition results due to issues such as finger tremors, touch ID reassignment in the operating system, and other problems when dragging, drawing precisely, or swiping quickly. This seriously affects the drawing quality and subsequent recognition accuracy. Furthermore, some systems use a global redrawing method to respond to interaction changes without effectively dividing and caching the layer state and change areas, resulting in wasted computing resources and reduced frame rate. This makes it difficult to support the real-time rendering requirements of high-frequency multi-finger interaction scenarios. Therefore, a real-time interactive image generation system based on a multi-touch canvas is designed. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time interactive image generation system based on a multi-touch canvas, in order to solve the problems mentioned in the background art, such as finger tremors and operating system touch point ID reassignment, which cause discontinuous touch trajectories and unstable recognition results when users are dragging, drawing accurately, or sliding quickly. This seriously affects the drawing quality and subsequent recognition accuracy. In addition, some systems use a global redrawing method to respond to interaction changes without effectively dividing and caching the layer state and the changed area, resulting in wasted computing resources and reduced frame rate, making it difficult to support the real-time rendering requirements of high-frequency multi-finger interaction scenarios.

[0005] To achieve the above objectives, the present invention aims to provide a real-time interactive image generation system based on a multi-touch canvas, including an input acquisition unit, which is used to acquire raw touch data from the operating system and preprocess the raw touch data.

[0006] The gesture recognition and parsing unit is used to perform high-level semantic behavior parsing on the touch point data sequence after the preprocessing module, and to automatically recognize and structurally classify user operation behavior by combining the preset gesture rule base and supervised learning model, and generate corresponding semantic events.

[0007] An interactive control and parameter mapping unit is configured to receive semantic events output by the gesture recognition and parsing unit, parse the semantic events into image control commands, and generate executable command sequences through state smoothing processing and an abnormality fallback mechanism;

[0008] An image generation unit is configured to receive executable command sequences output by the interactive control and parameter mapping unit, and perform image state updating, path drawing, transformation operations, and layer output based on an internal graphics state management mechanism and a rendering engine to generate real-time dynamically responsive interactive image content.

[0009] As a further improvement of the technical solution, the input collection unit comprises an input module and a preprocessing module.

[0010] The input module is configured to collect raw touch data, which includes touch point coordinates, quantity, touch point ID, pressure, size, direction, and time sequence.

[0011] The preprocessing module is configured to preprocess the raw touch point data based on an adaptive Kalman filtering algorithm and a trajectory prediction auxiliary mechanism.

[0012] As a further improvement of the technical solution, the specific steps involved in preprocessing the raw touch point data are as follows:

[0013] The adaptive Kalman filtering algorithm is used to perform dynamic smoothing processing on the touch point coordinates.

[0014] For each touch point, a state vector is constructed on the time sequence, which contains two-dimensional position and velocity information of the touch point, and the velocity estimation value between consecutive frames is obtained through time sequence buffering.

[0015] The process noise covariance matrix of the filter is dynamically adjusted according to the velocity size, which is used to realize adaptive adjustment of the filtering strength.

[0016] The steps involved in the adaptive adjustment are as follows:

[0017] When it is detected that the touch point is in a low-speed or stationary state, the filtering strength is enhanced through the adaptive Kalman filter to smooth fine jitter.

[0018] When it is detected that the touch point is in a fast-moving state, the filtering strength is reduced through the adaptive Kalman filter, thereby maintaining responsiveness and trajectory continuity.

[0019] In this way, the touch point trajectory can maintain high smoothness and continuity during dynamic operations, while not sacrificing the real-time responsiveness of the interaction, thereby effectively improving the input quality and user experience of the system in high-precision drawing, gesture control, and other fine-grained interaction tasks.

[0020] To further improve the consistency of touch point ID in multi-touch, a trajectory prediction auxiliary mechanism is introduced to correct the touch point ID consistency based on adaptive Kalman filtering, which is used to solve the problem of touch point ID jump caused by occlusion and system misidentification. The specific steps are as follows:

[0021] By analyzing the historical moving trajectory and current speed of each touch point, the expected touch point position of the next frame is predicted based on a linear dynamic model;

[0022] For all newly identified touch points in the current frame, the Euclidean distance and direction angle between them and the predicted position are calculated;

[0023] If the spatial distance between the touch point and the predicted point is less than the preset threshold and the motion direction of a new touch point in the current frame and the corresponding predicted touch point in the previous frame remains consistent, the system determines that it is the continuation of the same physical touch point, thereby inheriting the original touch point ID;

[0024] In the multi-point matching scene, an optimal matching strategy based on the Hungarian algorithm is adopted to achieve the optimal solution of global touch point ID allocation.

[0025] As a further improvement of the technical solution, the gesture recognition analysis unit includes a touch event sequence generation module, a behavior pattern recognition module, and a high-level semantic event module;

[0026] The touch event sequence generation module extracts the life cycle events of each touch point based on the structured touch point data output by the preprocessing module, and constructs a time sequence sliding window based on a time window buffering mechanism. For each time, a corresponding touch set state snapshot is generated, which includes the number, position, speed, and trajectory direction.

[0027] The behavior pattern recognition module is used to identify the behavior of typical gesture operations contained in the user operation;

[0028] Each identified gesture operation is encapsulated as a structured semantic event object through the high-level semantic event module.

[0029] As a further improvement of the technical solution, the specific steps involved in the behavior pattern recognition module to identify the behavior of typical gestures contained in the user operation are as follows:

[0030] In the time sliding window, spatial features, temporal features, and behavior common features are extracted for each touch point and its trajectory, and the above features are constructed into an input vector sequence;

[0031] A gesture recognition rule library is constructed, which includes a set of spatial / time constraint rules for typical touch gestures;

[0032] The gesture classification model based on supervised learning is deployed in the gesture recognition rule library in parallel, and is used for identifying the rule ambiguity and the behavior judgment of the uncollected new behavior recognition.

[0033] The gesture classification model based on supervised learning adopts a lightweight sequence classification network, the input of which is an extracted input vector sequence, and the output of which is a gesture category probability distribution, which is used for judging which gesture category the current operation is more likely to belong to.

[0034] When only a single rule in the gesture recognition rule library is matched and the unique corresponding gesture category is matched, the rule judgment result is directly used;

[0035] When there are multiple possible options in the rule matching result, the gesture classification model is called for auxiliary judgment, and the maximum probability is used.

[0036] In the output stage of the behavior pattern recognition module, a time series stability factor is introduced for output smoothing, and the category is output as the final recognition result, which is used to suppress the behavior misjudgment caused by single-frame recognition error.

[0037] As a further improvement of the technical solution, the interactive control and parameter mapping unit includes a semantic event analysis and command translation module, a state smoothing module, and an abnormality rollback module.

[0038] The semantic event analysis and command translation module is used to receive the semantic event object output by the gesture recognition analysis unit, and map different event types into corresponding image generation control commands and their original control parameters, and output an initial control command set CMD raw .

[0039] The state smoothing module optimizes the image generation control command based on a state buffer and interpolation mechanism to obtain a smoothed and interpolated control command sequence CMD smooth .

[0040] The abnormality rollback module is used to receive the interpolated control command sequence CMD smooth , and perform legality verification on the interpolated control command sequence CMD smooth based on a parameter legality verification mechanism, and finally output an executable command sequence CMD exec .

[0041] As a further improvement of the technical solution, the state buffer and interpolation mechanism is specifically:

[0042] A buffer pool of multi-frame continuous image control states is established to store the state parameters of the last N frames of images as a dynamic reference benchmark for subsequent interpolation calculation.

[0043] The initial control command set CMD rawThe continuous control parameters involved in the interpolation are generated by using a linear interpolation algorithm in the time dimension t to generate intermediate frame control parameters;

[0044] For instructions with sudden changes in the inter-frame variation, step-by-step transitions are forced to be performed and intermediate frame control parameters are inserted to construct a smooth interpolated control command sequence CMD smooth .

[0045] As a further improvement of the technical solution, the image generation unit includes an image state management module, a graphics rendering and synthesis module, and an image output module.

[0046] The image state management module is configured to store and maintain a data structure set of core control parameters in the image interaction process, and the core control parameters are mapped to the executable command sequence CMD output by the interactive control and parameter mapping unit. exec Bidirectional binding is used to dynamically update and record the image state changes generated by each gesture interaction in the image state management module.

[0047] The graphics rendering and synthesis module performs graphics rendering and layer synthesis processing on the drawing content generated by the current user interaction according to the state data provided by the image state management module, and transmits the rendering result as the final frame image to the image output module for caching and exporting.

[0048] The image output module is configured to receive the final frame image generated by the graphics rendering and synthesis module, and temporarily store the bitmap data of the current frame by establishing a real-time image frame buffer area.

[0049] When the system detects that the user triggers the export behavior, the image output module generates and formats the final output data based on the current frame bitmap data stored in the real-time image frame buffer area, in combination with the drawing path set and the layer structure provided by the image state management module.

[0050] As a further improvement of the technical solution, the specific steps involved in the generation of the final frame image by the graphics rendering and synthesis module are as follows:

[0051] The executable command sequence CMD output by the interactive control and parameter mapping unit is parsed exec , and each frame control instruction in the command sequence is parsed to schedule and change the state objects in the image state management module according to the command type;

[0052] The layer stack structure in the image state management module is traversed frame by frame, and the layer synthesis rendering is performed in turn according to the Z-order arrangement sequence of the layers to construct a composite image structure in a multi-layer superimposed scene.

[0053] During the layer rendering process, the system obtains the path set from the image state management module, renders each path in the path set as an image primitive, introduces the Bézier curve interpolation algorithm to achieve smooth trajectory reconstruction, and combines anti-aliasing rendering technology based on pixel coverage to improve the quality of path edges and overall visual smoothness.

[0054] The global view control parameters are encapsulated into a unified affine transformation matrix, which is applied to all graphic elements in the current frame and performs coordinate transformation on all drawing objects. This is used to control the synchronous displacement and scaling of graphic elements in the current canvas view. Furthermore, under the premise that the transformation has not changed, a transformation lazy refresh strategy is introduced to skip redundant calculations and trigger the matrix update and screen redraw process only when the state parameters change.

[0055] The executable command sequence CMD based on the current command frame exec By comparing the image state with the previous frame, the changed areas are dynamically detected. A dirty region caching strategy is used to identify the minimum boundary of the different areas, and local image redrawing is performed only on the areas in question.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] 1. In this real-time interactive image generation system based on a multi-touch canvas, by introducing an adaptive Kalman filter and trajectory prediction auxiliary mechanism in the input acquisition stage, the filtering intensity of touch data can be dynamically adjusted to achieve efficient suppression of finger tremors and consistent reconstruction of touch IDs. This significantly improves the input stability and data continuity under multi-touch operation, providing a high-quality input foundation for subsequent gesture recognition and image control.

[0058] 2. In this real-time interactive image generation system based on a multi-touch canvas, the interactive control and parameter mapping module supports high-precision mapping from semantic events to control commands. At the same time, combined with the state smooth interpolation mechanism and the abnormal rollback strategy, command jumps and control conflicts can be avoided, improving the continuity of interaction and the consistency of user experience.

[0059] Meanwhile, by constructing a behavior recognition module that integrates a rule base and a supervised learning model, and combining the spatial, temporal, and behavioral common features of touch point trajectories, it can effectively recognize a variety of typical touch behaviors, including drawing, zooming, rotating, and long pressing, while also supporting the recognition and expansion of new or blurred gestures.

[0060] 3、The real-time interactive image generation system based on multi-touch canvas, by introducing adaptive Kalman filtering and trajectory prediction auxiliary mechanism in the input acquisition stage, can dynamically adjust the filtering strength of touch data, realize efficient suppression of finger jitter and consistency reconstruction of touch ID, significantly improve the input stability and data continuity under multi-touch operation, and provide high-quality input basis for subsequent gesture recognition and image control. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 The overall flowchart of the present application is shown.

[0062] The meanings of the various numbers in the figure are as follows:

[0063] 1, input acquisition unit; 2, gesture recognition analysis unit; 3, interactive control and parameter mapping unit; 4, image generation unit. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0065] Embodiment: Please refer to Figure 1 As shown, a real-time interactive image generation system based on multi-touch canvas is provided, which includes an input acquisition unit 1, the input acquisition unit 1 is used to obtain original touch data from an operating system, and the original touch data is preprocessed;

[0066] In this embodiment, the input acquisition unit 1 includes an input module and a preprocessing module;

[0067] The input module is used to collect original touch data, and the original touch data includes touch coordinates, number, touch ID, pressure, size, direction and time sequence;

[0068] The preprocessing module preprocesses the original touch data based on an adaptive Kalman filtering algorithm and a trajectory prediction auxiliary mechanism.

[0069] Further, physiological jitter of a user's finger inevitably exists in the natural operation process, especially in fine dragging or drawing operation, and this high-frequency noise will significantly affect the stability of the touch trajectory;

[0070] At the same time, in the process of multi-point concurrent touch, due to factors such as palm shielding and rapid sliding, the touch ID is prone to jump or reallocation at the operating system level, which affects the subsequent gesture recognition and interaction stability;

[0071] Therefore, the original touch point data is preprocessed by using adaptive Kalman filtering and trajectory prediction assisted tracking, and the specific steps involved in the preprocessing of the original touch point data are as follows:

[0072] The touch point coordinates are dynamically smoothed by using an adaptive Kalman filtering algorithm:

[0073] For each touch point, a state vector is constructed on a time sequence, the state vector contains two-dimensional position and velocity information of the touch point, and the velocity estimation value between consecutive frames is obtained through a time sequence buffer;

[0074] The process noise covariance matrix of the filter is dynamically adjusted according to the velocity, for realizing adaptive adjustment of filtering strength;

[0075] The steps involved in the velocity adaptive adjustment are as follows:

[0076] When it is detected that the touch point is in a low-speed or stationary state, the filtering strength is enhanced through the adaptive Kalman filter to smooth the slight jitter;

[0077] When it is detected that the touch point is in a fast-moving state, the filtering strength is reduced through the adaptive Kalman filter, so as to maintain responsiveness and trajectory continuity;

[0078] In this way, the touch point trajectory can maintain high smoothness and continuity during dynamic operation, while the real-time responsiveness of the interaction is not sacrificed, thereby effectively improving the input quality and user experience of the system in high-precision drawing, gesture control and other fine-grained interaction tasks;

[0079] To further improve the consistency of touch point ID in multi-point touch, a trajectory prediction assisted mechanism is introduced on the basis of adaptive Kalman filtering to correct the touch point ID consistency, for solving the problem of ID jump caused by occlusion and system misrecognition:

[0080] By analyzing the historical moving trajectory and current velocity of each touch point, the expected touch point position of the next frame is predicted based on a linear dynamic model;

[0081] In this embodiment, the specific steps involved in predicting the expected touch point position of the next frame by using a linear dynamic model are as follows:

[0082] For a touch point i at time t, its state vector X i (t) contains its two-dimensional position and two-dimensional velocity :

[0083] ,

[0084] In the formula, X i(t) represents the state vector of the ith contact point at t frame; represents the x coordinate of contact i at t frame; represents the y coordinate of contact i at t frame; y represents the x coordinate of contact i at t frame; represents the velocity of contact i in x direction at t frame; represents the velocity of contact i in y direction at t frame; y represents the velocity of contact i in x direction at t frame; T represents the velocity of contact i in y direction at t frame;

[0085] wherein the inter-frame time interval is the predicted state of the next frame t+1 is calculated by the following state transition equation:

[0086] ,

[0087] wherein the state transition matrix F is:

[0088] ,

[0089] wherein F represents the state transition matrix; W t represents the process noise vector; represents the inter-frame time interval;

[0090] the predicted next frame contact position is:

[0091] ,

[0092] wherein represents the x coordinate of the t+1 frame prediction; represents the y coordinate of the t+1 frame prediction; wherein the two-dimensional velocity is obtained by the state estimation value of the Kalman filter or the velocity estimation value of the continuous inter-frame obtained by the time series buffer;

[0093] For all newly identified contact points in the current frame, the Euclidean distance and the direction angle between each prediction position are calculated;

[0094] If the spatial distance between the contact point and the prediction point is less than a preset threshold and the motion direction between a new contact point in the current frame and the corresponding prediction contact point in the previous frame remains consistent (direction consistency, which is specifically judged as: the angle between the motion direction vector of the contact point in the previous frame and the motion direction vector of the contact point in the current frame is within a set range (such as less than 25 degrees), that is, the cosine value of the angle of the two satisfies wherein is the direction consistency threshold (such as = 0.9), for determining whether the current contact is a track continuation of the last contact, the system determines it as a continuation of the same physical contact, thus inheriting the original contact ID;

[0095] In the multi-point matching scenario, an optimal matching strategy based on the Hungarian algorithm is adopted to achieve the optimal solution of global contact ID assignment, specifically: one-to-one matching between the predicted contact (the predicted position of the known contact in the last frame) and the newly observed contact in the current frame is performed to minimize the total matching cost (usually spatial distance) in the global range, and the optimal ID inheritance relationship is obtained;

[0096] In the multi-point touch operation, the user simultaneously slides, rotates, clicks, etc. on the screen with multiple fingers, and the system needs to continuously identify each contact and ensure that each contact ID does not jump in the life cycle;

[0097] Due to problems such as occlusion, finger leaving and returning, system misidentification leading to ID reallocation, etc. during the operation process, the contacts cannot be simply matched in order, but must be globally optimized and matched according to the motion trajectory prediction and current observation data.

[0098] Through the trajectory prediction auxiliary mechanism, even if the system level causes contact reallocation or ID confusion, the continuity and consistency of the contact track ID can still be maintained at the logical level, thereby significantly improving the stability and robustness of the downstream interaction recognition module;

[0099] On the basis of the above adaptive Kalman filtering algorithm and ID reconstruction, the system further determines the state of the contact as Down (pressed), Move (slid), or Up (lifted) according to the life cycle of the contact, in combination with the events of the first appearance, continuous movement, and disappearance of each contact in the time series, and combines the filtering result and the predicted trajectory to form a structured standard output for subsequent gesture recognition and image generation modules to call.

[0100] The real-time interactive image generation system based on the multi-point touch canvas further includes a gesture recognition and analysis unit 2, which is used for high-level semantic behavior analysis on the contact data sequence processed by the preprocessing module, automatically identifies and structurally classifies the user operation behavior in combination with a preset gesture rule library and a supervised learning model, and generates corresponding semantic events;

[0101] In this embodiment, the gesture recognition and analysis unit 2 includes a contact event sequence generation module, a behavior pattern recognition module, and a high-level semantic event module;

[0102] The contact event sequence generation module extracts the life cycle events of each contact, i.e. Down (pressing), Move (sliding), and Up (lifting), based on the structured contact data output by the preprocessing module, and constructs a time sequence sliding window based on a time window buffering mechanism, so as to perform subsequent behavior analysis in a local time period, and generate a corresponding contact set state snapshot for each time, which includes dynamic attributes such as quantity, position, speed, and trajectory direction.

[0103] The behavior pattern recognition module is configured to recognize the behaviors of typical gesture operations (including single-finger drawing, double-finger zooming, double-finger rotating, long pressing, and erasing) contained in the user operations.

[0104] Each recognized gesture operation is encapsulated as a structured semantic event object (including event type, target contact ID, duration, and related parameters (such as rotation angle and zoom coefficient)) by the high-level semantic event module.

[0105] The semantic event serves as an input of the subsequent interaction control and parameter mapping unit 3, and realizes accurate mapping and control of image generation instructions.

[0106] Further, in view of the poor adaptability and weak expansibility of the traditional gesture recognition method based on fixed threshold or hard-coded rules, and considering the superimposed features of time dynamics and spatial interaction of gesture behaviors, the specific steps involved in the behavior pattern recognition module for recognizing the behaviors of typical gestures contained in the user operations are as follows:

[0107] In the time sliding window, spatial features (trajectory length, contact distance change rate, trajectory curvature, and angle change), time features (contact event duration, speed change rate, and dwell time), and behavior common features (contact quantity change pattern, synchronicity, interlacing, and relative motion direction) are extracted for each contact and its trajectory, and the above features (spatial features, time features, and behavior common features) are constructed into an input vector sequence for joint calling of rule matching and model reasoning.

[0108] A gesture recognition rule library including a set of spatial / time constraint rules for typical touch gestures is constructed.

[0109] The gesture recognition rule library includes but is not limited to the following rules

[0110] If the number of contacts in the current frame is equal to 1 and the trajectory moves continuously and the moving direction changes smoothly, the rule R 绘图 is marked as a drawing gesture.

[0111] If the number of contacts in the current frame is equal to 2 and the Euclidean distance between the contacts continuously increases and the direction angle remains basically unchanged in a plurality of time frames, the rule R缩放 Zoom-in gesture is marked;

[0112] If the number of touch points in the current frame is equal to 2, the trajectories of the touch points are clockwise / counterclockwise rotation around the center point, the change of the trajectory direction angle is continuous, and the angle increment exceeds the set rotation angle threshold, then the matching rule R 旋转 Rotation gesture is marked;

[0113] If a touch point remains in a low-speed or stationary state in the same spatial region for a duration exceeding the set long-press time threshold, and the pressure value remains stable or shows an upward trend, then the matching rule R 长按 Long-press gesture is marked;

[0114] If the number of touch points in the current frame is greater than 2, and multiple touch points slide in similar directions at high speed, the trajectories are distributed closely in the horizontal direction, and the overall average speed is higher than the speed set threshold, then the matching rule R 擦除 Erase gesture is marked;

[0115] To adapt to individual operation differences and complex situation changes, a gesture classification model based on supervised learning (supervised learning model as a supplementary recognition path, used for fuzzy judgment, uncertain scenarios, and fault tolerance fallback) is deployed in the gesture recognition rule library to identify behavior judgments that are ambiguous and have not included new behavior recognition (i.e., to classify the extracted touch space-time feature sequence into multiple gesture categories);

[0116] Among them, the ambiguous rules include situations such as mixed scaling and rotation behaviors, unclear drawing and dragging boundaries, etc.

[0117] The new behavior recognition that has not been included includes situations such as fast scaling + rotation combination actions, etc.

[0118] The gesture classification model based on supervised learning uses a lightweight sequence classification network (such as LSTM or TCN), whose input is the extracted input vector sequence (such as touch point distance change, speed curve, duration, relative direction, etc.), and the output is the probability distribution of gesture categories (such as drawing, scaling, rotation, long-press, erase, etc.), which is used to determine which gesture category the current operation is more likely to belong to. By introducing different user datasets in the training stage, the model can gradually learn the operation patterns of different users, thereby achieving adaptive generalization of gesture recognition.

[0119] Specifically, in order to enable the gesture classification model based on supervised learning to effectively identify diversified touch gestures, the training sample structure is designed to cover spatial features, time features, topological relationships between fingers, and context features, specifically including: the number of touch points (the number of touch points involved in the current gesture (such as 1 finger, 2 fingers, 3 fingers, etc.)), the trajectory of each touch point, the trajectory speed vector, the touch point distance change curve (used to judge the zoom-in / zoom-out gesture), the included angle change rate (used to identify the rotation action), the trajectory duration (used to identify the rotation action), the average pressure value (used to judge the long press, pressure drawing, etc.), and the variance / energy index (representing the stability of the trajectory, used to determine whether there is jitter), wherein all sequence class features can use a sliding window structure to unify the length, facilitating network processing;

[0120] Further, for the multi-class gesture classification task, the supervised learning model uses a cross-entropy loss function to measure the difference between the predicted probability distribution and the true label.

[0121] And to support multi-class multi-gesture recognition, the label uses a one-hot encoding structure.

[0122] When only a single rule in the gesture recognition rule library is matched and uniquely corresponds to a certain gesture class, the rule determination result is directly used.

[0123] When the rule matching result has multiple possible options or low confidence, the gesture classification model is called for auxiliary judgment, and the maximum probability is used.

[0124] In the output stage of the behavior pattern recognition module, a time series stability factor (such as the number of continuously identified frames being greater than N, that is, setting a minimum continuous identification frame threshold, wherein N represents a minimum consistent frame threshold of a time sliding window, generally taking a value between 3-5 frames, determined according to the sampling rate (such as 30 frames per second) and the target response time (less than 0.5s)) is introduced for output smoothing, and the class is output as the final recognition result (only when a candidate gesture class remains consistent in the recognition result (or the maximum class of the classification model output probability is constant) for N consecutive frames, the output is performed), which is used to suppress behavior misjudgment caused by single-frame recognition error and improve the overall robustness and user experience consistency of the system.

[0125] In the present embodiment, the specific steps involved in introducing the time series stability factor for output smoothing are as follows:

[0126] A time sliding window with a length of N is set, N is the minimum continuous identification frame threshold, generally taking a value between 3 and 5 frames, determined in combination with the sampling rate (such as 30 frames / second) and the system response requirement, and the window stores the gesture recognition results and their corresponding classification probabilities for N consecutive frames;

[0127] At each new identified frame, the number of occurrences and probability distribution of each gesture category in the sliding window are counted. If the number of occurrences of a certain category in the window reaches or exceeds a preset threshold (e.g., N), and the average classification probability of the category is the highest, then the current output category is determined to be the gesture category;

[0128] If the current identified category is the same as the output category at the previous time, then the category is directly continued to be output;

[0129] If the category changes, then a delay confirmation mechanism is used to wait for N consecutive frames in the window to be consistent before switching the output, so as to avoid result jitter caused by single-frame misjudgment;

[0130] At the same time, the sliding window length N and the category occurrence threshold are dynamically adjusted according to the specific application scenario, so as to balance between recognition speed and stability.

[0131] The real-time interactive image generation system based on the multi-point touch canvas further includes an interactive control and parameter mapping unit 3, which is configured to receive semantic events output by the gesture recognition and analysis unit, analyze the semantic events into image control commands, and generate executable command sequences through state smoothing processing and an abnormal rollback mechanism;

[0132] In the embodiment, the interactive control and parameter mapping unit 3 includes a semantic event analysis and command translation module, a state smoothing module, and an abnormal rollback module.

[0133] The semantic event analysis and command translation module is configured to receive semantic event objects (including event types, touch point IDs, parameter values, etc.) output by the gesture recognition and analysis unit, and map different event types (drawing, scaling, rotating, dragging, erasing, etc.) into corresponding image generation control commands and original control parameters, and output an initial control command set CMD raw .

[0134] In the embodiment, the initial control command set CMD raw is an instruction sequence extracted from the semantic event objects output by the gesture recognition;

[0135] The mapping examples of different events are as follows:

[0136] Scaling type: The scaling factor (e.g., 1.2) in the scaling semantic event is translated into the instruction setScale(currentScale*1.2);

[0137] Dragging type: The trajectory offset (dx, dy) is translated into: translateCanvas(dx, dy);

[0138] Drawing type: The trajectory sequence is sequentially injected into a drawing path array to generate a drawing line segment;

[0139] Erase class: Finds the graphic elements in the intersection area based on the sliding trajectory and executes deletePath(*);

[0140] Long press class: Construct a right-click menu to call the command, such as: triggerContextMenu(x,y);

[0141] The state smoothing module optimizes the image generation control commands based on state buffering and interpolation mechanisms, resulting in a smoothed interpolated control command sequence (CMD). smooth ;

[0142] The exception fallback module is used to receive the interpolated control command sequence CMD. smooth And based on the parameter validity verification mechanism, the interpolated control command sequence CMD is... smooth Perform legality checks, including legality verification, state boundary constraints, conflict detection, and exception handling, and decide whether to execute, correct, or rollback. Finally, output the executable command sequence CMD. exec In addition to outputting executable commands (CMD), the exception rollback module also... exec In addition, it returns state constraint information to the state smoothing module to adjust subsequent interpolation strategies (such as dynamically shrinking the interpolation step size).

[0143] The parameter validity verification mechanism is as follows:

[0144] The parameters of each frame control command are validated for boundary conditions, such as the lower limit of the scaling factor being no less than 0.1 and the rotation angle not exceeding ±360 degrees.

[0145] Check for instruction conflicts or abnormal interaction logic in the command sequence (such as multiple events triggered by the same touch point), and perform state rollback or request confirmation (e.g., hover confirmation mechanism).

[0146] If an illegal parameter or abnormal conflict is found in a control frame, then:

[0147] Abort the execution of this frame and all subsequent frames;

[0148] Rollback to the most recent valid state or query the state stack;

[0149] Trigger the exception feedback mechanism.

[0150] Furthermore, the state buffering and interpolation mechanism is the core of the control parameter interpolation and state smoothing module. It is used to optimize the original control commands output by the semantic event parsing module, generating a smoother control command sequence with stronger temporal continuity and operational stability. Specifically, the state buffering and interpolation mechanism is as follows:

[0151] A buffer pool is established to store the state parameters (such as position, scale, rotation angle) of the last N frames of images as dynamic reference bases for subsequent interpolation calculation.

[0152] The initial control command set CMD raw The continuous control parameters (such as targetScale, rotationAngle, offsetX / Y) involved in the initial control command set CMD

[0153] The linear interpolation algorithm is used to perform linear transition on the scale, rotation and translation parameters at equal intervals along the time dimension between two consecutive state frames, generating an interpolation control parameter sequence.

[0154] The interpolation process satisfies the following constraints.

[0155] Time resolution: 33ms per frame.

[0156] Maximum step limit: ;

[0157] For instructions with sudden changes in inter-frame variation (such as instantaneously changing the scale from 1.0 to 2.5), step-by-step transition is enforced and intermediate frame control parameters are inserted to prevent abrupt changes in the interactive experience and to construct a smooth interpolation control command sequence. where each c i is a control frame obtained by smooth interpolation at a reasonable step.

[0158] The real-time interactive image generation system based on multi-point touch canvas also includes an image generation unit 4, which is used to receive the executable command sequence output by the interactive control and parameter mapping unit 3, and based on the internal graphics state management mechanism and rendering engine to perform image state update, path drawing, transformation operation and layer output, to generate real-time dynamic response interactive image content.

[0159] In this embodiment, the image generation unit 4 includes an image state management module, a graphics rendering and synthesis module, and an image output module.

[0160] The image state management module is used to store and maintain a set of data structures of core control parameters in the image interaction process, and the core control parameters are updated by the executable command sequence CMD execThe bidirectional binding (through the event-driven state synchronization mechanism (the event-driven state synchronization mechanism is a kind of reactive state management paradigm commonly used in front-end / interactive systems, and the representative system triggers state changes according to events (such as drawing, scaling, etc.)) dynamically manages the scaling ratio, rotation angle, offset position, path set, layer structure and drawing style parameters of the current canvas, wherein the scaling ratio, rotation angle and canvas offset constitute the global view control parameters, the drawing path set stores the trajectory data (including coordinate sequence, color, line width, brush type, etc.) generated by each user drawing operation, the layer structure adopts a stack type layer management manner, LayerStack supports path grouping, layer switching and visibility control, and the state snapshot stack (StateStack) records the state snapshot of the canvas after each operation, supports undo, redo, state rollback and other timing control operations) makes the image state change generated by each gesture interaction dynamically updated and recorded in the image state management module;

[0161] In the embodiment, the core control parameters include:

[0162] Global view control parameters: such as scaling ratio, rotation angle, canvas offset;

[0163] Drawing path parameters: such as coordinate points, color values, line widths, brush types, etc. of path sequence;

[0164] Layer control parameters: such as layer visibility flag, Z sequence order, transparency, blending mode, lock state, etc.;

[0165] Style control parameters: such as brush mode, dynamic line width rule, trajectory smoothness parameter, etc.;

[0166] The above core control parameters are used as the state basis for generating each frame of image in the system, are updated in real time under the driving of interactive instructions, and are mapped into the graphic rendering process, and the layer stack structure (LayerStack) corresponding to the layer control parameters adopts a stack type organization manner to arrange the layers in Z sequence, is used to record the state of each layer to support rendering traversal and image synthesis operation, and ensures the consistency of state ordered management and graphic structure construction of the system in a multi-layer environment;

[0167] The data structure set refers to the structured organization form for carrying the core control parameters and their timing change records, specifically including:

[0168] Global view state object (used to record the scaling, rotation and translation state of the current canvas);

[0169] Path set (used to store all user drawing trajectories, each path is an object unit containing coordinates, color, style);

[0170] Layer Stack Structure (used to manage the hierarchical relationship of image content, stack and organize independent drawing layers in Z-axis order, each layer can have its own graphical elements, transformation attributes and visibility state, and support composition and superposition between layers; for example, background layer, drawing layer, special effect layer, etc., each layer may contain multiple paths, shapes or other graphical elements, the stack structure means the layers have a top-down relationship (Z sequence), which determines their rendering order and coverage relationship);

[0171] State Snapshot Stack (used to record the complete image state snapshot after each interaction operation, to realize state backtracking, undo and redo) and other structured carriers, to maintain the consistency, traceability and dynamic responsiveness of the image state during interaction;

[0172] Among them, the global view control parameters are output by the interactive control and parameter mapping unit (3) exec Real-time state change, the system compares the parameters of the control command with the current canvas state after receiving each control command, and updates the state parameters through mapping strategies (such as incremental superposition, direct coverage or buffer interpolation), to ensure that the image state has high synchronization and consistency in multi-finger interaction, complex gestures and continuous drawing process;

[0173] The graphics rendering and synthesis module renders and synthesizes the drawing content generated by the user's current interaction based on the state data provided by the image state management module, and delivers the rendering result as the final frame image to the image output module for caching and exporting;

[0174] The image output module is used to receive the final frame image generated by the graphics rendering and synthesis module, and temporarily stores the bitmap data of the current frame through the establishment of a real-time image frame buffer (FrameBuffer), to support real-time preview, image snapshot and subsequent export operations;

[0175] When the system detects that the user triggers the export behavior (such as executing the save or export command), the image output module generates and formats the final output data based on the current frame bitmap data stored in the real-time image frame buffer, combined with the drawing path set and layer structure provided by the image state management module.

[0176] Among them, when the system detects that the user executes the save or export explicit output instruction, the image output module completes the packaging and exporting of the final output image data based on the global view control parameters, layer structure and path set stored in the current image state management module, combined with the bitmap frame image output by the graphics rendering and synthesis module;

[0177] The exported content includes but is not limited to: (1) bitmap data of the current frame image (saving the image output as a commonly used image format such as PNG, JPG, etc. for image saving and external display); (2) complete path set and layer structure (serializing all layers, path tracks and their style parameters into a structured data format such as JSON, SVG, etc. for image saving and external display) so as to support subsequent backtracking editing, cross-device migration or project-level archiving management.

[0178] Specifically, the specific steps involved in the generation of the final frame image by the graphic rendering and synthesis module are as follows:

[0179] Analyzing the executable command sequence CMD output by the interactive control and parameter mapping unit exec and parsing each frame control instruction in the command sequence to complete the scheduling and change of the state objects (related layers, paths, styles, etc.) in the image state management module according to the command type (such as setScale, translateCanvas, drawPath, deletePath);

[0180] Iterating through the layer stack structure (LayerStack) in the image state management module frame by frame and executing layer synthesis rendering in order according to the Z sequence arrangement (stacking order) of the layers. In the layer iteration process, the transparency adjustment, blending mode control and editable state identification of each layer are supported, thereby realizing the construction of a composite image structure in a multi-layer superimposed scene;

[0181] In the process of executing layer rendering, the system obtains the path set from the image state management module, renders each path (based on its track coordinates, color, line width, brush type, etc.) in the path set into an image primitive, supports pressure-sensitive driven brush width change and speed-driven brush deformation effect, and introduces a Bezier curve interpolation algorithm to realize track smoothing reconstruction, and combines an anti-aliasing rendering technology (such as MSAA or FXAA) based on pixel coverage to improve the path edge quality and overall visual fluency;

[0182] In this embodiment, when the graphic rendering and synthesis module executes path rendering, a Bezier curve interpolation algorithm is used to perform track smoothing reconstruction on the original coordinate point sequence in the user drawing path, and the specific steps involved are as follows:

[0183] Taking the continuous track points input by the user as control points, a quadratic or cubic Bezier curve segment is constructed to replace the original polyline track for path drawing, thereby improving the curve smoothness on the basis of maintaining the original gesture trend and avoiding the jaggedness and discontinuity caused by sparse sampling points or gesture jitter

[0184] Meanwhile, in the graphics rasterization rendering stage, an Anti-Aliasing rendering mechanism based on pixel coverage is introduced, which reduces the jaggedness of the path boundary and improves the overall visual quality by weighted mixing of the pixel gray scale in the edge area of the image primitive (such as using the MSAA or FXAA method).

[0185] Specifically, the Anti-Aliasing rendering mechanism based on pixel coverage dynamically adjusts the gray scale value or color weight of the edge pixel in the graphics rasterization process by judging the partial coverage degree of the edge pixel, so as to realize the smooth transition of the edge area. Specifically, when the path edge falls between multiple pixel units, the system calculates the output color value of the pixel according to the area ratio (coverage) of the pixel covered by the path primitive, realizes gray scale mixing or sub-pixel level filling, and avoids the jagged discontinuous stripes of the path edge. To further improve the rendering performance and picture quality, the system selects a multi-sampling Anti-Aliasing technology or a fast approximate Anti-Aliasing technology; wherein the multi-sampling Anti-Aliasing technology generates a more detailed edge mixing effect by performing coverage statistics on multiple sampling points of each pixel, and the fast approximate Anti-Aliasing technology performs post-processing after the image rendering is completed, and quickly smoothes the jagged area based on the edge detection algorithm, which has the advantages of small performance overhead and strong compatibility.

[0186] The global view control parameters (scaling, rotation, and translation, etc.) are encapsulated as a unified affine transformation matrix, which acts on all graphic elements in the current frame, and performs coordinate transformation processing on all drawing objects, which is used for synchronous displacement and scaling control of the graphic elements under the current canvas view. On the premise that the transformation does not change, a transformation lazy refresh strategy is introduced to skip redundant calculation, and only when the state parameter changes or a new command is received, the matrix update and picture redraw process is triggered.

[0187] Based on the executable command sequence CMD of the current command frame exec Compared with the state difference of the previous frame image, the changed area is dynamically detected, the dirty region cache strategy is used to identify the minimum boundary of the difference area, and only the local image redraw is performed on the area, so as to significantly reduce the invalid graphics calculation overhead and improve the frame rate and response speed of the graphics system.

[0188] In the embodiment, the graphics rendering and synthesis module adopts a local refresh strategy based on the dirty region cache strategy to realize dynamic identification and local image redraw of the difference area between the current frame and the previous frame image state, which specifically includes the following steps:

[0189] The system receives the executable command sequence CMD from the interactive control and parameter mapping unit in each rendering cycle exec , and compares the control parameters (such as scaling, translation offset, path trajectory, layer state, etc.) in the current command with the state parameters of the last frame cached in the image state management module, extracts the image elements that have changed and their affected range;

[0190] For the changed graphic objects (such as new paths, changed path styles, layer offsets, etc.), calculate their minimum bounding rectangle region in the canvas coordinate system according to their spatial boundaries (bounding box), and add them to the dirty region set; if it is a global view transformation (such as scaling, rotation), mark the entire canvas as dirty;

[0191] Perform spatial overlap detection and merging processing on the multiple dirty regions collected in the current cycle to avoid repeated drawing, form a set of minimized and non-overlapping dirty regions as the target region set for this time of local redrawing;

[0192] When the rendering engine performs the drawing task, it only traverses and redraws the target region content in the dirty region set, keeps the unchanged region image cache in the frame buffer, thereby reducing the GPU drawing instruction and calculation overhead;

[0193] After the redrawing is completed, the newly generated image frame and the dirty region are written into the real-time image frame buffer (FrameBuffer) in the image output module, ensuring that the next rendering cycle can continue to identify differences based on the latest image state and build a continuous region refresh chain;

[0194] By introducing the dirty region caching strategy, the system can significantly reduce the full-canvas redrawing operation, reduce the GPU resource consumption, improve the rendering efficiency, and improve the system frame rate and user operation smoothness in the scene of multi-finger high-frequency interaction, complex path drawing, and frequent view transformation.

[0195] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only preferred examples of the present application and are not intended to limit the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application.

Claims

1. A real-time interactive image generation system based on multi-touch canvas, characterized in that: Comprising an input acquisition unit (1) for obtaining raw touch data from an operating system and pre-processing the raw touch data; a gesture recognition and analysis unit (2) for performing high-level semantic behavior analysis on the touch data sequence processed by the pre-processing module, automatically recognizing and structurally classifying user operation behaviors in combination with a preset gesture rule library and a supervised learning model, and generating corresponding semantic events; an interactive control and parameter mapping unit (3) for receiving semantic events output by the gesture recognition and analysis unit, analyzing the semantic events into image control commands, and generating executable command sequences through state smoothing processing and an abnormality rollback mechanism; wherein the interactive control and parameter mapping unit (3) comprises a semantic event analysis and command translation module, a state smoothing module, and an abnormality rollback module; The semantic event analysis and command translation module is configured to receive the semantic event object output by the gesture recognition analysis unit, and map different event types into corresponding image generation control commands and original control parameters, and output an initial control command set CMD raw； The state smoothing module optimizes the image generation control command based on a state buffer and an interpolation mechanism to obtain a smooth interpolated control command sequence CMD smooth ; The abnormal rollback module is configured to receive the interpolated control command sequence CMD smooth , perform legality verification on the interpolated control command sequence CMD smooth based on a parameter legality verification mechanism, and finally output an executable command sequence CMD exec ; and the state buffering and interpolation mechanism specifically comprises: establishing a buffer pool for multiple consecutive image control states, storing image state parameters of the last N frames as a dynamic reference benchmark for subsequent interpolation calculation; The initial control command set CMD raw The continuous control parameters involved in the intermediate frame generation process are generated by using a linear interpolation algorithm in the time dimension t. a linear interpolation algorithm is used to perform linear transition on scaling, rotation, and translation parameters at equal intervals along the time dimension between two consecutive state frames, generating an interpolation control parameter sequence; wherein the interpolation process satisfies the following constraints; time resolution: Δt = 33 ms; Maximum step size limit: , ; For instructions with abrupt inter-frame change, step transition is enforced and intermediate frame control parameters are inserted to prevent abrupt change from causing jarring interactive experience, resulting in a smooth interpolated control command sequence where each c i is a control frame obtained by smooth interpolation at reasonable steps an image generation unit (4) for receiving executable command sequences output by the interactive control and parameter mapping unit (3), and performing image state updating, path drawing, transformation operations, and layer output based on an internal graphics state management mechanism and a rendering engine to generate real-time dynamic response interactive image content; wherein the image generation unit (4) comprises an image state management module, a graphics rendering and synthesis module, and an image output module; The image state management module is used for storing and maintaining a data structure set of core control parameters in the image interaction process, and the core control parameters are outputted through the executable command sequence CMD of the interactive control and parameter mapping unit (3) exec Bidirectional binding makes the image state change generated by each gesture interaction to be dynamically updated and recorded in the image state management module. the graphics rendering and synthesis module performs graphics rendering and layer synthesis processing on the drawing content generated by the user's current interaction according to the state data provided by the image state management module, and delivers the rendering result as the final frame image to the image output module for caching and exporting; the image output module is used to receive the final frame image generated by the graphics rendering and synthesis module, and temporarily stores the bitmap data of the current frame through the establishment of a real-time image frame buffer area; when the system detects that the user triggers the export behavior, the image output module generates and formats the final output data based on the current frame bitmap data stored in the real-time image frame buffer area, in combination with the drawing path set and layer structure provided by the image state management module; and the specific steps involved in the graphics rendering and synthesis module generating the final frame image are: parsing the executable command sequence CMD output by the interactive control and parameter mapping unit exec and parsing each frame control instruction in the command sequence, and completing scheduling and changing of the state object in the image state management module according to the command type; iterating through the layer stack structure in the image state management module frame by frame, and sequentially performing layer synthesis rendering according to the Z-order arrangement sequence of the layers to construct a composite image structure in a multi-layer superimposed scene; In the execution of the layer rendering process, the system obtains a path set from the image state management module, renders each path in the path set into an image primitive, introduces a Bezier curve interpolation algorithm to realize trajectory smooth reconstruction, and combines an anti-aliasing rendering technology based on pixel coverage to improve path edge quality and overall visual fluency. The global view control parameters are encapsulated into a unified affine transformation matrix, which is applied to all graphic elements in the current frame, and all drawing objects are subjected to coordinate transformation processing, which is used for synchronous displacement and scaling control of the graphic elements under the current canvas view, and on the premise that the transformation does not change, a transformation lazy refresh strategy is introduced to skip redundant calculation, and matrix updating and picture redrawing process is triggered only when the state parameters change. Executable command sequence CMD based on the current command frame exec In comparison with the difference of the previous frame image state, the changed region is dynamically detected, the dirty region cache strategy is adopted to identify the minimum boundary of the difference region, and only the local image redraw is performed on the region. 2.The multi-touch canvas based real-time interactive image generation system of claim 1, wherein: The input acquisition unit (1) comprises an input module and a preprocessing module. The input module is used for acquiring original touch data, and the original touch data comprises touch coordinates, quantity, touch ID, pressure, size, direction and time sequence. The preprocessing module pre-processes the original touch data based on an adaptive Kalman filtering algorithm and a trajectory prediction auxiliary mechanism.

3. The multi-touch canvas based real-time interactive image generation system of claim 2, wherein: The specific steps involved in the preprocessing of the original touch data are as follows: The adaptive Kalman filtering algorithm is used to dynamically smooth the touch coordinates: For each touch point, a state vector is constructed on the time sequence, which contains the two-dimensional position and velocity information of the touch point, and the velocity estimation value between consecutive frames is obtained through time sequence buffering; The process noise covariance matrix of the filter is dynamically adjusted according to the velocity, which is used to realize adaptive adjustment of the filtering strength; On the basis of the adaptive Kalman filtering, a trajectory prediction auxiliary mechanism is introduced to correct the consistency of the touch ID, and the specific steps are as follows: By analyzing the historical moving trajectory and current velocity of each touch point, the expected touch point position of the next frame is predicted based on a linear dynamic model; For all newly identified touch points in the current frame, the Euclidean distance and direction angle between them and the predicted positions are calculated; If the spatial distance between the touch point and the predicted point is less than a preset threshold and the motion direction of a new touch point in the current frame and the corresponding predicted touch point in the previous frame remains consistent, the system determines that it is the continuation of the same physical touch point, thereby inheriting the original touch point ID; In the multi-point matching scene, an optimal matching strategy based on the Hungarian algorithm is used to realize the optimal solution of global touch ID allocation.

4. The multi-touch canvas based real-time interactive image generation system of claim 1, wherein: The gesture recognition analysis unit (2) comprises a touch event sequence generation module, a behavior pattern recognition module and a high-level semantic event module; The touch event sequence generation module extracts the life cycle events of each touch point based on the structured touch data output by the preprocessing module, constructs a time sequence sliding window based on a time window buffering mechanism, generates a corresponding touch set state snapshot for each time, and the corresponding touch set state snapshot comprises quantity, position, velocity and trajectory direction; The behavior pattern recognition module is used to identify the typical gesture operation behaviors contained in the user operation; And each identified gesture operation is encapsulated into a structured semantic event object through the high-level semantic event module.

5. The multi-touch canvas based real-time interactive image generation system of claim 4, wherein: The specific steps involved in the behavior pattern recognition module recognizing the behavior of the typical gesture contained in the user operation are as follows: Within the time sliding window, for each touch point and its trajectory, spatial features, time features and behavior common features are extracted respectively, and the above features constitute an input vector sequence; A gesture recognition rule library including a set of space / time constraint rules for typical touch gestures is constructed; A gesture classification model based on supervised learning is deployed in the gesture recognition rule library in parallel, for identifying rule ambiguity and behavior judgment of unrecorded new behaviors; The gesture classification model based on supervised learning adopts a lightweight sequence classification network, the input of which is the extracted input vector sequence, and the output is a gesture category probability distribution, which is used to determine which gesture category the current operation is more likely to belong to; When only a single rule in the gesture recognition rule library is matched, and the unique corresponding gesture category is determined, the rule determination result is directly adopted; When there are multiple possible options in the rule matching result, the gesture classification model is called for auxiliary judgment, and the maximum probability is adopted; In the output stage of the behavior pattern recognition module, a time series stability factor is introduced for output smoothing, and the category is output as the final recognition result, which is used to suppress the behavior misjudgment caused by single-frame recognition error.

Images