Multi-platform fusion interaction method and system based on large touch screen

Abstract

The present invention provides a multi-platform fusion interaction method and system based on a large touch screen, which relates to the field of human-computer interaction technology, including dividing the main control and collaborative display areas, establishing a mapping relationship between touch operations and multi-platform interaction scenes, and using neural networks to recognize gesture intentions and generate touch event data. After receiving the touch event data, platform control instructions are generated according to the data dependency graph and the interaction rule library, the main control platform instructions are processed first, and the execution results are synchronized to the collaborative platform or vice versa. Finally, a real-time data synchronization mechanism is established between platforms, data is transmitted through a two-way data channel, and the transmission strategy is dynamically adjusted to ensure data consistency on each platform, thereby achieving efficient multi-platform fusion interaction, improving user experience, and reducing system latency.

Description

Technical Field

[0001] The present invention relates to human-computer interaction technology, and in particular to a multi-platform fusion interaction method and system based on a large touch screen. Background Art

[0002] At present, with the rapid development of multi-platform integration technology, users' demand for interaction between multiple devices and multiple systems is increasing. However, due to differences in hardware performance, software architecture and data format between different platforms, the implementation of cross-platform collaboration is complex and inefficient. Especially in scenarios involving large touch screens, there is still a technical bottleneck in how to achieve seamless collaboration of multiple platforms through natural touch interaction. For example, when users operate a large touch screen, they need to manually distribute instructions to each platform, making it difficult to achieve an intelligent and real-time interactive experience.

[0003] Traditional multi-platform interaction methods usually rely on fixed predefined rules, lack a deep understanding of users' complex touch operations, and are difficult to accurately identify operation intentions and cannot meet diverse interaction needs. In addition, the existing technology has limited ability to dynamically adjust data synchronization and transmission quality during inter-platform data transmission, and often has problems such as delays, data loss or inconsistency, which reduces the user experience and the overall reliability of the system.

[0004] In view of the above problems, a multi-platform fusion interaction method based on a large touch screen is urgently needed. Through the intelligent mapping of touch operations and multi-platform interaction scenarios, combined with the gesture intention recognition capability of the neural network model and the real-time data synchronization mechanism, the collaboration efficiency between platforms can be optimized. At the same time, by dynamically adjusting the data transmission strategy, the stability and consistency of data transmission can be guaranteed, providing users with a smooth and efficient cross-platform interaction experience. Summary of the invention

[0005] The embodiments of the present invention provide a multi-platform fusion interaction method and system based on a large touch screen, which can solve the problems in the prior art.

[0006] According to a first aspect of the embodiments of the present invention,

[0007] A multi-platform fusion interaction method based on a large touch screen is provided, comprising:

[0008] Obtaining display resolution parameters and processor performance parameters of the large touch screen, calculating an optimal display area division scheme based on the display resolution parameters, dividing the large touch screen into a main control display area and multiple collaborative display areas, establishing a mapping relationship library between touch operations and multi-platform interactive scenarios, collecting user touch operation information in each display area, inputting the touch operation information into a neural network model for gesture intention recognition, matching the recognition result with the mapping relationship library, and generating touch event data including touch position information, operation intention identification information, and interactive scenario type;

[0009] Receive the touch event data, determine the target display area according to the touch position information, establish a data dependency graph between platforms based on the interaction scenario type, match the corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and convert the touch event data into a plurality of related platform control instructions according to the data dependency graph, wherein when the target display area is the main control display area, give priority to processing the main control platform instructions and trigger the collaborative platform instructions according to the execution results, and when the target display area is the collaborative display area, process the corresponding collaborative platform instructions and synchronize the execution results to the main control platform, and generate interaction status information including the execution status of each platform and the data update identifier;

[0010] Based on the interactive status information, a real-time data synchronization mechanism is established between platforms, the interface parameters of the source platform and the target platform are obtained, a data format conversion strategy is generated and a two-way data channel is established, and the output data of the source platform is converted into a data format receivable by the target platform through the two-way data channel and then transmitted to the target platform. During the data transmission process, transmission quality parameters and data synchronization status are collected in real time, and the data transmission strategy is dynamically adjusted according to the transmission quality parameters and data synchronization status. The converted data is presented in the target display area, and data update messages are broadcast to all related platforms at the same time to ensure the consistency of data on each platform.

[0011] In an optional embodiment,

[0012] Calculating the optimal display area division scheme based on the display resolution parameter to divide the touch screen into a main control display area and a plurality of collaborative display areas includes:

[0013] Obtaining the physical resolution and display ratio parameters of the large touch screen, collecting the user's gaze point distribution data and the touch operation thermal distribution data, weightedly fusing the gaze point distribution data and the touch operation thermal distribution data to generate an interaction density distribution map, calculating the weight distribution coefficient of the display area based on the interaction density distribution map, setting the area threshold in combination with the visual perception characteristics of the human eye, and generating the initial display area division constraint conditions;

[0014] The interaction density distribution map and the initial display area division constraint conditions are input into an adaptive display area optimization network, a multi-objective loss function is constructed with visual attention distribution entropy, operation space continuity index and area switching cost as optimization targets, a deep reinforcement learning algorithm is used to train the adaptive display area optimization network, a reward value is calculated based on real-time user interaction feedback during the training process, and a weight coefficient of the multi-objective loss function is updated, the trained adaptive display area optimization network is applied to display area division optimization, and an optimal division scheme including area proportions and relative position relationships is output;

[0015] Based on the optimal division scheme, the position with the highest display area weight distribution coefficient is determined as the visual center point and a radial coordinate system is established. The boundary outline of the main control display area is calculated in the radial coordinate system. The layout position of the collaborative display area is determined according to the interaction density distribution map. The data correlation between the display areas is extracted and the width of the area interval band is adaptively adjusted. A division control instruction containing boundary description information and area scaling parameters is generated. The area division control instruction is sent to the display rendering module to execute display area division.

[0016] In an optional embodiment,

[0017] Based on the optimal division scheme, the position with the highest weight distribution coefficient of the display area is determined as the visual center point and a radial coordinate system is established, the boundary contour of the main control display area is calculated in the radial coordinate system, the layout position of the collaborative display area is determined according to the interactive density distribution map, the data association between the display areas is extracted and the width of the area interval band is adaptively adjusted, and the division control instruction containing the boundary description information and the area scaling parameter is generated, including:

[0018] Density clustering is performed on the display area weight distribution coefficients in the optimal division scheme, wherein the bandwidth parameter of the clustering is adaptively adjusted according to the display area size, the spatial coordinates of the point with the highest weight density are determined as the visual center point, a radial coordinate system is established with the visual center point as the origin, and a visual attention distribution model is constructed according to the user's line of sight movement trajectory;

[0019] In the radial coordinate system, the visual attention distribution model and the interactive density distribution map are weightedly fused to generate a display area density field, the reference radius of the parameter equation is determined based on the gradient distribution of the display area density field, and the fluctuation amplitude is calculated according to the local fluctuation characteristics of the display area density field. The period parameter is set based on the spatial distribution law of the display content, and the reference radius, fluctuation amplitude and period parameter are substituted into the parameter equation to generate an initial boundary contour, and the attention distribution entropy on the initial boundary contour is calculated. The initial boundary contour is iteratively optimized with minimizing the attention distribution entropy as the optimization goal, and the optimized boundary contour is subjected to curvature analysis and high curvature points are smoothed to generate the final contour of the main control display area;

[0020] Extract the spatial moment features of the interaction density distribution map, calculate the centroid moment feature vector, perform polar coordinate space mapping based on the visual attention distribution model, obtain the initial layout position of the collaborative display area, construct a correlation calculation model that integrates the interaction frequencies of multiple types of data to generate a data correlation matrix, and calculate and dynamically update the regional interval band width according to the data correlation matrix;

[0021] The final outline of the main control display area and the boundary of the collaborative display area are fitted with a Bezier curve to generate boundary description information, the area scaling parameters and the visual transition parameters are determined based on the data association matrix, and a division control instruction containing the boundary description information and the area scaling parameters is generated.

[0022] Establishing a mapping relationship library between touch operations and multi-platform interactive scenarios, collecting the user's touch operation information in each display area, inputting the touch operation information into a neural network model for gesture intention recognition, matching the recognition result with the mapping relationship library, and generating touch event data including touch position information, operation intention identification information, and interactive scenario type, including:

[0023] Build a mapping relationship library between touch operations and multi-platform interaction scenarios, extract the number of touch points, pressure value distribution and movement trajectory curvature to obtain gesture feature vectors, calculate the correlation between the gesture feature vectors and the operation intention identifier and scene type descriptor to generate mapping templates, and assign adaptive priority weights to each mapping template based on historical interaction frequency;

[0024] The coordinate sequence, pressure value gradient, velocity vector and acceleration vector of the touch point are collected to generate time series features, the Euclidean distance, angle change rate and relative velocity direction between the touch points are calculated to obtain a topological relationship feature matrix, and the time series features and the topological relationship feature matrix are spatially and temporally aligned and then combined to form multimodal touch feature data;

[0025] The multimodal touch feature data is input into a long short-term memory neural network, the temporal feature encoder of the long short-term memory neural network uses a bidirectional long short-term memory layer with skip connections to perform multi-scale encoding on the temporal features to obtain temporal coding features, the spatial feature fuser adaptively extracts the hierarchical representation of the topological relationship feature matrix through a dynamic graph convolutional network to obtain spatial fusion features, and the dual attention mechanism based on channel attention and spatial attention performs adaptive weight allocation on the temporal coding features and the spatial fusion features, and fuses and outputs the gesture intention recognition result;

[0026] Fuzzy reasoning is used to calculate the multi-dimensional similarity between the gesture intention recognition result and the mapping template, and the best matching mapping template is selected in combination with the adaptive priority weight. The touch position information in the multimodal touch feature data, the gesture intention recognition result and the scene type descriptor in the best matching mapping template are packaged to generate touch event data.

[0027] In an optional embodiment,

[0028] Receiving the touch event data, determining the target display area according to the touch position information, establishing a data dependency graph between platforms based on the interaction scenario type, matching a corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and converting the touch event data into a plurality of associated platform control instructions according to the data dependency graph includes:

[0029] receiving touch event data, performing coordinate transformation and normalization processing on touch position information in the touch event data to generate normalized touch coordinate data, determining a ray projection direction based on the normalized touch coordinate data, projecting detection rays in horizontal and vertical directions, and obtaining a set of intersection points between the detection rays and a display area boundary matrix;

[0030] Counting the number of intersections in the intersection set, determining the region attribution relationship of the normalized touch coordinate data according to an odd-even intersection determination rule, and determining a target display region, wherein the target display region includes a region identifier and a platform type identifier;

[0031] Extracting interaction scene information from the touch event data, associating and mapping the platform type identifier with the interaction scene information, constructing an initial set of platform nodes, calculating the data interaction frequency, data flow direction, and data structure characteristics between nodes in the initial set of platform nodes, generating a quantified value of data coupling between nodes, calculating a quantified value of functional coupling between nodes based on the calling relationship, interface mapping relationship, and state change requirements between nodes in the initial set of platform nodes, fusing the data coupling quantified value and the functional coupling quantified value according to a preset fusion weight, constructing a directed connection relationship between platform nodes, and outputting a data dependency graph;

[0032] Extracting operation intention information from the touch event data, performing semantic feature extraction on the operation intention information to generate an intention feature vector, retrieving a candidate instruction template set in a preset interactive rule library, calculating the semantic similarity between the intention feature vector and each template in the candidate instruction template set, and selecting the best matching instruction template according to the magnitude of the semantic similarity;

[0033] The optimal matching instruction template is decomposed into an atomic operation sequence, a priority identifier and an operation dependency identifier are added to each atomic operation in the atomic operation sequence, and an atomic operation sequence with identifiers is generated. Based on the topological structure of the data dependency graph, the processing capability parameters and load status parameters of each platform node are obtained, and the platform execution weight coefficient is calculated. According to the platform execution weight coefficient and the atomic operation sequence with identifiers, control instructions for different platforms are generated.

[0034] In an optional embodiment,

[0035] Establishing a real-time data synchronization mechanism between platforms based on the interaction status information, acquiring interface parameters of the source platform and the target platform, generating a data format conversion strategy and establishing a bidirectional data channel, and converting the output data of the source platform into a data format receivable by the target platform through the bidirectional data channel and then transmitting it to the target platform includes:

[0036] Acquire interaction state information, extract interface parameters of the source platform and the target platform from the interaction state information, the interface parameters including data structure information, data type information and data format information; extract data conversion features from the interface parameters, generate a feature weight matrix, perform feature weighting processing on the output data of the source platform according to the feature weight matrix, and obtain weighted feature data;

[0037] Performing format analysis on the output data according to the weighted feature data, performing data format mapping based on a pre-trained neural network model, and generating a data format conversion strategy, wherein the data format conversion strategy includes a target data structure, a target data type, and a target data format;

[0038] Establishing a bidirectional data channel between the source platform and the target platform based on the interaction status information, wherein the bidirectional data channel includes an uplink data channel and a downlink data channel;

[0039] Monitor the transmission status and system resource status of a bidirectional data channel, calculate a channel evaluation value according to the transmission status and the system resource status, wherein the channel evaluation value includes a transmission delay value and a resource consumption value; select an optimal data channel based on the channel evaluation value, and generate a channel configuration instruction, wherein the channel configuration instruction is used to configure data transmission parameters;

[0040] A plurality of synchronization checkpoints are set in the optimal data channel, a data version identifier is generated, and a data synchronization status is recorded at each synchronization checkpoint; the output data is format converted according to the data format conversion strategy to generate converted data; and the converted data is transmitted to a target platform through the optimal data channel based on the channel configuration instruction.

[0041] In an optional embodiment,

[0042] Collecting transmission quality parameters and data synchronization status in real time during data transmission, and dynamically adjusting the data transmission strategy according to the transmission quality parameters and data synchronization status include:

[0043] A multi-level monitoring node is set on the transmission path of the bidirectional data channel, and the multi-level monitoring node collects the transmission status information of the data packet, generates a monitoring data stream, calculates the network layer transmission index, the application layer processing index and the business layer synchronization index based on the monitoring data stream, and generates a real-time transmission status and data synchronization status;

[0044] The real-time transmission state and the data synchronization state are combined into a feature vector, the feature vector is input into a pre-trained deep reinforcement learning model, transmission state features are extracted, a transmission control strategy is generated based on the transmission state features, and transmission parameters are adjusted according to the transmission control strategy to form a transmission control instruction;

[0045] Based on the historical monitoring data collected by the multi-level monitoring nodes, a long short-term memory network is trained to obtain a transmission quality prediction model, the transmission quality parameters and data synchronization status within a specified time window are input into the transmission quality prediction model, the transmission quality change trend and abnormal warning information are generated, and the transmission control strategy is optimized according to the transmission quality change trend and abnormal warning information;

[0046] When the data synchronization state indicates a transmission abnormality, extract the abnormal feature pattern and match the abnormal type, execute the fault recovery strategy according to the abnormal type, generate recovery execution data and calculate the recovery effect index of the recovery execution data;

[0047] The recovery effect index is input into the deep reinforcement learning model to update the network parameters; the transmission control strategy is regenerated based on the updated network parameters to achieve closed-loop optimization of transmission quality.

[0048] A second aspect of an embodiment of the present invention provides a multi-platform fusion interaction system based on a large touch screen, comprising:

[0049] The first unit is used to obtain display resolution parameters and processor performance parameters of the large touch screen, calculate the optimal display area division scheme based on the display resolution parameters, divide the large touch screen into a main control display area and multiple collaborative display areas, establish a mapping relationship library between touch operations and multi-platform interactive scenarios, collect user touch operation information in each display area, input the touch operation information into a neural network model for gesture intention recognition, match the recognition result with the mapping relationship library, and generate touch event data including touch position information, operation intention identification information and interactive scenario type;

[0050] A second unit is used to receive the touch event data, determine the target display area according to the touch position information, establish a data dependency graph between platforms based on the interaction scenario type, match the corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and convert the touch event data into a plurality of related platform control instructions according to the data dependency graph, wherein when the target display area is the main control display area, the main control platform instructions are processed first and the collaborative platform instructions are triggered according to the execution results; when the target display area is the collaborative display area, the corresponding collaborative platform instructions are processed and the execution results are synchronized to the main control platform, and the interactive status information including the execution status of each platform and the data update identifier is generated;

[0051] The third unit is used to establish a real-time data synchronization mechanism between platforms based on the interaction status information, obtain interface parameters of the source platform and the target platform, generate a data format conversion strategy and establish a two-way data channel, convert the output data of the source platform into a data format receivable by the target platform through the two-way data channel, and then transmit it to the target platform, collect transmission quality parameters and data synchronization status in real time during data transmission, dynamically adjust the data transmission strategy according to the transmission quality parameters and data synchronization status, present the converted data in the target display area, and broadcast data update messages to all related platforms at the same time to ensure the consistency of data on each platform.

[0052] A third aspect of the embodiments of the present invention

[0053] An electronic device is provided, comprising:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] A fourth aspect of the embodiments of the present invention is:

[0058] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0059] In this embodiment, by dividing the large touch screen into multiple areas and establishing a mapping relationship library between touch operations and multi-platform interaction scenarios, parallel control and data interaction of multiple platforms are achieved, which reduces the user operation steps and improves the interaction efficiency. Based on the display resolution and processor performance parameters, display area division and platform instruction scheduling can dynamically adjust resource allocation according to device performance, make full use of hardware resources, avoid resource waste, and improve the overall performance of the system. A real-time data synchronization mechanism is established between platforms, and through a two-way data channel and data format conversion strategy, data consistency between different platforms is ensured, data conflicts and errors are avoided, and the reliability of multi-platform collaborative work is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flowchart of a multi-platform fusion interaction method based on a large touch screen according to an embodiment of the present invention;

[0061] Figure 2 It is a schematic diagram of the structure of a multi-platform integrated interactive system based on a large touch screen according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0064] Figure 1 FIG. 1 is a flow chart of a multi-platform fusion interaction method based on a touch screen according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0065] S101. Obtain display resolution parameters and processor performance parameters of the large touch screen, calculate the optimal display area division scheme based on the display resolution parameters, divide the large touch screen into a main control display area and multiple collaborative display areas, establish a mapping relationship library between touch operations and multi-platform interactive scenarios, collect user touch operation information in each display area, input the touch operation information into a neural network model for gesture intention recognition, match the recognition result with the mapping relationship library, and generate touch event data including touch position information, operation intention identification information, and interactive scenario type;

[0066] S102. Receive the touch event data, determine the target display area according to the touch position information, establish a data dependency graph between platforms based on the interaction scenario type, match the corresponding operation instruction template in the preset interaction rule library according to the operation intention identification information, and convert the touch event data into multiple related platform control instructions according to the data dependency graph, wherein when the target display area is the main control display area, give priority to the main control platform instructions and trigger the collaborative platform instructions according to the execution results; when the target display area is the collaborative display area, process the corresponding collaborative platform instructions and synchronize the execution results to the main control platform, and generate interactive status information including the execution status of each platform and the data update identifier;

[0067] S103. A real-time data synchronization mechanism is established between platforms based on the interaction status information, interface parameters of the source platform and the target platform are obtained, a data format conversion strategy is generated and a bidirectional data channel is established, output data of the source platform is converted into a data format receivable by the target platform through the bidirectional data channel and then transmitted to the target platform, transmission quality parameters and data synchronization status are collected in real time during the data transmission process, the data transmission strategy is dynamically adjusted according to the transmission quality parameters and data synchronization status, the converted data is presented in the target display area, and data update messages are broadcast to all related platforms at the same time to ensure the consistency of data on each platform.

[0068] In an optional implementation, calculating an optimal display area division scheme based on the display resolution parameter, dividing the large touch screen into a main control display area and a plurality of collaborative display areas comprises:

[0069] Obtaining the physical resolution and display ratio parameters of the large touch screen, collecting the user's gaze point distribution data and the touch operation thermal distribution data, weightedly fusing the gaze point distribution data and the touch operation thermal distribution data to generate an interaction density distribution map, calculating the weight distribution coefficient of the display area based on the interaction density distribution map, setting the area threshold in combination with the visual perception characteristics of the human eye, and generating the initial display area division constraint conditions;

[0070] The interaction density distribution map and the initial display area division constraint conditions are input into an adaptive display area optimization network, a multi-objective loss function is constructed with visual attention distribution entropy, operation space continuity index and area switching cost as optimization targets, a deep reinforcement learning algorithm is used to train the adaptive display area optimization network, a reward value is calculated based on real-time user interaction feedback during the training process, and a weight coefficient of the multi-objective loss function is updated, the trained adaptive display area optimization network is applied to display area division optimization, and an optimal division scheme including area proportions and relative position relationships is output;

[0071] Based on the optimal division scheme, the position with the highest display area weight distribution coefficient is determined as the visual center point and a radial coordinate system is established. The boundary outline of the main control display area is calculated in the radial coordinate system. The layout position of the collaborative display area is determined according to the interaction density distribution map. The data correlation between the display areas is extracted and the width of the area interval band is adaptively adjusted. A division control instruction containing boundary description information and area scaling parameters is generated. The area division control instruction is sent to the display rendering module to execute display area division.

[0072] Exemplarily, first obtain the physical resolution and display ratio parameters of the large touch screen, and collect the user's gaze point distribution data and touch operation thermal distribution data. Assume a 4K touch screen with a resolution of 3840x2160 pixels and a display ratio of 16:9. Use an eye tracking device to collect the user's gaze points on the screen. For example, if the user stares at the center of the screen for a long time, the gaze point data of the center point will be higher. At the same time, the user's touch operation thermal distribution data is collected through a touch device. For example, if the user frequently touches a button in the lower left corner of the screen, the thermal value of this area will be higher.

[0073] Perform weighted fusion of gaze point distribution data and touch operation thermal distribution data to generate an interaction density distribution map. Perform weighted fusion of the collected gaze point data and touch operation thermal distribution data. For example, give the gaze point data a 70% weight and the touch operation thermal distribution data a 30% weight, and superimpose the two data to generate an interaction density distribution map. On the interaction density distribution map, the higher the value, the more frequent the user interacts with the area.

[0074] The weight distribution coefficient of the display area is calculated based on the interaction density distribution map. According to the interaction density distribution map, the weight distribution coefficient of each area is calculated. For example, areas with high interaction density are assigned higher weight coefficients. Assuming that the interaction density in the center of the screen is the highest, its weight distribution coefficient is 0.8, and the weight coefficients of other areas decrease in turn.

[0075] The area threshold is set in combination with the visual perception characteristics of the human eye to generate the initial display area division constraint. Considering the visual perception characteristics of the human eye, the area threshold is set. Among them, the minimum area of ​​the area cannot be less than 5% of the total screen area, and the maximum area of ​​the area cannot exceed 60% of the total screen area. At the same time, in order to avoid the area being too fragmented, the upper limit of the number of areas is set to 8. These conditions together constitute the initial display area division constraint.

[0076] The generated interaction density distribution map and initial display area division constraints are input into an adaptive display area optimization network. The generated interaction density distribution map and initial display area division constraints are input into a pre-trained adaptive display area optimization network. The network is trained using a deep reinforcement learning algorithm.

[0077] A multi-objective loss function is constructed with the visual attention distribution entropy, the operation space continuity index and the region switching cost as optimization targets. The optimization target of this network is to maximize the visual attention distribution entropy, maximize the operation space continuity index, and minimize the region switching cost. These three targets together constitute the multi-objective loss function.

[0078] The deep reinforcement learning algorithm is used to train the adaptive display area optimization network. During the training process, the interaction between the user and the screen is simulated, and the reward value is calculated based on the user's real-time interaction feedback, and the weight coefficient of the multi-objective loss function is updated based on this. For example, if the partitioning scheme causes the user to frequently switch between different areas, the weight of the area switching cost in the loss function is increased.

[0079] During the training process, the reward value is calculated based on the user's real-time interactive feedback and the weight coefficient of the multi-objective loss function is updated. Through continuous iterative training, a network model is finally obtained that can output the optimal partitioning solution based on the interaction density distribution map and constraint conditions.

[0080] The trained adaptive display area optimization network is applied to the display area division optimization, and the optimal division scheme including the area ratio and relative position relationship is output. The trained network is applied to the actual display area division optimization, and the optimal division scheme is output. For example, the network outputs that the main control display area occupies 60% of the area in the center of the screen, and the other collaborative display areas are located in the four corners of the screen, each occupying 10% of the area.

[0081] Based on the optimal partitioning scheme, the position with the highest weight distribution coefficient of the display area is determined as the visual center point and a radial coordinate system is established. Based on the output optimal partitioning scheme, the position with the highest weight distribution coefficient is determined as the visual center point and a radial coordinate system is established based on it.

[0082] The boundary contour of the main control display area is calculated in the radial coordinate system. In the radial coordinate system, the boundary contour of the main control display area is calculated. For example, the boundary of the main control display area is a circle with the visual center point as the center.

[0083] Determine the layout position of the collaborative display area according to the interactive density distribution map. Determine the layout position of the collaborative display area according to the interactive density distribution map. For example, place the area with the second highest interactive density around the main control display area.

[0084] Extract the data correlation between the displayed areas and adjust the area interval width adaptively. Extract the data correlation between the displayed areas and adjust the area interval width adaptively. For example, if the data correlation between two areas is high, reduce the interval width between them, otherwise increase it.

[0085] Generate a division control instruction including boundary description information and area scaling parameters. Generate a division control instruction including boundary description information and area scaling parameters. For example, the instruction includes circular boundary description information of the main control display area, and rectangular boundary description information and scaling parameters of each collaborative display area.

[0086] The area division control instruction is sent to the display rendering module to perform the display area division. The generated division control instruction is sent to the display rendering module to perform the display area division, and finally the dynamic division of the touch screen is realized.

[0087] In this embodiment, by dynamically adjusting the layout and size of the display area according to the user's interaction habits, the user's operation burden is reduced and the user's interaction experience is improved. By displaying important information in the area that the user is concerned about, the number of times the user switches between different areas is reduced, and the display efficiency is improved. According to the interaction habits of different users, personalized display area division can be achieved to meet the needs of different users.

[0088] In an optional implementation, based on the optimal division scheme, the position with the highest weight distribution coefficient of the display area is determined as the visual center point and a radial coordinate system is established, the boundary contour of the main control display area is calculated in the radial coordinate system, the layout position of the collaborative display area is determined according to the interactive density distribution map, the data correlation between the display areas is extracted and the width of the area interval band is adaptively adjusted, and the division control instruction containing the boundary description information and the area scaling parameter is generated, including:

[0089] Density clustering is performed on the display area weight distribution coefficients in the optimal division scheme, wherein the bandwidth parameter of the clustering is adaptively adjusted according to the display area size, the spatial coordinates of the point with the highest weight density are determined as the visual center point, a radial coordinate system is established with the visual center point as the origin, and a visual attention distribution model is constructed according to the user's line of sight movement trajectory;

[0090] In the radial coordinate system, the visual attention distribution model and the interactive density distribution map are weightedly fused to generate a display area density field, the reference radius of the parameter equation is determined based on the gradient distribution of the display area density field, and the fluctuation amplitude is calculated according to the local fluctuation characteristics of the display area density field. The period parameter is set based on the spatial distribution law of the display content, and the reference radius, fluctuation amplitude and period parameter are substituted into the parameter equation to generate an initial boundary contour, and the attention distribution entropy on the initial boundary contour is calculated. The initial boundary contour is iteratively optimized with minimizing the attention distribution entropy as the optimization goal, and the optimized boundary contour is subjected to curvature analysis and high curvature points are smoothed to generate the final contour of the main control display area;

[0091] Extract the spatial moment features of the interaction density distribution map, calculate the centroid moment feature vector, perform polar coordinate space mapping based on the visual attention distribution model, obtain the initial layout position of the collaborative display area, construct a correlation calculation model that integrates the interaction frequencies of multiple types of data to generate a data correlation matrix, and calculate and dynamically update the regional interval band width according to the data correlation matrix;

[0092] The final outline of the main control display area and the boundary of the collaborative display area are fitted with a Bezier curve to generate boundary description information, the area scaling parameters and the visual transition parameters are determined based on the data association matrix, and a division control instruction containing the boundary description information and the area scaling parameters is generated.

[0093] Exemplarily, first, the weight distribution coefficients of the display area are density clustered. The bandwidth parameter of the cluster is adaptively adjusted according to the size of the display area. A larger display area uses a larger bandwidth parameter, and a smaller display area uses a smaller bandwidth parameter to ensure the accuracy of the clustering results. Find the highest point of weight density, and determine its spatial coordinates as the visual center point. Assuming that the weight distribution coefficients of the four display areas are 0.4, 0.3, 0.2, and 0.1, respectively, after density clustering, the coordinates of the highest point of weight density are determined to be (100, 200). A radial coordinate system is established with the visual center point as the origin. At the same time, according to the movement trajectory of the user's line of sight, the movement trajectory data of the user's line of sight on the screen can be recorded by an eye tracking device to construct a visual attention distribution model. This model can represent the degree of user attention in different screen areas.

[0094] Next, in the radial coordinate system, the visual attention distribution model and the interaction density distribution map are weighted fused to generate the display area density field. Assuming that the weight of a certain area in the visual attention distribution model is 0.8 and the weight of the same area in the interaction density distribution map is 0.6, the weight of the area in the fused density field is the weighted average of the two. The reference radius of the parameter equation is determined based on the gradient distribution of the display area density field. The area with a larger gradient indicates that the density changes violently, and the reference radius is correspondingly larger. At the same time, the fluctuation amplitude is calculated according to the local fluctuation characteristics of the display area density field, and the fluctuation amplitude reflects the degree of fluctuation of the density field. The fluctuation amplitude is larger in the area where the density field fluctuates violently. The period parameter is set according to the spatial distribution law of the display content. If the display content is distributed in a grid shape, the period parameter can be set to the width or height of the grid. Substitute the reference radius, fluctuation amplitude and period parameter into the parameter equation to generate the initial boundary contour. Calculate the attention distribution entropy on the initial boundary contour, and iteratively optimize the initial boundary contour with minimizing the attention distribution entropy as the optimization goal, for example, iteratively optimize using the gradient descent method. Perform curvature analysis on the optimized boundary contour and smooth the high curvature points to generate the final contour of the main control display area.

[0095] Then, the spatial moment features of the interaction density distribution map are extracted, such as calculating the second-order and third-order moments of the image and the centroid moment feature vector. Polar coordinate space mapping is performed based on the visual attention distribution model to obtain the initial layout position of the collaborative display area. A correlation calculation model that integrates the interaction frequencies of multiple types of data is constructed, and the frequency of user clicks, drags, zooms, and other operations is considered to generate a data correlation matrix. If the data interaction frequency between two display areas is very high, their corresponding values ​​in the data correlation matrix are large. The area interval band width is calculated and dynamically updated according to the data correlation matrix. The interval band width between areas with high data correlation is narrower, and the interval band width between areas with low data correlation is wider.

[0096] Finally, the Bezier curve is used to fit the final outline of the main display area and the boundary of the collaborative display area to generate boundary description information. For example, a cubic Bezier curve is used to fit the outline and the coordinates of the control points are recorded. The regional scaling parameters and visual transition parameters are determined based on the data association matrix. Smooth transition animations can be used between areas with high data association. A partitioning control instruction containing boundary description information and regional scaling parameters is generated, and the boundary description information and regional scaling parameters are encoded into an instruction string and sent to the display control system.

[0097] In this embodiment, by adaptively adjusting the layout and size of the display area, the displayed content is more in line with the user's visual habits and focus, reducing the user's visual search and cognitive burden, thereby improving the user experience. By intelligently dividing the display area, the multi-screen display space can be effectively utilized, information redundancy and overlap can be avoided, the display efficiency of information can be improved, and users can obtain the required information more quickly. By dynamically adjusting the width of the area interval band and the visual transition parameters, the transition between different display areas can be made more natural and smooth, enhancing the naturalness and comfort of user interaction.

[0098] In an optional implementation, a mapping relationship library between touch operations and multi-platform interaction scenarios is established, the user's touch operation information in each display area is collected, the touch operation information is input into a neural network model for gesture intention recognition, the recognition result is matched with the mapping relationship library, and touch event data including touch position information, operation intention identification information and interaction scenario type is generated, including:

[0099] Build a mapping relationship library between touch operations and multi-platform interaction scenarios, extract the number of touch points, pressure value distribution and movement trajectory curvature to obtain gesture feature vectors, calculate the correlation between the gesture feature vectors and the operation intention identifier and scene type descriptor to generate mapping templates, and assign adaptive priority weights to each mapping template based on historical interaction frequency;

[0100] The coordinate sequence, pressure value gradient, velocity vector and acceleration vector of the touch point are collected to generate time series features, the Euclidean distance, angle change rate and relative velocity direction between the touch points are calculated to obtain a topological relationship feature matrix, and the time series features and the topological relationship feature matrix are spatially and temporally aligned and then combined to form multimodal touch feature data;

[0101] The multimodal touch feature data is input into a long short-term memory neural network, the temporal feature encoder of the long short-term memory neural network uses a bidirectional long short-term memory layer with skip connections to perform multi-scale encoding on the temporal features to obtain temporal coding features, the spatial feature fuser adaptively extracts the hierarchical representation of the topological relationship feature matrix through a dynamic graph convolutional network to obtain spatial fusion features, and the dual attention mechanism based on channel attention and spatial attention performs adaptive weight allocation on the temporal coding features and the spatial fusion features, and fuses and outputs the gesture intention recognition result;

[0102] Fuzzy reasoning is used to calculate the multi-dimensional similarity between the gesture intention recognition result and the mapping template, and the best matching mapping template is selected in combination with the adaptive priority weight. The touch position information in the multimodal touch feature data, the gesture intention recognition result and the scene type descriptor in the best matching mapping template are packaged to generate touch event data.

[0103] Exemplarily, first, a mapping relationship library between touch operations and multi-platform interaction scenarios is established.

[0104] Specifically, feature vectors of multiple gestures are extracted. For example, for a two-finger zoom gesture, the pressure value distribution of the two contact points is extracted, the rate of change of the distance between the two contact points, and the rate of change of the angle between the two contact points and the horizontal direction are calculated, and these features are combined into a vector. Operation intention identifiers such as "zoom", "rotate", "translate", etc. Scene type descriptors such as "map", "image browser", "game interface", etc. The extracted gesture feature vectors are associated with the corresponding operation intention identifiers and scene type descriptors to generate a mapping template. The association calculation can be achieved by calculating the similarity between the feature vector and the identifier. For example, cosine similarity can be used to measure the similarity between the feature vector and the identifier. Based on the historical interaction frequency, an adaptive priority weight is assigned to each mapping template. For example, if the interaction frequency of "two-finger zoom in the map scene" is very high, the corresponding mapping template will be assigned a higher priority weight. Assuming that the interaction frequency of "two-finger zoom in the map scene" is 0.8 and the interaction frequency of "two-finger zoom in the image browser scene" is 0.5, their corresponding priority weights are 0.8 and 0.5 respectively.

[0105] Next, the user's touch operation information in each display area is collected.

[0106] Specifically, when the user performs a touch operation, the coordinate sequence, pressure value gradient, velocity vector and acceleration vector of the touch point are collected to generate a time series feature. For example, a single-finger sliding operation can collect a series of coordinate points (x1, y1), (x2, y2), (x3, y3)..., as well as the pressure value, velocity and acceleration of each point. The Euclidean distance, angle change rate and relative velocity direction between the touch points are calculated to obtain a topological relationship feature matrix. For example, for a two-finger rotation operation, the change in the distance between the two touch points, the rate of change of the angle between the two touch points and the horizontal direction, and the direction of the relative velocity of the two touch points can be calculated, and this information can be organized into a matrix. The time series feature and the topological relationship feature matrix are aligned in time and space and then combined to form multimodal touch feature data. For example, the collected time series features such as coordinate sequence, pressure value, velocity and acceleration are aligned in time with the calculated topological relationship features such as the distance between the touch points, the angle change rate and the relative velocity direction, and then they are combined into a multimodal feature data. Suppose there are two contact points A and B, the coordinate sequence of A is (1, 1), (2, 2), (3, 3), and the coordinate sequence of B is (4, 4), (5, 5), (6, 6), then the distance sequence between the contact points is 5.66, 5.66, 5.66.

[0107] Then, the multimodal touch feature data is input into the long short-term memory neural network for gesture intention recognition.

[0108] Specifically, the temporal feature encoder of the long short-term memory neural network uses a bidirectional long short-term memory layer with skip connections to perform multi-scale encoding of temporal features to obtain temporal coding features. The spatial feature fuser adaptively extracts the hierarchical representation of the topological relationship feature matrix through a dynamic graph convolutional network to obtain spatial fusion features. The dual attention mechanism based on channel attention and spatial attention performs adaptive weight allocation on the temporal coding features and spatial fusion features and fuses them to output the gesture intention recognition result. For example, for a two-finger zoom gesture, the neural network may output the intention recognition result of "zoom".

[0109] Finally, the recognition result is matched with the mapping relationship library to generate touch event data.

[0110] Specifically, fuzzy reasoning is used to calculate the multi-dimensional similarity between the gesture intention recognition result and the mapping template. For example, the similarity between the "zoom" and "two-finger zoom in map scene" templates is calculated. The best matching mapping template is selected in combination with the adaptive priority weight. For example, if the similarity between the "zoom" and "two-finger zoom in map scene" templates is the highest, and the priority weight of the template is also high, then the template is selected as the best match. The touch position information in the multimodal touch feature data, the gesture intention recognition results, and the scene type descriptor in the best matching mapping template are packaged to generate touch event data. For example, the touch event data finally generated may include: touch position information (x, y), operation intention identifier "zoom", and scene type "map".

[0111] In this embodiment, through multimodal feature fusion and deep learning models, the user's gesture intention can be recognized more accurately and the misrecognition rate can be reduced. By establishing a mapping relationship library, the same touch operation can be mapped to different interaction scenarios and platforms, improving the cross-platform adaptability of the system. Through adaptive priority weights and fuzzy reasoning matching, the user's touch operation can be responded to more quickly and accurately, providing a smoother and more natural interaction experience.

[0112] In an optional implementation, receiving the touch event data, determining a target display area according to the touch position information, establishing a data dependency graph between platforms based on the interaction scenario type, matching a corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and converting the touch event data into a plurality of associated platform control instructions according to the data dependency graph includes:

[0113] receiving touch event data, performing coordinate transformation and normalization processing on touch position information in the touch event data to generate normalized touch coordinate data, determining a ray projection direction based on the normalized touch coordinate data, projecting detection rays in horizontal and vertical directions, and obtaining a set of intersection points between the detection rays and a display area boundary matrix;

[0114] Counting the number of intersections in the intersection set, determining the region attribution relationship of the normalized touch coordinate data according to an odd-even intersection determination rule, and determining a target display region, wherein the target display region includes a region identifier and a platform type identifier;

[0115] Extracting interaction scene information from the touch event data, associating and mapping the platform type identifier with the interaction scene information, constructing an initial set of platform nodes, calculating the data interaction frequency, data flow direction, and data structure characteristics between nodes in the initial set of platform nodes, generating a quantified value of data coupling between nodes, calculating a quantified value of functional coupling between nodes based on the calling relationship, interface mapping relationship, and state change requirements between nodes in the initial set of platform nodes, fusing the data coupling quantified value and the functional coupling quantified value according to a preset fusion weight, constructing a directed connection relationship between platform nodes, and outputting a data dependency graph;

[0116] Extracting operation intention information from the touch event data, performing semantic feature extraction on the operation intention information to generate an intention feature vector, retrieving a candidate instruction template set in a preset interactive rule library, calculating the semantic similarity between the intention feature vector and each template in the candidate instruction template set, and selecting the best matching instruction template according to the magnitude of the semantic similarity;

[0117] The optimal matching instruction template is decomposed into an atomic operation sequence, a priority identifier and an operation dependency identifier are added to each atomic operation in the atomic operation sequence, and an atomic operation sequence with identifiers is generated. Based on the topological structure of the data dependency graph, the processing capability parameters and load status parameters of each platform node are obtained, and the platform execution weight coefficient is calculated. According to the platform execution weight coefficient and the atomic operation sequence with identifiers, control instructions for different platforms are generated.

[0118] Exemplarily, touch event data is first received. A touch screen or other touch device generates touch event data, which includes touch position information (x, y coordinates), timestamp, touch type (press, move, lift), etc. For example, a user clicks the screen coordinates (200, 300), and the touch type is press.

[0119] The touch position information in the touch event data is transformed and normalized to generate normalized touch coordinate data. Different devices have different screen sizes and resolutions. In order to process them uniformly, the original touch coordinates need to be converted to normalized coordinates, which usually range from 0 to 1. Assuming the screen width is 1080 pixels and the height is 1920 pixels, the normalized coordinates are (200 / 1080, 300 / 1920) = (0.185, 0.156).

[0120] Determine the ray projection direction based on the normalized touch coordinate data. Determine the ray projection direction based on the type of touch event (press, move, lift). For example, the projection direction of a press event is from outside the screen to inside the screen.

[0121] Project detection rays in the horizontal and vertical directions to obtain the intersection points of the detection rays and the display area boundary matrix. The display area boundary matrix defines the boundaries of different display areas on the screen. Suppose there is a button on the screen, and its boundary matrix is ​​(0.2, 0.3) in the upper left corner and (0.4, 0.5) in the lower right corner. The ray emitted from the normalized touch coordinates (0.185, 0.156) has no intersection with the button area.

[0122] Count the number of intersections in the intersection set, determine the region attribution of the normalized touch coordinate data according to the odd-even intersection determination rule, and determine the target display area. Odd-even intersection determination rule: If the number of intersections between the ray and the region boundary is an odd number, the touch point is located in the region; if the number of intersections is an even number, the touch point is outside the region. Since the number of intersections in the previous example is 0 (even number), the touch point is not in the button area. Assume that the user clicks the screen coordinates (300, 400), the normalized coordinates are (0.278, 0.208), which is located in the button area, and the ray has two intersections with the button area boundary. Therefore, the target display area is determined to be the button, its region identifier is "button1", and the platform type identifier is "Android".

[0123] Extract interaction scene information from touch event data. Extract interaction scene information based on the context of the application and the current user operation. For example, the current scene is the "video playback interface".

[0124] The platform type identifier is associated and mapped with the interactive scene information to construct the initial set of platform nodes. The "Android" platform and the "video playback interface" scene are associated to construct the initial set of platform nodes {("Android", "video playback interface")}.

[0125] Calculate the data interaction frequency, data flow direction, and data structure characteristics between nodes in the initial set of platform nodes to generate a quantitative value of data coupling between nodes. Assume that the "Android" platform needs to send a video playback request to the "Server" platform, the data interaction frequency is high, the data flow direction is from "Android" to "Server", and the data structure is in JSON format. Then the quantitative value of data coupling between "Android" and "Server" is high.

[0126] Based on the calling relationship, interface mapping relationship and state change requirements between nodes in the initial set of platform nodes, the functional coupling metric value between nodes is calculated. The "Android" platform calls the video playback interface of the "Server" platform, and the functional coupling metric value is high.

[0127] The data coupling metric value and the functional coupling metric value are fused according to the preset fusion weights to construct a directed connection relationship between platform nodes and output a data dependency graph. The data coupling metric value and the functional coupling metric value are weighted averaged to obtain the final coupling metric value. According to the coupling metric value, a directed connection relationship between platform nodes is constructed to form a data dependency graph, for example, "Android" points to "Server".

[0128] Extract operation intention information from touch event data. Extract operation intention information based on the user's touch operation and application context. For example, when the user clicks the "Play" button, the operation intention information is "Start playing the video."

[0129] Extract semantic features of the operation intention information and generate an intention feature vector. Use natural language processing technology to extract the semantic features of "start playing the video" and generate an intention feature vector.

[0130] A candidate instruction template set is retrieved in a preset interaction rule library. Various operation instruction templates are predefined in the interaction rule library, for example, "playVideo(videoUrl)" means playing a video.

[0131] Calculate the semantic similarity between the intent feature vector and each template in the candidate instruction template set. Calculate the semantic similarity between the intent feature vector and "playVideo(videoUrl)".

[0132] The best matching instruction template is selected according to the semantic similarity. The instruction template "playVideo(videoUrl)" with the highest semantic similarity is selected as the best matching instruction template.

[0133] Decompose the best matching instruction template into an atomic operation sequence. Decompose "playVideo(videoUrl)" into an atomic operation sequence: {"get the video URL", "send a play request"}.

[0134] Add a priority flag and an operation dependency flag to each atomic operation in the atomic operation sequence to generate an atomic operation sequence with flags. Add a high priority flag and a no dependency flag to "Get Video URL"; add a low priority flag and a flag that depends on "Get Video URL" to "Send Play Request".

[0135] Based on the topological structure of the data dependency graph, the processing capacity parameters and load status parameters of each platform node are obtained to calculate the platform execution weight coefficient. According to the processing capacity and load status of the "Android" and "Server" platforms, their execution weight coefficients are calculated.

[0136] Generate control instructions for different platforms based on the platform execution weight coefficient and the atomic operation sequence with identifiers. Send the "get video URL" instruction to the "Android" platform, and send the "send play request" instruction to the "server" platform.

[0137] In this embodiment, by constructing a data dependency graph and optimizing the instruction execution order, the communication overhead between platforms is reduced and the efficiency of cross-platform interaction is improved. A unified touch event processing mechanism and instruction template are provided to shield the underlying platform differences and simplify the cross-platform application development process. Dynamic generation of control instructions is supported according to different interaction scenarios and operation intentions, enhancing the flexibility of cross-platform interaction.

[0138] In an optional implementation, a real-time data synchronization mechanism is established between platforms based on the interaction status information, interface parameters of the source platform and the target platform are obtained, a data format conversion strategy is generated and a bidirectional data channel is established, and output data of the source platform is converted into a data format receivable by the target platform through the bidirectional data channel and then transmitted to the target platform, including:

[0139] Acquire interaction state information, extract interface parameters of the source platform and the target platform from the interaction state information, the interface parameters including data structure information, data type information and data format information; extract data conversion features from the interface parameters, generate a feature weight matrix, perform feature weighting processing on the output data of the source platform according to the feature weight matrix, and obtain weighted feature data;

[0140] Performing format analysis on the output data according to the weighted feature data, performing data format mapping based on a pre-trained neural network model, and generating a data format conversion strategy, wherein the data format conversion strategy includes a target data structure, a target data type, and a target data format;

[0141] Establishing a bidirectional data channel between the source platform and the target platform based on the interaction status information, wherein the bidirectional data channel includes an uplink data channel and a downlink data channel;

[0142] Monitor the transmission status and system resource status of a bidirectional data channel, calculate a channel evaluation value according to the transmission status and the system resource status, wherein the channel evaluation value includes a transmission delay value and a resource consumption value; select an optimal data channel based on the channel evaluation value, and generate a channel configuration instruction, wherein the channel configuration instruction is used to configure data transmission parameters;

[0143] A plurality of synchronization checkpoints are set in the optimal data channel, a data version identifier is generated, and a data synchronization status is recorded at each synchronization checkpoint; the output data is format converted according to the data format conversion strategy to generate converted data; and the converted data is transmitted to a target platform through the optimal data channel based on the channel configuration instruction.

[0144] For example, first, through the state acquisition module deployed on the source platform and the target platform, the platform operation status, network connection status and system resource status are collected in real time. Taking the industrial control system as an example, the source platform uses the PLC controller to generate sensor data in binary format, and the target platform requires JSON format data for the monitoring system. The state acquisition module will obtain information such as the PLC's communication port, data type definition and transmission protocol.

[0145] Then, the interface parameters are extracted from the interaction status information, including data structure information, data type information, and data format information. For industrial control systems, the interface parameters include PLC data block structure, variable type definition, data encoding format, etc. The data structure information describes the organization of data, the data type information defines the data type of the variable, and the data format information specifies the encoding and decoding rules of the data.

[0146] After obtaining the interface parameters, the feature extraction module is used to analyze the data conversion characteristics. This module uses deep learning technology to identify key features in the data through a multi-layer feature extraction network. For industrial control data, the core features such as timestamps, measurements, and status identification are extracted. After feature extraction, a feature weight matrix is ​​generated based on the attention mechanism to assign importance weights to different features. For example, a higher weight is assigned to sensor measurements and a lower weight is assigned to descriptive information.

[0147] The feature weight matrix is ​​applied to the data output by the source platform to perform feature weighting processing. Weighted processing highlights the important features in the data, which facilitates subsequent format conversion. In industrial control systems, the weighted feature data retains the high-precision information of sensor measurements while compressing non-critical information.

[0148] Based on weighted feature data, a pre-trained neural network model is used for data format mapping. The model is trained with a large amount of historical data and can accurately identify the conversion rules between different data formats. The model analyzes the structure and content of the weighted feature data and generates a data format conversion strategy. The strategy includes the structure definition of the target data, type mapping rules, and format conversion methods.

[0149] When establishing a bidirectional data channel, configure the data uplink channel and downlink channel based on the interactive status information. The uplink channel is responsible for transmitting the converted data, and the downlink channel transmits control instructions and confirmation information. The transmission status and resource status are continuously monitored on the channel, including indicators such as network latency, bandwidth utilization, and processor occupancy.

[0150] The channel evaluation value is calculated by comprehensively evaluating the transmission delay value and resource consumption value. The channel with a higher evaluation value is selected as the optimal data channel, and the system generates a channel configuration instruction containing transmission parameter configuration based on it. Multiple synchronization checkpoints are set on the optimal channel, and each checkpoint is responsible for monitoring the data synchronization status of a specific segment.

[0151] According to the data format conversion strategy, the binary data of the source platform is converted into JSON format. The conversion process includes data structure reorganization, type conversion and format reconstruction. Finally, based on the channel configuration instructions, the converted data is transmitted to the target platform through the optimal channel.

[0152] In this embodiment, by analyzing the data conversion characteristics and establishing a bidirectional data channel, real-time synchronization of data between platforms is achieved, data delay is reduced, and data synchronization efficiency is improved. By monitoring the channel status and system resource status, selecting the optimal data channel, and configuring data transmission parameters, system resource consumption can be reduced and system stability can be improved. By setting synchronization checkpoints and data version identifiers, data synchronization status can be monitored, data consistency can be ensured, and data rollback operations can be supported to improve data reliability.

[0153] In an optional implementation, collecting transmission quality parameters and data synchronization status in real time during data transmission, and dynamically adjusting the data transmission strategy according to the transmission quality parameters and data synchronization status includes:

[0154] A multi-level monitoring node is set on the transmission path of the bidirectional data channel, and the multi-level monitoring node collects the transmission status information of the data packet, generates a monitoring data stream, calculates the network layer transmission index, the application layer processing index and the business layer synchronization index based on the monitoring data stream, and generates a real-time transmission status and data synchronization status;

[0155] The real-time transmission state and the data synchronization state are combined into a feature vector, the feature vector is input into a pre-trained deep reinforcement learning model, transmission state features are extracted, a transmission control strategy is generated based on the transmission state features, and transmission parameters are adjusted according to the transmission control strategy to form a transmission control instruction;

[0156] Based on the historical monitoring data collected by the multi-level monitoring nodes, a long short-term memory network is trained to obtain a transmission quality prediction model, the transmission quality parameters and data synchronization status within a specified time window are input into the transmission quality prediction model, the transmission quality change trend and abnormal warning information are generated, and the transmission control strategy is optimized according to the transmission quality change trend and abnormal warning information;

[0157] When the data synchronization state indicates a transmission abnormality, extract the abnormal feature pattern and match the abnormal type, execute the fault recovery strategy according to the abnormal type, generate recovery execution data and calculate the recovery effect index of the recovery execution data;

[0158] The recovery effect index is input into the deep reinforcement learning model to update the network parameters; the transmission control strategy is regenerated based on the updated network parameters to achieve closed-loop optimization of transmission quality.

[0159] Exemplarily, first, a multi-level monitoring node is set on the transmission path of the bidirectional data channel. For example, monitoring nodes are set at the sending end, the receiving end, and the intermediate nodes on the transmission path. These multi-level monitoring nodes are responsible for collecting the transmission status information of the data packet, such as the delay, packet loss rate, disorder rate, etc. of the data packet, and generating a monitoring data stream with this information. Assuming that the delay of a data packet collected by the sending end monitoring node is 10ms, the packet loss rate is 0%, and the disorder rate is 0%, then this information is packaged into a monitoring data record and added to the monitoring data stream.

[0160] Next, the network layer transmission indicators, application layer processing indicators, and business layer synchronization indicators are calculated based on the monitoring data flow to generate real-time transmission status and data synchronization status. Network layer transmission indicators include average delay, peak delay, packet loss rate, etc. Application layer processing indicators include data processing rate, error rate, etc. Business layer synchronization indicators include data synchronization delay, data consistency, etc. Assuming that the average network layer delay is 15ms, the packet loss rate is 1%, the application layer data processing rate is 10Mbps, the error rate is 0.1%, and the business layer data synchronization delay is 20ms, these indicators are integrated to generate real-time transmission status and data synchronization status.

[0161] Then, the real-time transmission status and data synchronization status are combined into a feature vector. For example, the average delay, packet loss rate, data processing rate, data synchronization delay and other indicators calculated above are combined into a feature vector: [15, 1, 10, 20]. The feature vector is input into the pre-trained deep reinforcement learning model. The model can be a multi-layer neural network, which is trained with a large amount of historical data to learn how to generate the optimal transmission control strategy based on the transmission status characteristics. The deep reinforcement learning model extracts the transmission status characteristics and generates a transmission control strategy based on these characteristics. For example, if the model determines that the current network condition is poor, it will generate a control strategy to reduce the transmission rate. According to the generated transmission control strategy, the transmission parameters are adjusted to form a transmission control instruction. For example, the transmission rate is adjusted to 5Mbps and the instruction is sent to the transmitter.

[0162] At the same time, a long short-term memory network is trained based on historical monitoring data collected by multi-level monitoring nodes to obtain a transmission quality prediction model. For example, a long short-term memory network is trained using monitoring data from the past week to predict the transmission quality in the future. The transmission quality parameters and data synchronization status within the specified time window are input into the transmission quality prediction model to generate transmission quality change trends and abnormal warning information. For example, the transmission quality parameters of the past hour are input into the prediction model to predict the transmission quality change trend in the next 10 minutes and determine whether there are abnormal conditions. Assuming that the prediction model predicts that the packet loss rate will rise to 5% in the next 10 minutes, an abnormal warning message is generated. The transmission control strategy is optimized based on the transmission quality change trend and abnormal warning information. For example, according to the predicted rising trend of the packet loss rate, the sending strategy is adjusted in advance, such as enabling the forward error correction mechanism.

[0163] When the data synchronization status indicates a transmission anomaly, such as when the data synchronization delay exceeds a threshold, the anomaly feature pattern is extracted and the anomaly type is matched. For example, if the data synchronization delay continues to increase, it is judged to be a network congestion anomaly. A fault recovery strategy is executed according to the anomaly type. For example, for network congestion anomalies, strategies such as reducing the sending rate and enabling flow control can be adopted. Recovery execution data is generated and recovery effect indicators of the recovery execution data are calculated. For example, indicators such as data synchronization delay and packet loss rate after executing the fault recovery strategy are recorded, and recovery effect indicators such as the decrease in data synchronization delay are calculated.

[0164] Finally, the recovery effect index is input into the deep reinforcement learning model to update the network parameters. If the recovery effect index indicates that the fault recovery strategy is effective, the parameters of the deep reinforcement learning model are adjusted accordingly so that it can make correct decisions faster when encountering similar situations in the future. The transmission control strategy is regenerated based on the updated network parameters to achieve closed-loop optimization of transmission quality.

[0165] In this embodiment, by real-time monitoring and dynamic adjustment of transmission strategies, transmission delay and packet loss rate can be effectively reduced, thereby improving data transmission efficiency. By predicting the transmission quality change trend and abnormal warning information, measures can be taken in advance to avoid or mitigate the impact of transmission anomalies, thereby enhancing transmission stability. Through deep reinforcement learning and closed-loop feedback mechanisms, the system can continuously learn and optimize transmission strategies, thereby achieving adaptive optimization and adapting to different network environments and business needs.

[0166] Figure 2 FIG. 1 is a schematic diagram of a multi-platform fusion interactive system based on a touch screen according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0167] The first unit is used to obtain display resolution parameters and processor performance parameters of the large touch screen, calculate the optimal display area division scheme based on the display resolution parameters, divide the large touch screen into a main control display area and multiple collaborative display areas, establish a mapping relationship library between touch operations and multi-platform interactive scenarios, collect user touch operation information in each display area, input the touch operation information into a neural network model for gesture intention recognition, match the recognition result with the mapping relationship library, and generate touch event data including touch position information, operation intention identification information and interactive scenario type;

[0168] A second unit is used to receive the touch event data, determine the target display area according to the touch position information, establish a data dependency graph between platforms based on the interaction scenario type, match the corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and convert the touch event data into a plurality of related platform control instructions according to the data dependency graph, wherein when the target display area is the main control display area, the main control platform instructions are processed first and the collaborative platform instructions are triggered according to the execution results; when the target display area is the collaborative display area, the corresponding collaborative platform instructions are processed and the execution results are synchronized to the main control platform, and the interactive status information including the execution status of each platform and the data update identifier is generated;

[0169] The third unit is used to establish a real-time data synchronization mechanism between platforms based on the interaction status information, obtain interface parameters of the source platform and the target platform, generate a data format conversion strategy and establish a two-way data channel, convert the output data of the source platform into a data format receivable by the target platform through the two-way data channel, and then transmit it to the target platform, collect transmission quality parameters and data synchronization status in real time during data transmission, dynamically adjust the data transmission strategy according to the transmission quality parameters and data synchronization status, present the converted data in the target display area, and broadcast data update messages to all related platforms at the same time to ensure the consistency of data on each platform.

[0170] According to a third aspect of the embodiments of the present invention,

[0171] An electronic device is provided, comprising:

[0172] processor;

[0173] a memory for storing processor-executable instructions;

[0174] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0175] A fourth aspect of the embodiments of the present invention is:

[0176] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0177] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-platform fusion interaction method based on a large touch screen, characterized in that: include: Obtaining display resolution parameters and processor performance parameters of the large touch screen, calculating an optimal display area division scheme based on the display resolution parameters, dividing the large touch screen into a main control display area and multiple collaborative display areas, establishing a mapping relationship library between touch operations and multi-platform interactive scenarios, collecting user touch operation information in each display area, inputting the touch operation information into a neural network model for gesture intention recognition, matching the recognition result with the mapping relationship library, and generating touch event data including touch position information, operation intention identification information, and interactive scenario type; Receive the touch event data, determine the target display area according to the touch position information, establish a data dependency graph between platforms based on the interaction scenario type, match the corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and convert the touch event data into a plurality of related platform control instructions according to the data dependency graph, wherein when the target display area is the main control display area, give priority to processing the main control platform instructions and trigger the collaborative platform instructions according to the execution results, and when the target display area is the collaborative display area, process the corresponding collaborative platform instructions and synchronize the execution results to the main control platform, and generate interaction status information including the execution status of each platform and the data update identifier; Based on the interactive status information, a real-time data synchronization mechanism is established between platforms, the interface parameters of the source platform and the target platform are obtained, a data format conversion strategy is generated and a two-way data channel is established, and the output data of the source platform is converted into a data format receivable by the target platform through the two-way data channel and then transmitted to the target platform. During the data transmission process, transmission quality parameters and data synchronization status are collected in real time, and the data transmission strategy is dynamically adjusted according to the transmission quality parameters and data synchronization status. The converted data is presented in the target display area, and data update messages are broadcast to all related platforms at the same time to ensure the consistency of data on each platform.

2. The method according to claim 1, characterized in that Calculating the optimal display area division scheme based on the display resolution parameter to divide the touch screen into a main control display area and a plurality of collaborative display areas includes: Obtaining the physical resolution and display ratio parameters of the large touch screen, collecting the user's gaze point distribution data and the touch operation thermal distribution data, weightedly fusing the gaze point distribution data and the touch operation thermal distribution data to generate an interaction density distribution map, calculating the weight distribution coefficient of the display area based on the interaction density distribution map, setting the area threshold in combination with the visual perception characteristics of the human eye, and generating the initial display area division constraint conditions; The interaction density distribution map and the initial display area division constraint conditions are input into an adaptive display area optimization network, a multi-objective loss function is constructed with visual attention distribution entropy, operation space continuity index and area switching cost as optimization targets, a deep reinforcement learning algorithm is used to train the adaptive display area optimization network, a reward value is calculated based on real-time user interaction feedback during the training process, and a weight coefficient of the multi-objective loss function is updated, the trained adaptive display area optimization network is applied to display area division optimization, and an optimal division scheme including area proportions and relative position relationships is output; Based on the optimal division scheme, the position with the highest display area weight distribution coefficient is determined as the visual center point and a radial coordinate system is established. The boundary outline of the main control display area is calculated in the radial coordinate system. The layout position of the collaborative display area is determined according to the interaction density distribution map. The data correlation between the display areas is extracted and the width of the area interval band is adaptively adjusted. A division control instruction containing boundary description information and area scaling parameters is generated, and the area division control instruction is sent to the display rendering module to execute the display area division.

3. The method according to claim 2, characterized in that Based on the optimal division scheme, the position with the highest weight distribution coefficient of the display area is determined as the visual center point and a radial coordinate system is established, the boundary contour of the main control display area is calculated in the radial coordinate system, the layout position of the collaborative display area is determined according to the interactive density distribution map, the data association between the display areas is extracted and the width of the area interval band is adaptively adjusted, and the division control instruction containing the boundary description information and the area scaling parameter is generated, including: Density clustering is performed on the display area weight distribution coefficients in the optimal division scheme, wherein the bandwidth parameter of the clustering is adaptively adjusted according to the display area size, the spatial coordinates of the point with the highest weight density are determined as the visual center point, a radial coordinate system is established with the visual center point as the origin, and a visual attention distribution model is constructed according to the user's line of sight movement trajectory; In the radial coordinate system, the visual attention distribution model and the interactive density distribution map are weightedly fused to generate a display area density field, the reference radius of the parameter equation is determined based on the gradient distribution of the display area density field, and the fluctuation amplitude is calculated according to the local fluctuation characteristics of the display area density field. The period parameter is set based on the spatial distribution law of the display content, and the reference radius, fluctuation amplitude and period parameter are substituted into the parameter equation to generate an initial boundary contour, and the attention distribution entropy on the initial boundary contour is calculated. The initial boundary contour is iteratively optimized with minimizing the attention distribution entropy as the optimization goal, and the optimized boundary contour is subjected to curvature analysis and high curvature points are smoothed to generate the final contour of the main control display area; Extract the spatial moment features of the interaction density distribution map, calculate the centroid moment feature vector, perform polar coordinate space mapping based on the visual attention distribution model, obtain the initial layout position of the collaborative display area, construct a correlation calculation model that integrates the interaction frequencies of multiple types of data to generate a data correlation matrix, and calculate and dynamically update the regional interval band width according to the data correlation matrix; The final outline of the main control display area and the boundary of the collaborative display area are fitted with a Bezier curve to generate boundary description information, the area scaling parameters and the visual transition parameters are determined based on the data association matrix, and a division control instruction containing the boundary description information and the area scaling parameters is generated.

4. The method according to claim 1, characterized in that: Establishing a mapping relationship library between touch operations and multi-platform interactive scenarios, collecting the user's touch operation information in each display area, inputting the touch operation information into a neural network model for gesture intention recognition, matching the recognition result with the mapping relationship library, and generating touch event data including touch position information, operation intention identification information, and interactive scenario type, including: Build a mapping relationship library between touch operations and multi-platform interaction scenarios, extract the number of touch points, pressure value distribution and movement trajectory curvature to obtain gesture feature vectors, calculate the correlation between the gesture feature vectors and the operation intention identifier and scene type descriptor to generate mapping templates, and assign adaptive priority weights to each mapping template based on historical interaction frequency; The coordinate sequence, pressure value gradient, velocity vector and acceleration vector of the touch point are collected to generate time series features, the Euclidean distance, angle change rate and relative velocity direction between the touch points are calculated to obtain a topological relationship feature matrix, and the time series features and the topological relationship feature matrix are spatially and temporally aligned and then combined to form multimodal touch feature data; The multimodal touch feature data is input into a long short-term memory neural network, the temporal feature encoder of the long short-term memory neural network uses a bidirectional long short-term memory layer with skip connections to perform multi-scale encoding on the temporal features to obtain temporal coding features, the spatial feature fuser adaptively extracts the hierarchical representation of the topological relationship feature matrix through a dynamic graph convolutional network to obtain spatial fusion features, and the dual attention mechanism based on channel attention and spatial attention performs adaptive weight allocation on the temporal coding features and the spatial fusion features, and fuses and outputs the gesture intention recognition result; Fuzzy reasoning is used to calculate the multi-dimensional similarity between the gesture intention recognition result and the mapping template, and the best matching mapping template is selected in combination with the adaptive priority weight. The touch position information in the multimodal touch feature data, the gesture intention recognition result and the scene type descriptor in the best matching mapping template are packaged to generate touch event data.

5. The method according to claim 1, characterized in that Receiving the touch event data, determining the target display area according to the touch position information, establishing a data dependency graph between platforms based on the interaction scenario type, matching a corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and converting the touch event data into a plurality of associated platform control instructions according to the data dependency graph includes: receiving touch event data, performing coordinate transformation and normalization processing on touch position information in the touch event data to generate normalized touch coordinate data, determining a ray projection direction based on the normalized touch coordinate data, projecting detection rays in horizontal and vertical directions, and obtaining a set of intersection points between the detection rays and a display area boundary matrix; Counting the number of intersections in the intersection set, determining the region attribution relationship of the normalized touch coordinate data according to an odd-even intersection determination rule, and determining a target display region, wherein the target display region includes a region identifier and a platform type identifier; Extracting interaction scene information from the touch event data, associating and mapping the platform type identifier with the interaction scene information, constructing an initial set of platform nodes, calculating the data interaction frequency, data flow direction, and data structure characteristics between nodes in the initial set of platform nodes, generating a quantified value of data coupling between nodes, calculating a quantified value of functional coupling between nodes based on the calling relationship, interface mapping relationship, and state change requirements between nodes in the initial set of platform nodes, fusing the data coupling quantified value and the functional coupling quantified value according to a preset fusion weight, constructing a directed connection relationship between platform nodes, and outputting a data dependency graph; Extracting operation intention information from the touch event data, performing semantic feature extraction on the operation intention information to generate an intention feature vector, retrieving a candidate instruction template set in a preset interactive rule library, calculating the semantic similarity between the intention feature vector and each template in the candidate instruction template set, and selecting the best matching instruction template according to the magnitude of the semantic similarity; The optimal matching instruction template is decomposed into an atomic operation sequence, a priority identifier and an operation dependency identifier are added to each atomic operation in the atomic operation sequence, and an atomic operation sequence with identifiers is generated. Based on the topological structure of the data dependency graph, the processing capability parameters and load status parameters of each platform node are obtained, and the platform execution weight coefficient is calculated. According to the platform execution weight coefficient and the atomic operation sequence with identifiers, control instructions for different platforms are generated.

6. The method according to claim 1, characterized in that Establishing a real-time data synchronization mechanism between platforms based on the interaction status information, acquiring interface parameters of the source platform and the target platform, generating a data format conversion strategy and establishing a bidirectional data channel, and converting the output data of the source platform into a data format receivable by the target platform through the bidirectional data channel and then transmitting it to the target platform includes: Acquire interaction state information, extract interface parameters of the source platform and the target platform from the interaction state information, the interface parameters including data structure information, data type information and data format information; extract data conversion features from the interface parameters, generate a feature weight matrix, perform feature weighting processing on the output data of the source platform according to the feature weight matrix, and obtain weighted feature data; Performing format analysis on the output data according to the weighted feature data, performing data format mapping based on a pre-trained neural network model, and generating a data format conversion strategy, wherein the data format conversion strategy includes a target data structure, a target data type, and a target data format; Establishing a bidirectional data channel between the source platform and the target platform based on the interaction status information, wherein the bidirectional data channel includes an uplink data channel and a downlink data channel; Monitor the transmission status and system resource status of a bidirectional data channel, calculate a channel evaluation value according to the transmission status and the system resource status, wherein the channel evaluation value includes a transmission delay value and a resource consumption value; select an optimal data channel based on the channel evaluation value, and generate a channel configuration instruction, wherein the channel configuration instruction is used to configure data transmission parameters; A plurality of synchronization checkpoints are set in the optimal data channel, a data version identifier is generated, and a data synchronization status is recorded at each synchronization checkpoint; the output data is format converted according to the data format conversion strategy to generate converted data; and the converted data is transmitted to a target platform through the optimal data channel based on the channel configuration instruction.

7. The method according to claim 1, characterized in that Collecting transmission quality parameters and data synchronization status in real time during data transmission, and dynamically adjusting the data transmission strategy according to the transmission quality parameters and data synchronization status include: A multi-level monitoring node is set on the transmission path of the bidirectional data channel, and the multi-level monitoring node collects the transmission status information of the data packet, generates a monitoring data stream, calculates the network layer transmission index, the application layer processing index and the business layer synchronization index based on the monitoring data stream, and generates a real-time transmission status and data synchronization status; The real-time transmission state and the data synchronization state are combined into a feature vector, the feature vector is input into a pre-trained deep reinforcement learning model, transmission state features are extracted, a transmission control strategy is generated based on the transmission state features, and transmission parameters are adjusted according to the transmission control strategy to form a transmission control instruction; Based on the historical monitoring data collected by the multi-level monitoring nodes, a long short-term memory network is trained to obtain a transmission quality prediction model, the transmission quality parameters and data synchronization status within a specified time window are input into the transmission quality prediction model, the transmission quality change trend and abnormal warning information are generated, and the transmission control strategy is optimized according to the transmission quality change trend and abnormal warning information; When the data synchronization state indicates a transmission abnormality, extract the abnormal feature pattern and match the abnormal type, execute the fault recovery strategy according to the abnormal type, generate recovery execution data and calculate the recovery effect index of the recovery execution data; The recovery effect index is input into the deep reinforcement learning model to update the network parameters; the transmission control strategy is regenerated based on the updated network parameters to achieve closed-loop optimization of transmission quality.

8. A multi-platform fusion interactive system based on a large touch screen, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to obtain display resolution parameters and processor performance parameters of the large touch screen, calculate the optimal display area division scheme based on the display resolution parameters, divide the large touch screen into a main control display area and multiple collaborative display areas, establish a mapping relationship library between touch operations and multi-platform interactive scenarios, collect user touch operation information in each display area, input the touch operation information into a neural network model for gesture intention recognition, match the recognition result with the mapping relationship library, and generate touch event data including touch position information, operation intention identification information and interactive scenario type; A second unit is used to receive the touch event data, determine the target display area according to the touch position information, establish a data dependency graph between platforms based on the interaction scenario type, match the corresponding operation instruction template in a preset interaction rule library according to the operation intention identification information, and convert the touch event data into a plurality of related platform control instructions according to the data dependency graph, wherein when the target display area is the main control display area, the main control platform instructions are processed first and the collaborative platform instructions are triggered according to the execution results; when the target display area is the collaborative display area, the corresponding collaborative platform instructions are processed and the execution results are synchronized to the main control platform, and the interactive status information including the execution status of each platform and the data update identifier is generated; The third unit is used to establish a real-time data synchronization mechanism between platforms based on the interaction status information, obtain interface parameters of the source platform and the target platform, generate a data format conversion strategy and establish a two-way data channel, convert the output data of the source platform into a data format receivable by the target platform through the two-way data channel, and then transmit it to the target platform, collect transmission quality parameters and data synchronization status in real time during data transmission, dynamically adjust the data transmission strategy according to the transmission quality parameters and data synchronization status, present the converted data in the target display area, and broadcast data update messages to all related platforms at the same time to ensure the consistency of data on each platform.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images