Multi-screen layout adaptive generation method based on space-time attention mechanism

Abstract

The application relates to the technical field of electric data processing, in particular to a multi-screen layout adaptive generation method based on a space-time attention mechanism, which comprises the following steps: firstly, a layout signal is sent out when a layout switching operation is performed, current interface layout information of a main screen and a secondary screen is acquired, and historical interaction and activity data related to the main screen are acquired; then, the acquired data is processed through a space-time attention mechanism in combination with position coding and a correlation matrix, fusion is carried out by using a correlation attention mechanism, and a first layout scheme is obtained by decoding the fused data; next, the first layout scheme is evaluated by using a graph evolution method and a time sequence layout comparison method, the first layout scheme is adjusted by dynamically weighting the evaluation results, a second layout scheme is obtained, and a layout instruction is sent to apply the second layout scheme to the display layout of the main screen and the secondary screen. The application significantly improves the accuracy of multi-screen adaptive generation and improves the fluency of multi-screen use.

Description

A Multi-Screen Layout Adaptive Generation Method Based on Spatiotemporal Attention Mechanism Technical Field

[0001] This invention relates to the field of electrical data processing technology, specifically to a multi-screen layout adaptive generation method based on a spatiotemporal attention mechanism. Background Technology

[0002] With increasingly diverse computing environments, user interaction is no longer limited to a single screen. Multi-screen systems are widely used in scenarios ranging from personal workstations to professional collaborative environments such as research laboratories, classrooms, remote conference rooms, and command and control centers. In these complex multi-screen environments, users may need to process data from multiple sources simultaneously, switch focus between different screens, and even collaborate with other users. Traditional user interfaces, often using static or simple responsive layouts, struggle to meet the demands for efficient data presentation, smooth task execution, and state synchronization.

[0003] In recent years, deep learning technology has been widely used in adaptive multi-screen layout generation, automating or assisting the generation of multi-screen layouts by analyzing historical user interaction data and user data. However, it struggles to understand user needs when generating multi-screen layouts, leading to extensive manual adjustments by users and impacting user experience.

[0004] To address this, a multi-screen layout adaptive generation method based on a spatiotemporal attention mechanism is proposed. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-screen layout adaptive generation method based on a spatiotemporal attention mechanism. By utilizing a spatiotemporal attention mechanism and an evaluation mechanism to adaptively generate multi-screen layouts, the accuracy of generation and the smoothness of multi-screen use are significantly improved. First, when performing a layout switching operation, a layout signal is emitted to obtain the current interface layout information of the main screen and the secondary screen, and to obtain historical interaction and activity data related to the main screen. Then, the position encoding and the main-secondary screen relationship representation are input into the spatiotemporal attention mechanism and fused using an associative attention mechanism. The fused data is then decoded to obtain a first layout scheme. Next, the first layout scheme is evaluated using a graph evolution method and a temporal layout comparison method, respectively. The first layout scheme is adjusted based on the dynamically weighted evaluation results to obtain a second layout scheme. Finally, a layout command is sent to apply the second layout scheme to the display layout of the main screen and the secondary screen.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A multi-screen layout adaptive generation method based on spatiotemporal attention mechanism includes:

[0008] When performing a layout switching operation, a layout signal is emitted to obtain the first spatial data consisting of the current interface layout information of the main screen and the secondary screen, and the first temporal data consisting of the historical interaction and activity data related to the main screen.

[0009] The first spatial data, the first temporal data, the position encoding, and the relationship representation of the main and secondary screens are processed through a spatiotemporal attention mechanism to obtain the second spatial data and the second temporal data; the second spatial data and the second temporal data are input into an associative attention mechanism for fusion, wherein the associative attention mechanism is a cross-attention mechanism that introduces a bias for adjustment, and the fused data is decoded to obtain the first layout scheme;

[0010] The first layout scheme is evaluated using graph evolution and temporal layout comparison methods. The dynamic weighted evaluation results are used to adjust the first layout scheme to obtain the second layout scheme, which is then applied to the main screen and the secondary screen.

[0011] Furthermore, the acquisition of the first spatial data includes: representing the relationship between the main and secondary screens as an association matrix, and acquiring the current interface layout information of the main and secondary screens through the network. The layout information includes the interface layout parameters of the main screen, the list of device identifiers of the secondary screen, the connection status of the secondary screen, and a vectorized representation of the interface layout information of the secondary screen.

[0012] Furthermore, the acquisition of the first time data includes: acquiring historical interaction and activity data of the main screen, performing noise reduction processing on the historical interaction and activity data, and performing serialization and event encoding processing on the noise-reduced historical interaction and activity data to obtain the first time data.

[0013] Furthermore, the second spatial data obtained by processing the first spatial data includes:

[0014] The location code is replaced with enhanced location code, and the enhanced location code is used to generate a location code with the same dimension as the first spatial data. The enhanced location code integrates the unique identifier of the data element, the data element type, the logical grouping information of the data element and the coordinate information of the main screen element. The location code is embedded into a high-dimensional feature vector to obtain a high-dimensional encoding vector. All the high-dimensional encoding vectors are combined into a high-dimensional encoding sequence. The high-dimensional encoding sequence is calculated using a self-attention mechanism and an association matrix to obtain the second spatial data.

[0015] Furthermore, the second spatial data and the second temporal data are input into the associative attention mechanism for fusion, including:

[0016] A cross-attention mechanism is used to calculate the mutual attention score between the second temporal data and the second spatial data; a guidance bias matrix is ​​added to the mutual attention score using an association guidance module, and the data is normalized to generate the second spatiotemporal data.

[0017] Furthermore, the evaluation process of the graph evolution method includes:

[0018] The content describing the main screen interface within the first spatial data is represented as a first layout diagram. The nodes in the diagram represent main screen interface elements, and the edges in the diagram represent the spatial relationships between main screen elements. The main screen layout diagram generated after simulating the first layout scheme is used to obtain a second layout diagram. The structural difference between the first layout diagram and the second layout diagram is calculated, and a quantified structural change metric is output.

[0019] Furthermore, the evaluation process of the temporal layout comparison method includes: for operation events in the first time data, performing spatial enhancement processing by combining the spatial data corresponding to the occurrence of the operation event, and combining it with the encoded representation of the operation event itself, and grouping them according to user tasks to form an enhanced event database; applying a sequence mining algorithm and setting a minimum support threshold to mine the enhanced event database; mining and outputting frequent sequences that satisfy the minimum support threshold and are composed of composite events, as a first sequence set; mapping the sequences in the first sequence set to obtain the execution path from the current layout to the first layout generation scheme; evaluating the change in interaction cost based on the execution path; and calculating the temporal contrast value based on the evaluation results.

[0020] Furthermore, the adjustment of the first layout scheme based on the dynamic weighted evaluation results includes:

[0021] The structural change metric and the temporal contrast metric are normalized. The current layout complexity is determined based on the first spatial data. The frequency of recent events is obtained based on the first sequence set. The preset weights are dynamically adjusted using the layout complexity and the event frequency to obtain the final weights. The structural change metric and the temporal contrast metric are then fused using the final weights to obtain an evaluation score. The evaluation score is then compared with preset high and low thresholds to dynamically adjust the first layout scheme.

[0022] Furthermore, the multi-screen layout adaptive generation method based on the spatiotemporal attention mechanism includes: establishing and maintaining a communication connection between the secondary screen and the main screen; when receiving a layout signal from the main screen, reporting its own status and interface layout information according to the request of the main screen; continuously listening to and receiving layout instructions sent from the main screen that point to the secondary screen; parsing the layout instructions to obtain specific interface adjustment parameters; and automatically updating the user interface display layout of the secondary screen according to the interface adjustment parameters.

[0023] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0024] 1. By processing the temporal and spatial data of multi-screen display devices through a spatiotemporal attention mechanism, and dynamically adjusting the spatiotemporal fusion data through enhanced positional changes and the main and secondary screen association matrix, the complex spatial structure and inter-element relationships of the multi-screen layout are effectively captured. The long-distance temporal dependencies and contextual information in the user's historical interaction sequence are accurately modeled, and the user's deep-level interaction needs are understood, thereby improving the user experience.

[0025] 2. By adopting an associative attention mechanism and introducing an associative guidance module, the system intelligently integrates the accurate location perception and historical interaction data from the previous step, enabling it to more accurately understand the user's real-time operational intent in a specific spatial environment. This significantly improves the accuracy of adaptive layout generation for multi-screen display devices and enhances the user experience.

[0026] 3. The layout adaptive generation scheme was evaluated from multiple dimensions using graph evolution method and temporal layout comparison method. This led to a deeper understanding of the potential impact of layout, which can significantly reduce the layout adjustment scheme generated by external interference, improve the robustness of multi-screen layout adaptive generation, reduce the number of manual adjustments by users, and improve user experience. Attached Figure Description

[0027] Figure 1 is a schematic diagram of the method flow provided in an embodiment of the present invention;

[0028] Figure 2 is a flowchart of the spatiotemporal attention mechanism for processing spatial data provided in an embodiment of the present invention.

[0029] Figure 3 is a layout generation flowchart provided by an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Please refer to Figures 1 to 3. This invention provides a multi-screen layout adaptive generation method based on a spatiotemporal attention mechanism. The technical solution is as follows:

[0032] Example 1

[0033] With the advancement of digital education, multi-screen interactive systems, as an effective tool to enhance teaching interactivity and enrich data presentation dimensions, have been deployed and applied in numerous schools. School A introduced a large number of multi-screen interactive systems to improve teaching quality. However, the systems rely on preset templates and simple rule engines for layout generation and mechanisms, lacking the ability to perceive dynamic teaching scenarios and adapt to real-time user needs. This leads to problems such as low screen space utilization, chaotic data distribution logic, inefficient cross-screen operation processes, and slow or erroneous layout generation responses in complex teaching interactions. These problems seriously affect the flow of the teaching process and hinder the improvement of overall teaching quality. Therefore, to overcome the limitations of existing technologies, it is urgent to develop a new adaptive multi-screen layout generation method to achieve a more intelligent, efficient, and flexible multi-screen teaching environment.

[0034] Figure 1 is a schematic diagram of the process provided in this embodiment.

[0035] As shown in Figure 1, the multi-screen layout adaptive generation method based on the spatiotemporal attention mechanism includes:

[0036] When generating an adaptive layout, this invention first sends a layout signal during layout switching to obtain the current interface layout information of the main screen and the secondary screen, and acquires historical interaction and activity data related to the main screen. Then, it processes the acquired data using a spatiotemporal attention mechanism by combining position encoding and association matrix, and fuses the data using an association attention mechanism. The fused data is then decoded to obtain a preliminary layout scheme. Next, the layout scheme is evaluated using a graph evolution method and a temporal layout comparison method, and the preliminary layout scheme is adjusted based on the dynamically weighted evaluation results. Finally, a layout command is sent to apply the layout scheme to the display layout of the main screen and the secondary screen.

[0037] Specifically, when the main screen is switched, a layout signal is sent to obtain the current interface layout information of the main screen and the secondary screen to obtain the first spatial data, and the relevant historical interaction and activity data of the main screen to obtain the first time data.

[0038] Furthermore, the acquisition of the first spatial data includes:

[0039] A unique ID is identified and assigned to the teacher's main screen, which performs core processing, designated as the master node. All currently connected student sub-screens are identified through a network discovery mechanism (obtaining a list from the teaching management platform), and a unique ID is assigned to each student. Let the total number of participating nodes be N. An N*N zero matrix is ​​created as the association matrix A. This matrix is ​​filled based on the real-time network connection status and teaching logic. Matrix elements A[T,Si] and A[Si,T] represent the valid connection status between the teacher's main screen and student sub-screen i (1=connected, 0=disconnected). Matrix element R[Si, Sj] represents whether student i and student j belong to the same predefined group on the teacher's main screen (1=same group / collaborating, 0=no). This matrix reflects the reachability of the network topology, communication quality, and the logical grouping and collaboration relationships among participants, and it is dynamically updated as the teacher's main screen is adjusted.

[0040] Furthermore, the interface layout parameters and current state of the teacher's main screen are obtained. These layout parameters include the layout, size, coordinates, and screen resolution of each element on the main screen. Student-side secondary screen data is collected via the network. This data includes the student's unique ID, connection status, screen resolution, device performance level, and interface layout information (e.g., interface element layout, window type of main content). This secondary screen data is encoded into a vectorized representation. The collected teacher-side layout parameters and state information, along with the vectorized interface layout and state information obtained from the student-side, are converted into a unified format high-dimensional feature vector. This high-dimensional feature vector and the correlation matrix together constitute the first spatial data.

[0041] By using the association matrix to support logical grouping of the teacher's main screen, it can support complex instructional design and collaborative needs. By obtaining spatial data through high-dimensional feature vectors, it can comprehensively, dynamically, and structurally capture key spatial information in networked teaching scenarios.

[0042] Furthermore, historical interaction and activity data related to the main screen are obtained to acquire real-time data, including:

[0043] Obtain the raw logs related to the current teaching session and the main screen, including: teacher-side interaction logs (recording teacher operations), student-side interaction logs reported via the network (recording key student interactions), and central server logs (recording global events such as user connection and mode switching commands).

[0044] Clean the raw logs. Filter out incomplete, incorrectly formatted, or irrelevant event types (such as low-level system events or unnecessary mouse movements). Merge repetitive or jittery operations on the same element within a short period (such as rapid clicks). Check the unique ID of the event source, and in conjunction with the screen resolution of the event source, check and remove event records whose coordinates fall outside the valid screen boundaries (such as click coordinates that are negative or exceed the screen width and height).

[0045] Furthermore, the denoised events are organized. The denoised events are strictly sorted from earliest to latest according to their timestamps to preserve the complete cross-screen operation sequence. Then, based on the continuity of user interaction, the sorted long event stream is divided into multiple independent session sequences using landmark events of the teaching phase (teacher's main screen switching mode).

[0046] Furthermore, each session sequence is converted into a sequence of numerical vectors. For each event in the sequence, its key attributes are extracted, including event type, target screen ID, logical region, time difference with the previous event, and coordinates at the time of the event. The categorical attributes (event type, target screen ID, logical region) are converted into vectors using one-hot encoding; the numerical attributes (time difference, event coordinates) are standardized. These encoded numerical values ​​and vectors are concatenated into a single feature vector, and missing optional data is padded with 0 to ensure that event vectors in the same sequence and even all sequences have a uniform dimension, thus obtaining the first-time data.

[0047] By denoising the original logs of the main screen and secondary screen, noisy events in the original logs were removed. Then, the denoised events were divided according to the landmark events of the teaching stage, which reduced the computational load of subsequent sequence calculations and ensured the relevance and coherence of the context when analyzing specific teaching activities. This made the behavioral patterns within the stage clearer and easier for subsequent algorithms to effectively identify.

[0048] Furthermore, the first spatial data is combined with enhanced location encoding and an association matrix and input into a spatiotemporal attention mechanism to obtain the second spatial data. The flowchart of this process is shown in Figure 2, and includes:

[0049] For each data element in the first spatial data, representing a graphical element on the teacher's interface, a multi-component fused enhanced positional code pi is generated. The enhanced positional code pi is generated by fusing the participant's unique identifier (ID) corresponding to the data element, the entity type of the data element (distinguishing between teacher interface elements and student state vectors), and the logical grouping information to which the data element belongs (the teaching group to which the student belongs). Furthermore, the enhanced positional vector pi is conditionally processed: when the data element represents a teacher interface element, its two-dimensional coordinate positional code within the teacher's screen is further fused into pi; when the data element represents a student state vector, a zero vector is used to replace the two-dimensional coordinate positional code portion and integrate it into pi. The generated enhanced positional code pi is combined with the original high-dimensional feature vector of the corresponding element in the first spatial data, and a final encoded vector is obtained through vector concatenation and linear projection. The final encoded vectors of all data elements are organized into a high-dimensional encoded sequence according to their type IDs. Using three sets of independent, learnable linear transformations, the high-dimensional encoded sequence is mapped into three new sequences: a query sequence Q, a key sequence K, and a value sequence V. Each element i in the sequence now has a corresponding query vector Qi, key vector Ki, and value vector Vi. These sequences Q, K, and V are represented and processed in matrix form during computation.

[0050] Furthermore, by querying the transpose K of matrix Q and key matrix K... T Interactive computation is performed to generate a first spatial matrix S1. Each element in the first scoring matrix S1 quantifies the initial association strength between a query vector from Q and a key vector from K. Next, the elements in the first spatial matrix S1 are normalized by dividing by the square root of the dimension of the key vector K, thus obtaining a second spatial matrix S2. The positions of elements in the non-associative screens in the second scoring matrix are determined according to the predefined association matrix A, and a large negative bias is applied to them, thus obtaining a third spatial matrix S3. This operation aims to significantly reduce the attention interaction strength between elements in the non-associative screens. Finally, Softmax is used to normalize the third scoring matrix S3, transforming it into a probability distribution matrix. The second spatial data is obtained by multiplying the probability distribution matrix with the value matrix V and performing a weighted summation of V.

[0051] By generating an enhanced positional code for each data element in the first spatial data, it becomes possible to integrate these two types of data, which come from different sources and have different properties, into a single self-attention model framework for unified processing. This solves the core problem of inconsistent coordinate systems in network scenarios, enabling the capture of a complete spatial state containing information from both teachers and students. By applying a negative bias using the correlation matrix A, unnecessary attention weights are significantly reduced in the model, which allocates them to unrelated nodes, thereby improving the model's learning efficiency and the accuracy of the generated layout.

[0052] Furthermore, the first-time data is processed to obtain the second-time data, including:

[0053] First, the first-time data is input into the Transformer encoder. First, the first-time data is processed through an embedding layer to add relative positions, resulting in first-time encoded data, which provides the relative position data of events within the sequence for modeling. Then, the first-time encoded data is input into the Transformer encoder for processing. Using three independent, learnable linear transformations, the first-time encoded data is mapped into three new sequences: a query sequence Q, a key sequence K, and a value sequence V. Each element i in the sequence now has a corresponding query vector Qi, a key vector Ki, and a value vector Vi. These sequences Q, K, and V are represented and processed using matrix form in computation; this is achieved by transforming the query matrix Q with the transpose K of the key matrix K. T Interactive computation is performed to generate a first time matrix T1. Each element in the first time matrix T1 quantifies the initial association strength between a query vector from Q and a key vector from K. Next, the elements in the first time matrix T1 are normalized by dividing by the square root of the dimension of the key vector K, thus obtaining a second time matrix T2. Softmax is used to normalize the second time matrix T2, transforming it into a probability distribution matrix. The probability distribution matrix is ​​then multiplied by the value matrix V to achieve a weighted summation of V, resulting in a third time matrix T3.

[0054] Further, the third time matrix T3 is residually concatenated with the first time-coded data to obtain the second time-coded data, which is then normalized. Next, the second time-coded data is input into a position-wise feedforward network containing two layers of linear transformation and activation functions to obtain the third time-coded data. Finally, the third time-coded data is residually concatenated with the second time-coded data again and normalized to obtain the fourth time-coded data.

[0055] Furthermore, the above steps are repeated, and the second time data is obtained after processing by N Transformer encoders.

[0056] Furthermore, the second spatial data and the second temporal data are input into the associative attention mechanism for fusion, including: first, using the second temporal data as the query vector Q and the second spatial data as the key vector K and value vector V, calculating the basic mutual attention score between them to obtain the first scoring matrix, where each element of the matrix initially represents the degree of attention to the temporal data and the spatial data;

[0057] Furthermore, an association guidance module is introduced, which is a multilayer perceptron (MLP). This module concatenates the second temporal data and the second spatial data in pairs. The association guidance module is trained end-to-end with the overall network, outputting a guidance bias value for each pair of spatiotemporally concatenated data, forming a guidance bias matrix. This guidance bias matrix is ​​added to the first scoring matrix to obtain a second scoring matrix. This generation ensures that the attention score reflects the estimated task relevance. The second scoring matrix is ​​then Softmax normalized to obtain a third scoring matrix. The third scoring matrix is ​​then weighted and summed with the second spatial data to obtain the second spatiotemporal data. Finally, a Transformer decoder is used to decode the second spatiotemporal data to generate a first layout generation scheme. The Transformer decoder operates in an autoregressive manner, predicting one element in the output sequence at each decoding time step until a complete first layout generation scheme is generated.

[0058] By introducing an association guidance module to fuse the second time data and the second spatial data, the task relevance is estimated based on the user intent inferred from the current time data and the key layout elements identified from the spatial data, and guidance bias terms are generated. This more intelligently focuses on the spatiotemporal feature interactions that are most relevant and important to the generated layout scheme, thereby effectively handling the potential feature ambiguity problem.

[0059] Furthermore, by processing the first layout generation scheme and the first spatial data using the graph evolution method, structural change metrics are obtained, including:

[0060] Traverse the graphic elements and area information describing the current layout of the teacher's interface in the first spatial data, and create a node for each key interface element (e.g., student preview window, function panel, control button, content display area). The node stores unique identifier data, element type, bounding box coordinates in the teacher's virtual desktop coordinate system, and the parent node ID of the hierarchical structure. Create directed edges from the parent node to the child node according to the hierarchical structure. Set a first distance threshold T1 based on the average size of the teacher's interface elements, and calculate the Euclidean distance L between the center points of the bounding boxes of any two element nodes i and j on the teacher's interface. If the distance L is less than the first distance threshold T1, add an undirected edge between element nodes i and j to represent their visual proximity on the teacher's interface, thereby constructing the current teacher's layout graph Gc.

[0061] Furthermore, based on the specific adjustment suggestions for the teacher's interface in the first layout scheme, the adjusted teacher's interface layout is simulated and generated. Using the same node definitions and edge construction rules (including hierarchical edges and screen-based proximity edges based on T1) as the current layout graph Gc, a generated layout graph Gp is ​​constructed. The number of nodes in the generated layout graph Gp that have changed (type, attribute changes, additions, and deletions) compared to the current layout graph Gc is counted. The average distance the center point moves for nodes whose center point movement distance is greater than a preset movement change threshold is calculated. The proportion of nodes whose area (boundary box size) change rate is greater than a preset area change threshold is calculated. The Jaccard similarity coefficient J (defined as intersection size divided by union size) between the edge set Ec of the current layout graph Gc and the edge set Ep of the generated layout graph Gp is ​​calculated, and 1-J is used as a measure of the difference between the edge sets. The specific calculation formula is as follows:

[0062] ;

[0063] in, Indicates the edge difference degree. This represents the set of edges in the current layout graph. This represents the set of edges used to generate the layout graph. express and The Jaccard similarity coefficient;

[0064] Furthermore, the number of changed nodes, the average distance, the node ratio, and the edge difference are normalized to the range [0,1]. The normalized number of changed nodes, the average distance, the node ratio, and the edge difference are then weighted and summed using preset weights to obtain the structural change metric.

[0065] By using a graph evolution method to calculate a structural change metric for multi-screen layouts, we can effectively quantify the changes in layout at the relationship network level. This allows for a better understanding of the complex structural changes that may result from the generation of the main screen layout at the spatial level, thus maintaining the stability of the relationship between the main screen and the secondary screen.

[0066] Furthermore, a sequence mining algorithm is used to mine the first-time data to obtain a first sequence set. The mining algorithm aims to deeply bind user actions to their specific spatial environment, thereby uncovering more context-meaningful behavioral patterns. The algorithm first performs spatial context enhancement processing on the first-time data. Firstly, by combining the spatial data corresponding to each operation event, key spatial context features for understanding user behavior are extracted, such as: the coordinates of elements on the screen, the multi-view layout mode in effect at that time, and the interaction mode with the student. These features are selected to capture macroscopic and microscopic layout data that may significantly influence user decisions and operational processes. The extracted spatial features are then combined with the original encoded representation of the operation event through vector concatenation to form a composite event with higher dimensions and richer spatial background data. Then, the long sequence composed of the composite events, ordered by time, is segmented using teaching stage events.

[0067] Furthermore, the PrefixSpan data mining algorithm is selected, and a minimum support threshold is preset. The data mining algorithm and threshold are used to process and enhance the event sequence database, thereby mining and outputting frequent sequences composed of compound events that occur frequently in the dataset, resulting in a first sequence set.

[0068] Furthermore, a temporal layout comparison method is used to evaluate the first sequence set and the first layout generation scheme to obtain a temporal contrast value. This includes: for the first sequence set, firstly, mapping the execution paths of the sequences within the first sequence set in the current layout and the first layout generation scheme, and determining the elements and their states involved in each operation within the sequence set. Then, comparing the execution process of the sequences within the set under the two layouts, the interaction cost change introduced by layout generation is quantitatively evaluated. The interaction cost change includes the total pointer movement distance required to complete the sequence, the number of necessary switches made by the teacher between different interface functional areas and teaching task focus, and the accessibility change of key elements in the sequence on the teacher's interface. Based on these quantitative indicators, the difference between the current layout and the suggested layout is judged, and the difference is weighted using preset weights to obtain a perturbation score, which reflects the potential interference degree of layout generation to this specific user flow. Finally, the perturbation scores of the sequences within the set are averaged and summed to obtain the temporal contrast value.

[0069] The timing contrast value is calculated by using the timing layout comparison method. It objectively reflects the actual impact of the first layout generation scheme on the user's established operating habits and common processes. It can better reflect the user's behavior patterns in the real world and is closer to the user's actual experience.

[0070] Furthermore, the dynamic weighted evaluation results are used to adjust the first layout scheme. The process is shown in Figure 3. First, the structural change metric and the temporal contrast metric are normalized. The current layout complexity is determined based on the first spatial data. A complexity score is generated based on the current layout complexity using a preset lookup table. The higher the complexity, the higher the complexity score. The frequency of recent events is obtained based on the first sequence set. A frequency score is generated based on the frequency of recent events using a preset lookup table. The higher the frequency, the lower the frequency score. A preset weight ratio is used, for example, the structural change metric has a weight of 0.7 and the temporal contrast metric has a weight of 0.3. Then, the preset weights are adjusted according to the ratio of the complexity score and the frequency score to obtain the final weighted weight. The structural change metric and the temporal contrast metric are then weighted and fused together with the final weighted weight to obtain the evaluation score.

[0071] Furthermore, a low threshold and a high threshold are preset. An evaluation score is obtained by weighting the structural change metric and temporal contrast metric according to preset weights. This evaluation score is compared with both the low and high thresholds. If the evaluation score is less than the low threshold, it is considered a high-quality layout, and the first layout generation scheme is fully adopted for adaptive multi-screen layout generation. The first layout generation scheme is directly determined as the second layout generation scheme. If the evaluation score is greater than the low threshold but less than the high threshold, it is considered a medium-quality layout. In this case, an output signal is generated to request external confirmation of whether to adopt the first layout generation scheme. If a confirmation input signal is received, the first layout generation scheme is determined as the second layout generation scheme. If a rejection input signal is received or there is no response, the current layout scheme is determined as the second layout adjustment scheme. If the evaluation score is greater than the high threshold, the generated scheme is considered a low-quality layout, and the first layout generation scheme is automatically rejected. The current layout scheme is directly determined as the second layout generation scheme. Finally, a layout command is sent to apply the second layout scheme to the display layout of the main screen and secondary screen.

[0072] By dynamically adjusting the weighted average of change metrics and temporal contrast values, an evaluation score is generated to adjust the first layout scheme. This effectively utilizes time and spatial data, ensuring the safety and rationality of automatic layout generation and making multi-screen layout generation more intelligent.

[0073] Furthermore, the operation of the secondary screen includes: when the student application starts and joins the teaching session, it actively establishes a stable, bidirectional communication link with the teacher's main screen. Throughout the session, the student secondary screen maintains the connection's activity through a heartbeat mechanism, ensuring it can receive instructions and signals from the main screen at any time. When the student secondary screen receives a layout signal from the main screen, it collects its current key status information (such as device ID, network connection quality indicators, screen resolution, and performance level) and interface layout information (such as the current active window type, and the relative position / size / visibility status of each main content window), and encodes it into a vectorized representation. After collection and processing, the student sends this information back to the main screen through the established communication link. After reporting, it continuously listens for and receives layout instructions from the main screen pointing to its unique ID. It parses the layout instructions to obtain specific interface adjustment parameters and automatically updates the user interface display layout of the secondary screen according to the interface adjustment parameters.

[0074] By establishing a stable communication link between the student's secondary screen and the teacher's main screen, the system can monitor the instructions and signals sent by the main screen in real time. This ensures that the main screen can obtain information from the secondary screen in real time and can receive layout instructions from the main screen pointing to the secondary screen for real-time parsing and updating the screen layout.

[0075] To verify the advantages of the multi-screen adaptive generation method based on spatiotemporal attention mechanism proposed in this invention, a set of comparative experiments was conducted. The experimental groups used the proposed multi-screen adaptive generation method based on spatiotemporal attention mechanism. Control group 1 used only temporal data for multi-screen adaptive generation, control group 2 used only spatial data, and control group 3 used both temporal and spatial data, but did not utilize graph evolution methods or temporal comparison methods to evaluate the layout scheme. The same operations were performed in a specific scenario to determine the task success rate, the number of user corrections, and the window overlap conflict rate. The experiment involved 100 operations. Specific comparison results are shown in Table 1. As can be seen from Table 1, compared with single spatiotemporal data, single spatial data, and single spatiotemporal data, the method of this invention significantly improves the task success rate, reduces the window overlap conflict rate, and reduces the number of user corrections.

[0076] Table 1 Comparative Experiment Results

[0077] Experimental / Control Group Task Success Rate Window Overlap Conflict Rate Number of User Corrections Experimental Group 95% 1.7% 5 Control Group 179% 9.7% 21 Control Group 273% 8.5% 27 Control Group 385% 2.5% 15 surface

[0078] This invention provides a multi-screen layout adaptive generation method based on a spatiotemporal attention mechanism. First, it fully integrates temporal and spatial data using this mechanism to generate a preliminary layout scheme. Then, it evaluates the preliminary scheme from multiple dimensions using a temporal layout comparison method. Finally, it adjusts the layout scheme based on the evaluation and sends the final scheme to different screens for further adjustments. This method fully integrates temporal and spatial data, avoiding the influence of single data sources on the generation effect. The temporal layout comparison method improves the accuracy of the layout generation scheme. This significantly enhances the accuracy of multi-screen adaptive generation using spatiotemporal data in complex scenarios, effectively improving the user experience.

[0079] Example 2

[0080] To improve the quality of laboratory teaching, School B introduced a multi-screen system in the laboratory. However, the current multi-screen adaptive generation method of the multi-screen system is difficult to achieve accurate generation that meets user needs in the complex environment and under interference conditions of the laboratory.

[0081] This reduced the smoothness of experimental teaching. To address this, School B introduced the multi-screen adaptive generation method based on spatiotemporal attention mechanism provided by this invention.

[0082] For a specific example of layout generation, evaluation, and decision-making, refer to Table 2. Initially, a low threshold of 0.4 and a high threshold of 0.9 are set. When the teacher switches the mouse focus from the main presentation screen to the question list, the main screen sends a layout signal. This signal acquires the current layout information of the main and secondary screens, along with historical interaction and activity data related to the main screen. The layout information and historical interaction data are processed using a spatiotemporal attention mechanism, fused using an associative attention mechanism, and then decoded. This results in a layout scheme where the question is centered and enlarged, the question list is displayed on the side for selection on the main screen, and the question is displayed on the secondary screen. First, a graph evolution method is used to evaluate the layout scheme, determining that changes in nodes and edges are small, resulting in a low structural change metric of 0.2. Then, the user's interaction history is mined, revealing that the question list on the secondary screen is frequently highlighted or enlarged after switching from the main screen. A temporal layout comparison method is used to evaluate the layout scheme and determine the temporal layout execution path. The changes are small, resulting in a low temporal contrast value of 0.15. This indicates low layout complexity. Based on the lookup table, a low complexity score of 20 is assigned. Then, the frequency of recent events is obtained. It is found that after switching the question list in the current main screen interaction history, the questions are displayed in the center and enlarged, while the question list on the side has a high frequency. The frequency is matched with the preset lookup table to obtain a low frequency score of 30. The weights of the complexity score and frequency score are compared to calculate the structural change metric weight of 0.61 and the temporal contrast value of 0.39. Finally, the evaluation score is calculated to be 0.18, which is less than the set low threshold. The evaluation result is judged as a high-quality layout. The questions are displayed in the center and enlarged, and the question list on the side is applied to the main screen. The questions displayed on the secondary screen are also applied to the secondary screen.

[0083] When collaborating in a group, student B is selected as the group leader on the main screen to initiate the presentation and operations. A layout is generated that centers and enlarges the analysis tool window currently being used by student B on the main screen, while shrinking or moving the windows of students A and C to the sides. The layout is evaluated using a graph evolution method, and the evaluation result is a medium-quality layout. At this point, an output signal is generated to request external confirmation on whether to adopt the layout. Specifically, a pop-up window asks, "Set student B as focus?" If a confirmation input signal is received, the analysis tool window currently being used by student B is placed in the center of the main screen, and the windows of students A and C are shrunk or moved to the sides. If a rejection input signal is received or there is no response, the original screen layout is maintained and no further generation is performed.

[0084] The teacher is conducting a complex teaching activity on the main screen that requires simultaneous reference to multiple information sources. The main screen displays and arranges: a teaching document, a real-time student answer data dashboard, and a whiteboard area for temporary notes or drawings. The screen layout is currently complex, but all elements are necessary for the current teaching segment. The secondary screen displays the students' normal view. While attempting to resize a window, the teacher drags the mouse too quickly, accidentally performing a gesture to reset the desktop layout in a blank area of ​​the screen. The system receives the layout signal, generates a layout scheme that resets the desktop layout of both the main and secondary screens, evaluates the layout scheme using a graph evolution method, determines the evaluation result to be a low-quality layout, rejects the layout generation scheme, and maintains the current layout.

[0085] Table 2 Examples of Layout Generation, Evaluation, and Decision-Making

[0086] User Operation Suggested Layout Scheme Score / Evaluation Layout Generation Teacher switches mouse focus from the main presentation screen to the question list on the secondary screen. The questions are centered and enlarged, with the question list on the side. The secondary screen displays the questions. 0.18 / High Quality Directly apply the layout generation scheme for adaptive generation. Teacher switches teaching mode from whole-class demonstration to group collaborative monitoring. The system detects that student B has started to lead the discussion and operation. Suggestion: Place the analysis tool window that student B is using in the center of the main screen, and shrink or move the windows of students A and C to the side, keeping the student screen unchanged. 0.56 / Medium Quality A pop-up window asks whether to set student B as the focus? Wait for teacher confirmation. If the teacher confirms, apply the suggested layout; if the teacher refuses or there is no response, maintain the current layout. Teacher dragged the mouse too fast and accidentally performed a gesture operation to reset the desktop layout in a blank area of ​​the screen. Suggestion: Reset the desktop layout of both the main screen and the secondary screen. 0.94 / Low Quality Reject applying the layout generation scheme and maintain the current display layout. surface

[0087] To demonstrate the effectiveness of the method of this invention, a comparison was made with a time-series-based multi-screen generation method introduced by School B. The comparison indicators included the number of manual adjustments by users, window overlap conflict rate, and user satisfaction score. User satisfaction was assessed using a satisfaction survey questionnaire, evaluating the smoothness of multi-screen teaching, the rationality of layout generation, the convenience of cross-screen operation, the stability of layout generation, and overall user satisfaction. A random sample of 200 users who had used both methods were surveyed, and scores were given using a satisfaction rating scale (out of 10). Table 3 shows the method evaluation comparison. The method of this invention had an average of 0.7 manual adjustments by users, a window overlap conflict rate of 2.3%, and a user satisfaction score of 9.3. In contrast, School B's original method had 2.6 manual adjustments by users, a window overlap conflict rate of 10.2%, and an average user satisfaction score of 8.1. This indicates that the method of this invention has a significant advantage in adaptive multi-screen layout generation and significantly improves the user experience.

[0088] Table 3 Comparison of Method Evaluation

[0089] Method: Manually adjust average number of times, window overlap, conflict rate, user satisfaction. School B: Existing method: 2.6%, 10.2%, 8.1%. Invention method: 0.7%, 2.3%, 9.3%. surface

[0090] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-screen layout adaptive generation method based on a spatiotemporal attention mechanism, characterized in that, include: When performing a layout switching operation, a layout signal is emitted to obtain first spatial data composed of the current interface layout information of the main screen and the secondary screen, and first temporal data composed of historical interaction and activity data related to the main screen; the first spatial data, the first temporal data, the position code and the relationship representation of the main and secondary screens are processed through a spatiotemporal attention mechanism to obtain second spatial data and second temporal data. The second spatial data and the second temporal data are input into the associative attention mechanism for fusion. The associative attention mechanism is an adjustment that introduces a bias into the cross-attention mechanism. The fused data is then decoded to obtain the first layout scheme. The first layout scheme is evaluated using both graph evolution and temporal layout comparison methods. The graph evolution method evaluation process includes: representing the content describing the main screen interface within the first spatial data as a first layout graph, where nodes represent main screen interface elements and edges represent spatial relationships between main screen elements; simulating the main screen layout generated by the first layout scheme to obtain a second layout graph; calculating the structural difference between the first and second layout graphs, and outputting a quantified structural change metric. The temporal layout comparison method evaluation process includes: for operation events in the first temporal data, performing spatial enhancement processing in conjunction with the spatial data corresponding to the occurrence of the operation event, and comparing... The operation events are encoded and grouped according to user tasks to form an enhanced event database. A sequence mining algorithm is applied to the enhanced event database and a minimum support threshold is set. Frequent sequences consisting of composite events that satisfy the minimum support threshold are mined and output as a first sequence set. The sequences in the first sequence set are mapped to obtain the execution path from the current layout to the first layout generation scheme. The interaction cost change is evaluated based on the execution path. Based on the evaluation result, the time contrast value is calculated, and the first layout scheme is adjusted dynamically with weighted evaluation results to obtain a second layout scheme. The second layout scheme is then applied to the main screen and the secondary screen.

2. The multi-screen layout adaptive generation method based on spatiotemporal attention mechanism according to claim 1, characterized in that, The acquisition of the first spatial data includes: representing the relationship between the main screen and the secondary screen as an association matrix, and obtaining the current interface layout information of the main screen and the secondary screen through the network. The layout information includes the interface layout parameters of the main screen, the list of device identifiers of the secondary screen, the connection status of the secondary screen, and the vectorized representation of the interface layout information of the secondary screen.

3. The multi-screen layout adaptive generation method based on spatiotemporal attention mechanism according to claim 1, characterized in that, The acquisition of the first time data includes: acquiring historical interaction and activity data of the main screen, performing noise reduction processing on the historical interaction and activity data, and performing serialization and event encoding processing on the noise-reduced historical interaction and activity data to obtain the first time data.

4. The multi-screen layout adaptive generation method based on spatiotemporal attention mechanism according to claim 2, characterized in that, The first spatial data processing to obtain the second spatial data includes: replacing the position code with an enhanced position code; generating a position code with the same dimension as the first spatial data using the enhanced position code; fusing the unique identifier of the data element, the data element type, the logical grouping information of the data element, and the coordinate information of the main screen element in the enhanced position code; embedding the position code into a high-dimensional feature vector to obtain a high-dimensional encoding vector; forming a high-dimensional encoding sequence from all the high-dimensional encoding vectors; and calculating the high-dimensional encoding sequence using a self-attention mechanism and an association matrix to obtain the second spatial data.

5. The multi-screen layout adaptive generation method based on spatiotemporal attention mechanism according to claim 4, characterized in that, The second spatial data and the second temporal data are input into the association attention mechanism for fusion, which includes: using a cross attention mechanism to calculate the mutual attention score between the second temporal data and the second spatial data; using an association guidance module to add a guidance bias matrix to the mutual attention score and normalize it to generate the second spatiotemporal data.

6. The multi-screen layout adaptive generation method based on spatiotemporal attention mechanism according to claim 1, characterized in that, The adjustment of the first layout scheme by the dynamic weighted evaluation result includes: normalizing the structural change metric and the temporal contrast metric; determining the current layout complexity based on the first spatial data; obtaining the frequency of recent events based on the first sequence set; dynamically adjusting the preset weights using the layout complexity and the event frequency to obtain the final weights; fusing the structural change metric and the temporal contrast metric using the final weights to obtain an evaluation score; and comparing the evaluation score with preset high and low thresholds to dynamically adjust the first layout scheme.

Images