Digital large screen multi-view collaborative display and interaction method based on visual focus tracking

Abstract

The application belongs to the technical field of screen display and interaction, and particularly relates to a digital large-screen multi-view collaborative display and interaction method based on visual focus tracking, which realizes dynamic quantification and real-time updating of information correlation between digital large-screen multi-views through a view semantic correlation calculation mode of fusion of time sequence similarity, time sequence response correlation, event synchronization correlation and user historical gaze behavior, deeply fuses user potential viewing intention recognition, multi-dimensional display weight generation and visual focus tracking technology, forms a complete logical collaborative closed loop of content perception-focus recognition-intention prediction-collaborative adjustment-feedback optimization, and realizes deep matching of multi-view display effect and user information acquisition demand.

Description

Technical Field

[0001] This invention belongs to the field of screen display and interaction technology, specifically relating to a method for multi-view collaborative display and interaction of digital large screens based on visual focus tracking. Background Technology

[0002] With the rapid development of big data, IoT, and AI technologies, digital large screens have been widely applied in smart cities, traffic management, energy management, and emergency command, becoming a core carrier for information aggregation, monitoring and early warning, and decision support. Modern digital large screen systems typically employ multi-view collaborative display, simultaneously presenting various types of information such as maps, videos, charts, and text on the same screen to meet users' needs for multi-source data fusion and perception in complex scenarios. Visual focus tracking technology, as a human-computer interaction method capable of real-time perception of user gaze points, captures the user's gaze position and duration through devices such as eye trackers and cameras, allowing the system to infer the user's attention area and intent.

[0003] In terms of display control, existing technologies have attempted to use visual focus information for dynamic adjustment of screen display. For example, while Chinese patent application CN121148316A combines deep reinforcement learning with eye tracking for display optimization, its application scenario is the overall brightness adjustment of a single display device, without addressing differentiated weight allocation in multi-view scenarios. On the other hand, US patent US11269403B2 discloses an adaptive multi-window configuration method based on gaze tracking, but its adjustment is based solely on the user's current gaze position, without considering the semantic relationship between the content of each window and the view. This results in other windows closely related to the content of the gaze window not being able to be adjusted collaboratively.

[0004] In summary, existing technologies still have the following shortcomings in the context of multi-view collaborative display on digital large screens: First, in digital large screens with multi-view collaborative displays, different views often carry interrelated information. For example, in traffic monitoring scenarios, the congested road sections displayed in the map view are strongly correlated with the real-time traffic flow presented in the vehicle trajectory view, while the device status information recorded in the text log view is less correlated with the current focus. When a user focuses on a particular view, existing systems typically only make local adjustments to the display parameters of that view, while the display status of other views remains unchanged. This forces users to actively perform visual searches when switching between the focus view and other views, increasing their cognitive load.

[0005] Second, existing technologies typically employ a simple mapping rule of gaze point-brightness adjustment, which means that after detecting that a user is gazing at a certain area, a preset brightness increase operation is performed on that area. However, they lack contextual awareness of the user's intentions. This rigid human-computer interaction fails to combine the content features of the view with the user's gaze behavior patterns for differentiated processing.

[0006] Therefore, there is an urgent need to propose a technical solution that combines the view semantic association matrix with deep reinforcement learning, which can solve the technical problems of "related views cannot be linked and adjustment strategies cannot be self-optimized" in multi-view collaborative display. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a method for multi-view collaborative display and interaction of digital large screens based on visual focus tracking, which solves the technical problems of existing digital large screen systems lacking a linkage adjustment mechanism for related views and having an inability to self-optimize adjustment strategies in multi-view collaborative display.

[0008] The objective of this invention can be achieved through the following technical solution: a method for multi-view collaborative display and interaction on a digital large screen based on visual focus tracking, comprising the following steps: S1: Obtain the content type of each view in the currently playing sequence on the screen and extract the content feature sequence of each view; S2: Calculate the view semantic correlation degree between any two views based on the content feature sequence of each view and the user's gaze shift data. Construct a view semantic correlation degree matrix based on multiple view semantic correlation degrees as matrix elements. The view semantic correlation degree matrix is ​​used to characterize the strength of correlation between different views in terms of information content. S3: Collect and preprocess the user's gaze point data on the screen, and determine the current focus view based on the gaze point data. The gaze point data includes gaze position coordinates, gaze duration, gaze point trajectory, and saccade frequency. S4: Based on the current focused view, obtain the view semantic association degree between other views and the current focused view from the view semantic association degree matrix; S5: Dynamically generate the display weight of each view based on the semantic relevance, the content type of the view, and the gaze point data, and adjust the display parameters of each view according to each display weight; S6: Collect user gaze behavior feedback data after display parameter adjustment, and calculate reinforcement learning reward value based on the feedback data; construct reinforcement learning model with deep Q network, take the current display weight as model state and weight adjustment amount as execution action, optimize display weight allocation strategy by maximizing cumulative discount reward; use the trained reinforcement model to infer the weight adjustment amount online, and update the display weight of each view in real time.

[0009] Preferably, in S2, the semantic relationship between each view is obtained, and a semantic relationship matrix is ​​constructed, including the following sub-steps: S201: Within a preset time window, calculate the dynamic semantic correlation between the content feature sequences of any two views; S202: Collect user gaze shift data in historical playback sequences, calculate the gaze shift probability matrix between views, and calculate the historical correlation based on user behavior through a collaborative filtering algorithm; S203: The dynamic semantic correlation degree and historical correlation degree are weighted and fused using an attention mechanism to obtain the view semantic correlation degree between the two corresponding views. The view semantic correlation degree matrix is ​​constructed using all view semantic correlation degrees as matrix elements.

[0010] Preferably, in S201, calculating the dynamic semantic correlation between the content feature sequences of any two views includes the following steps: The cosine similarity of the content feature vectors of two views is calculated to form a temporal similarity sequence, and the mean of the temporal similarity sequence is taken as the temporal similarity mean. Calculate the cross-correlation function of the content feature sequences of the two views, and extract the cross-correlation peak of the two cross-correlation functions as the correlation degree of the time series response; Get the number of times that two views simultaneously experience scene switching or content abrupt changes within a sliding time window, and calculate the event synchronization correlation. The mean temporal similarity, temporal response correlation, and event synchronization correlation are weighted and fused to obtain the dynamic semantic correlation between the two views in the current playback sequence.

[0011] Preferably, in S203, the view semantic relevance matrix is ​​also updated in real time using an exponential decay weighting method, and the fusion formula is as follows:

[0012] In the formula, Let be the semantic association matrix of the view at time t. The semantic association matrix of the view is calculated for the current time window. As the attenuation factor, Let be the view semantic association matrix at time t-1.

[0013] Preferably, in step S3, determining the current focus view based on the gaze point data includes the following sub-steps: S301: For the gaze position coordinates, use the DBSCAN clustering algorithm to perform cluster analysis and identify at least one dense region of gaze points; S302: Calculate the user's fixation duration, fixation trajectory, and saccade frequency in each fixation dense region, and generate a visual attention heatmap using kernel density estimation; S303: Based on the peak area of ​​the visual attention heatmap and the display area of ​​each view, perform position mapping and determine the current focus view by calculating the intersection-union ratio.

[0014] Preferably, in S303, the current focus view is determined by calculating the intersection-union ratio, and the calculation formula is as follows:

[0015] In the formula, Let i be the intersection-union ratio corresponding to the i-th view. The pixel area of ​​the matrix region. Let be the outermost matrix of the display area of ​​the i-th view. It is the minimum outer matrix corresponding to the peak region with the highest heat value extracted from the visual attention heatmap.

[0016] Preferably, in step S4, obtaining the view semantic relevance of other views from the view semantic relevance matrix based on the current focused view includes the following sub-steps: S401: Extract the row vector of the currently focused view from the view semantic association matrix to obtain the association degree between this view and all other views; S402: Sort the elements in the row vector (excluding themselves) in descending order of their relevance values, and filter out the set of views with significant relevance based on a preset threshold. S403: When causal correlation directionality measurement is enabled, extract edge correlation degree and ingress edge correlation degree from the directed correlation matrix to distinguish between the dominant view and the follower view. S404: When multi-scale time window fusion is enabled, the correlation vectors are extracted from the correlation matrices of the three scales respectively, and the comprehensive correlation is obtained by weighted fusion of multi-scale attention weights. S405: Finally, the correlation degree value, sorting results, filtered correlation view set, and directed correlation direction information are encapsulated into structured data and passed to the next step.

[0017] Preferably, in step S5, the step of dynamically generating the display weight of each view based on semantic relevance, view content type, and gaze point data includes the following sub-steps: S501: Generate the first weight factor vector based on the semantic relevance between the current focused view and other views; S502: Determine the corresponding basic display weight range based on the content type of each view, and generate the second weight factor vector; S503: Construct a gaze behavior feature vector based on the user's gaze duration, gaze point trajectory, and saccade frequency in each view area. Input the gaze behavior feature vector into a pre-trained attention concentration evaluation network to generate a third weight factor vector. S504: Generate a gaze sequence based on the user's gaze point data, input it into a pre-trained intent recognition model, output the user's potential intent type, and generate an intent feature vector based on the potential intent category; S505: The first weight factor vector, the second weight factor vector, the third weight factor vector, and the intention feature vector are concatenated to form a single-view fusion feature vector; S506: Input the multiple single-view fusion feature vectors into a pre-trained multilayer perceptron network, and output the attention score of each view as the final display weight of each view.

[0018] Preferably, in S5, the display parameters of each view are adjusted according to each display weight, and a grouping and collaboration strategy driven by the relevance threshold is adopted, including the following execution process: Based on the view semantic relevance matrix, the current focus view and views with a relevance greater than 0.6 are divided into a strong relevance group, those with a relevance between 0.3 and 0.6 are divided into a weak relevance group, and those with a relevance less than 0.3 are divided into an independent group. All views in a strongly related group adopt a linked adjustment strategy. When the display weight of the focus view is increased, the weights of other views in the group are increased synchronously according to the degree of relevance, and the adjustment time is synchronized with a delay of less than 100 milliseconds. Views in a weakly related group are increased by 50% of the degree of relevance. When the focus view is clear, the display weight of the independent group of views is suppressed to the lowest level, and the degree of suppression is negatively correlated with the gaze duration of the focus view; Based on the weights determined by the above group collaboration strategy, specific display parameter adjustments are performed: for views with display weights lower than the first preset threshold, the display parameters are adjusted to the preset minimum display level; for views with display weights higher than the second preset threshold, the display parameters are adjusted to the preset maximum display level; for views with display weights greater than the first preset threshold and less than the second preset threshold, the display parameters are adjusted according to a linear ratio of the display weights.

[0019] Preferably, in S6, the deep Q-network employs the following innovative design: Differential encoding of state space: The graph embedding features obtained by encoding the display weight vectors of each view and the semantic association matrix of the view through a graph convolutional network are concatenated and used as the state input of the deep Q network, so that the agent can perceive the semantic association topology between views; Action space constraint design: The action adopts the Softmax action space, so that the output weight adjustment amount satisfies the probability distribution constraint that the sum of the weights of all views is always 1, thus avoiding invalid adjustments; Multi-objective decomposition of the reward function: The reward function is decomposed into two sub-objectives: information acquisition efficiency reward and visual comfort reward. A multi-objective deep reinforcement learning framework is used to balance the conflict between the two objectives through a Pareto optimal policy. Among these, the information acquisition efficiency reward F... info and visual comfort reward F comfort They can be represented as:

[0020]

[0021] Among them, T n B is the normalized value of fixation duration. n P is the normalized value of the number of revisits. n S is the normalized value of the rate of change of pupil diameter. n λ1~λ4 are the normalized values ​​of gaze shift speed, and the weight coefficients of each index are 1.

[0022] The beneficial effects of this invention are as follows: This application constructs a dynamically updated view semantic relevance matrix to quantify the strength of the correlation between information content in different views. It also employs a correlation threshold-driven grouping and collaboration strategy, breaking the limitation of existing technologies that can only adjust the gaze view unidirectionally. When a user gazes at the focus view, the system can automatically adjust the display parameters of strongly correlated views based on semantic relevance, achieving collaborative display between multiple views. This allows users to perceive the overall visual linkage effect when their gaze shifts, effectively reducing the visual search cost and cognitive load for users acquiring cross-view information in multi-view scenarios. By integrating a multi-dimensional display weight generation system that considers view semantic relevance, content type, user gaze behavior characteristics, and potential viewing intentions, it replaces the fixed "gaze point-display parameter" single mapping rule in existing technologies, achieving context-aware differentiated display adjustments tailored to users' actual needs. Simultaneously, a reinforcement learning mechanism based on deep Q-networks is introduced to construct a self-optimizing closed loop, enabling the system to continuously iterate and optimize the display weight allocation strategy based on user gaze behavior feedback. This solves the technical problems of poor adaptability and inability to self-optimize caused by fixed rules in existing technologies, achieving an upgrade of the display strategy from passive response to personalized adaptive adjustment. Attached Figure Description

[0023] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0024] Figure 1This is a schematic diagram of the steps of a digital large screen multi-view collaborative display and interaction method based on visual focus tracking provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the sub-steps for constructing a view semantic relevance matrix according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the sub-steps for calculating dynamic semantic relevance provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the sub-steps for generating the display weights of each view according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the sub-steps for calculating the reinforcement learning reward value according to an embodiment of the present invention. Detailed Implementation

[0025] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0026] Figure 1 This is a schematic diagram illustrating the steps of a digital large-screen multi-view collaborative display and interaction method based on visual focus tracking, according to an embodiment of the present invention. Please refer to [link / reference]. Figure 1 As shown, this embodiment provides a method for multi-view collaborative display and interaction of digital large screens based on visual focus tracking, which is suitable for mainstream digital large screen application scenarios such as smart cities, traffic dispatching, energy management, and emergency command. The method includes the following steps: S1: Obtain the content type of each view in the currently playing sequence on the screen and extract the content feature sequence of each view; S2: Calculate the view semantic correlation degree between any two views based on the content feature sequence of each view and the user's gaze shift data. Construct a view semantic correlation degree matrix based on multiple view semantic correlation degrees as matrix elements. The view semantic correlation degree matrix is ​​used to characterize the strength of correlation between different views in terms of information content. S3: Collect and preprocess the user's gaze point data on the screen, and determine the current focus view based on the gaze point data. The gaze point data includes gaze position coordinates, gaze duration, gaze point trajectory, and saccade frequency. S4: Based on the current focused view, obtain the view semantic association degree between other views and the current focused view from the view semantic association degree matrix; S5: Dynamically generate the display weight of each view based on the semantic relevance, the content type of the view, and the gaze point data, and adjust the display parameters of each view according to each display weight; S6: Collect user gaze behavior feedback data after display parameter adjustment, input it into reinforcement learning model, train the model with deep Q network and optimize the display weight allocation strategy by maximizing cumulative discount reward, and then generate adjustment amounts online through the trained model to update the display weight of each view.

[0027] The detailed execution process for each step is as follows: S1: Obtain the content type of each view displayed on the screen and extract the content feature sequence of each view, specifically including the following steps: The system acquires real-time content data of each view in the current playback sequence of the digital screen. The playback sequence contains multiple frames or segments within a continuous time window. At the same time, it identifies the content type of each view. The content type can be determined based on the content type displayed on the screen. In this embodiment, it is set to three types, including dynamic video view, static text view, and map view. Temporal features are extracted from the content of each view in the current playback sequence to obtain a content feature sequence for each view. These temporal features include at least one of keyframe image features, scene transition points, motion vector changes, and text content changes. Different feature extraction methods are used for views with different content types, specifically: For dynamic video views, depth features of each frame of video image are extracted using a pre-trained ResNet50 network to form a temporal image feature sequence; motion vectors between adjacent frames are calculated using optical flow, and the mean and variance of the motion vectors are statistically analyzed to form a motion change feature sequence; scene switching points are identified using the inter-frame pixel difference thresholding method, and the timestamps of scene switching are recorded. For map views: obtain vector data of map views through geographic information system interfaces, extract features of road, landmark, and event point elements within the current display range, record the timestamps and content change features of map zooming, panning, and layer switching operations, and form a time-series feature sequence of map elements.

[0028] For static text views, OCR is used to extract the text content in the view, and a pre-trained BERT model is used to extract the semantic features of the text. The timestamps of text content updates and semantic change features are recorded to form a text semantic temporal feature sequence.

[0029] The content feature sequences of all views are length-aligned and L2-normalized to obtain standardized content feature sequences.

[0030] S2: Based on the content feature sequence of each view and the user's gaze shift data, calculate the view semantic correlation degree between any two views. Construct a view semantic correlation degree matrix using multiple view semantic correlation degrees as matrix elements. This matrix characterizes the strength of the correlation between different views in terms of information content. The view semantic correlation degree matrix M is an N×N square matrix, where N is the total number of views displayed on the screen, and the matrix elements are... This represents the semantic association degree between the i-th view and the j-th view, with a value ranging from 0 to 1. The larger the value, the stronger the association.

[0031] Figure 2 This is a schematic diagram illustrating the sub-steps for constructing a view semantic relevance matrix according to an embodiment of the present invention. Please refer to [link / reference]. Figure 2 Specifically, the semantic association matrix is ​​constructed using the following sub-steps: S201: Within a preset time window, calculate the dynamic semantic correlation between the content feature sequences of any two views. Figure 3 This is a schematic diagram of the sub-steps for calculating dynamic semantic relevance according to an embodiment of the present invention. Please refer to [link / reference]. Figure 3 As shown, the calculation of the dynamic semantic relevance degree also includes the following innovative sub-steps: (a) Hierarchical Semantic Association Decomposition: The view content feature sequence is decomposed into two parts: a global semantic basis vector and local temporal residuals. The global semantic basis vector is obtained through online incremental principal component analysis, reflecting the stable semantic association skeleton between views; the local temporal residuals are obtained through temporal difference calculation, reflecting the dynamic changes in the current playback sequence. The final dynamic semantic association degree is a weighted synthesis of the global basis vector association degree and the local residual association degree. The weight coefficient is adaptively adjusted according to the content change rate of the playback sequence. The higher the content change rate, the greater the weight of the local residual association degree.

[0032] (b) Causal Relationship Direction Measurement: When calculating the cross-correlation function of two views, the skewness coefficient and temporal delay direction consistency index of the cross-correlation function are further extracted to determine the causal direction of information flow between views, i.e., which view's content change leads the other view in time. The causal direction information is encoded as a directed correlation degree to construct a directed view semantic correlation graph instead of an undirected correlation matrix.

[0033] (c) Multi-scale time window fusion: Simultaneously using short time windows (2 seconds), medium time windows (10 seconds) and long time windows (60 seconds) to calculate the dynamic semantic relevance at three scales, and adaptively fusion through a multi-scale attention mechanism, so that the relevance can not only respond quickly to content changes, but also maintain the stability of long-term semantic relationships.

[0034] (d) Based on steps (a) to (c), calculate the cosine similarity of the content feature vectors of the two views at the same time, forming a temporal similarity sequence. Take the arithmetic mean of the similarity values ​​at all times in this temporal similarity sequence, and use the result as the temporal similarity mean. The formula for calculating the cosine similarity is: ; In the formula, Let be the cosine similarity between the i-th view and the j-th view at time t. Let be the content feature vectors of the i-th view and the j-th view at time t, respectively.

[0035] (e) Calculate the cross-correlation function of the feature sequences of the two views, and extract the cross-correlation peak of the two cross-correlation functions as the correlation degree of the time series response. The formula for calculating the cross-correlation function is as follows: ; In the formula, This represents the time delay, with a range of values. The cross-correlation value under time delay is taken as... The maximum value is taken as the correlation degree of the time series response, and N is the total length of the feature sequence within the time window.

[0036] (f) Obtain the number of times N occurs when two views simultaneously experience scene switching or content abrupt changes within the sliding time window. sync And count the number of content mutations N in each of the two views. i and N j Calculate N sync With N i and N j The smaller value in min(N) i， N j The ratio of the event synchronization correlation (Sync) is used as the event synchronization correlation degree. This ratio reflects the proportion of mutations in the view with fewer mutations that occur synchronously with the other view, thus fairly measuring the degree of synchronization between the two views in terms of content changes.

[0037] (g) The mean temporal similarity, temporal response correlation and event synchronization correlation are weighted and fused to obtain the dynamic semantic correlation between the two views in the current playback sequence. The value range of the dynamic semantic correlation is [0-1].

[0038] S202: Collect user gaze shift data in historical playback sequences, calculate the gaze shift probability matrix between views, and calculate the historical correlation based on user behavior through a collaborative filtering algorithm; S203: The dynamic semantic correlation degree and historical correlation degree are weighted and fused using an attention mechanism to obtain the view semantic correlation degree between two corresponding views. The view semantic correlation degree matrix is ​​constructed using the view semantic correlation degree between all pairs of views as matrix elements.

[0039] Furthermore, an attention mechanism is used to weightedly fuse dynamic semantic relevance and historical relevance to obtain view semantic relevance (matrix elements). The implementation process includes: Obtain the context features of the current playback sequence. The context features are the mean vector of the content features of all views within the current time window, which is used as the query vector q for the attention mechanism. Query vector q, dynamic semantic relevance and historical correlation As two value vectors input to the attention layer, the attention weights for dynamic semantic relevance and historical relevance are calculated. The formula for calculating the attention weights is as follows: ; In the formula, These are trainable weight matrices, and parameter optimization is achieved through pre-training. Used for normalizing weight values ​​to ensure .

[0040] The matrix elements (dynamic semantic relevance) of the view semantic relevance matrix are calculated based on the two attention weights. The calculation formula is as follows: ; Furthermore, this includes using an exponential decay weighted formula to update the view semantic relevance matrix in real time, including the following processes: Set the sliding time window length to L and the update step size to S. Recalculate the current time window after S time units. In-view semantic association matrix ; An exponentially decaying weighted method is used to fuse historical semantic relevance and currently calculated dynamic semantic relevance: ; In the formula, Let be the semantic association matrix of the view at time t. The semantic association matrix of the view is calculated for the current time window. As the attenuation factor, Let be the view semantic association matrix at time t-1.

[0041] S3: Collect and preprocess the user's gaze point data on the screen, and determine the current focus view based on the gaze point data. Collect the user's gaze point data through the infrared eye tracker integrated above the screen. The gaze point data includes gaze position coordinates, gaze duration, gaze point trajectory and eye saccade frequency. The gaze position coordinates are based on the upper left corner of the screen as the origin, with the positive X-axis pointing to the right and the positive Y-axis pointing downwards. Preprocessing of fixation data includes the following sub-steps: Wavelet transform is used to denoise the original gaze point coordinate sequence, remove high-frequency jitter signals, and retain the main trend of gaze point change. The denoised gaze coordinate sequence is smoothed using a Kalman filter to estimate the viewer’s true gaze position and correct coordinate offsets caused by head movements or device noise. The smoothed gaze point data is used to identify gaze points, and continuous gaze points with a gaze duration exceeding a preset threshold are extracted as valid gaze points, while intermediate points in the saccade process are filtered out.

[0042] Determining the current focus view based on the preprocessed foveation data includes the following sub-steps: S301: For the gaze position coordinates of effective gaze points, use the DBSCAN clustering algorithm or the mean-shift clustering algorithm to perform cluster analysis and identify dense regions of gaze points.

[0043] Preferably, the neighborhood radius of the DBSCAN algorithm is set to 30 pixels, the minimum number of neighborhood samples MinPts is set to 5, discrete noisy gaze points are eliminated, and at least one gaze point dense region is output. S302: Calculate the user's fixation duration, fixation trajectory, and saccade frequency in each dense fixation area, and generate a visual attention heatmap using kernel density estimation or a Gaussian mixture model.

[0044] Among them, saccade frequency is the number of saccades per unit time, and the formula for calculating kernel density estimation is as follows: ; In the formula, screen coordinates The kernel density value at a given location, i.e., the attention heat value, where n is the total number of effective fixations and h is the bandwidth coefficient. For Gaussian kernel function, Let be the coordinates of the i-th valid gaze point.

[0045] S303: Based on the peak region of the visual attention heatmap and the display area of ​​each view, perform position mapping and determine the current focus view through intersection-over-union (IoU) calculation. Specifically, this includes the following process: Extract the peak region with the highest heat value from the visual attention heatmap and obtain the minimum bounding rectangle of that peak region. Get each view in advance The outer matrix of the display area is Calculate each and The crossover ratio is calculated using the following formula: ; In the formula, For the i-th view The corresponding crossover / union ratio, For the pixel area of ​​the matrix region, the largest The corresponding view is determined as the currently focused view; if it is all calculated... If all values ​​are below the set threshold of 0.3, it is determined that there is no valid focus view at present, and the display parameters of each view remain unchanged.

[0046] S4: Based on the currently focused view, obtain the view semantic relevance between other views and the currently focused view from the view semantic relevance matrix. This includes the following sub-steps: S401: Index Positioning. Let M be the view semantic association matrix constructed in step S2, which is an N×N square matrix, where N is the total number of views displayed on the screen. If the current focus view determined in step S3 is the k-th view (k∈[1,N]), then extract all elements from the k-th row (or k-th column, since both are equivalent when the matrix is ​​symmetric) of matrix M to obtain the vector rk=[M]. k1 M k2 ,...,M kN ], where M ki This represents the semantic association between view k and view i, with a value ranging from [0,1]. A larger value indicates a stronger association. Since the association M between views themselves... kk It is always 1, so it will be excluded in subsequent processing.

[0047] S402: Relevance Ranking and Filtering for Vector r k All elements except the k-th element are sorted in descending order of their relevance values ​​to obtain a sorted relevance list. Based on a preset relevance threshold (e.g., 0.3), views with a relevance greater than this threshold are selected as the set of views significantly related to the currently focused view. The relevance value for each related view is recorded for use in generating the first weight factor vector later.

[0048] S403: Directed Relationship Processing (when causal relationship directionality measurement is enabled): If causal relationship directionality measurement was enabled in step S201 (i.e., a directed view semantic relationship graph was constructed), then the outgoing edge relationship degree of view k pointing to other views, and the incoming edge relationship degree of other views pointing to view k are extracted from the directed relationship matrix. The outgoing edge relationship degree represents the degree of influence of changes in the content of the focus view on other views, and the incoming edge relationship degree represents the degree of influence of changes in the content of other views on the focus view. The outgoing edge relationship degree and the incoming edge relationship degree are recorded separately for distinguishing between the dominant view and the following view during subsequent collaborative adjustments.

[0049] S404: Multi-scale correlation fusion (when multi-scale time window fusion is enabled). If multi-scale time window fusion was enabled in step S201, then the corresponding correlation vectors are extracted from the correlation matrices of the three scales (short window 2 seconds, medium window 10 seconds, long window 60 seconds) respectively, and weighted and fused by the multi-scale attention weights trained in step S201 to obtain the final comprehensive correlation vector for use in subsequent steps.

[0050] S405: Encapsulate the correlation information between the current focus view and other views obtained from the above processing (including correlation values, sorting results, filtered set of related views, directed correlation information, etc.) into structured data and pass it to the subsequent steps as the basis for generating the first weight factor vector.

[0051] S5: Dynamically generate the display weight of each view based on the dynamic semantic correlation, the content type of the view, and the gaze point data, and adjust the display parameters of each view according to each display weight.

[0052] The display weight of each view is dynamically generated based on dynamic semantic relevance, view content type, and gaze point data. Figure 4 This is a schematic diagram illustrating the sub-steps for generating display weights for each view according to an embodiment of the present invention. Please refer to [link / reference]. Figure 4 Specifically, it includes the following sub-steps: S501: Generate the first weight factor vector based on the semantic relevance between the current focus view and other views.

[0053] S502: Based on the content type of each view, determine the corresponding basic display weight range and generate a second weight factor vector. In this embodiment, basic display weight ranges are pre-set for views with different content types: the basic weight range for dynamic video views is [0.6, 1.0], the basic weight range for map views is [0.5, 0.9], and the basic weight range for static text views is [0.3, 0.7]. Based on the content type of each view, the median value of the corresponding range is taken as the second weight factor of that view.

[0054] S503: Based on fixation duration, fixation point trajectory, and saccade frequency, construct a fixation behavior feature vector, input it into a pre-trained attention concentration evaluation network, and generate a third weight factor vector. Specifically, extract three features: the cumulative fixation duration ratio, the dispersion of the fixation point trajectory, and the saccade frequency of the user in each view area, normalize them, and construct a 3-dimensional fixation behavior feature vector. Input this feature vector into the pre-trained attention concentration evaluation network, which is a 3-layer fully connected neural network with an input dimension of 3 and an output dimension of 1. The output value is the user's attention concentration score for the view, with a value range of [0,1], which serves as the third weight factor for the view.

[0055] The training details of the attention concentration assessment network are as follows: Training dataset construction: Gazing behavior data from multiple users in multi-view scenarios on a digital large screen were collected. For each view region, three features were extracted: cumulative gaze duration percentage, gaze trajectory dispersion, and saccade frequency, forming a 3D feature vector. Simultaneously, experts scored the user's attention concentration in each view region based on task completion, subjective questionnaire scores, and subsequent information recall accuracy. The scores ranged from [0,1] and served as supervision labels. The dataset contained no fewer than 5000 samples, divided into training, validation, and test sets in an 8:1:1 ratio.

[0056] Network Structure and Training Configuration: The attention concentration evaluation network employs a 3-layer fully connected neural network, with the number of neurons in each layer decreasing sequentially from 3 to 64 to 32 to 1. Except for the last layer, which uses the Sigmoid function to restrict the output to the [0,1] interval, all other hidden layers use the ReLU activation function. The loss function is the mean squared error (MSE), used to measure the difference between the predicted score and the expert-annotated score. The optimizer is Adam, with an initial learning rate of 0.001, a batch size of 32, and 200 training epochs. An early stopping strategy is used (training stops when the validation set loss does not decrease for 10 consecutive epochs) to prevent overfitting.

[0057] Regularization and Data Augmentation: To prevent overfitting, Dropout layers are added between fully connected layers with a dropout rate of 0.2. Simultaneously, small Gaussian noise (standard deviation 0.01) is applied to the input features during training to augment the data and improve the network's generalization ability.

[0058] Model Deployment and Online Adaptation: After training, the attention evaluation network is deployed to the edge computing unit of the digital large screen system. In actual use, the system can collect subjective feedback from users on the display adjustments (such as satisfaction ratings) as weak supervision signals, and use online gradient descent to fine-tune the network parameters, gradually adapting to the user's personalized gaze behavior pattern.

[0059] S504: Generate gaze sequence based on user gaze point data, input it into a pre-trained intent recognition model, output the user's potential intent type, and generate an intent feature vector based on the potential intent category.

[0060] The intent recognition model employs a hierarchical temporal attention network, and its innovative design includes the following sub-steps: (a) Spatiotemporal encoding of gaze sequence: The gaze point coordinate sequence, gaze duration sequence and saccade sequence of the user in the past 3 seconds are encoded by three parallel temporal encoders (bidirectional LSTM), and then the features of the three modalities are fused by a cross-modal attention mechanism to generate a comprehensive temporal representation of gaze behavior.

[0061] (b) Hierarchical Definition of Intent Types: Intent types are not limited to the static four-category classification, but rather a hierarchical intent tree is constructed. The top level consists of two main categories: "Information Acquisition" and "Monitoring and Observation." The bottom level is further subdivided into specific intents, such as "Detailed Inquiry of Congested Road Sections" and "Confirmation of Abnormal Device Status." The model first predicts the top-level category, then predicts the bottom-level subcategories, ensuring consistency in predictions through a hierarchical conditional random field. Weight preferences for each view are pre-set for different intent types, for example: Monitoring and alert intent: The weighting preference for dynamic video view is 1.0, map view is 0.8, chart view is 0.5, and text view is 0.3; Data query intent: Chart view weighting is 1.0, text view is 0.8, map view is 0.5, and video view is 0.3; Detailed viewing intent: The focus view has a weight preference of 1.0, and other views decrease in weight according to their relevance; Global browsing intent: The weight preference for all views is 0.6, with no significant difference; (c) Intent shift prediction: Based on the identification of the current intent, the intent shift prediction model is further trained based on the historical intent sequence to predict the user's intent change trend in the next 3 seconds, so that the adjustment of display weight is forward-looking rather than just responding to the gaze behavior that has already occurred.

[0062] Based on the potential intent types (including top-level major categories and bottom-level subcategories) output by the hierarchical intent recognition model and the future intent trends output by the intent transfer prediction model, the weight preference values ​​corresponding to each view are obtained comprehensively to generate an intent feature vector.

[0063] The training details of the intent recognition model are as follows: Training dataset construction: Gazing behavior data from multiple users in various digital large-screen multi-view scenarios was collected, including gaze point coordinate sequences, gaze duration sequences, saccade sequences, and corresponding screen content snapshots. Experts labeled user intent within each time window using hierarchical label pairs (top-level broad category label, bottom-level sub-category label), while also annotating the time points of intent transitions. The dataset was divided into training, validation, and test sets in an 8:1:1 ratio.

[0064] Training of the hierarchical intent recognition network: The hierarchical temporal attention network employs an end-to-end joint training approach. The loss function consists of three parts: Top-level classification loss: The cross-entropy loss function is used to optimize the classification accuracy of the top-level major categories (information acquisition type / monitoring and observation type); Low-level classification loss: The cross-entropy loss function is used to optimize the classification accuracy of the specific intent at the low level; Hierarchical consistency loss: The negative log-likelihood loss of the hierarchical conditional random field is used to penalize inconsistencies between the top-level and bottom-level predictions (e.g., the top-level prediction is "monitoring and observation" while the bottom-level prediction is "data query").

[0065] The total loss function is a weighted sum of the three factors, and the weight coefficients are determined through optimization on the validation set. During training, the Adam optimizer is used, with an initial learning rate of 0.001, a batch size of 64, and 100 training epochs. An early stopping strategy is employed to prevent overfitting.

[0066] Training of the Intent Transfer Prediction Model: The intent transfer prediction model employs an independent LSTM predictor. During training, historical intent sequences (N=5 time steps in length) are used as input features, and actual intent changes within the next 3 seconds are used as supervision labels. The loss function is cross-entropy loss, and the optimizer is also Adam, with a learning rate of 0.0005. After the hierarchical intent recognition network is trained, its parameters are fixed, and then the model is trained independently.

[0067] Model Deployment and Online Updates: After training, the hierarchical intent recognition network and intent transfer prediction model are deployed to the edge computing unit of the digital large screen system. During actual use, user gaze behavior data and corresponding system feedback (such as user manual view adjustments) are collected. Incremental learning is used to periodically fine-tune the model parameters to adapt to the personalized gaze behavior patterns of different users.

[0068] S505: Concatenate the first weight factor vector, the second weight factor vector, the third weight factor vector, and the intent feature vector to form a single-view fusion feature vector. By concatenating the first weight factor vector, the second weight factor vector, the third weight factor vector, and the intent feature vector corresponding to each view, a single-view fusion feature vector is formed. Combine all the single-view fusion feature vectors into a total fusion feature matrix.

[0069] S506: The fused feature vectors from multiple single images are input into a pre-trained multilayer perceptron (MLP) network. The training details of the MLP network are as follows: Training Dataset Construction: Complete interaction data of multiple users in a multi-view scenario on a digital large screen was collected, including content features of each view, user gaze behavior data, intent recognition results, etc. Single-view fusion feature vectors were generated for each view according to the processing flow from S501 to S505. The single-view fusion feature vectors of all views were combined into a total fusion feature matrix, which was used as the input to the MLP network. Simultaneously, experts labeled each view with an ideal attention score (range 0-1, and the sum of all view scores is 1) based on the user's actual information acquisition efficiency and subjective satisfaction, serving as supervision labels. The dataset contained no fewer than 8000 samples, divided into training, validation, and test sets in an 8:1:1 ratio.

[0070] Network Structure and Training Configuration: The MLP network consists of an input layer, two hidden layers, and an output layer. The input layer dimension is equal to the dimension of the fused feature vector from each view (assumed to be D). The first hidden layer has 128 neurons, and the second hidden layer has 64 neurons. ReLU is used as the activation function for both hidden layers. The output layer uses the Softmax activation function, and its output dimension is equal to the total number of views N. The output value is the attention score for each view. Cross-entropy loss is used as the loss function to measure the difference between the predicted attention score distribution and the ideal score distribution annotated by experts. The optimizer is Adam, with an initial learning rate of 0.0005, a batch size of 32, and 150 training epochs. An early stopping strategy is employed (training stops when the validation set loss does not decrease for 15 consecutive epochs) to prevent overfitting.

[0071] Regularization and Data Augmentation: To prevent overfitting, a Dropout layer is added between the two hidden layers with a dropout rate of 0.3. Simultaneously, small Gaussian noise (standard deviation 0.02) is applied to the input features during training to augment the data and improve the network's generalization ability. Furthermore, L2 weight decay regularization is employed with a coefficient of 1e-4.

[0072] Model Deployment and Online Adaptation: After training, the MLP network is deployed to the edge computing unit of the digital large screen system. In actual use, the system can construct weak supervision signals based on the user's implicit feedback on the display effect (such as changes in gaze behavior) or explicit feedback (such as manual view adjustment), and fine-tune the MLP network parameters using online learning to gradually adapt to the user's personalized preferences.

[0073] The attention scores of each view output by the MLP network output layer are used as the final display weights of each view. The display parameters of each view are adjusted according to each display weight. The display parameters include brightness, contrast, refresh rate, transparency, resolution, display level, and sharpening intensity.

[0074] When adjusting the display parameters of each view based on its display weight, a grouping and collaboration strategy driven by an associativity threshold is adopted, specifically including: (a) Grouping of related views: Based on the semantic relevance matrix of views, the current focus view and views with a relevance greater than 0.6 are divided into a strong relevance group, those with a relevance between 0.3 and 0.6 are divided into a weak relevance group, and those with a relevance less than 0.3 are divided into an independent group.

[0075] (b) Enhanced intra-group collaboration: All views in a strongly related group adopt a linked adjustment strategy—when the display weight of a view increases due to user gaze, the weights of other views in the group increase synchronously according to the degree of correlation, and the adjustment time is synchronized (delay less than 100ms), so that users can perceive the overall visual linkage effect when their gaze shifts.

[0076] (c) Intergroup differential inhibition: When the focal view is clear, the display weight of the independent group is suppressed to the lowest level, and the degree of inhibition is negatively correlated with the fixation time of the focal view—the longer the fixation time, the stronger the inhibition. This mechanism simulates the competitive inhibition phenomenon of human visual attention.

[0077] Based on the above group collaboration strategy, the display parameters of each view are adjusted according to the final display weight of each view, including the execution process: For views whose display weight is lower than the first preset threshold, adjust their display parameters to the preset minimum display level; For views whose display weight is higher than the second preset threshold, adjust their display parameters to the preset highest display level (the highest display level of the display parameter value corresponding to the view), and set the display level to the top layer for front display; For views whose display weight is greater than the first preset threshold but less than the second preset threshold, the display parameters are adjusted linearly according to the display weight. The calculation formula is as follows: ; In the formula, Display parameter values ​​for the target corresponding to this view. Displays the lowest and highest preset values ​​for the target parameter.

[0078] In this embodiment, the specific numerical ranges of each display parameter and the definitions of the minimum / maximum display levels are as follows: brightness range 0 to 255, minimum 30 and maximum 220; contrast range 0% to 100%, minimum 20% and maximum 90%; refresh rate range 30Hz to 144Hz, minimum 30Hz and maximum 120Hz; transparency range 0% to 100%, minimum 80% and maximum 0%; resolution range 50% to 100% of the original view resolution, minimum 50% and maximum 100%; display level range 0 to 10, minimum 0 and maximum 10; sharpening intensity range 0.0 to 1.0, minimum 0.0 and maximum 0.8.

[0079] At the same time, for the focus view of different content types, differentiated and refined display parameters are adjusted, and the adjustment rules are as follows: When the focus view is a dynamic video view, motion vector fields in the video are extracted using optical flow. A motion region mask is calculated, and the refresh rate and dynamic contrast are increased for the motion region, while the contrast is decreased for the static background region. The matching degree between the motion vector direction and the gaze point shift direction is calculated, and the enhancement level of the motion region is dynamically adjusted based on the matching degree. Specifically, the refresh rate of the motion region is adjusted to the highest display level (120Hz), and the contrast is adjusted to 80%~90%; the refresh rate of the static background region is reduced to the lowest display level (30Hz), and the contrast is reduced to 30%~40%. When the focus view is a static text view, the text lines are detected by combining the maximum stable extreme value region algorithm with a convolutional neural network to generate text region bounding boxes. Adaptive histogram equalization is used to enhance the contrast of the text region to 70%~90%, and a Gaussian low-pass filter is applied to the non-text region to soften the background, reducing the contrast to 20%~30%. The sharpening intensity is dynamically adjusted according to the relative position of the gaze point and the text region. The closer the gaze point is to the center of the text region, the higher the sharpening intensity (maximum 0.8), and vice versa (minimum 0.0). When the focus view is a map view, the vector data of the map view is obtained through the geographic information system interface. Based on the position of the gaze point, the road and landmark elements within the preset range around the gaze point are dynamically identified. The identified elements are highlighted (brightness is increased to 180~220, contrast is increased to 80%~90%), and the color saturation of elements in non-gaze areas is reduced (saturation is reduced to 30%~50%) and the brightness is reduced to 50~80.

[0080] The specific values ​​in the above differential adjustment rules are all within the range of the lowest / highest display level and are compatible with the linear scaling formula: when the view display weight is in the middle value, each parameter is further subdivided linearly within the sub-range specified by the differential adjustment rules.

[0081] S6: Collect user gaze behavior feedback data after display parameter adjustment, and calculate reinforcement learning reward value based on the feedback data; construct a reinforcement learning model using a deep Q-network, and optimize the display weight allocation strategy by maximizing the cumulative discount reward; use the trained model for online inference to obtain the weight adjustment amount, and update the display weight of each view in real time. The deep Q-network adopts the following innovative design: (a) Differentiated encoding of the state space: The display weight vectors of each current view and the graph structure encoding of the view semantic association matrix are used together as the state input. Specifically, the view semantic association matrix constructed in step S2 is input into the graph convolutional network for encoding, and the graph embedding features of the relationship between views are extracted. These graph embedding features are then concatenated with the display weight vectors of each current view to serve as the state representation of the deep Q-network. This allows the reinforcement learning agent to not only perceive the current weight allocation values ​​but also the semantic association topology between views, thus considering the intrinsic connections between views when adjusting weights.

[0082] (b) Action Space Constraint Design: Actions are not independent weight adjustments, but rather normalized adjustment vectors that satisfy the condition that the sum of the weights of all views is always 1. A Softmax action space design is adopted, whereby after the deep Q-network outputs the original adjustment scores for each view, it is converted into a probability distribution form of adjustment values ​​through a Softmax function. This ensures that the adjusted weights always satisfy the probability distribution constraint (the sum of the weights of all views is 1), avoiding invalid adjustments where all weights increase or decrease simultaneously, while also ensuring that the adjusted weights remain within a reasonable range.

[0083] (c) Multi-objective decomposition of the reward function: The reward function is decomposed into two sub-objectives: information acquisition efficiency reward and visual comfort reward. The information acquisition efficiency reward combines fixation dwell time and number of regressions, while the visual comfort reward combines pupil diameter change rate and fixation point shift speed. A multi-objective deep reinforcement learning framework is adopted, balancing the conflict between the two objectives through a Pareto optimal policy, rather than a simple weighted summation. Specifically, the reward value for each sub-objective is calculated separately, and the deep Q-network maintains two independent Q-heads, corresponding to the two sub-objectives respectively. The optimal action is selected through an action selection policy based on Pareto dominance. As a specific example of multi-objective reward decomposition, the information acquisition efficiency reward F... info and visual comfort reward F comfort They can be represented as:

[0084]

[0085] Among them, T n B is the normalized value of fixation duration. n P is the normalized value of the number of revisits. n S is the normalized value of the rate of change of pupil diameter. n λ1 represents the normalized value of the gaze shift speed, and λ4 represents the weight coefficients of each indicator, with a sum of 1. During training, the model automatically balances the two sub-objectives through Pareto front search to avoid over-optimization of a single objective.

[0086] Figure 5 This is a schematic diagram illustrating the sub-steps for calculating the reinforcement learning reward value according to an embodiment of the present invention. Please refer to [link / reference]. Figure 5 Specifically, the execution of step S6 includes the following sub-steps: S601: Collects feedback data on the viewer's gaze behavior after the display parameters are adjusted, including gaze duration, number of regressions, pupil diameter changes, and gaze point shift speed; S602: Concatenate the display weight vectors of each current view with the graph embedding features encoded by the graph convolutional network to form the current state vector; S603: Input the state vector into the deep Q network and output the Q value of each candidate action (each of the two sub-objectives has a Q value). Select the optimal action (i.e., weight adjustment) through Pareto dominance. S604: Execute the action, update the display weight of each view, collect user gaze behavior feedback data at the next moment, and calculate the reward value of the two sub-goals; S605: Store the experience tuple (state, action, reward, next state) into the experience replay pool, train a deep Q network using a fixed target network and an experience replay mechanism, and optimize the distribution strategy of the display weights by maximizing the cumulative discount reward; S606: Utilizes a trained deep Q-network to generate online adjustments to display weights and updates the display weights of each view in real time.

[0087] Furthermore, when multiple viewers are involved, collaborative processing of the multiple viewers' perspectives is required, including the following sub-steps: The eye tracker collects fixation data from each viewer in front of the screen, performs preprocessing steps, and generates a visual attention heatmap for each viewer. The visual attention heatmaps of each viewer are weighted and fused, with the weighting coefficients determined based on the viewer's identity priority or historical interaction activity. Based on the fused attention heatmap, calculate the group attention score for each view; The group attention score is used as an additional weighting factor, which, together with semantic relevance, view content type, and individual gaze point data, generates the display weight of each view. When multiple viewers focus on the same view, the display effect of that view is enhanced; when the focus is dispersed across different views, a split-screen optimization strategy is adopted to personalize the display of each view.

[0088] Furthermore, when the system is initialized or when a user uses it for the first time, the content features of the current playback sequence are collected, and the initial semantic relationships of each view are identified through a content feature clustering algorithm. During system initialization, the content features of the current playback sequence are collected, and the content features of each view are clustered using the K-means clustering algorithm to identify the initial semantic associations of each view and generate an initial semantic association matrix. A content-based hot start strategy is adopted, and the initial display weight is preset according to the similarity of the content types of the views. The initial relevance of the views with the same content type is set to 0.8, the initial relevance of the views with similar content types is set to 0.5, and the initial relevance of the views with large differences in content types is set to 0.2. After the user's gaze behavior data accumulates to a preset threshold, behavior-based collaborative filtering correlation is gradually introduced to correct the initial semantic correlation. By employing an online learning mechanism, the semantic association matrix and weight generation model are updated in real time, enabling a smooth transition from cold start to personalized customization.

[0089] This invention first extracts content feature sequences for different content types of views using differentiated temporal feature extraction methods. It then integrates content temporal semantic correlation with user historical gaze behavior to construct a dynamically updatable view semantic correlation matrix, achieving precise quantitative representation of the information correlation between different views. Next, through cluster analysis of user gaze point data, generation of visual attention heatmaps, and region intersection-union matching, it accurately identifies the user's current focus view and obtains related view information based on the view semantic correlation matrix, breaking the limitation of existing technologies that can only unidirectionally adjust the gaze view. Simultaneously, by integrating view semantic correlation, content type, user gaze behavior characteristics, and potential viewing intentions into a multi-dimensional display weight generation system, it replaces the fixed gaze point-display parameter mapping rule in existing technologies, achieving context-aware differentiated display adjustments that align with actual user needs. Finally, by constructing a self-optimizing closed loop through a reinforcement learning mechanism based on user gaze behavior feedback, it achieves continuous iterative optimization of the display weight allocation strategy, completing a deep match between multi-view display effects and user information acquisition needs.

[0090] Meanwhile, for different content types such as dynamic videos, static text, and maps, differentiated and refined display parameter adjustment strategies were designed to achieve targeted enhancement of core information areas and appropriate weakening of non-critical areas within the view. In addition, a deep Q-network reinforcement learning mechanism based on user gaze behavior feedback was introduced, enabling the system to upgrade from a passive response to fixed rules to a personalized adaptive adjustment that can adapt to different users and different scenarios, thus achieving continuous self-optimization of the display strategy.

[0091] Furthermore, in terms of user experience, by enhancing the collaboration of views strongly associated with the focus view and appropriately weakening views weakly associated with it, the visual search cost and cognitive load for users to obtain cross-view information in multi-view scenarios are significantly reduced. The refined display adjustments for different content types effectively improve the user's visual comfort while highlighting core information, and significantly improve the efficiency of information acquisition for users in complex business scenarios. In terms of system performance, the adaptive adjustment based on user intent and the reinforcement learning self-optimization closed loop solve the problem of poor adaptability of fixed rules in existing technologies.

[0092] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for multi-view collaborative display and interaction on a digital large screen based on visual focus tracking, characterized in that, Includes the following steps: Based on the acquired sequence of content features of each view and user gaze shift data, the dynamic semantic correlation degree and the historical correlation degree based on user behavior between any two views are calculated. The two are then weighted and fused to construct a view semantic correlation degree matrix, and the view semantic correlation degree matrix is ​​updated in real time. The construction of the view semantic association matrix includes the following sub-steps: Within a preset time window, calculate the dynamic semantic correlation between the content feature sequences of any two views; Collect user gaze shift data in historical playback sequences, calculate the gaze shift probability matrix between views, and calculate the historical correlation based on user behavior using a collaborative filtering algorithm; The dynamic semantic correlation and historical correlation are weighted and fused using an attention mechanism to obtain the view semantic correlation between the two corresponding views. The view semantic correlation matrix is ​​constructed using all view semantic correlations as matrix elements. Calculating the dynamic semantic correlation between any two view content feature sequences includes the following steps: The cosine similarity of the content feature vectors of two views is calculated to form a temporal similarity sequence, and the mean of the temporal similarity sequence is taken as the temporal similarity mean. Calculate the cross-correlation function of the content feature sequences of the two views, and extract the cross-correlation peak of the two cross-correlation functions as the correlation degree of the time series response; Get the number of times that two views simultaneously experience scene switching or content abrupt changes within a sliding time window, and calculate the event synchronization correlation. The mean temporal similarity, temporal response correlation, and event synchronization correlation are weighted and fused to obtain the dynamic semantic correlation between the two views in the current playback sequence. A visual attention heatmap is generated based on the collected user gaze point data. The current focus view is determined by mapping the view to the position of each view display area and calculating the intersection-union ratio. Based on the current focused view, obtain the view semantic association between other views and the focused view from the view semantic association matrix; By integrating the semantic relevance of views, the content type of views, user gaze behavior characteristics, and potential viewing intentions, the display weight of each view is dynamically generated, and the display parameters of each view are adjusted according to the display weight. A reinforcement learning model is constructed using a deep Q-network. The current display weight is taken as the state and the weight adjustment amount is taken as the action. The multi-objective reward value is calculated based on the user gaze behavior feedback data after the display parameters are adjusted. The display weight allocation strategy is optimized by maximizing the cumulative discount reward, so as to realize the online update of the display weight.

2. The method for multi-view collaborative display and interaction of digital large screens based on visual focus tracking according to claim 1, characterized in that, It also includes updating the view semantic relevance matrix in real time using an exponential decay weighting method, with the fusion formula being: In the formula, Let be the semantic association matrix of the view at time t. The semantic association matrix of the view is calculated for the current time window. As the attenuation factor, Let be the semantic association matrix of the view at time t-1.

3. The method for multi-view collaborative display and interaction of a digital large screen based on visual focus tracking according to claim 1, characterized in that, Determining the current focus view includes the following sub-steps: Cluster analysis was performed using the DBSCAN clustering algorithm on the gaze position coordinates to identify at least one dense region of gaze points. Calculate the user's fixation duration, fixation trajectory, and saccade frequency in each fixation dense region, and generate a visual attention heatmap using kernel density estimation; Position mapping is performed based on the peak region of the visual attention heatmap and the display area of ​​each view, and the current focus view is determined by cross-union ratio calculation.

4. The method for multi-view collaborative display and interaction of a digital large screen based on visual focus tracking according to claim 3, characterized in that, The current focus view is determined by calculating the intersection-union ratio, and the calculation formula is as follows: In the formula, Let i be the intersection-union ratio corresponding to the i-th view. The pixel area of ​​the matrix region. Let be the outermost matrix of the display area of ​​the i-th view. This is the minimum outer matrix corresponding to the peak region with the highest heat value extracted from the visual attention heatmap.

5. The method for multi-view collaborative display and interaction of a digital large screen based on visual focus tracking according to claim 1, characterized in that, The process of fusing view semantic relevance, view content type, user gaze behavior characteristics, and potential viewing intent to dynamically generate the display weight of each view includes the following sub-steps: Generate a first weight factor vector based on the semantic relevance between the current focused view and other views; Based on the content type of each view, determine the corresponding basic display weight range and generate a second weight factor vector; A gaze behavior feature vector is constructed based on the user's gaze duration, gaze point trajectory, and saccade frequency in each view area. The gaze behavior feature vector is then input into a pre-trained attention concentration evaluation network to generate a third weight factor vector. A gaze sequence is generated based on the user's gaze point data, input into a pre-trained hierarchical intent recognition model, the user's potential intent type is output, and an intent feature vector is generated based on the potential intent type. The first weight factor vector, the second weight factor vector, the third weight factor vector, and the intent feature vector are concatenated to form a single-view fusion feature vector; The multiple single-view fused feature vectors are input into a pre-trained multilayer perceptron network, and the attention score of each view is output as the final display weight of each view.

6. The method for multi-view collaborative display and interaction of a digital large screen based on visual focus tracking according to claim 5, characterized in that, This includes generating an intent feature vector by obtaining the weight preference value corresponding to each view based on the weight preference set for different potential intent types and the output potential intent type; It also includes predicting intent shift trends based on historical intent sequences, and obtaining weight preference values ​​for each view by combining the potential intent types and intent shift trends.

7. The method for multi-view collaborative display and interaction of a digital large screen based on visual focus tracking according to claim 1, characterized in that, The display parameters of each view are adjusted according to its display weight, and a grouping and collaboration strategy driven by the relevance threshold is adopted, including: Based on the view semantic relevance matrix, the current focus view and views with a relevance greater than the first preset value are divided into a strong relevance group, the relevance between the first preset value and the second preset value is divided into a weak relevance group, and the relevance less than the second preset value is divided into an independent group. For views within strongly related groups, a linked adjustment strategy is adopted: when the display weight of the focus view is increased, the weights of other views in the group are increased synchronously according to the degree of relevance; for views within weakly related groups, the weights are increased according to a preset percentage of the degree of relevance; for views in independent groups, the display weight is suppressed to the lowest level when the focus view is clearly defined. Based on the weights determined by the above group collaboration strategy, specific display parameters are adjusted.

8. The method for multi-view collaborative display and interaction of a digital large screen based on visual focus tracking according to claim 1, characterized in that, The calculation of the multi-objective reward value includes the information acquisition efficiency reward F of the deep Q-network. info and visual comfort reward F comfort The calculation formulas are expressed as follows: Among them, T n B is the normalized value of fixation duration. n P is the normalized value of the number of revisits. n S is the normalized value of the rate of change of pupil diameter. n λ1~λ4 are the normalized values ​​of gaze shift speed, and the weight coefficients of each index are 1.

Images