Visual attention prediction method oriented to man-machine cooperation and task driving

Abstract

The invention relates to the technical field of man-machine interaction and computer vision, and discloses a visual attention prediction method oriented to man-machine cooperation and task driving. The method is realized by constructing a multi-layer mapping structure in which a visual scene analysis layer, an operation intention decoding layer and an attention generation layer are coupled. According to the method, a task instruction and a visual flow are fused to generate a structured visual field, meanwhile, real-time physical operation of a user and a staring behavior flow are accessed to carry out cross-modal intention alignment, and a potential observation intention path is analyzed. And carrying out iterative focus deduction by combining attention transfer priori knowledge extracted from a historical cooperation mode, and generating a dynamic attention prediction map. According to the method, the model parameters are dynamically optimized by verifying the loop in real time, so that the accuracy and the time sequence continuity of predicting the visual attention transfer path of the operator in a complex collaborative task are improved.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and computer vision technology, specifically to a visual attention prediction method for human-computer collaboration and task-driven approaches. Background Technology

[0002] Existing visual attention prediction techniques are mostly based on deep neural networks, directly learning and generating saliency maps from the input visual scene. After introducing task instructions, mainstream methods typically encode the task text into vectors and fuse or perform cross-attention calculations with image features to drive the model to focus on task-related regions. These methods mainly rely on static image content and abstract task descriptions; their prediction process is a statistical fit of the task-visual association in the training data, lacking an understanding of the dynamic behavioral logic during task execution.

[0003] The drawback of existing technical solutions lies in their general neglect of the inherent coupling between the operator's real-time physical interaction behavior and visual attention in human-computer collaboration scenarios. The shift in visual attention is not only guided by the task objective but also closely related to the current specific operational steps, gesture pointing, and tool grasping state. Relying solely on visual and textual modalities, the system struggles to accurately infer the potential target the operator will be observing to achieve the next action. Furthermore, the temporal patterns of attention shifts are personalized and task-specific. Conventional methods lack the ability to model these complex shift patterns using long-term collaboration history, resulting in predicted attention paths that may conform to task semantics but not to actual operational habits and consistency.

[0004] The problem this invention aims to solve is how to introduce real-time physical operation behavior of users as a key modality into the prediction model to decode the task intent path behind it, and how to use the attention transfer patterns formed in historical collaboration as prior knowledge to generate dynamic attention predictions that are more in line with human operation logic and have task coherence. Summary of the Invention

[0005] The purpose of this invention is to provide a visual attention prediction method for human-computer collaboration and task-driven approaches, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, this invention provides a visual attention prediction method for human-computer collaboration and task-driven approaches, the method comprising: A multi-layer mapping structure that includes task perception and behavior coupling is constructed. The multi-layer mapping structure is composed of a visual scene parsing layer, an operation intention decoding layer, and an attention generation layer coupled sequentially. In the visual scene parsing layer, the input raw visual stream is fused with the current task instructions to generate a structured visual field under task constraints. In the operation intent decoding layer, the user's real-time operation behavior flow is synchronously accessed. The operation behavior flow includes physical operation vectors and gaze point sequences. The structured visual field and the operation behavior flow are aligned across modal intents to parse out the task-driven potential observation intent path. In the attention generation layer, based on the potential observation intention path, iterative attention focus deduction is performed in the structured visual field. The iterative attention focus deduction process introduces prior knowledge of attention transfer extracted from historical collaboration patterns, and finally synthesizes a dynamic attention prediction map for the current human-computer collaboration task. A real-time prediction performance verification loop is established, and the dynamic attention prediction map is deeply compared with the actual attention distribution obtained through independent perception channels. Based on the error vector generated by the comparison, the coupling weight parameters in the multi-layer mapping structure are dynamically adjusted.

[0007] Preferably, the step of generating a structured visual field under task constraints specifically includes: Perform multi-granularity visual entity segmentation on the input raw visual stream to identify the basic visual elements and their spatial relationships in the scene, thus forming the original visual field; At the same time, the current task instruction is semantically deconstructed to extract the target object set, operation action sequence and spatial constraints in the task instruction to form a task semantic framework; The original visual field and the task semantic framework are fed into a bidirectional matching network, which performs reverse retrieval from the task target to the visual entity and forward projection from the salient visual entity to the possible task target. Through the calculation of the bidirectional matching network, task relevance weights and expected interaction probabilities are labeled for each visual element in the original visual field, generating a structured visual field with attached task semantic labels and spatial topological relationships. The structured visual field is a dynamically updated data structure.

[0008] Preferably, the step of parsing the task-driven potential observation intent path includes: Continuously receive and cache the user's real-time operation behavior stream, and separate the physical operation vector representing limb movements and the gaze point sequence representing visual gaze points from the operation behavior stream; Within the operation intent decoding layer, a time-varying behavior-intent association model is established to jointly analyze the change pattern of the physical operation vector and the transition pattern of the gaze point sequence in order to infer the user's real-time operation goal and cognitive state. The inferred immediate operational goals and cognitive states are fused with the task semantic tags from the structured visual field to identify the sequence of visual objects and the order of observation that the user is most likely to observe in subsequent time segments in order to achieve the current task instruction. The sequence of most likely visual objects to be observed and the order of observation are formally defined as a task-driven potential observation intention path and passed as an intermediate output to the attention generation layer.

[0009] Preferably, the step of performing iterative attention focus deduction is specifically implemented as follows: The attention generation layer maintains an updatable attention state matrix, which is initialized with the baseline attention values ​​of all visual elements in the structured visual field. Visual objects to be observed are extracted sequentially from the potential observation intention path as the current inference target. The probability distribution of shifting attention from the current focus to the current inference target is calculated based on the prior knowledge of attention shift. The prior knowledge of attention shift encodes the statistical regularity of observation pattern shift in the history of similar tasks. The attention state matrix is ​​updated based on the calculated probability distribution, the process of shifting attention focus is simulated, and the visual object that has completed the shift is taken as the new attention focus. This process is repeated until the potential observation intention path has been traversed. In each iteration of the deduction, the attention competition mechanism from spatially adjacent visual elements in the structured visual field is considered simultaneously to correct the probability distribution; Finally, the attention state matrix, after all iterations and updates, is transformed into a dynamic attention prediction graph that can be visualized.

[0010] Preferably, establishing a real-time predictive performance verification loop includes the following interactive steps: Through a perception channel independent of the multi-layer mapping structure, such as a high-precision eye tracker, the actual visual gaze point data of the user when performing the same task is collected synchronously and processed into an actual attention distribution map. A differential comparison unit is set up to receive the dynamic attention prediction map from the attention generation layer and the actual attention distribution map from the independent perception channel. The difference between the two maps is measured at the pixel level or the region level, and a multidimensional error vector is generated. The multidimensional error vector represents the deviation of the prediction in terms of spatial distribution, intensity and temporal accumulation. The multidimensional error vector is fed back to the internal coupling nodes of the multi-layer mapping structure, specifically including the information fusion node between the visual scene parsing layer and the operation intent decoding layer, and the path generation node between the operation intent decoding layer and the attention generation layer; At the coupling node, the weight coefficients and threshold parameters of cross-layer information transmission are dynamically adjusted according to the direction and magnitude of each component of the error vector, so that the multi-layer mapping structure can be adaptively fine-tuned based on real-time performance feedback to reduce the error of subsequent predictions.

[0011] Preferably, the pixel-level or region-level difference measurement of the two images employs a hierarchical difference fusion strategy, specifically including: At the pixel level, the difference in attention intensity at corresponding coordinate points between the dynamic attention prediction map and the actual attention distribution map is calculated to form the original difference field; Based on the task-related regions defined in the structured visual field, the original difference field is divided into multiple sub-regions. The mean, variance, and spatial gradient of the differences are calculated in each sub-region to form a region difference descriptor. The pixel-level original difference field and the region-level difference descriptor are concatenated, and then dimensionality reduction and fusion are performed through a fully connected network to output a compact multidimensional error vector. The multidimensional error vector retains both detailed difference and macro-regional difference information.

[0012] Preferably, the time-varying behavior-intent association model is implemented through a recurrent neural network, and its specific workflow is as follows: Align and concatenate the physical operation vectors and gaze point sequences on the time axis to form a multimodal behavior time stream, which serves as the input to the recurrent neural network. The hidden state of a recurrent neural network is used to represent the user's potential intention state as it evolves over time. The network receives new behavioral data at each time step and updates its hidden state accordingly. The goal of training the network is to predict the user's action target or gaze point location at the next moment based on the historical behavior sequence, so that its hidden state can effectively encode intent information. In online applications, the behavioral time sequence stream within the current and a short historical time window is input into the recurrent neural network, and the hidden state at the final moment is taken as the inference representation of the user's real-time operation goal and cognitive state.

[0013] Preferably, the attention transfer prior knowledge is encapsulated and invoked through a probability transfer matrix library, and the specific construction and usage methods are as follows: A large number of successful task execution instances were extracted from historical human-computer collaboration task data. Each instance contains the actual visual gaze shift sequence of the user during the task. Statistical modeling is performed on all gaze shift sequences to analyze the frequency and conditional probability of shifting from one type of visual element to another, forming multiple probability shift matrices associated with different task types or sub-task stages, thus creating a probability shift matrix library. During the inference process of the attention generation layer, one or more probability transition matrices with the highest matching degree are retrieved from the probability transition matrix library according to the type of the current task instruction and the steps already executed. The retrieved probability transition matrix is ​​used as prior knowledge of attention transition to calculate the transition probability from the current attention focus to the next target in the potential observation intention path. The values ​​in the probability transition matrix are used as prior probabilities in the calculation.

[0014] Preferably, the method further includes an offline optimization phase, which periodically enhances the multi-layer mapping structure using accumulated collaborative task data, including the following steps: Collect data on all executed task instances within a cycle. Each instance contains the original visual flow, task instructions, user action flow, and the actual attention distribution map obtained through independent perceptual channels. Using the task instance data as the training set, with the optimization objective of maximizing the overall similarity between the predicted dynamic attention map and the actual attention distribution map, end-to-end global optimization is performed on all trainable parameters in the multi-layer mapping structure. After optimization, the updated parameters are deployed to the online multi-layer mapping structure to replace the original parameters, thereby achieving a gradual performance improvement of the prediction model during long-term task execution.

[0015] Preferably, the offline optimization phase employs a course learning strategy to organize the training set, specifically: Based on the complexity of the task instructions and the user's proficiency in operation, the collected task instance data is graded by difficulty to form multiple sets of training data from easy to difficult. During the global optimization process, the multi-layer mapping structure is first trained using the set of training data with the lowest difficulty, so that it can master the basic task-attention mapping relationship. By gradually introducing more challenging training data sets and further optimizing the parameters obtained from previous training, the model is forced to learn to handle more complex and ambiguous visual scenes and operational intentions. This progressive learning strategy aims to improve the generalization ability and prediction robustness of the multi-layer mapping structure under tasks of varying difficulty.

[0016] Compared with the prior art, the beneficial effects of the present invention are: By using the user's real-time physical operation vectors and gaze point sequences as key inputs and aligning them cross-modally with the task-structured visual field, the system directly resolves the task-driven potential observation intention path. This transforms discrete operational behaviors into continuous visual exploration plans, enabling the system to understand the next visual requirement implied by the current action. This allows for causal inference of attention shifts, rather than simply predicting correlations based on visual features. This makes the prediction results more purposeful and logically forward-looking, enabling more accurate prediction of the key areas the user is about to gaze at in dynamic collaborative tasks.

[0017] During attention generation, iterative focus deduction is performed based on the parsed intent path, incorporating prior knowledge of attention transfer extracted from historical collaborative data. This mechanism ensures that the evolution of each focus is driven not only by the current instantaneous state but also follows the transfer patterns formed over long-term collaboration. The resulting dynamic prediction map exhibits temporal coherence and task logic that align with human operational habits, avoiding ineffective jumps in focus between semantically related but operationally irrelevant areas. This makes the prediction path more closely resemble the efficient and natural visual scanning rhythm of experienced operators performing tasks.

[0018] A real-time prediction performance verification loop is established, dynamically adjusting the internal coupling weights of the model by comparing the error between the prediction results and the actual attention distribution. This closed-loop mechanism enables the system to adapt online to different user operating styles and task variations, continuously optimizing the accuracy of intent decoding and focus inference. The entire multi-layer mapping structure thus possesses the ability to self-evolve in real-world collaborative scenarios, continuously improving the prediction model's generalization performance to new task situations and individual differences. Attached Figure Description

[0019] Figure 1 This is a schematic diagram illustrating the working principle of the visual attention prediction method for human-computer collaboration and task-driven approaches described in this invention. Figure 2 A flowchart for generating a structured visual field under task constraints; Figure 3 A flowchart for parsing the potential observation intent path driven by the task; Figure 4 A regional difference measurement analysis diagram for human-computer collaborative visual attention prediction; Figure 5 A radar chart showing the multi-dimensional performance of a visual attention prediction model in human-machine collaborative tasks. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1This invention provides a visual attention prediction method for human-computer collaboration and task-driven tasks. The method includes a multi-layer mapping structure coupling task perception and behavior, consisting of a visual scene analysis layer, an operation intent decoding layer, and an attention generation layer sequentially coupled. In the visual scene analysis layer, the input raw visual stream is fused with the current task instructions to generate a structured visual field under task constraints. In the operation intent decoding layer, the user's real-time operation behavior stream, containing physical operation vectors and gaze point sequences, is synchronously accessed. The structured visual field and the operation behavior stream are aligned across modal intentions to resolve the potential observation intent path driven by the task. In the attention generation layer, based on the potential observation intent path, iterative attention focus deduction is performed within the structured visual field. This deduction process incorporates prior knowledge of attention transfer extracted from historical collaboration patterns, ultimately synthesizing a dynamic attention prediction map for the current human-computer collaboration task. A real-time prediction performance verification loop is established, performing a deep comparison between the dynamic attention prediction map and the actual attention distribution obtained through independent perception channels. Based on the error vector generated by the comparison, the coupling weight parameters in the multi-layer mapping structure are dynamically adjusted.

[0022] In one embodiment of the present invention, see [reference] Figure 2 In the visual scene parsing layer, the specific operation for generating a structured visual field under task constraints involves performing multi-granularity visual entity segmentation on the input raw visual stream to identify the basic visual elements and their spatial relationships in the scene, thus constructing the raw visual field. Simultaneously, the current task instruction is semantically deconstructed, extracting the target object set, operation sequence, and spatial constraints to form a task semantic framework. The raw visual field and the task semantic framework are then fed into a bidirectional matching network, which performs reverse retrieval from the task target to visual entities and forward projection from salient visual entities to the task target. Through the computation of the bidirectional matching network, each visual element in the raw visual field is labeled with task relevance weights and expected interaction probabilities, generating a structured visual field with task semantic labels and spatial topological relationships. This structured visual field is a dynamically updated data structure.

[0023] In specific implementations, the visual scene parsing layer receives the raw visual stream from the camera device and the current task instruction text from the task planning module. The raw visual stream is input into a pre-trained multi-granularity visual entity segmentation model. This model identifies all basic visual elements in the scene, such as robotic arm ends, screws, tool trays, and assembly panels, through convolutional and region proposal networks. Simultaneously, it analyzes the spatial positions and inclusion relationships between these basic visual elements using graph neural networks, constructing a raw visual field containing a list of elements and a relationship graph. In some embodiments, the semantic deconstruction of the current task instruction is performed by a lightweight language model. This language model parses the instruction text, using noun phrases as a set of target objects, verb phrases as sequences of operational actions, and prepositional phrases as spatial constraints. All extracted semantic components are organized into a hierarchical task semantic framework.

[0024] In implementation, the original visual field and the task semantic framework are fed into a bidirectional matching network, which consists of two parallel attention sub-networks. The first attention sub-network receives the task semantic framework as a query, using basic visual elements from the original visual field as keys and values, and performs a reverse retrieval from the task target to the visual entity, calculating the semantic relevance score of each basic visual element relative to the task target. The second attention sub-network receives salient visual elements from the original visual field as queries, using semantic components from the task semantic framework as keys and values, and performs a forward projection from salient visual entities to the task target, calculating the probability that each task semantic component is activated by the current visual scene. The bidirectional matching network integrates the outputs of the two sub-networks through a fusion layer, fusing the calculations to generate task relevance weights and expected interaction probabilities for each visual element in the original visual field. For example, in a task of "picking up a red screw with a robotic arm," the red screw in the scene will receive a high task relevance weight and a high expected interaction probability, while the toolbox in the background will receive a lower weight and probability.

[0025] In some embodiments, the computation process of the bidirectional matching network annotating each visual element can be formally represented by a matching function. This matching function takes into account both semantic matching degree and spatial suitability, and its expression is as follows: in: Indicates the first A composite scalar of the task relevance weights of each visual element and the expected interaction probability. It is a normalization function. Representing visual elements With task semantic framework The semantic matching degree is obtained by calculating the weighted sum of the cosine similarities between the visual element category and the embedding vectors of the task target object and the operation action. Representing visual elements Compared to task semantic framework Medium space constraints and current operation context Spatial suitability score. Coefficient and It is a learnable balancing parameter used to adjust the contribution ratio of semantic and spatial factors.

[0026] Understandably, through the computation of the bidirectional matching network, the original visual field is transformed into a structured visual field with accompanying task semantic labels and spatial topological relationships. Each visual element node in the structured visual field carries its original visual features, spatial coordinates, task relevance weights, expected interaction probabilities, and links pointing to related task semantic components. Optionally, the structured visual field is designed as a dynamically updated data structure. When a new visual frame is input or a sub-step of the task instruction changes, the bidirectional matching network recalculates and updates the annotation information of all visual elements, ensuring that the structured visual field is synchronized with the current task state in real time.

[0027] In one embodiment of the present invention, see [reference] Figure 3 In the operation intent decoding layer, the steps for parsing the task-driven potential observation intent path include continuously receiving and caching the user's real-time operation behavior stream, and separating the physical operation vectors representing limb movements and the gaze point sequence representing visual fixation points from the operation behavior stream. Inside the operation intent decoding layer, a time-varying behavior-intent association model is established, implemented through a recurrent neural network. The physical operation vectors and gaze point sequences are aligned and concatenated on the time axis to form a multimodal behavior time-series stream, which serves as the input to the recurrent neural network. The hidden state of the recurrent neural network represents the user's potential intent state evolving over time. The network receives new behavior data at each time step and updates its hidden state. The training objective of the network is to predict the user's operation target or gaze point position at the next moment based on historical behavior sequences, thereby enabling its hidden state to effectively encode intent information. In online applications, the behavior time-series stream within the current and a short historical time window is input into the recurrent neural network, and the hidden state at the final moment is used as the inferred representation of the user's immediate operation target and cognitive state. The inferred immediate operational goal and cognitive state are fused with task semantic tags from the structured visual field to identify the sequence and order of visual objects observed by the user in subsequent time segments to achieve the current task instruction. This sequence and order of observed visual objects is formally defined as a task-driven potential observation intent path and is passed as an intermediate output to the attention generation layer.

[0028] In its implementation, the operation intent decoding layer continuously receives and caches real-time user operation behavior streams from the data glove and eye tracker. These streams include physical operation vectors representing hand movement and grip posture, and a gaze point sequence representing the screen coordinates of the visual fixation point. A time-varying behavior-intent association model is built within the operation intent decoding layer using a Long Short-Term Memory (LSTM) recurrent neural network (RNN). The input units of the LSM RNN are designed to receive multimodal features. Specifically, the time-varying behavior-intent association model is implemented using an LSM RNN, whose input units are designed to receive multimodal features. These multimodal features are constructed by strictly aligning and concatenating the physical operation vectors and gaze point sequences along the time axis. The physical operation vectors include the user's three-dimensional spatial position, movement speed, and joint angles of the hand, while the gaze point sequence covers the two-dimensional screen coordinates and dwell time of the visual fixation point. The aligned multimodal behavior time-series serves as the unified input to the LSM RNN. Long Short-Term Memory (LSTM) recurrent neural networks update their hidden states at each time step to represent the user's evolving potential intentions over time. During training, the network predicts the next action target or gaze point position based on historical behavior sequences, enabling the hidden states to effectively encode intention information. In online applications, the behavioral time-series streams within the current and a short historical time window are input into the trained LSM recurrent neural network. The hidden state at the final moment is used as an inference representation of the user's immediate action target and cognitive state. This inference result is then fused with the task semantic labeling of the structured visual field to resolve the task-driven potential observation intention path. In some embodiments, the physical operation vectors and gaze point sequences are strictly aligned and concatenated on the time axis before being input into the LSM recurrent neural network. The physical operation vectors include three-dimensional spatial position, velocity, and joint angles, while the gaze point sequences include two-dimensional screen coordinates and dwell time. After alignment and concatenation, a unified multimodal behavioral time-series stream is formed.

[0029] In practical implementation, the hidden states of a Long Short-Term Memory (LSTM) recurrent neural network (RNN) are used to represent the user's potential intention state evolving over time. At each time step, the LSM RNN receives new multimodal behavioral time-series data points and updates its hidden states according to its internal gating mechanism. The training objective of the LSM RNN is to predict the user's target category or gaze point location coordinates at the next moment based on historical behavior sequences. This training objective enables the hidden states of the LSM RNN to effectively encode dynamic information related to task intent. The hidden state update process of the LSM RNN can be represented as follows: in: Indicates at time step The updated hidden state vector is a continuous encoding of the user's immediate action goals and cognitive state. Indicates at time step The input is multimodal behavioral time-series data, namely the aligned and concatenated physical operation vectors and gaze point sequence features. This represents the hidden state of the previous time step. This represents all trainable parameters of a Long Short-Term Memory Recurrent Neural Network, including the weights and biases of the input gate, forget gate, output gate, and cell state. This represents the forward computation function for the Long Short-Term Memory unit.

[0030] Understandably, during the online application phase, the operation intent decoding layer inputs the multimodal behavior time-series stream from the current moment and a short historical time window into a pre-trained Long Short-Term Memory (LSTM) recurrent neural network (RNN). The hidden state of the LSM RNN at the final moment is used as the inferred representation of the user's immediate operation goal and cognitive state. In some embodiments, the length of this short time window is set to match the duration of typical task sub-steps, such as data from the most recent 2 seconds, to balance the real-time nature of intent inference with the amount of historical contextual information. The operation intent decoding layer then fuses the immediate operation goal and cognitive state represented by the hidden state vector output by the LSM RNN with the task semantic labels from the structured visual field of the visual scene parsing layer. This fusion is performed through a fully connected network that concatenates and maps the hidden state vector with the task semantic label vectors of relevant visual elements in the structured visual field, thereby identifying the sequence of visual objects and the order of observation observed by the user in subsequent time segments to achieve the current task instruction. Optionally, the sequence of visual objects identified can be directly selected from the structured visual field according to their task relevance weights and spatial topological relationships, while the observation order is sorted by combining the operational stage information implied in the hidden state vector. For example, in a "grab and place" task, when the hidden state inference of the long short-term memory recurrent neural network indicates that the user has just completed the grabbing action, the fusion judgment module outputs the "target placement area" as the next visual object to be observed.

[0031] In one embodiment of the present invention, the iterative attention focus deduction operation in the attention generation layer is specifically implemented as follows: the attention generation layer maintains an updatable attention state matrix, which is initialized with the baseline attention values ​​of all visual elements in the structured visual field. Visual objects to be observed are sequentially extracted from the potential observation intention path as the current deduction target, and the probability distribution of shifting from the current attention focus to the current deduction target is calculated based on prior knowledge of attention shifts. This prior knowledge of attention shifts is encapsulated and invoked through a probability transition matrix library. Specifically, its construction and usage involve extracting a large number of successful task execution instances from historical human-computer collaboration task data. Each instance contains the actual visual gaze shift sequence of the user during the task. Statistical modeling is performed on all gaze shift sequences to analyze the frequency and conditional probability of shifting from one type of visual element to another, forming multiple probability transition matrices associated with different task types or sub-task stages, constituting the probability transition matrix library. During deduction in the attention generation layer, one or more probability transition matrices with the highest matching degree are retrieved from the probability transition matrix library according to the type of the current task instruction and the executed steps. The retrieved probability transition matrix is ​​used as prior knowledge for attention transition, calculating the probability of shifting from the current focus of attention to the next target in the potential observation intent path. The values ​​in the probability transition matrix are used as prior probabilities in the calculation. The attention state matrix is ​​updated based on the calculated probability distribution, simulating the attention focus transition process. The visual object that has completed the transition is taken as the new focus of attention, and this process is repeated until the entire potential observation intent path has been traversed. In each iteration, attention competition mechanisms from spatially adjacent visual elements in the structured visual field are considered, and the probability distribution is corrected accordingly. Finally, the attention state matrix, updated after all iterations, is converted into a dynamic attention prediction map for visualization output.

[0032] In practical implementation, the attention generation layer maintains an updatable attention state matrix. This matrix is ​​initialized with the baseline attention values ​​of all visual elements in the structured visual field. These baseline attention values ​​can be set based on the saliency or uniform distribution of low-level features of the visual elements. Visual objects to be observed are sequentially extracted from the task-driven potential observation intent paths received from the operation intent decoding layer as the current inference target. The probability distribution of shifting attention from the current focus to the current inference target is calculated based on prior knowledge of attention transfer. This prior knowledge of attention transfer is encapsulated and invoked through a probability transfer matrix library. The library is constructed and used as follows: A large number of successful task execution instances are extracted from historical human-computer collaboration task data. Each instance contains the actual visual gaze transfer sequence during the task. Statistical modeling is performed on all gaze transfer sequences to analyze the frequency and conditional probability of shifting from one type of visual element to another, forming multiple probability transfer matrices associated with different task types or sub-task stages, thus constituting the probability transfer matrix library.

[0033] In some embodiments, during iterative attention focus deduction in the attention generation layer, one or more probability transition matrices with the highest matching degree are retrieved from the probability transition matrix library based on the type of the current task instruction and the executed steps. The matching degree is determined by comparing the metadata tags associated with the current task description and the probability transition matrix, as well as the similarity between the currently completed sub-steps and the historical step sequences recorded in the matrix. The retrieved probability transition matrices are used as prior knowledge for attention transition to calculate the transition probability from the current attention focus to the next target in the potential observation intention path, with the values ​​in the probability transition matrices serving as prior probabilities in the calculation. In a specific implementation, when calculating the probability distribution from the current attention focus to the currently deduced target, an attention competition mechanism from spatially adjacent visual elements in the structured visual field is introduced to correct the probability distribution. The calculation of the probability distribution can be expressed as: in: Indicates the visual element currently in focus of attention. Transfer to target visual elements The final probability, This represents the visual element category retrieved from the probability transition matrix library. To Category The prior transition probability, This represents the correction factor for the attention competition mechanism based on spatial adjacency. With visual elements and The differences in spatial distance and task relevance between them are related. This indicates the visual element in the structured visual field that is currently the focus of attention. A collection of visually adjacent elements in space.

[0034] It is understandable that, based on the calculated final probability distribution Update the attention state matrix and simulate the shift of attention focus from the current visual element. To target visual elements The transfer process. The update operation will target the visual element. The attention value at the corresponding position in the attention state matrix is ​​increased, while the attention values ​​of other adjacent visual elements are decreased according to the probability distribution. After the transfer is complete, the target visual element... The new focus of attention is set, and this process is repeated until all visual objects in the potential observation intention path have been traversed. Optionally, in each iteration of the inference mechanism, an attention competition mechanism correction factor is applied. The calculation takes into account the visual element currently in focus of attention. The probability distribution is dynamically corrected by considering the diversion effect of other high task-relevance visual elements in the vicinity of the attention shift. In some embodiments, the attention state matrix after all iterations is finally converted into a dynamic attention prediction map for visualization output through a normalization and rendering operation. The brightness or color intensity of each pixel or region in the dynamic attention prediction map corresponds to its predicted attention weight.

[0035] In one embodiment of the present invention, establishing a real-time prediction performance verification loop includes the following interactive steps: Simultaneously acquiring actual visual gaze point data of a user performing the same task through a perception channel independent of the multi-layer mapping structure, such as a high-precision eye tracker, and processing it into an actual attention distribution map. A difference comparison unit is established to receive the dynamic attention prediction map from the attention generation layer and the actual attention distribution map from the independent perception channel, performing pixel-level or region-level difference measurement on the two images to generate a multi-dimensional error vector. This multi-dimensional error vector characterizes the deviation of the prediction in terms of spatial distribution, intensity, and temporal accumulation. The pixel-level or region-level difference measurement of the two images employs a hierarchical difference fusion strategy, specifically including calculating the attention intensity difference between corresponding coordinate points in the dynamic attention prediction map and the actual attention distribution map at the pixel level to form an original difference field. Based on the task-related regions defined in the structured visual field, the original difference field is divided into multiple sub-regions, and the mean, variance, and spatial gradient of the difference are statistically analyzed within each sub-region to form a region difference descriptor. The pixel-level original difference field and the region-level difference descriptor are concatenated, and then dimensionality reduction and fusion are performed through a fully connected network to output a compact multidimensional error vector. This multidimensional error vector preserves both detailed differences and macroscopic regional differences. The multidimensional error vector is fed back to the internal coupling nodes of the multi-layer mapping structure, specifically the information fusion node between the visual scene parsing layer and the operation intent decoding layer, and the path generation node between the operation intent decoding layer and the attention generation layer. At the coupling nodes, the weight coefficients and threshold parameters for cross-layer information transmission are dynamically adjusted based on the direction and magnitude of each component of the error vector. This allows the multi-layer mapping structure to adaptively fine-tune based on real-time performance feedback, thereby reducing the error in subsequent predictions.

[0036] In practical implementation, a real-time prediction performance verification loop is established, comprising the following interactive steps: Through a perception channel independent of the multi-layer mapping structure, such as a high-precision eye tracker, actual visual gaze point data of the user performing the same task is synchronously collected. This actual visual gaze point data is processed into an actual attention distribution map, which has the same spatial resolution and coordinate reference as the dynamic attention prediction map output by the attention generation layer. A differential comparison unit receives the dynamic attention prediction map from the attention generation layer and the actual attention distribution map from the independent perception channel, and performs pixel-level or region-level difference measurement on the two maps, generating a multi-dimensional error vector. The pixel-level or region-level difference measurement of the two maps employs a hierarchical difference fusion strategy. Specifically, at the pixel level, the strategy calculates the attention intensity difference at corresponding coordinate points in the dynamic attention prediction map and the actual attention distribution map, forming an original difference field. Based on the task-related regions defined in the structured visual field, the original difference field is divided into multiple sub-regions. Within each sub-region, the mean, variance, and spatial gradient of the differences are statistically analyzed to form a region difference descriptor. See Table 1.

[0037] Table 1: Sub-table describing regional differences Region ID Regional Categories Mean difference (μ) Variance (σ²) Spatial gradient mean (∇) R1 Operation target area 0.12 0.05 0.08 R2 Tool Selection Area 0.25 0.12 0.15 R3 Status feedback area 0.08 0.03 0.05 R4 Background area 0.02 0.01 0.01 In practice, the pixel-level original difference field and the region-level difference descriptor are concatenated. The original difference field can be flattened into a long vector, and all statistics in the region difference descriptor are sequentially combined into another vector. After concatenation, the two vectors are input into a fully connected network for dimensionality reduction and fusion, outputting a compact multidimensional error vector. The multidimensional error vector retains both detailed difference and macroscopic regional difference information. The generation process of the multidimensional error vector can be represented as follows: in: This represents a compact, multidimensional error vector of the output. This represents the dimensionality reduction and fusion transformation function performed on a fully connected network. Represents the original difference field. This represents the vector obtained by flattening the original difference field. This represents a subset of regional differences descriptors consisting of statistical measures such as the mean, variance, and spatial gradient of differences across all regions. This indicates that the subset of regional differences is flattened into a vector. (Symbol) This indicates a vector concatenation operation.

[0038] It is understandable that the multidimensional error vector characterizes the deviation of the prediction in terms of spatial distribution, intensity, and temporal accumulation. In some embodiments, the differential comparison unit also performs a moving average of the multidimensional error vector over multiple consecutive time steps to generate a smoother error feedback signal that better reflects the trend. The multidimensional error vector is fed back to the internal coupling nodes of the multi-layer mapping structure, specifically including the information fusion node between the visual scene parsing layer and the operation intent decoding layer, and the path generation node between the operation intent decoding layer and the attention generation layer. The information fusion node is responsible for adjusting the weights of the task semantic information flowing from the structured visual field to the operation intent decoding layer, and the path generation node is responsible for adjusting the path confidence weights flowing from the potential observation intent path to the attention generation layer. At the coupling nodes, the weight coefficients and threshold parameters of cross-layer information transmission are dynamically adjusted according to the direction and magnitude of each component of the error vector. The adjustment of the weight coefficients and threshold parameters follows a gradient-based adaptive rule. In some embodiments, the dynamic adjustment process can be specified as follows: after the information fusion node receives the error component related to the task semantic region in the multidimensional error vector, if the component is continuously positive, the fusion weight of the task semantic tag extracted from the structured visual field corresponding to the region is reduced, thereby weakening the influence of the region information in the subsequent intent decoding.

[0039] See Figure 4 This is a regional difference measurement analysis chart for human-computer collaborative visual attention prediction. The R² (tool selection area) has the highest values ​​for all three difference measures, representing the main source of error in attention prediction. Regions with higher mean difference values ​​also have greater variance and spatial gradients, indicating concentrated and unevenly distributed errors. This chart serves the "hierarchical difference fusion stage" of visual attention prediction, its value lying in identifying R² as the key attention prediction module to optimize; for regions with high variance / gradient in R², the visual feature parsing weights can be enhanced; regional difference data is the core input for subsequent multi-dimensional error vectors, providing a basis for adaptive fine-tuning of the model.

[0040] In one embodiment of the invention, the method further includes an offline optimization phase, which periodically enhances the multi-layer mapping structure using accumulated collaborative task data. The steps include collecting all executed task instance data within a period, each instance containing the original visual flow, task instructions, user operation behavior flow, and the actual attention distribution map obtained through independent perceptual channels. Using the task instance data as a training set, with the optimization objective of maximizing the overall similarity between the predicted dynamic attention map and the actual attention distribution map, end-to-end global optimization is performed on all trainable parameters in the multi-layer mapping structure. After optimization, the updated parameters are deployed to the online multi-layer mapping structure, replacing the original parameters, thereby achieving a gradual improvement in the performance of the prediction model during long-term task execution. This offline optimization phase employs a course learning strategy to organize the training set, specifically by classifying the collected task instance data into difficulty levels according to the complexity of the task instructions and the user's proficiency, forming multiple sets of training data from easy to difficult. During the global optimization process, the multi-layer mapping structure is initially trained using the lowest difficulty set of training data to master the basic task-attention mapping relationship. Then, increasingly challenging training datasets are gradually introduced, and the parameters obtained from previous training are further optimized, forcing the model to learn to handle more complex and ambiguous visual scenes and operational intentions. This step-by-step learning strategy aims to improve the generalization ability and prediction robustness of the multi-layer mapping structure under tasks of varying difficulty.

[0041] In its implementation, the method further includes an offline optimization phase, which periodically enhances the multi-layer mapping structure using accumulated collaborative task data. The offline optimization phase involves collecting all executed task instance data within a period. Each task instance data set includes the original visual flow, task instructions, user operation behavior flow, and the actual attention distribution map obtained through independent perceptual channels. Using the collected task instance data as a training set, and with the optimization objective of maximizing the overall similarity between the predicted dynamic attention map and the actual attention distribution map, end-to-end global optimization is performed on all trainable parameters in the multi-layer mapping structure. Maximizing the overall similarity is achieved by minimizing the loss function between the predicted map and the actual map. The loss function can be defined as: in: This represents the total loss that needs to be minimized. This indicates the number of task instances in a training batch. Represents a multi-layer mapping structure for the first... Dynamic attention prediction graph for each task instance Indicates the first Actual attention distribution map for each task instance It is a function that measures the similarity between two images, such as the structural similarity index or the integral of cosine similarity in image space. This represents the set of all trainable parameters in a multi-layer mapping structure. It is a regularization term for the parameters, used to prevent overfitting. It is a coefficient that controls the strength of regularization.

[0042] In some embodiments, after optimization, the updated parameters are deployed to the online multi-layer mapping structure to replace the original parameters. It is understood that parameter replacement is typically performed during system idle or maintenance periods to ensure the continuity of the online prediction service. The offline optimization phase employs a curriculum learning strategy to organize the training set. Specifically, this strategy categorizes the collected task instance data into difficulty levels based on the complexity of the task instructions and the user's proficiency, forming multiple sets of training data from easy to difficult. The complexity of the task instructions is quantified by the number of target objects involved, the nesting level of the operation steps, and the strictness of spatial constraints. The user's proficiency is measured by the efficiency of completing the task, the smoothness of the operation path, and the concentration of the actual attention distribution map. During global optimization, the multi-layer mapping structure is initially trained using the lowest-difficulty set of training data to master the basic task-attention mapping relationship, such as establishing a basic attention pattern for the target object and placement area in a simple pick-and-place task. Then, increasingly challenging training data sets are gradually introduced, further optimizing the parameters obtained from previous training. This forces the model to learn to handle more complex and ambiguous visual scenes and operational intentions, such as resolving more nuanced observation sequences in assembly tasks with obstacles or requiring multi-tool collaboration. This progressive learning strategy aims to improve the generalization ability and predictive robustness of the multi-layered mapping structure across tasks of varying difficulty. Optionally, the boundaries of difficulty groupings and the conditions for transitioning between groups in the learning strategy can be automatically determined by the predictive performance on the validation set. Once the model's validation loss on the current difficulty group converges to a stable plateau, it automatically switches to the next difficulty group.

[0043] See Figure 5 This is a multi-dimensional performance radar chart of a visual attention prediction model in human-machine collaborative tasks. Tasks with shorter response times typically show higher prediction accuracy, reflecting the characteristic of "more stable model performance in low-complexity tasks." This chart serves the "task adaptability evaluation phase" of visual attention prediction models. Its value lies in prioritizing the deployment of models in low-complexity scenarios such as simple pick-and-place and assembly tasks; strengthening the feature parsing and intent decoding modules for high-complexity tasks to address performance shortcomings in maintenance and complex operations; and providing a basis for classifying human-machine collaborative tasks into automated / human intervention categories based on model performance differences.

[0044] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0045] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A visual attention prediction method for human-computer collaboration and task-driven learning, characterized in that, include: A multi-layer mapping structure that includes task perception and behavior coupling is constructed. The multi-layer mapping structure is composed of a visual scene parsing layer, an operation intention decoding layer, and an attention generation layer coupled sequentially. In the visual scene parsing layer, the input raw visual stream is fused with the current task instructions to generate a structured visual field under task constraints. In the operation intent decoding layer, the user's real-time operation behavior flow is synchronously accessed. The operation behavior flow includes physical operation vectors and gaze point sequences. The structured visual field and the operation behavior flow are aligned across modal intents to parse out the task-driven potential observation intent path. In the attention generation layer, based on the potential observation intention path, iterative attention focus deduction is performed in the structured visual field. The iterative attention focus deduction process introduces prior knowledge of attention transfer extracted from historical collaboration patterns, and finally synthesizes a dynamic attention prediction map for the current human-computer collaboration task. A real-time prediction performance verification loop is established, and the dynamic attention prediction map is deeply compared with the actual attention distribution obtained through independent perception channels. Based on the error vector generated by the comparison, the coupling weight parameters in the multi-layer mapping structure are dynamically adjusted.

2. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 1, characterized in that, The specific steps for generating a structured visual field under task constraints are as follows: Perform multi-granularity visual entity segmentation on the input raw visual stream to identify the basic visual elements and their spatial relationships in the scene, thus forming the original visual field; At the same time, the current task instruction is semantically deconstructed to extract the target object set, operation action sequence and spatial constraints in the task instruction to form a task semantic framework; The original visual field and the task semantic framework are fed into a bidirectional matching network, which performs reverse retrieval from the task target to the visual entity and forward projection from the salient visual entity to the possible task target. Through the calculation of the bidirectional matching network, task relevance weights and expected interaction probabilities are labeled for each visual element in the original visual field, generating a structured visual field with attached task semantic labels and spatial topological relationships. The structured visual field is a dynamically updated data structure.

3. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 2, characterized in that, The steps for parsing the potential observation intent path driven by the task include: Continuously receive and cache the user's real-time operation behavior stream, and separate the physical operation vector representing limb movements and the gaze point sequence representing visual gaze points from the operation behavior stream; Within the operation intent decoding layer, a time-varying behavior-intent association model is established to jointly analyze the change pattern of the physical operation vector and the transition pattern of the gaze point sequence in order to infer the user's real-time operation goal and cognitive state. The inferred immediate operational goals and cognitive states are fused with the task semantic tags from the structured visual field to identify the sequence of visual objects and the order of observation that the user is most likely to observe in subsequent time segments in order to achieve the current task instruction. The sequence of most likely visual objects to be observed and the order of observation are formally defined as a task-driven potential observation intention path and passed as an intermediate output to the attention generation layer.

4. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 3, characterized in that, The specific implementation of the iterative attention focus deduction step is as follows: The attention generation layer maintains an updatable attention state matrix, which is initialized with the baseline attention values ​​of all visual elements in the structured visual field. Visual objects to be observed are extracted sequentially from the potential observation intention path as the current inference target. The probability distribution of shifting attention from the current focus to the current inference target is calculated based on the prior knowledge of attention shift. The prior knowledge of attention shift encodes the statistical regularity of observation pattern shift in the history of similar tasks. The attention state matrix is ​​updated based on the calculated probability distribution, the process of shifting attention focus is simulated, and the visual object that has completed the shift is taken as the new attention focus. This process is repeated until the potential observation intention path has been traversed. In each iteration of the deduction, the attention competition mechanism from spatially adjacent visual elements in the structured visual field is considered simultaneously to correct the probability distribution; Finally, the attention state matrix, after all iterations and updates, is transformed into a dynamic attention prediction graph that can be visualized.

5. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 1, characterized in that, The establishment of a real-time predictive performance verification loop includes the following interactive steps: Through a perception channel independent of the multi-layer mapping structure, such as a high-precision eye tracker, the actual visual gaze point data of the user when performing the same task is collected synchronously and processed into an actual attention distribution map. A differential comparison unit is set up to receive the dynamic attention prediction map from the attention generation layer and the actual attention distribution map from the independent perception channel. The difference between the two maps is measured at the pixel level or the region level, and a multidimensional error vector is generated. The multidimensional error vector represents the deviation of the prediction in terms of spatial distribution, intensity and temporal accumulation. The multidimensional error vector is fed back to the internal coupling nodes of the multi-layer mapping structure, specifically including the information fusion node between the visual scene parsing layer and the operation intent decoding layer, and the path generation node between the operation intent decoding layer and the attention generation layer; At the coupling node, the weight coefficients and threshold parameters of cross-layer information transmission are dynamically adjusted according to the direction and magnitude of each component of the error vector, so that the multi-layer mapping structure can be adaptively fine-tuned based on real-time performance feedback to reduce the error of subsequent predictions.

6. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 5, characterized in that, The pixel-level or region-level difference measurement of the two images employs a hierarchical difference fusion strategy, specifically including: At the pixel level, the difference in attention intensity at corresponding coordinate points between the dynamic attention prediction map and the actual attention distribution map is calculated to form the original difference field; Based on the task-related regions defined in the structured visual field, the original difference field is divided into multiple sub-regions. The mean, variance, and spatial gradient of the differences are calculated in each sub-region to form a region difference descriptor. The pixel-level original difference field and the region-level difference descriptor are concatenated, and then dimensionality reduction and fusion are performed through a fully connected network to output a compact multidimensional error vector. The multidimensional error vector retains both detailed difference and macro-regional difference information.

7. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 3, characterized in that, The time-varying behavior-intent association model is implemented through a recurrent neural network, and its specific workflow is as follows: Align and concatenate the physical operation vectors and gaze point sequences on the time axis to form a multimodal behavior time stream, which serves as the input to the recurrent neural network. The hidden state of a recurrent neural network is used to represent the user's potential intention state as it evolves over time. The network receives new behavioral data at each time step and updates its hidden state accordingly. The goal of training the network is to predict the user's action target or gaze point location at the next moment based on the historical behavior sequence, so that its hidden state can effectively encode intent information. In online applications, the behavioral time-series stream within the current and a short historical time window is input into the recurrent neural network, and the hidden state at the final moment is taken as the inference representation of the user's real-time operation goal and cognitive state.

8. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 4, characterized in that, The attention transfer prior knowledge is encapsulated and invoked through a probability transfer matrix library, and the specific construction and usage methods are as follows: A large number of successful task execution instances were extracted from historical human-computer collaboration task data. Each instance contains the actual visual gaze shift sequence of the user during the task. Statistical modeling is performed on all gaze transition sequences to analyze the frequency and conditional probability of transitioning from one type of visual element to another, forming multiple probability transition matrices associated with different task types or sub-task stages, thus creating a probability transition matrix library. During the inference process of the attention generation layer, one or more probability transition matrices with the highest matching degree are retrieved from the probability transition matrix library according to the type of the current task instruction and the steps already executed. The retrieved probability transition matrix is ​​used as prior knowledge of attention transition to calculate the transition probability from the current attention focus to the next target in the potential observation intention path. The values ​​in the probability transition matrix are used as prior probabilities in the calculation.

9. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 1, characterized in that, The method further includes an offline optimization phase, which periodically enhances the multi-layer mapping structure using accumulated collaborative task data. The steps include: Collect data on all executed task instances within a cycle. Each instance includes the original visual flow, task instructions, user action flow, and the actual attention distribution map obtained through independent perceptual channels. Using the task instance data as the training set, with the optimization objective of maximizing the overall similarity between the predicted dynamic attention map and the actual attention distribution map, end-to-end global optimization is performed on all trainable parameters in the multi-layer mapping structure. After optimization, the updated parameters are deployed to the online multi-layer mapping structure to replace the original parameters, thereby achieving a gradual performance improvement of the prediction model during long-term task execution.

10. The visual attention prediction method for human-computer collaboration and task-driven learning according to claim 9, characterized in that, The offline optimization phase employs a course learning strategy to organize the training set, specifically: Based on the complexity of the task instructions and the user's proficiency in operation, the collected task instance data is graded by difficulty to form multiple sets of training data from easy to difficult. During the global optimization process, the multi-layer mapping structure is first trained using the set of training data with the lowest difficulty, so that it can master the basic task-attention mapping relationship. By gradually introducing more challenging training data sets and further optimizing the parameters obtained from previous training, the model is forced to learn to handle more complex and ambiguous visual scenes and operational intentions. This progressive learning strategy aims to improve the generalization ability and prediction robustness of the multi-layer mapping structure under tasks of varying difficulty.

Images