A data labeling method and system based on user behavior and attention tracking

Abstract

This invention discloses a data annotation method and system based on user behavior and attention tracking. It synchronously and in real-time collects multi-source behavioral signals from doctors, including mouse, keyboard, and eye movements. Combined with identity and interface metadata, the data undergoes standardization, anomaly removal, and short-term behavioral unit segmentation. Unsupervised clustering is used to extract individual behavioral micro-patterns, and a behavioral profile library is constructed. Through multimodal temporal modeling and an adaptive spatiotemporal attention mechanism, behavioral features, interface regions, and report text are deeply integrated to output multi-level correlation probabilities, achieving high-precision automatic labeling of content and image regions.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and in particular to a data annotation method and system based on user behavior and attention tracking. Background Technology

[0002] Currently, the fields of pathological image annotation and medical artificial intelligence data management are rapidly developing towards intelligence and multimodal interaction. With the widespread application of digital pathology slide reading and pathology information systems, the personalized and intelligent association of multi-source behavioral data (such as mouse, keyboard, and eye-tracking interaction signals) generated by doctors during slide reading and reporting with multimodal data such as pathological images and text content has become a key technical issue for improving the quality and efficiency of automatic annotation.

[0003] Current technology has significant shortcomings in the following aspects:

[0004] (1) The modeling of doctors’ short-term micro-behavioral patterns is insufficient, and the temporal dependence and individual differences between behavioral signals such as mouse, keyboard and eye movement are ignored;

[0005] (2) When faced with differences in doctors’ operating styles, changes in procedures or switching between professional environments, existing correlation discrimination models are easily sensitive to the drift of behavioral feature distribution.

[0006] (3) The ability to capture the dynamic and hierarchical relationship between user behavior and actual content expression is still limited, as is the model’s ability to learn and optimize itself with the user, resulting in poor generalization adaptability and a heavy burden of manual verification and correction.

[0007] (4) There is a lack of modeling schemes that can extract individualized behavioral profiles and dynamic clustering features from doctors' operational data and adapt to their personalized habits in real time when judging correlation. Summary of the Invention

[0008] In order to solve the above-mentioned technical problems, the present invention provides a data annotation method and system based on user behavior and attention tracking.

[0009] The technical solution of this invention is implemented as follows: a data annotation method based on user behavior and attention tracking, comprising:

[0010] S1: Synchronously collect raw user behavior data from multiple sources, such as mouse operation signals, keyboard input signals, and eye tracking signals from different doctors, and mark the timeline with the image reading interface area and report content input status to reflect the environmental differences in multidimensional doctor behavior characteristics.

[0011] S2: Based on the multi-source raw user behavior data, normalize, remove outliers, and segment short-term windows for signals such as mouse trajectory, dwell distribution, eye gaze points, and keyboard input intervals to generate a time-series short-term micro-behavioral unit sequence, thereby achieving standardized preprocessing of basic user behavior features.

[0012] S3: For the segmented short-term micro-behavioral unit sequence, unsupervised clustering algorithms (including but not limited to DBSCAN and deep clustering networks) are used to extract behavioral clustering results from the operation records of different doctors, and construct a behavioral profile library oriented towards individual differences to characterize the distribution of micro-pattern differences among multiple doctors in different image reading scenarios.

[0013] S4: Utilizing the clustering identifiers and behavioral features of the behavioral profile database, combined with the area labels of the image reading interface and the report content summary, a multi-level temporal modeling approach (such as hierarchical attention-Transformer structure) is adopted to jointly model the micro-pattern sequence of individual doctor behavior, image area features, and report text information, and extract high-dimensional correlation features across modalities and time domains.

[0014] S5: Based on the high-dimensional correlation features obtained by joint modeling, the feature fusion weights between behavioral micro-patterns, gaze events, report text and image regions are allocated through an adaptive spatiotemporal attention mechanism. The fusion parameters are dynamically adjusted to adapt to doctors' professional preferences and scene changes, thereby improving the diversity and adaptability of correlation inference.

[0015] S6: The multimodal high-dimensional feature input correlation discrimination network after the spatiotemporal attention mechanism is integrated calculates the multi-level correlation probability output between the current report content and the screen image region (including global, local and patch fine-grained levels) to achieve content-region high matching based on personalized operation of doctors.

[0016] S7: For the multi-level correlation probability results of the output, set an adaptive threshold judgment criterion. When the correlation probability exceeds a certain threshold, automatically establish a label pairing relationship between the report content and the corresponding image area. Otherwise, maintain the manual or semi-automatic verification state to realize dynamic label generation.

[0017] S8: Based on continuously collected doctor feedback and review annotation information, the behavioral profile library and correlation discrimination network parameters are periodically updated. An online fine-tuning strategy is adopted to optimize the behavioral feature extraction and correlation discrimination process, thereby achieving dynamic self-learning and model self-adaptation under different professional habits and reading conditions, ensuring the long-term robustness and generalization ability of label annotation.

[0018] This invention also provides a data annotation system based on user behavior and attention tracking, which uses the above-mentioned data annotation method based on user behavior and attention tracking for standardization.

[0019] Beneficial effects

[0020] The present invention provides a data annotation method and system based on user behavior and attention tracking, which has the following beneficial effects:

[0021] (1) Significantly improved the accuracy and generalization ability of correlation discrimination under different doctors and different operating styles. This invention constructs a multimodal behavioral data foundation with rich context and identity information by real-time high-precision acquisition of multi-source behavioral signals (mouse, keyboard, eye movement, etc.) and binding of various environmental metadata, effectively eliminating data bias caused by differences in operating habits and hardware environment. By introducing standardized modeling of short-time window micro-behavioral units, the system can capture fine-grained operational micro-patterns and achieve high-resolution analysis of doctors' behavioral responses under complex object content, report input, and image reading scenarios. Therefore, the system can be compatible with the diverse behavioral characteristics of doctors across the entire hospital and all specialties under different workflow modes, and the generalization ability of the correlation discrimination network is significantly better than that of traditional static feature classifiers.

[0022] (2) Existing technologies are mostly based on coarse-grained fixed behavioral statistics or simple content-region similarity matching, ignoring the deep dynamic connection between behavioral micro-patterns and the actual reported content. This leads to significant fluctuations in the relevance output once the doctor's operating style or workflow changes, which can easily result in label matching errors. This invention effectively enhances the model's ability to represent the potential structure between doctors' personalized fragmented behaviors and content expression through unsupervised dynamic clustering, adaptive aggregation of multimodal inputs (behavior, image, text), and hierarchical temporal joint encoding. This makes the relevance probability output not only consider spatial-visual factors, but also actively incorporate individual behavioral styles, scene switching, and historical feedback. This achieves effective modeling of relevance under disease heterogeneity and operational diversity, and greatly reduces the probability of misclassification.

[0023] (3) The multi-level (global, local, and patch-level) correlation discrimination and cross-level decision fusion method designed in this invention can dynamically distinguish the complex subjective cognitive mapping relationship between report text and multi-scale image regions. Combined with an adaptive spatiotemporal attention mechanism, the network automatically assigns weights to different types of behavioral signals (e.g., some doctors rely more on mouse trajectories, while others have higher weights on gaze signals), achieving hierarchical high-confidence automatic pairing of labels, content, and regions. In actual clinical annotation scenarios, after processing by this invention, the average label positioning error is reduced by more than 30% compared to traditional methods, supporting high-resolution region annotation with a delay of seconds, which can significantly improve common problems such as insufficient label resolution and frequent manual intervention.

[0024] (4) This invention establishes a closed loop of label feedback and behavior distribution, supports the periodic use of structured labels and verification data to fine-tune the behavior profile library and discrimination network online. The system has the ability to instantly correct new doctor styles and absorb new scenario operation templates, effectively overcoming the shortcomings of traditional models that rely on manual batch retraining and have poor adaptability. Attached Figure Description

[0025] Figure 1 This is a flowchart of a data annotation method based on user behavior and attention tracking according to the present invention;

[0026] Figure 2 This is a sub-flowchart of a data annotation method based on user behavior and attention tracking according to the present invention;

[0027] Figure 3 This is a sub-flowchart of a data annotation method based on user behavior and attention tracking according to the present invention. Detailed Implementation

[0028] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0029] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0030] When used herein, the singular forms of “a,” “an,” and “the” may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having,” etc., specify the presence of the stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not preclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0031] As attached Figure 1 As shown, this application provides a data annotation method based on user behavior and attention tracking, specifically including:

[0032] S1: Synchronously collect raw user behavior data from multiple sources, such as mouse operation signals, keyboard input signals, and eye-tracking signals (optional), from different doctors. Combine this with the image reading interface area and report content input status to mark the timeline, so as to reflect the environmental differences in multidimensional doctor behavior characteristics.

[0033] S2: Based on the multi-source raw user behavior data, normalize, remove outliers, and segment short-term windows for signals such as mouse trajectory, dwell distribution, eye gaze points, and keyboard input intervals to generate a time-series short-term micro-behavioral unit sequence, thereby achieving standardized preprocessing of basic user behavior features.

[0034] S3: For the segmented short-term micro-behavioral unit sequence, unsupervised clustering algorithms (including but not limited to DBSCAN and deep clustering networks) are used to extract behavioral clustering results from the operation records of different doctors, and construct a behavioral profile library oriented towards individual differences to represent the micro-pattern differences distribution of multiple doctors in different image reading scenarios.

[0035] S4: Utilizing the clustering identifiers and behavioral features of the behavioral profile database, combined with the area labels of the image reading interface and the report content summary, a multi-level temporal modeling approach (such as hierarchical attention-Transformer structure) is adopted to jointly model the micro-pattern sequence of individual doctor behavior, image area features, and report text information, and extract high-dimensional correlation features across modalities and time domains.

[0036] S5: Based on the high-dimensional relevance features obtained by joint modeling, the feature fusion weights between behavioral micro-patterns, gaze events, report text and image regions are allocated through an adaptive spatiotemporal attention mechanism. The fusion parameters are dynamically adjusted to adapt to doctors' professional preferences and scene changes, thereby improving the diversity and adaptability of relevance inference.

[0037] S6: The multimodal high-dimensional feature input correlation discrimination network after the spatiotemporal attention mechanism is integrated calculates the multi-level correlation probability output between the current report content and the screen image region (including global, local and patch fine-grained levels) to achieve high content-region matching based on personalized operation of doctors.

[0038] S7: For the multi-level correlation probability results of the output, set an adaptive threshold judgment criterion. When the correlation probability exceeds a specific threshold, automatically establish a label pairing relationship between the report content and the corresponding image area. Otherwise, maintain the manual or semi-automatic verification state to achieve dynamic label generation.

[0039] S8: Based on continuously collected doctor feedback and review annotation information, the behavioral profile library and correlation discrimination network parameters are periodically updated. An online fine-tuning strategy is adopted to optimize the behavioral feature extraction and correlation discrimination process, thereby achieving dynamic self-learning and model self-adaptation under different professional habits and reading conditions, ensuring the long-term robustness and generalization ability of label annotation.

[0040] This invention also provides a data annotation system based on user behavior and attention tracking, which uses the above-mentioned data annotation method based on user behavior and attention tracking for standardization.

[0041] Step S1 involves synchronously and in real-time collecting raw user behavior data from multiple sources, including mouse operation signals, keyboard input signals, and eye-tracking signals from different doctors. This data is then combined with timeline annotations based on the image viewing interface area and report content input status to reflect the environmental differences in multidimensional doctor behavior characteristics. Specifically, this includes:

[0042] S1.1: Authentication and session identifier assignment are performed on the identity information of active doctors in the pathology slide reading system to ensure the uniqueness of the ownership and consistency of the labels of subsequent multi-source original user behavior data, and to provide basic user metadata for subsequent multi-dimensional data mapping.

[0043] The system authenticates and assigns session identifiers to active doctors within the pathology slide reading system to ensure the uniqueness of ownership and consistency of labels for subsequent multi-source original user behavior data, and to provide basic metadata for multi-dimensional data mapping.

[0044] A multi-factor authentication method (parameters include doctor's employee ID, password, biometrics, or two-factor authentication) is employed to achieve high-confidence uniqueness in doctor identification. Furthermore, a session management algorithm (parameters include system timestamp, client device identifier, and session hash token) assigns a unique session identifier to each doctor login operation, achieving isolation and traceability between operational behavior and identity mapping. Additionally, a real-time identity-session mapping table registration mechanism is used to bidirectionally associate the unique doctor's identity ID with its current session ID, and maintains concurrent processing warnings and session weighting rules for multiple sessions of the same doctor to ensure the label normalization and temporal consistency of the collected data.

[0045] Furthermore, by integrating the business role management module of the image reading system, professional metadata of doctors is extracted and structured according to their qualification level, department affiliation, and specific permission tags, forming a structured user basic information dictionary. Further, by binding the aforementioned structured user basic information with subsequently collected multi-source raw behavioral data streams such as mouse, keyboard, and eye-tracking data, consistency in identity indexing of all behavioral data is achieved during storage and subsequent modeling processes, improving the accuracy and scalability of subsequent multi-dimensional signal step alignment and individual difference modeling.

[0046] This identity authentication and session identifier allocation process binds doctors' login operations, user basic metadata, and multi-source behavioral event data at multiple levels, providing a data foundation for the unique tracking, label filling, and personalized analysis of multimodal behavioral characteristics. It effectively prevents error propagation problems such as data confusion and incorrect identity attribution, achieving the expected technical effects of automated behavior aggregation and individualized modeling.

[0047] For example, a pathology slide reading system in a tertiary-level Class A hospital uses Single Sign-On (SSO) authentication based on the Active Directory domain. Each time a doctor logs in, they enter an 8-digit physician ID and a personal password. The system automatically assigns a 128-bit random hash session token. The doctor's login information, along with their department code, years of practice, and subspecialty, is synchronously and structurally written into an identity information table. The platform assigns a unique session ID to each doctor's parallel multi-window operations and generates a unique mapping table in real time to mark data ownership. When doctors are simultaneously interacting on different terminals, the system assigns temporary session weights based on operation priority to prevent data out-of-order processing. After this step, all collected mouse tracks, keyboard inputs, and (if any) eye-tracking signals carry a unified doctor ID, session ID, department, and specialty label in the data table, enabling personalized analysis based on high-confidence identifiers in subsequent behavioral pattern clustering and association discrimination. Initial statistics from the hospital's initial implementation showed that the system achieved 100% accuracy in identity attribution within 10,000 data sampling periods, with a data mismatch rate of less than one in ten thousand. This fully met the subsequent needs for personalized behavior mining and tag standardization, significantly improving the automation and intelligence of the data chain.

[0048] S1.2: Based on an embedded acquisition agent, parallel real-time acquisition of multiple sources of raw user behavior data, such as mouse operation signals (including mouse trajectory point sequences, click events, drag intervals, and scroll wheel operations) and keyboard input signals (including keystroke timestamps, input character content, and report input window context), is performed to ensure the timing synchronization accuracy of various behavior events.

[0049] The input data consists of the doctor's identity information and session identifier, which have been authenticated and assigned, as well as the mouse sensor signals and keyboard input signals output in real time by the front end of the pathology reading system. The operation events are collected by the embedded acquisition agent activated or resident by the system control command.

[0050] An embedded acquisition agent module (with parameters including the number of acquisition threads, buffer length, signal sampling frequency, and system call priority) is deployed at the front end of the pathology slide reading system to achieve parallel real-time acquisition of mouse operation signals and keyboard input signals. Mouse signals include mouse trajectory point sequences (composed of sampling time and screen coordinates), click events (mouse click with event code), drag intervals (mouse drag with start-end points), and scroll wheel operation data (scroll wheel direction and number of rotation steps). Keyboard signals encompass keystroke timestamps, input character content (character code), and report input window context (focus window ID and active area identifier).

[0051] Furthermore, a high-precision system hardware and software co-working timer (with parameters including high-resolution clock and synchronization callback period) is used to uniformly timestamp the sampling time points of the operation signals of the acquisition thread at the microsecond level, ensuring that the time series data of various behavioral events have global alignment. An event-driven non-blocking synchronization method and a double-buffered asynchronous queue management device (buffer swapping + event loop) are employed to effectively alleviate acquisition congestion and packet loss caused by high-frequency concurrent signal streams, improving the smoothness and robustness of real-time acquisition.

[0052] Furthermore, a multi-channel data structure (such as the RingBuffer structure) is used to independently enqueue and index the parallel acquisition streams of the mouse and keyboard in multiple threads. This records the unique signal type identifier, the original time of event triggering, and the arrival time of system processing for each event source, forming a multi-source event parallel buffer. This provides basic support for subsequent accurate timing alignment and multimodal behavior analysis.

[0053] Furthermore, through a protocol-based event packaging mechanism, additional metadata is automatically added to all raw behavioral signals captured in real time, including fields such as doctor ID, session ID, terminal device serial number, operating system type, and foreground window status. This enables the synchronous binding of behavioral data with identity-environment metadata and outputs a structured multi-source raw behavioral event package with identity index and session tag.

[0054] Through the above-mentioned technical processing chain, the mouse trajectory data, click and drag events, scroll wheel changes and keyboard input signals collected synchronously are integrated into a unified structured set of behavioral events with strict time baselines and identity and session dimension indexes. This enables high-throughput, low-latency, lossless concurrent real-time acquisition of multimodal raw behavioral data, laying a precise data foundation for downstream behavioral signal synchronization, feature standardization and time-series segmentation processing.

[0055] For example, in a practically deployed pathology slide annotation platform, the front-end system loads an embedded acquisition agent implemented in C++. Parameters are set as follows: mouse and keyboard signal sampling period of 1 millisecond, single signal buffer length of 100,000 records, and polling synchronization period of 0.5 seconds. A doctor performs slide reading operations on a PC, including continuous mouse cursor movement (1000 trajectory points collected per second), high-frequency local area clicks (single click event code, 20 times per second), local image zooming (scroll wheel up 20 increments, down 15 increments), and frequent focus switching in the report text input window (800 characters entered, average keystroke interval of 120ms). The multi-channel acquisition stream is synchronously bound with the identity ID "D10345" and session ID "SID20240622ABC," and each acquisition event embeds the acquisition time, window status code, and device serial number. The system underwent a one-hour signal stream test, with no packet loss during total acquisition, an acquisition latency of <5ms, and a single-source signal and global behavioral event tracing error of <0.2ms. This fully supports the accurate serial tracing of high-density operational events and behavioral patterns. In contrast, a conventional acquisition strategy without a high-precision acquisition proxy solution exhibits an event packet loss rate as high as 4%, with signal overflow and out-of-order issues being highly likely, failing to meet the stringent requirements of subsequent fine-grained segmentation of micro-behavioral units and individualized feature tracking.

[0056] S1.3: Optionally, for doctors equipped with eye-tracking devices, eye-tracking signals are collected, including raw temporal eye-tracking data such as fixation point coordinate sequence, fixation duration, and saccade path distribution, to achieve multi-source signal synchronization with mouse and keyboard data.

[0057] For doctor terminals with eye-tracking capabilities, the input conditions are the doctor's basic metadata that has been authenticated and assigned a session identifier, the real-time status of the pathology reading system front end, and the system clock synchronized with mouse and keyboard signals.

[0058] An eye-tracking device interface connection method (parameters include access protocol type, device refresh frequency, and data accuracy level) is adopted to achieve real-time capture of doctors' eye-tracking signals.

[0059] Furthermore, the eye-tracking acquisition driver module (parameters including fixation point sampling period, saccade path continuity breakpoint threshold, and data buffer length) acquires the doctor's original fixation point coordinate sequence (x) in real time. t ,y t ), duration of each fixation d t And the distribution of saccade paths between consecutive fixation points.

[0060] A multi-source synchronization mechanism based on a high-precision system clock is adopted to synchronize the gaze point data output by the eye-tracking acquisition driver module with the mouse and keyboard signals at the queue level using the sampling time as the index. Through a timestamp alignment algorithm (parameters include the maximum allowable synchronization deviation and the sliding window matching length), millisecond-level synchronization of multi-source signals is achieved.

[0061] Furthermore, an eye-tracking saccade path structured analysis algorithm is used to analyze and identify fixation hotspots, saccade jumps, and fixation order in the fixation point sequence, extract temporal feature data reflecting the dynamics of the doctor's visual attention, and supplement relevant signal-to-noise ratio and data integrity indicators.

[0062] By using a protocol-based behavioral event packaging mechanism, meta-tags such as doctor ID, session ID, device serial number, and terminal type are uniformly added to the raw eye-tracking signals and structured features, enabling full-field correspondence and synchronous output with other behavioral signals.

[0063] Through the above series of methods, the original eye-tracking signals of individual doctors and their structured analysis results are strictly synchronized into a unified behavioral data stream, realizing high-precision, multimodal, and multi-temporal baseline behavioral data integration, providing a precise gaze behavior basis for subsequent micro-behavioral unit segmentation, pattern clustering, and cross-modal temporal modeling.

[0064] For example, a Tobii Pro Spectrum eye tracker was deployed on a reading terminal in a tertiary hospital, with a sampling frequency set to 1200Hz and a fixation spatial resolution of 0.1°. The system synchronously collects the doctor's eye movement data along with mouse trajectories and keyboard event streams in 1ms increments. During pathology slide reading, the average duration of a single sustained fixation area is 800ms, with saccade jump intervals between 50-300ms. Complete fixation heatmaps, saccade paths, and fixation point sets are collected. Through a timestamp alignment algorithm, the time synchronization error between eye movement data and mouse / keyboard signals is less than 2ms, and the data integrity rate is 99.98%. The structured output event packets include fixation points (x... t ,y t ), fixation duration d t The system collected data on the start and end times of eye scans and corresponding session metadata, and no data corruption was found during the data collection process of ten doctors during peak hours within a day. The output high-resolution eye-tracking heatmap was perfectly aligned with the mouse operation behavior timeline, effectively supporting the high-expression requirements of downstream micro-behavioral unit analysis and personalized temporal modeling, and significantly improving the accuracy of annotation correlation judgment and the system's generalization ability.

[0065] S1.4: Real-time analysis and collection of metadata about the business area where the doctor is located on the image reading interface, including the pathological image display area, magnification, screen coordinate system, and the status of the report content input module. By associating interface area labels with behavioral events, the binding of behavioral signals with the physical operating environment is realized.

[0066] S1.5: The raw acquisition results of mouse operation signals, keyboard input signals, eye tracking signals and interface area labels are synchronized with timestamps and aligned with time sequence based on a high-precision system clock to obtain a multi-dimensional behavioral event time sequence stream with a unified time baseline.

[0067] The input data consists of the doctor's identity ID and session ID assigned after identity authentication and session identification, the collected multi-source raw behavioral signals (including mouse operation signals, keyboard input signals, and eye tracking signals), and the interface area meta tags. All collected signals are accompanied by an independent collection timestamp.

[0068] A high-precision system clock (parameters include master clock resolution <1ms, arbitration synchronization callback period 0.5ms, and cross-device NTP time source synchronization threshold <2ms) is used to globally timestamp the raw mouse, keyboard, and eye-tracking signals output by each acquisition thread, thereby merging the master clocks of all behavioral event data.

[0069] Furthermore, a hierarchical timing alignment algorithm (parameters include a single signal flow tolerance window ΔT = 1ms, a multi-signal synchronization buffer window length N = 1000, and a maximum synchronization offset ε = 2ms) is used to perform sliding window-level timing alignment on the event queues collected by each signal source, thereby achieving millisecond-level time baseline unification between event types.

[0070] An event-time recoding method is adopted, in which a global clock sequence number (GlobalSequence Index) is appended to each behavioral event to synchronize and reorder the original signals. For concurrent events with high density across signal sources, an optimized conflict resolution algorithm (such as based on timestamp priority weight, with parameter λ graded according to event priority) is used to ensure strict temporal consistency of the final multi-source event stream.

[0071] Furthermore, for timing anomalies or signal missingness detected within the window buffer, the anomaly detection logic marks the data verification status in real time and provides automatic correction (such as linear interpolation and event completion) for over-threshold delays or repetitive events, ensuring the integrity and high confidence of the event stream.

[0072] By using the above-mentioned high-precision clock synchronization and multi-source timing alignment processing methods, the original behavioral signals and interface meta tags from different sources are transformed into a multi-dimensional behavioral event timing stream with a unified time baseline. This achieves precise consistency of the timing relationships of all signals across modalities, threads, and devices, laying a rigorous data timing foundation for subsequent standardized feature extraction, timing micro-unit segmentation, and behavioral clustering modeling.

[0073] For example, in a pathology annotation platform at a tertiary-level Class A hospital, an Intel precision master clock module and NTP network time synchronization protocol are used. The master clock resolution is set to 0.5ms, and the front-end embedded acquisition agent completes real-time acquisition of mouse trajectories, keyboard keystrokes, and eye movement signals at a sampling period of 0.5ms. All acquired events are timestamped locally with a precision of 1ms, and cross-terminal network latency is corrected on the server side via the NTP protocol. For the 1000 mouse, 500 keyboard, and 2000 eye movement event streams of the acquisition thread, a buffered queue sliding window synchronization is used, and the actual measured maximum time offset error between adjacent events is 0.8ms. If a segment of eye movement data is found to have lost frames, the event verification logic automatically performs linear interpolation according to the coordinates of adjacent gaze points and re-enters the queue. The final output is a multi-source event stream with a unified Global Sequence Index, supporting all subsequent behavioral micro-patterns and scene analyses to iterate under the same time base and sequence. Actual statistics show that this step can improve the alignment accuracy of cross-signal source behavioral events to the sub-millisecond level, and reduce the data overflow and misorder rate to below 10^(-5), thereby improving data reliability and reproducibility for back-end time series modeling and multimodal fusion.

[0074] S1.6: Using doctor identification, multi-source behavioral signals, and interface region meta-labels as input, construct a structured raw dataset of user behavior with complete temporal, spatial, interface, and identity labels, laying the foundation for subsequent feature standardization and behavioral pattern modeling.

[0075] Step S2: Based on the multi-source raw user behavior data, normalize, remove outliers, and segment short-term windows for signals such as mouse trajectory, dwell distribution, eye gaze points, and keyboard input intervals to generate a time-seriesed sequence of short-term micro-behavioral units, achieving standardized preprocessing of basic user behavior features. Specifically, this includes:

[0076] S2.1: Perform unified time baseline calibration and synchronous timing alignment on the collected multi-source raw user behavior data (including mouse trajectory point coordinate sequence, dwell duration distribution, eye movement gaze point spatiotemporal sequence, and keyboard input interval timestamp sequence) to construct a consistent time domain dataset across signal sources and ensure the consistency of subsequent timing analysis.

[0077] S2.2: Based on the time-synchronized cross-signal source time-domain dataset, standard normalization algorithms, including zero-mean normalization and range normalization, are used to perform unified normalization mapping of feature values ​​such as mouse trajectory point coordinates, dwell time length, eye movement gaze coordinates and duration, and keyboard input interval cooldown cycle, in order to improve the compatibility of behavioral feature data of different signal dimensions.

[0078] The input is a multi-source cross-signal user behavior feature dataset that has been calibrated with a unified time baseline and synchronized with the timing, covering timing features such as mouse trajectory point coordinates, dwell time length, eye movement gaze coordinates and duration, and keyboard input time interval.

[0079] A standard normalization algorithm (including zero-mean normalization and range normalization) is used to perform linear expansion mapping of the feature space on various behavioral feature data sequences, uniformly normalizing the spatial features to the [0,1] interval. Zero-mean normalization is applied to feature X using the following formula:

[0080]

[0081] Where μ is the mean of feature X and σ is the standard deviation, achieving centering and variance standardization of each sequence data.

[0082] Furthermore, for features with absolute value limits (such as screen coordinates and eye fixation points), a range normalization algorithm is used to standardize them based on the maximum and minimum value intervals:

[0083]

[0084] Among them, X min With X max These are the minimum and maximum values ​​of feature X within the observation interval, respectively, thus mapping the feature value to the standard interval [0,1].

[0085] Furthermore, considering the spatial distribution characteristics of mouse trajectory points and eye-tracking fixation points, their coordinate data are uniformly normalized to a coordinate system relative to the screen. That is, after standardizing the screen width and height, the coordinate point P(x, y) is normalized as follows:

[0086]

[0087] W and H represent the screen width and height, respectively, effectively eliminating spatial scale variations between different resolutions and terminals.

[0088] Furthermore, by normalizing time-related features such as dwell time, gaze duration, and keystroke interval, consistent representation is achieved across doctors, devices, and scenarios, laying a highly compatible feature foundation for subsequent outlier detection and micro-behavioral unit segmentation.

[0089] By standardizing the mapping, a set of user behavior data with all feature dimensions normalized is generated, providing a high-confidence and highly compatible data prerequisite for downstream robust outlier detection and windowed time series segmentation.

[0090] For example, in a pathology slide annotation system, the input is the mouse trajectory point (x) collected at a sampling frequency of 1000Hz. t y t ), where x t ∈[0, 1920]、y t ∈[0,1080], eye-tracking fixation coordinates Within the same resolution range, dwell time t stay Within the interval [0.025.00] seconds, the keystroke interval Δt key The time interval is [302000] milliseconds. Mouse trajectory and gaze coordinates are uniformly normalized to the [0, 1] interval. The normalized standard deviation of the average coordinates is 0.13, and the mean dwell time after range processing is 0.41. The keyboard interval is evenly distributed after normalization. The normalized feature matrix input outlier detection and window segmentation processing modules, in actual tests, processed 1 million event data points. The normalization execution time was less than 1 second, and the numerical distribution was not distorted, significantly improving the data compatibility and discrimination accuracy of subsequent behavior segmentation and clustering discrimination stages.

[0091] S2.3: Robust outlier detection algorithms (such as the IQR interquartile range method and the Local Outlier Factor (LOF) method) are used to identify and remove outliers from the normalized behavioral feature data sequences, outputting a set of regular feature data with high confidence, thereby improving the confidence strength of subsequent time series analysis and behavioral discrimination.

[0092] S2.4: Based on the normalized high-confidence behavioral feature data set, time-series segmentation is performed according to the fixed-length or adaptive variable-length sliding window algorithm to divide the long sequence into serialized short-time window behavioral segments, ensuring the complete expression of the spatiotemporal distribution of micro-behavioral units and laying the structural foundation for subsequent behavioral micro-unit feature extraction.

[0093] S2.5: For all behavioral fragments generated in short time windows, further utilize signal type and temporal distribution characteristics to construct and output composite fine-series behavioral units containing mouse trajectory segmentation distribution, dwell heat distribution, eye movement gaze path, keyboard input events, etc., providing a high-precision unified input feature template for subsequent behavior clustering and personalized doctor profile construction.

[0094] The input data is a collection of short-term window behavior fragments after window segmentation, covering multi-source features such as normalized and anomaly-removed mouse trajectories, dwell distributions, eye-tracking fixations, and keyboard input events.

[0095] A type separation and time-domain aggregation method (parameters include signal type identifier and segment time span) is adopted to achieve independent extraction and feature aggregation of various types of signals in each short-time window behavior segment.

[0096] Furthermore, a trajectory space segmentation algorithm (parameters including segmentation step size Δd and spatial region grid number m×n) is used to reconstruct the sub-trajectory distribution of the mouse trajectory signal, thereby extracting the mouse trajectory segmentation distribution. This processing uses the normalized trajectory coordinate point set {(x i y i Taking )} as input, calculate the count distribution and movement vector of each trajectory point falling into the spatial grid, and output the spatial distribution density matrix T. mouse .

[0097] Furthermore, a time distribution mapping algorithm (parameters including hotspot detection radius R and dwell time threshold θ) is employed to filter dwell points and model their thermal distribution for signals such as mouse trajectory and eye gaze points. Specifically, the time intensity of each dwell point is calculated using the following formula:

[0098]

[0099] Among them, S j Let Δt be the dwell time at the j-th dwell position, N be the total number of coordinate points within the window, and Δt be the dwell time at the j-th dwell position. i Let i be the dwell time at the i-th coordinate point. This is an indicator function. It outputs the mouse hover heatmap and eye-tracking gaze heatmap at the window level.

[0100] Furthermore, through path reconstruction and curvature analysis algorithms (parameters being curvature threshold κ and minimum jump point interval τ), the eye movement saccade path is reconstructed for the eye movement fixation point signal within the entire short time window based on the spatial and temporal coherence of adjacent fixation points. The trajectory is then segmented by curvature and labeled with fixation-saccade-jump point events, outputting window-level eye movement fixation path and event sequence features.

[0101] Furthermore, a keystroke event extraction and timing statistics algorithm (with parameters being the number of keystrokes K and the average input duration T_{key} within the window) is used to extract the keystroke event sequence, single keystroke timestamp distribution, and character input rate within each window from the keyboard input signal. These are then organized into window-level keyboard input event features, and statistical quantities such as keystroke frequency, input peak value, and cooling cycle are output.

[0102] By combining multiple signal features and using label fusion, the above mouse trajectory segmentation distribution T is obtained. mouse Residence heat distribution S mouse Eye movement fixation thermal distribution S eye Eye-tracking fixation path P eyeKeyboard input event statistics K key Together, they form the window-level composite fine-grained temporal behavioral unit feature vector F. unit The data is then uniformly output to the feature dataset, providing high-precision structured input for subsequent unsupervised clustering and personalized doctor profile modeling.

[0103] Through the above step-by-step processing, the original short-term window behavior fragments are transformed into high-confidence fine-grained behavior units carrying multiple signals and diverse features, thereby achieving high expressiveness and cross-modal uniformity in behavior pattern decomposition.

[0104] For example, in a typical tertiary hospital pathology slide reading platform, behavior segments are processed after being segmented into short windows of 36ms each. Configuration parameters include a 10×10 grid for mouse trajectory spatial segmentation, a hotspot detection radius R of 0.03 (normalized coordinate system), a dwell time threshold θ of 50ms, an eye movement curvature segmentation threshold κ of 0.02, and a keyboard keystroke peak window width of 18ms. For a window segment containing 800 sampling points, the system first separates and acquires each type of signal; the spatial distribution matrix is ​​reconstructed from the mouse trajectory, and the hotspot grid count peak within the window reaches 15 times, identifying the corresponding area as a high-interest area. Dwell heat distribution modeling captures the difference in dwell intensity between the entry and exit areas, forming a boundary discrimination template. Eye movement gaze path analysis obtains three high-curvature jump points, distinguishing them as saccade events and labeling them as gaze-saccade sequences. Keyboard event statistics show 22 keystrokes within the window, with an input rate of 0.61 keys / ms. Two cooldown cycles >300ms were detected and considered as report writing pause features. Ultimately, the integrated output of this window segment includes 109 feature items: a 10×10 matrix of mouse distribution, a dwell heatmap, a gaze heatmap, a gaze path sequence, and keystroke statistics (7 dimensions). This accurately reflects the doctor's complex interactive behavior characteristics within that time slice, providing consistent and high-resolution feature input for subsequent doctor profiling and micro-behavioral pattern clustering. In 10,000 randomly selected window segment processing cycles, the segment-level signal recognition accuracy reached 99.6%, and the correspondence between segment labels and actual doctor operations exceeded 98%. The highly expressive feature unit achieved the expected technical goals of behavior decomposition, behavior aggregation, and cross-modal template learning.

[0105] Step S3: For the segmented short-term micro-behavioral unit sequence, unsupervised clustering algorithms (including but not limited to DBSCAN and deep clustering networks) are used to extract behavioral clustering results from the operation records of different doctors, constructing a behavioral profile library oriented towards individual differences to characterize the distribution of micro-pattern differences among multiple doctors in different image reading scenarios. Figure 2 As shown, it specifically includes:

[0106] S3.1: Perform feature vectorization processing on the input short-term micro-behavioral unit sequence, and generate a standardized set of micro-behavioral feature vectors using behavioral event statistical parameters (such as mouse movement speed, fixed eye gaze duration, keyboard typing timing distribution, etc.) to support the modeling requirements of subsequent unsupervised clustering algorithms.

[0107] The input is a set of composite fine behavioral unit features after sliding window segmentation, normalization and outlier removal, including multi-source temporal features such as mouse trajectory segmentation distribution, dwell heat distribution, eye movement gaze heat distribution, eye movement gaze path features, and keyboard input event statistics.

[0108] A feature statistical analysis method (parameters including mean trajectory velocity μ_v, eye-tracking fixation duration μ_fix, keystroke interval distribution μ_k, etc.) is used to achieve fine-grained statistical feature extraction of the signal for each short-term window behavior unit.

[0109] Furthermore, through a temporal aggregation algorithm (parameters including signal type identifier S_type and time span ΔT), unified feature extraction across signal types is achieved, such as calculating the average moving speed, moving acceleration, and expected trajectory curvature within the window for mouse trajectory points, and extracting the average duration of gaze points and the total length of saccade paths for eye-tracking signals.

[0110] Furthermore, regarding the mouse trajectory segmentation distribution T mouse Using the spatial density normalization method, generate Where t i N is the normalized number of trajectory points within a single grid. grid To divide the grid and achieve vectorization of spatial distribution; for the residence thermal distribution S mouse With S eye The average value of hotspots, peak heat value, and distribution entropy index within the window are calculated to form the window's thermal statistical characteristics.

[0111] Furthermore, using a curvature segmentation algorithm (parameters being the gaze sequence length L and the segmented curvature threshold κ), the eye-tracking gaze path P is segmented. eye The process is decomposed into continuous fixation and saccade events. For each segment, the average duration, spatial span, and maximum curvature value are calculated, and the temporal statistical characteristics of fixation / saccade events are output.

[0112] Furthermore, a keyboard input event statistics method is used to count the number of keystrokes K within the window. count Average input rate V of the window key =K count / ΔT, maximum pause period D max The indicators are collected and standardized to form a keystroke timing behavior feature vector.

[0113] The various statistical and time-series features extracted above are standardized (e.g., min-max normalization, z-score normalization) and unified into a fixed-length, dimensionless behavioral feature vector F. unit Specifically, it includes:

[0114]

[0115] in, σ is the average speed of the mouse trajectory. mouse The standard deviation of the trajectory velocity. σ represents the average duration of fixation. fix Let M1 be the standard deviation of fixation duration, M1 be the mean of the spatial grid distribution, and H be the standard deviation of fixation duration. mouse With H eye These are the entropy values ​​for the thermal distribution of the mouse and eye movements, respectively. D represents the average keystroke rate. max Maximum keystroke pause.

[0116] By combining and standardizing features, a complete and structured set of short-term window standardized micro-behavioral feature vectors is formed, which serves as the modeling input for subsequent unsupervised clustering algorithms, thereby achieving a refined vectorized representation of doctors' micro-behavioral units.

[0117] For example, in the actual pathology slide reading workstation acquisition environment of a tertiary-level Class A hospital, the system segments behavior segments with a window width of 36ms. For each window behavior unit, the z-score normalization method is used to process the average mouse trajectory speed (μ=0.15, σ=0.04), fixation duration (average 0.17 seconds, σ=0.05 seconds), and keystroke rate statistics (average rate 0.65key / ms). For the spatial distribution of the mouse trajectory, the mean hotspot density is calculated to be 0.098 and the heat distribution entropy is 2.15 based on a 10×10 grid. For the eye-tracking fixation heat distribution, the entropy value is 1.32, and the maximum fixation duration is 0.41 seconds; the total number of keystrokes obtained from keyboard input statistics is in the range [0,24], and the maximum window pause period is 360ms. All the above statistical results are uniformly mapped to the [0,1] interval using min-max normalization, and finally, each behavior unit outputs a 23-dimensional structured feature vector. After manual annotation and verification, the resulting feature vectors can accurately distinguish the differences in the operation methods of the same doctor in different image reading scenarios such as sliding zoom, region marking, and text supplementation. The distribution pattern is highly consistent with subjective operation habits, providing a standard input feature set with high expressive power and good complementarity for downstream doctors' individual behavior clustering and profile construction.

[0118] S3.2: Based on the standardized micro-behavioral feature vector set, an unsupervised clustering algorithm (such as DBSCAN, deep clustering network) is used to perform clustering analysis on different behavioral units from the same doctor, outputting the behavioral pattern category labels and their distribution information under the individual doctor, and realizing multi-mode classification of micro-behavioral units.

[0119] S3.3: Associate and map the obtained behavioral pattern category labels with the original micro-behavioral feature vectors to construct a doctor individual behavior clustering meta-dataset containing elements such as cluster labels, category center features, and time series indexes, providing refined data support for subsequent individualized analysis and cross-time domain modeling.

[0120] The input consists of a set of category labels for physician microbehavioral units output by unsupervised clustering (such as DBSCAN, deep clustering networks), and a standardized microbehavioral feature vector corresponding to each unit.

[0121] A label index mapping algorithm (parameters: category label L, micro-behavioral feature vector set F) is used to achieve a complete association between the category label and the original feature vector of each micro-behavioral unit under the same doctor, and to cluster the label L. i With the corresponding eigenvector F i One-to-one correspondence, forming a mapping set {(L i F i )}.

[0122] Furthermore, by using the category center calculation method (parameter: number of category labels K), all feature vectors of each category label L are aggregated, and the mean center and covariance are used to express the result, thus obtaining the category center feature C. k With the category feature distribution matrix ∑ k ,Right now:

[0123]

[0124] Where, N k Let C be the number of samples in class k. k The mean center of the feature vectors of this category, ∑ k Let be the covariance matrix of the features of this category.

[0125] Furthermore, using the behavioral unit time-series index backtracking method (parameter: original fragment timestamp T), the original short-time window start and end timestamps (t) corresponding to each cluster label are retrieved. start ,t end ), construct the metadata item of the category label-central feature-time index triple.

[0126] Furthermore, a structured behavioral clustering metadata organization method is adopted to assemble the above-mentioned related data into a structured behavioral clustering metadata dataset for individual doctors. The data structure includes cluster labels L and category centers C. k , characteristic distribution ∑ k The time-series index T forms M. doc ={(L, C k , ∑ k ,T)}.

[0127] By integrating high-dimensional vectorization and managing indexes, the original clustering output is mapped into a fine-grained individualized behavior clustering meta-dataset that can be efficiently retrieved, modeled across time domains, and compared by category. This provides the foundation for subsequent dynamic archiving of behavior profiles and individualized time-series analysis.

[0128] For example, in an actual pathology slide reading case at a tertiary-level Class A hospital, a single doctor collected 84,000 short-term window behavior units in one day. After modeling with a deep clustering network, K=7 behavior pattern category labels were output. For each category label, a category center C was constructed from all feature vectors. k (e.g., the center mean vector of category 3 [0.15, 0.48, ..., 0.07]), and the largest element of the main diagonal of the covariance matrix is ​​0.09. The behavioral unit time index T points to typical operation intervals such as gaze hotspot focusing, high-frequency mouse dragging, and high-density keyboard input. The systematically output doctor individual clustering meta-dataset contains 7 items, each containing a category label, a 23-dimensional center mean vector, a 23×23 covariance matrix, and a unit start and end timestamp index. In the test set, the backtracking retrieval efficiency was compared, and the mapping accuracy between cluster labels and time-series indexes reached 100%, with a 13.2% improvement in the cross-window behavioral pattern continuity recognition rate. This structured clustering meta-dataset directly serves subsequent behavioral pattern tracing, behavioral profile archiving, and cross-modal modeling input, realizing the quantitative expression and dynamic tracking of individual doctor behavioral differences.

[0129] S3.4: Perform behavioral pattern distribution statistics on the cross-doctor behavior clustering metadata, and combine it with image reading scenario labels (such as high-resolution area dwell time, peak report writing time, etc.) to archive patterns and characterize differences, forming a behavioral profile sub-library indexed by doctor identity, systematically revealing the micro-pattern hierarchical characteristics of individuals in different situations.

[0130] S3.5: Integrate multiple sub-databases of doctors' behavioral profiles, and use principal component analysis (PCA) based on the similarity of behavioral categories and the spectrum of operational styles to achieve high-dimensional feature integration of behavioral clustering results, and establish a total behavioral profile database for multiple doctors and cross-scenarios, so as to provide structured input for downstream hierarchical time series modeling and multimodal interaction alignment.

[0131] Step S4: Utilizing the clustering identifiers and behavioral features of the behavioral profile database, combined with the image viewing interface area labels and report content summaries, a multi-level temporal modeling approach (such as a hierarchical attention-Transformer structure) is employed to jointly model the micro-pattern sequences of individual doctor behaviors, image region features, and report text information, extracting high-dimensional correlation features across modalities and time domains. For example... Figure 3 As shown, it specifically includes:

[0132] S4.1: Based on the clustering identifiers and micro-pattern sequences of doctor behavior from the behavioral profile library, the behavioral clustering labels are used as input, and the label encoding technology is used to vectorize the behavioral clustering labels to generate behavioral label embedding sequences, providing a unified structure of behavioral label embedding features for cross-modal feature collaborative modeling.

[0133] S4.2: A hierarchical temporal modeling structure is used to extract features from the micro-pattern sequence of doctor behavior (including mouse trajectory, dwell distribution, keyboard operation segments and eye movement gaze feature embedding). Multi-scale dynamic spatiotemporal modeling is performed using behavior embedding vectors and original temporal signals to obtain high-dimensional behavioral representation features with multi-granularity temporal dependencies.

[0134] The input consists of a sequence of behavioral micro-patterns from individual doctors, including mouse trajectory signals, dwell distribution data, keyboard operation segment signals, and eye-tracking gaze feature embedding vectors, which have been normalized and labeled in previous steps.

[0135] A hierarchical sequence modeling structure (such as a stacked temporal neural network or a hierarchical Transformer model, with parameters including window length l_win, feature dimension d_feat, and number of layers N_layer) is adopted to achieve multi-scale spatiotemporal feature extraction of behavioral micro-pattern sequences.

[0136] Furthermore, by combining behavioral label embedding with the original behavioral temporal signal as input, and using temporal coding algorithms (such as positional coding and sliding window embedding), the temporal features of the input sequence are adjusted, enabling the model to capture dynamic dependencies across time scales and enhance the complementarity and interaction between different types of behavioral features.

[0137] Furthermore, a multi-scale temporal convolution module (parameters including kernel length k_1, k_2, ..., stride s) is used to extract local and global dynamic features of behavioral micro-pattern sequences under different behavioral persistence and change rates, respectively obtaining multi-granularity behavioral temporal feature representation vectors such as short-term changes (e.g., high frequency range of mouse clicks), medium-term dwell time (e.g., high gaze duration in local areas), and long-term trends (e.g., continuous keyboard input).

[0138] Furthermore, by utilizing hierarchical attention mechanisms (such as local window attention and global cross-window attention), the relative weights of different modal features (mouse, eye movement, keyboard) within the same time step are adaptively modeled to achieve information flow and weight fusion between behavioral embedding vectors, and output the integrated multidimensional behavioral representation features.

[0139] By employing residual connections and normalization methods (such as LayerNorm and residual short-circuiting), the stability of gradient propagation and the consistency of numerical distribution of extracted high-dimensional spatiotemporal features between network layers are ensured, thereby improving the robustness of deep feature representation.

[0140] Through the above steps, the model outputs a high-dimensional behavioral representation feature sequence with aligned behavioral category labels, temporal dependencies, and modal correlations as the output of this sub-step. This provides a structured, multi-granular, and multi-modal input foundation for subsequent regional feature fusion and cross-modal joint modeling modules, enabling the full expression and correlation modeling of heterogeneous behavioral signal features.

[0141] For example, in a tertiary-level hospital image reading scenario, doctors can collect 1800 mouse operation, eye movement, and keyboard event sequences per minute. The sliding window width is set to 64 (sampling interval 20ms), and the feature dimension is configured to 28. A two-layer stacked Transformer Encoder is used as the backbone structure for hierarchical temporal modeling. The first layer has a window length of 8, handling short-term mouse-gaze module associations; the second layer has a window length of 48, focusing on long-term temporal dependencies of behaviors. The multi-scale convolutional kernels are k_1=3, k_2=7, and k_3=13. Hierarchical attention adopts a local / global weight adaptive allocation method, with an average attention distribution range of [0.18, 0.42] across different behavioral modalities. After residual connections and LayerNorm normalization, the output behavioral representation vector has a dimension of 64. In actual verification, this high-dimensional representation feature improves the representation ability of subsequent relevance discrimination by 26.3% and the pattern discrimination between doctors with different professional backgrounds by 18.7%, laying a high-discriminative and highly expressive input foundation for backend regional feature alignment and semantic content fusion.

[0142] S4.3: For the labels in the image viewing interface, the visual features of the labeled areas are extracted using an image feature encoding module (such as a visual Transformer backbone network) to obtain regional image feature embeddings. Then, using behavioral representation features and label embeddings as synergistic conditions, the behavioral features and regional features are initially aligned and fused, laying the foundation for subsequent cross-modal modeling.

[0143] The input conditions include high-dimensional behavioral representation feature sequences obtained through multi-level time series modeling, doctor behavior clustering label encoding results, image reading system interface region labels, and the original pathological image data or region candidate windows to be processed.

[0144] The method employs region localization and interface label mapping (parameters: interface coordinates X_{viewport}, region label R, current zoom S_{zoom}) to achieve spatial alignment between the pathological image and the doctor's current operating interface area, accurately extracts candidate windows of the pathological image region currently displayed on the screen, and establishes a mapping index from the interface to the pixel blocks of the original image.

[0145] Furthermore, through the image feature encoding module (which employs a visual Transformer backbone network with parameters set to embedding dimension d_emb, number of encoding layers N_layer, window size w_patch, and global / local attention mechanism), the extracted region candidate windows are input into the Transformer backbone to achieve the encoding of region-level visual feature sequences.

[0146] Furthermore, using a multi-head self-attention computation method, feature extraction is performed on fine-grained patches (small regions) within the candidate window to generate patch-level feature representations F. patch And through position encoding P pos Ensure the preservation of spatial hierarchy information.

[0147] Global pooling and region aggregation algorithms are used to reduce and summarize the patch feature sequences, outputting a multi-scale region feature embedding F that represents the global and local features of the current candidate region. region .

[0148] Furthermore, through the feature alignment fusion module (parameters: alignment weight W_{align}, fusion strategy F_{fuse}), the obtained behavioral representation feature sequence and cluster label embedding are concatenated with the region feature embedding, and a label embedding consistency loss L is introduced. align This achieves initial alignment between behavioral features and image region features.

[0149] By employing residual connections and LayerNorm normalization strategies, the behavior-region joint embedding vector is normalized at the module output, forming a high-dimensional, multimodal, and structured region feature embedding F. region-joint This serves as the basic input for subsequent cross-modal feature joint modeling and correlation discrimination.

[0150] The processing method described above transforms the behavioral pattern features and regional visual features obtained in the previous step into spatiotemporally aligned high-dimensional structured embeddings, thereby achieving preliminary labeling and fusion of individual behavioral features and pathological block content, and providing a unified input basis for multimodal correlation modeling.

[0151] For example, in a digital workstation scenario for pathology image reading in a tertiary-level Class A hospital, a doctor selects a suspected tumor area at 40X magnification. The reading system automatically maps the displayed coordinates X_{viewport} = [220, 320, 640, 840] to the original pathology image area. A visual Transformer backbone is used, with an embedding dimension d_emb = 192, a cumulative encoding layer N_layer = 8, and a patch window size w_patch = 16x16 pixels. Each region's candidate window is divided into 40×40 = 1600 patches. Visual features are extracted using a 3-layer multi-head self-attention mechanism, with patch-level feature mean values ​​ranging from -0.013 to 0.17. After global pooling, the region-level embedding F is obtained. region The dimension is 384. The corresponding doctor behavioral characteristics F... behavior (Length 64) is concatenated with the clustering label code (Length 7), and the weight parameter W is fused. align =0.42 weighted, alignment loss optimization for joint embedding L align The final output is a high-dimensional region-behavior joint embedding F. region-joint This is used for subsequent text feature embedding and multimodal correlation discrimination. In real-world testing, this fusion method improved the consistency of regional features between fine-grained pathological content in the high-density mouse trajectory area and the high-gaze-heat area of ​​the doctor by 14.6%, improved the discriminativeness of regional interest points across doctor models by 18.2%, and improved the recognition accuracy of the joint embedding layer output feature distribution in subsequent correlation discrimination tasks by 12.5%.

[0152] S4.4: Text encoding is performed on the report content summary. A text Transformer encoder is used to perform contextual modeling on the core content and keywords, outputting text embedding feature vectors. Modality normalization is performed simultaneously with behavior label embedding and image region embedding to achieve unified value range of multimodal feature space.

[0153] S4.5: Based on the obtained behavioral representation features, region feature embeddings, and text feature embeddings, a cross-modal joint modeling network is constructed. The multi-source inputs are concatenated into a high-dimensional feature sequence. A hierarchical attention-Transformer structure is introduced to adaptively capture the multi-level temporal dependencies of behavior, region, and text. This enables dynamic correlation modeling of multi-modal cross-temporal high-dimensional features and outputs a comprehensive high-dimensional correlation feature vector, providing an input basis for correlation inference.

[0154] Step S5: Based on the high-dimensional relevance features obtained from joint modeling, an adaptive spatiotemporal attention mechanism is used to allocate feature fusion weights among behavioral micro-patterns, gaze events, report text, and image regions. The fusion parameters are dynamically adjusted to adapt to physician professional preferences and scene changes, improving the diversity and adaptability of relevance inference. Specifically, this includes:

[0155] S5.1: Using the high-dimensional relevance feature vector of joint modeling as input, construct a multimodal feature set including behavioral micro-pattern features, gaze event features, report text features, and image region features. Use a feature embedding layer to perform dimensional alignment and feature abstraction on various features to obtain multimodal feature representations under a unified coding space for subsequent weight allocation.

[0156] Using the high-dimensional relevance feature vector output by joint modeling as input, the feature decomposition and type indexing method is adopted. Based on the preset data structure of behavioral micro-pattern features, gaze event features, report text features and image region features, the input features are split and mapped into multiple unimodal feature subsets according to type.

[0157] Furthermore, for each modal feature subset, a feature embedding layer (parameters: embedding output dimension d_embed, activation function f_act, dropout probability p_drop) is used. Through fully connected mapping and normalization operations, linear transformation and nonlinear activation are performed on the original input features respectively, realizing dimension matching and numerical domain normalization between different features, and obtaining multimodal feature embedding representation.

[0158] Furthermore, by using a fusion algorithm of position encoding and category embedding, relative temporal position weights are assigned to temporal features, and category label encoding is embedded into each modal feature, thereby enhancing the input features with temporal context and business label semantics, forming a multimodal feature embedding with spatiotemporal linkage semantics.

[0159] Furthermore, a multimodal joint normalization algorithm (such as LayerNorm or BatchNorm, with parameters: normalization axis a_norm and scaling parameter gamma) is used to normalize the feature embeddings of each modality within the same spatial range, calibrating the mean and variance between different features, and improving the numerical consistency and comparability of multimodal representations.

[0160] Through the above processing method, the high-dimensional correlation features generated in the previous step are transformed into a unified encoding feature set with dimension alignment, numerical normalization, and multimodal fusion. This provides a standardized and structured input basis for the subsequent feature weight allocation and dynamic attention mechanism, and enables the support capability for collaborative modeling among cross-modal and multi-type features.

[0161] For example, in a tertiary-level Class A hospital's digital image annotation system, the input is a high-dimensional feature vector collected during a pathologist's 60-second image reading process. The behavioral micro-modal feature dimension is 64, the gaze event feature dimension is 24, the report text feature dimension is 128, and the image region feature dimension is 384. The system uses a feature embedding layer to perform linear transformations on the above four types of features, with the embedding dimension uniformly set to d_embed = 96, the activation function being GELU, and the Dropout probability being set to 0.15. Subsequently, sine and cosine positional codes are fused on a time-step basis, and the doctor's behavior category embedding (7 doctor cluster labels) is concatenated into each feature sequence. Multimodal normalization uses the LayerNorm algorithm, with the normalization axis being the feature dimension and the initial scaling value being gamma = 1.0. After the above processing, the output multimodal feature set size is [batch = 64, time = 100, feature = 96], and the mean of each modality sample is normalized to -0.05 to 0.05. Test results show that, with the effective support of the downstream spatiotemporal attention weight allocation module, this feature set improves the feature utilization rate of the relevance discrimination network for behavior-content-region matching tasks by 13.9%, and significantly improves the expressiveness and generalization performance of the model under multi-source information fusion.

[0162] S5.2: Based on multimodal feature representation under a unified coding space, an adaptive spatiotemporal attention mechanism is used to assign correlation weights among micro-pattern features, gaze event features, report text features and image region features for each time step. By calculating the attention distribution matrix among each feature, adaptive fusion weights are obtained to achieve dynamic information flow and highlighting of key points among features.

[0163] The input is a set of multimodal features in a unified coding space after multimodal feature embedding and normalization, including behavioral micro-modal feature embeddings F. bhv gaze event feature embedding F gaze Report text feature embedding F txt and image region feature embedding F img .

[0164] An adaptive spatiotemporal attention mechanism is adopted (parameter: number of attention heads h). att Hidden layer dimension d att The attention normalization method (softmax) enables dynamic correlation weight allocation among four types of features—behavioral micro-patterns, gaze events, report text, and image regions—in each time step.

[0165] Furthermore, by constructing a multimodal self-attention distribution matrix, the relationship distribution between different feature vectors at each time step t is dynamically modeled. Specifically, the attention score is calculated using the following formula:

[0166]

[0167] in, The original attention weights between features i and j at time step t. Let i be the query vector for feature i at time t. Let d be the key vector of feature j. k The dimension of the query / key vector.

[0168] Furthermore, through softmax normalization, the attention distribution corresponding to each type of feature in each time step is as follows:

[0169]

[0170] Where M is the number of feature types (M = 4 in this step), This refers to the adaptive fusion weight of feature i for feature j at time t.

[0171] Furthermore, based on attention distribution For each type of feature component, a weighted feature transfer is performed, and Feature states projected across modalities Highlighting key signals that drive the dynamic interactions:

[0172]

[0173] in, Let j be the value vector of feature j. The result is the multimodal weighted fusion at time t.

[0174] By applying the above calculations step by step across multiple time steps, the output includes an attention matrix and a weighted feature set that contain the fusion weight distribution of each time step and the dynamic interaction and flow of features.

[0175] By using adaptive spatiotemporal attention allocation, the multimodal feature vectors from the previous stage are transformed into adaptive fusion weighted data with feature transfer capabilities and prominent master control signals, thereby achieving the expected technical effects of dynamic information transfer between features and highlighting key phenotypic features.

[0176] For example, in a digital image reading scenario at a tertiary-level Class A hospital, the input includes behavioral micro-modal features (64 dimensions), gaze event features (24 dimensions), report text features (128 dimensions), and image region features (384 dimensions) processed by feature embedding. After being uniformly mapped to the encoding space, each modal feature is 96 dimensions. The system is configured with a multi-head attention mechanism, with h attention heads. att =8, hidden layer dimension d att=128. Each modal feature is mapped independently to obtain the Q / K / V matrix, forming a 4×4 attention distribution matrix for the four types of features within each time step. In practical applications, the average attention weight for the cross-correlation between behavioral micro-patterns and gaze events is 0.34, the average weight for gaze events and image regions is 0.46, and the weight for report text and behavioral micro-patterns is 0.23, reflecting the dominant information channel in the current physician workflow. The final output temporal fusion feature weight sequence is used for subsequent personalized preference modeling and relevance inference, improving the information expression power under scene adaptation and the model's fusion and discrimination capabilities. The AUC performance of the actual relevance inference task can be improved by approximately 11.2%.

[0177] S5.3: Using adaptive fusion weights and multimodal feature representations as input, feature-level weighted fusion is performed. Behavioral micro-pattern features, gaze event features, report text features, and image region features are weighted and summed according to the acquired attention weights to obtain a scene-aware driven fusion feature vector, providing a highly expressive input basis for subsequent personalized preference modeling and relevance inference.

[0178] S5.4: For the fused feature vector, combined with the doctor's professional preference label and scene change detection signal, the weight adjustment rules of the adaptive spatiotemporal attention network are optimized in real time through the dynamic parameter adjustment module. The optimization rules are fine-tuned based on the doctor's behavior profile library and historical feedback data to achieve dynamic adaptation to different doctors' professional habits and real-time scene status, and continuously improve the personalized expression ability of correlation inference.

[0179] S5.5: Based on the feature weighted fusion output optimized by dynamic parameters, it implements diversity relevance expression enhancement. Through a multi-head attention mechanism, it further deconstructs and reorganizes the fused feature vector, extracts multi-angle and multi-granularity relevance sub-features, and outputs multi-level and multi-view feature fusion results, providing rich input for subsequent multi-level relevance discrimination networks, and significantly improving the diversity and adaptability of content-region matching.

[0180] Step S6: Input the multimodal high-dimensional features fused by the spatiotemporal attention mechanism into the relevance discrimination network, calculate the multi-level relevance probability output between the current report content and the screen image region (including global, local, and patch-level fine-grained levels), to achieve high-level content-region matching based on personalized doctor operations. Specifically, this includes:

[0181] S6.1: The multimodal high-dimensional feature set (including behavioral micro-pattern features, gaze event features, report text features and image region features) after fusion by the adaptive spatiotemporal attention mechanism is preprocessed. Through normalization and feature recoding techniques, the consistency of feature distribution of the input correlation discrimination network is ensured so as to obtain a unified multimodal high-dimensional feature vector.

[0182] S6.2: Based on the above unified multimodal high-dimensional feature vector, a multi-layered nested correlation discrimination network (such as a multi-level Transformer discrimination structure or a multi-scale convolutional neural network) is used to perform feature mapping and discrimination at three levels: global, local, and patch fine-grained, in order to extract deep correlation representations under the combination of each level.

[0183] S6.3: For the high-dimensional features of the global layer of the entire set, the global attention weighting mechanism is used to calculate the overall correlation probability between the report content and the entire pathological image region to obtain the global content-region matching probability distribution.

[0184] S6.4: For the high-dimensional features of the local layer of the region, through region division and local attention mechanism, calculate the correlation probability between the report content and the currently displayed local region on the screen to obtain a correlation probability heatmap of local content-region matching.

[0185] The input is a multimodal high-dimensional feature set fused by an adaptive spatiotemporal attention mechanism, including behavioral micro-pattern features, gaze event features, report text features, and image region features. To meet the discrimination requirements of the local layer, the correlation probability distribution between the report content and the currently displayed local image region is calculated through region segmentation and local attention mechanism.

[0186] The region partitioning algorithm (parameters: region segmentation scheme S_reg, region granularity G, such as according to the standard visible partition of medical images or adaptive thermal regions) is used to divide the currently displayed image on the screen into several spatially continuous local regions, and each region generates an independent image region index and mask.

[0187] Furthermore, image region features corresponding to each segmented local region are extracted and embedded. Combine behavioral micro-pattern features aligned to the same duration gaze event characteristics and report text features Constructing a regional multimodal input feature matrix Where l∈[1,G] represents the region index.

[0188] Employing a local self-attention mechanism (parameter: number of attention heads h) reg Hidden layer dimension d reg For each local region Relevance modeling is performed. Specifically, the attention weights for local regions are calculated using the following formula:

[0189]

[0190] in, The original attention weights between features i and j in local region l. Let i be the query vector for feature i in region l. Let j be the key vectors in the same region as feature j.

[0191] By normalizing using softmax, an adaptive weight distribution within the local region is obtained:

[0192]

[0193] normalized weights and value vectors of various features The weighted fusion result is calculated based on the following:

[0194]

[0195] Furthermore, a regional-level correlation probability output network (such as a regional local discrimination subnet or classification head) is constructed to... As input, a fully connected layer with a sigmoid activation function is used to output the region correlation probability.

[0196] After iterative processing of all local regions, all are pieced together. As a result, a spatially distributed regional correlation probability heatmap is formed.

[0197] Through the aforementioned region segmentation, local feature fusion, attention normalization, and correlation probability output network, the multimodal high-dimensional features of behavior-text-image are transformed into a spatially precise and nuanced regional content-region correlation probability distribution. This enables accurate subjective correlation judgment between report content and local areas displayed on the screen under personalized doctor operations, providing strong support for downstream label generation and visual interpretation.

[0198] For example, in a tertiary-level hospital's image reading scenario, the system uses an 8×8 grid partitioning scheme to divide the screen display area into 64 independent local regions (G=64). The image features extracted from each region have a dimension of 128, with behavioral micro-patterns and gaze events having dimensions of 32 and 16, respectively. The local self-attention module is configured with 4 heads, each with a hidden layer dimension of 64. The region local correlation probability output network structure is a two-layer fully connected network (with 32 and 1 hidden units), connected to a sigmoid function at the end. In practical applications, for the same report content and a 60-second image reading period, the system outputs a 64-dimensional spatial correlation probability vector, which is then reconstructed into an 8×8 heatmap. Regions that doctors focus on deeply, operate on frequently, or are directly mentioned in the text have a mean correlation probability of over 0.85; non-focused regions have a probability below 0.15. This heatmap intuitively reflects the subjective correspondence between the report content and the image reading area. In actual verification, the region positioning accuracy rate improved by over 14%, providing a high-precision discrimination basis for subsequent segmentation and annotation loop closure.

[0199] S6.5: Based on the fine-grained patch layer's high-dimensional features, and using fine-grained patch segmentation and sequence modeling, dynamically assigns relevance weights to each patch, outputting a refined relevance probability matrix between the report content and each patch, achieving accurate mapping and discrimination results between content and image micro-regions.

[0200] S6.6: The relevance probability outputs at three levels—global, local, and patch—obtained through execution are aggregated using a multi-level fusion strategy (such as weighted average or confidence integration algorithm) to form the final hierarchical content-region relevance discrimination output, providing accuracy assurance for tag pairing decisions and automated annotation.

[0201] Step S7: For the output multi-level correlation probability results, an adaptive threshold judgment criterion is set. When the correlation probability exceeds a specific threshold, a label pairing relationship between the report content and the corresponding image region is automatically established; otherwise, the manual or semi-automatic verification state is maintained, thus realizing dynamic label generation. Specifically, this includes:

[0202] S7.1: Normalize the multi-level correlation probabilities (including global probability of the entire set, local probability of the region, and fine-grained probability of the patch) output by the correlation discrimination network to ensure that the probability distribution scale of different levels is uniform, which facilitates the dynamic comparison of subsequent adaptive threshold criteria.

[0203] S7.2: Based on the standardized multi-level correlation probability, the optimal adaptive threshold is dynamically calculated for different doctors and different workflows by using probability density estimation algorithms (such as Gaussian kernel density estimation) combined with historical manual verification data, so as to adapt to individual behavior profiles and scene changes.

[0204] S7.3: Standardized multi-level correlation probabilities are discriminated based on adaptive threshold criteria. Report content exceeding the threshold is automatically paired with the corresponding image region to establish a label pairing relationship and output structured pairing labels for subsequent semantic keyword and region mask mapping generation.

[0205] The input is the standardized multi-level correlation probability result, including the global probability of the entire set, the local probability of the region, and the fine-grained probability of the patch. The correlation probabilities of each level have been uniformly distributed by the previous steps.

[0206] An adaptive threshold criterion (parameters: dynamically set thresholds θ_glo, θ_reg, and θ_patch for each level and adapted to the behavior profile library and historical verification tags) is used to perform discrimination processing on the relevance probability of each level, thereby automatically determining the tag matching eligibility.

[0207] Furthermore, by traversing multi-level correlation probability vectors Calculate the discrimination result of each element and its corresponding adaptive threshold, based on the following discrimination formula:

[0208]

[0209] in, For the k-th level discriminant label (binary), Let θ be the correlation probability at level k. (k) For the k-th level adaptive threshold, This is an indicator function.

[0210] Furthermore, by discriminant vectors All paired units with a score of 1 are selected, and their index positions are mapped to create a pairing list. For each pairing result, the context text number of the relevant report content, the identification keywords, the spatial coordinates of the paired image region and the region mask index, as well as the relevance discrimination confidence score are extracted to generate structured pairing label data.

[0211] Furthermore, a structured tag auto-encoding template (parameters: tag fields include report_id, keyword, region_id, mask_index, match_score, level_tag) is used to uniformly encode each item in the pairing list, forming a tag object set that meets the downstream semantic keyword and region mask fusion requirements.

[0212] Through the above chain processing, standardized multi-level relevance probability data is transformed into a structured tag set that can be used for subsequent automatic keyword-region mask mapping, achieving efficient and intelligent tag matching between content and image regions.

[0213] For example, in a tertiary-level Class A hospital's digital pathology image reading system, a standardized regional local correlation probability heatmap is generated based on a doctor's complete report text and a 60-second viewing period. The image is 64-dimensional, with each dimension corresponding to an 8×8 grid of sub-regions. An adaptive threshold θ_reg = 0.80 is configured. Traversing the heatmap, nine sub-regions with probabilities p_corr^{(l)} exceeding the threshold are identified. These nine sub-regions are designated as high-confidence content-region pairing units. For these units, pathological keywords (such as "adenocarcinoma" and "Grade G3") in the current report text are automatically associated, and the corresponding mask region index and relevance probability are extracted and uniformly encoded into structured pairing labels. For example, the label [report_id = 1021, keyword = 'adenocarcinoma', region_id = 22, mask_index = 4, match_score = 0.87, level_tag = 'reg'] is generated. In this case, the actual automatic label pairing accuracy reaches 95%, and the label generation latency is less than 30 milliseconds, supporting a highly efficient automated process for subsequent image segmentation and semantic fusion.

[0214] S7.4: For correlation probability results that do not reach the threshold, they are automatically marked as pending manual verification and a corresponding list of manual verification tasks is generated to achieve seamless integration between automatic model judgment and manual interactive review.

[0215] S7.5: Based on structured matching tags and feedback from manual verification, periodically collect the correctness data of the verification process, continuously optimize the adaptive threshold calculation model using online statistical methods, dynamically adjust the threshold parameters, and iteratively improve the accuracy and robustness of the system's tag generation.

[0216] Step S8: Based on continuously collected doctor feedback and review annotation information, the behavioral profile library and correlation discrimination network parameters are periodically updated. An online fine-tuning strategy is adopted to optimize the behavioral feature extraction and correlation discrimination process, thereby achieving dynamic self-learning and model self-adaptation under different professional habits and reading conditions, ensuring the long-term robustness and generalization ability of label annotation.

[0217] Specifically, it includes:

[0218] S8.1: Continuously collect the correctness data of the generated structured matching labels and the output of the manual verification process in real time. Use the user label feedback collection mechanism and the review annotation data flow interface to collect the label feedback metadata and review operation log data of individual doctors in order to obtain high confidence feedback signals.

[0219] S8.2: Based on the high-confidence feedback signal of periodic aggregation, the annotation quality assessment algorithm (such as consistency measure, F1 score, etc.) is applied to perform statistical analysis on the established behavioral cluster labels and cluster center features in the behavioral profile library, identify the offset and abnormal distribution, and realize the feedback loop correction of the individual doctor behavioral profile library.

[0220] Using periodically collected high-confidence feedback signals (including user tag feedback and manually reviewed log data) as input, it provides a data foundation for the statistical analysis of the established behavioral clustering tags and cluster center features of the behavioral profile database.

[0221] A labeling quality assessment algorithm (parameters: consistency metric algorithm, including label consistency rate, cluster purity index, and classification performance evaluation index such as accuracy, recall, F1 score, etc.) is adopted to quantify the quality of different behavior cluster labels in the behavior profile database.

[0222] Furthermore, for each type of behavior cluster label and its corresponding cluster center features, the labeling consistency rate with the high-confidence feedback label is calculated using the following formula:

[0223]

[0224] Among them, C agree N represents the consistency rate between the behavioral cluster categories and the feedback labels. matched N is the number of instances under this cluster label that match the feedback label. total This represents the total number of instances in this cluster.

[0225] Furthermore, the F1 score is used to comprehensively evaluate the labeling precision and recall of each cluster label. The F1 score calculation formula is as follows:

[0226]

[0227] Among them, Precision represents the proportion of true labels in the behavioral cluster labels, and Recall represents the proportion of all positive labels that are correctly identified as belonging to the cluster, which is obtained through confusion matrix statistics.

[0228] Furthermore, by statistically analyzing the above indicators for all behavioral cluster labels, the distribution shift of behavioral cluster labels and category center features is identified, including decreased consistency, slippage of F1 score, and category center drift. An abnormal distribution detection threshold is set to determine cluster units with weak labeling, abnormal distribution, or deviation from historical patterns.

[0229] Through the above algorithm, the periodic feedback signal is transformed into a statistical index of annotation quality, thereby enabling the identification of abnormal distributions of behavioral cluster labels and cluster center features in the behavioral profile database, and completing the feedback loop correction of the individual doctor behavioral profile database.

[0230] For example, in the digital pathology annotation platform of a tertiary medical institution, periodically collected high-confidence feedback includes structured paired labels and review logs of 50 doctors from the past 24 hours. For the 12 cluster labels obtained from DBSCAN clustering in the behavioral profile database, the corresponding manually verified labels are compared with the sample set contained in each cluster, and the consistency rate C_{agree} is calculated, with a maximum value of 0.97 and a minimum value of 0.83. Further analysis shows that the precision of each cluster ranges from 0.94 to 0.80, and the recall ranges from 0.92 to 0.81. The F1 score is calculated using the above formula, and cluster labels with a score below 0.85 are automatically marked as quality-offset units. For three of the 12 cluster centers, significant fluctuations in label consistency were detected (a decrease of >8% compared to the previous period), and the system automatically proceeded to the subsequent anomaly correction stage. This method achieves dynamic inspection of cluster label quality, ensuring high adaptability of each unit in the profile database to feedback changes, and providing a solid statistical basis for subsequent cluster metadata screening and adaptive optimization of correlation discrimination parameters.

[0231] S8.3: The behavioral profile distribution offset results produced by the annotation quality assessment are compared with the existing multimodal feature template library. Using similarity measurement algorithms and reconstruction error threshold conditions, behavioral micro-patterns with strong relevance to the current task scenario and user tag feedback are selected to generate dynamically adjusted behavioral clustering meta-datasets.

[0232] S8.4: Using the dynamically adjusted behavior clustering metadata as input, an online fine-tuning algorithm (such as cumulative incremental gradient descent) is adopted for the behavior feature extraction layer and parameter mapping layer in the correlation discrimination network to update the network weights and optimizer state in small steps, so as to ensure that the discrimination network has real-time adaptability to changes in the distribution of behavior micro-patterns.

[0233] S8.5: The fine-tuned and optimized correlation discrimination network is evaluated through cross-validation mechanism and historical task backtracking simulation to assess its generalization performance under changes in the professional habits and reading conditions of target doctors. A continuous performance monitoring report is output to provide decision-making basis for subsequent optimization of the image database and discrimination process.

[0234] S8.6: Based on continuous performance monitoring reports and feedback fusion, the model structure adaptive module is periodically activated to adjust the parameters and reconfigure the structure of the behavioral feature extraction process and joint discrimination strategy, realizing the dynamic closed-loop upgrade of the behavioral profile library and the correlation discrimination network parameters, and steadily improving the long-term robustness and generalization ability of label annotation.

[0235] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0236] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and rules of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data labeling method based on user behavior and attention tracking, characterized in that, Includes the following steps: S1: Synchronously and in real time collects raw user behavior data from different doctors from multiple sources, and marks the timeline by combining the image reading interface area and the report content input status. S2: Based on the multi-source original user behavior data, normalization, outlier removal and short-term window segmentation are performed to generate a time-series short-term micro-behavioral unit sequence; S3: For the segmented short-term micro-behavioral unit sequence, an unsupervised clustering algorithm is used to extract behavioral clustering results from the operation records of different doctors to construct a behavioral profile library oriented towards individual differences; S4: Utilizing the clustering identifiers and behavioral features of the aforementioned behavioral profile database, combined with the image viewing interface area labels and report content summaries, a multi-level temporal modeling approach is employed to jointly model the micro-pattern sequences of individual doctor behaviors, image region features, and report text information. This extracts high-dimensional correlation features across modalities and time domains, specifically: Based on the clustering identifiers and micro-pattern sequences of doctor behavior from the behavioral profile database, the behavioral clustering labels are used as input, and the label encoding technology is employed to vectorize the behavioral clustering labels to generate a behavioral label embedding sequence. We use a hierarchical temporal modeling structure to extract features from the micro-pattern sequence of physician behavior, and use behavior embedding vectors and original temporal signals to perform multi-scale dynamic spatiotemporal modeling to obtain high-dimensional behavioral representation features. For the labels in the viewing interface, the visual features of the labeled areas are extracted using an image feature encoding module to obtain regional-level image feature embeddings. The report content summary is text-encoded, and a text Transformer encoder is used to perform contextual modeling of the core content and keywords, outputting text embedding feature vectors. Based on the obtained behavioral representation features, region feature embeddings and text feature embeddings, a cross-modal joint modeling network is constructed to concatenate multi-source inputs into a high-dimensional feature sequence and introduce a hierarchical attention-Transformer structure to adaptively capture the multi-level temporal dependencies of behavior, region and text. S5: Based on the high-dimensional correlation features obtained by joint modeling, the feature fusion weights between behavioral micro-patterns, gaze events, report text and image regions are assigned through an adaptive spatiotemporal attention mechanism, and the fusion parameters are dynamically adjusted. S6: Input the multimodal high-dimensional features after fusing the spatiotemporal attention mechanism into the correlation discrimination network, calculate the multi-level correlation probability output between the current report content and the screen image region; S7: For the output multi-level correlation probability results, set an adaptive threshold judgment criterion. When the correlation probability exceeds the adaptive threshold, automatically establish a label pairing relationship between the report content and the corresponding image region. Otherwise, maintain the manual or semi-automatic verification state. The adaptive threshold is dynamically calculated based on the multi-level correlation probability results and combined with historical manual verification data.

2. The data annotation method based on user behavior and attention tracking according to claim 1, characterized in that, The process following step S7 also includes: S8: Based on continuously collected doctor feedback and review annotation information, the behavioral profile database and correlation discrimination network parameters are periodically updated, and an online fine-tuning strategy is adopted to optimize the behavioral feature extraction and correlation discrimination process.

3. The data annotation method based on user behavior and attention tracking according to claim 1, characterized in that, The multi-source raw user behavior data acquisition in step S1 includes the parallel real-time acquisition of mouse trajectory point sequences, click events, drag intervals, scroll wheel operations and keyboard input signals by an embedded acquisition agent. At the same time, a high-precision system clock is used to mark microsecond-level timestamps, and a multi-channel data structure is used to complete the synchronization and buffering of concurrent data.

4. The data annotation method based on user behavior and attention tracking according to claim 1, characterized in that, Step S2 specifically includes: A unified time baseline calibration and synchronization timing alignment are performed on the collected multi-source raw user behavior data to construct a consistent time-domain dataset across signal sources. Based on the time-synchronized cross-signal source time-domain dataset, a standard normalization algorithm is used to perform a unified normalization mapping of the feature space of the cross-signal source time-domain dataset. A robust outlier detection algorithm is used to identify and remove outliers from the normalized behavioral feature data sequences, and output a set of regular feature data with high confidence. Based on the normalized high-confidence behavioral feature data set, time-series segmentation is performed according to a fixed-length or adaptive variable-length sliding window algorithm to divide the long sequence into serialized short-time window behavioral segments. For all behavioral segments generated by short time windows, the signal type and temporal distribution characteristics are further utilized to construct and output composite fine-sequence behavioral units.

5. The data annotation method based on user behavior and attention tracking according to claim 4, characterized in that, The unified normalization mapping of the feature space is as follows: the collected data is normalized to zero mean and range, and the spatial features are uniformly normalized to the interval [0,1].

6. The data annotation method based on user behavior and attention tracking according to claim 1, characterized in that, Step S3 specifically includes: The input short-term micro-behavioral unit sequence is processed by feature vectorization, and a standardized set of micro-behavioral feature vectors is generated using the statistical parameters of behavioral events. Based on the standardized micro-behavioral feature vector set, an unsupervised clustering algorithm is used to perform cluster analysis on different behavioral units from the same doctor, and output the behavioral pattern category labels and their distribution information for individual doctors. The behavioral pattern category labels are associated and mapped with the original micro-behavioral feature vectors to construct a doctor individual behavior clustering meta-dataset containing elements such as clustering labels, category center features, and time series indexes; For the cross-doctor behavior clustering metadata, perform behavior pattern distribution statistics, combine image reading scenario labels to archive patterns and characterize differences, and form a behavior profile sub-library indexed by doctor identity; By integrating multiple sub-databases of doctors' behavioral profiles and using principal component analysis to integrate high-dimensional features of behavioral clustering results based on behavioral category similarity and operational style spectrum, a comprehensive behavioral profile database for multiple doctors and across scenarios is established.

7. The data annotation method based on user behavior and attention tracking according to claim 1, characterized in that, In step S4, the joint modeling of the behavioral micro-pattern sequence adopts a multi-level temporal structure and a hierarchical attention Transformer model, which integrates behavioral label embedding, region label, image visual features and text features, and enhances the feature representation capability of the model through residual connection and normalization.

8. The data annotation method based on user behavior and attention tracking according to claim 2, characterized in that, Step S8 specifically involves: Based on periodically collected structured paired labels and feedback signals, and through labeling quality assessment methods, the distribution shifts and anomalies of behavioral clustering results are identified, and the behavioral profile database is continuously adjusted in a closed loop.

9. A data annotation system based on user behavior and attention tracking, characterized in that: Data annotation is performed using the data annotation method based on user behavior and attention tracking as described in any one of claims 1-8.

Images