Calligraphy and painting appreciation locking method and device based on eye movement capture

Abstract

This invention discloses a method and device for appreciating and locking calligraphy and paintings based on eye-tracking, relating to the fields of human-computer interaction and digital image processing. It acquires user gaze data and determines coordinates on a digital model of the calligraphy and painting, then uses a time-window-based moving average algorithm to smooth the gaze trajectory, effectively filtering out eye-tracking noise. Next, it extracts gaze duration, pupil changes, and local image features to construct multi-source heterogeneous feature data. A pre-trained intelligent interest recognition model is then used to analyze this multi-source heterogeneous feature data to obtain an eye-tracking interest score. Finally, based on this score, the local area of ​​the calligraphy or painting that the user is interested in is determined and locked. The fusion of multi-dimensional heterogeneous features improves the accuracy of interest recognition, and the coupled resonant particle swarm optimization algorithm is used to optimize model parameters. The pairwise interaction strategy between particles enhances the global optimization capability, significantly improving the model's convergence speed and locking accuracy, thus achieving a precise and intelligent interactive experience for appreciating calligraphy and paintings.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction and digital image processing technology, and more specifically, to a method and apparatus for appreciating and locking calligraphy and paintings based on eye-tracking. Background Technology

[0002] With the development of digital museums and virtual reality technology, the digital appreciation of calligraphy and painting works has become an important way of cultural experience. In traditional human-computer interaction, users usually select areas of interest in calligraphy and painting through mouse clicks or gestures. This method is not only cumbersome and disrupts the immersive experience, but also makes it difficult to capture the user's instantaneous, subconscious points of interest. Eye-tracking technology offers a solution to these problems. However, existing eye-tracking interaction technologies have the following drawbacks when applied to calligraphy and painting appreciation: the inherent physiological tremors and microsaccades of the human eye cause high-frequency jitter in the coordinates of the collected gaze points, resulting in a jagged gaze trajectory, making it difficult to accurately locate fine strokes or specific parts in the calligraphy and painting works. Existing methods mostly rely solely on gaze duration or gaze point density to determine the region of interest, ignoring the user's psychological cognitive state reflected by pupil dilation and contraction, as well as the visual semantic features of the calligraphy and painting images themselves, resulting in low accuracy in interest judgment. When using deep learning models for interest recognition, the model parameter settings often rely on experience, easily getting trapped in local optima, leading to poor adaptability of the model when faced with calligraphy and painting works of different styles. Summary of the Invention

[0003] The present invention provides a method and device for appreciating and locking calligraphy and painting based on eye-tracking, which aims to solve the problems of large eye-tracking trajectory jitter, single interest judgment features and / or insufficient accuracy of recognition models in the prior art.

[0004] The first aspect of this invention provides a method for appreciating and locking calligraphy and paintings based on eye-tracking, comprising: Acquire user gaze data, and determine model gaze coordinates on a pre-constructed digital model of calligraphy and painting based on the user gaze data; A time window-based moving average algorithm is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time, eliminating gaze jitter noise and obtaining a smooth gaze trajectory. Based on the smooth gaze trajectory and user gaze data, gaze duration features, pupil dilation variation features, and local image features are obtained to obtain multi-source heterogeneous feature data; A pre-trained intelligent interest recognition model is used to analyze the multi-source heterogeneous feature data to obtain the user's eye movement interest score for the current gaze region; Based on the eye-tracking interest score, the local area of ​​the painting or calligraphy that the user is currently focusing on is determined and locked, thus completing the art appreciation and locking based on eye-tracking capture.

[0005] Further, acquiring user gaze data and determining model gaze coordinates on a pre-constructed digital model of calligraphy and painting based on the user gaze data includes: The user's gaze data is obtained by acquiring the user's infrared image, gaze duration characteristics, and pupil dilation variation characteristics. Based on the user's infrared image, the model gaze coordinates are determined on a pre-constructed digital model of calligraphy and painting.

[0006] Furthermore, a time-window-based moving average algorithm is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time, eliminating gaze jitter noise and obtaining a smooth gaze trajectory, including: Define a fixed-length time window T, and extract the time window [t] at time t. All model gaze coordinate sequences within [T,t]; The coordinate sequence is assigned a weighting coefficient based on time decay, with the weights decreasing as they approach the current time. t The larger the coordinate weight, the greater the sum of all coordinate weights is to be; The weighted mean of the coordinates was calculated and used as the smoothed gaze coordinates at time t to filter out high-frequency eye movement microsaccades and physiological tremors. Smoothing is performed on the gaze coordinates of multiple consecutive models to obtain a smooth gaze trajectory.

[0007] Furthermore, based on the smooth gaze trajectory and user gaze data, gaze duration features, pupil dilation variation features, and local image features are obtained to acquire multi-source heterogeneous feature data, including: Based on the user gaze data, determine the gaze duration characteristics and pupil dilation change characteristics corresponding to each smooth gaze coordinate in the smooth gaze trajectory; Based on the smooth gaze trajectory, on the pre-constructed digital model of calligraphy and painting, the smooth gaze coordinates are used as the center and local image features of a fixed size are cropped. Smooth gaze trajectory, gaze duration features, pupil dilation variation features, and local image features are used together as multi-source heterogeneous feature data.

[0008] Furthermore, a pre-trained intelligent interest recognition model is used to analyze the multi-source heterogeneous feature data to obtain the user's eye-tracking interest score for the current gaze region, including: The smooth gaze trajectory, gaze duration features, and pupil dilation variation features in the multi-source heterogeneous feature data are combined into a vector, and the temporal feature extraction module in the pre-trained interest degree intelligent recognition model is used to extract features from the vector to obtain temporal features. The visual feature extraction module in the pre-trained interest-based intelligent recognition model is used to extract local image features from the multi-source heterogeneous feature data to obtain visual features. The temporal features and the visual features are fused using the feature fusion module in a pre-trained interest-based intelligent recognition model to obtain fused features. The fused features are analyzed using the feature analysis module in the pre-trained interest-based intelligent recognition model to obtain the user's eye movement interest score for the current gaze region.

[0009] Furthermore, the temporal feature extraction module is configured as a long short-term memory network, the visual feature extraction module is configured as a convolutional neural network, the feature fusion module is used to concatenate the visual features and the fused features into a vector, and the feature analysis module is configured as a backpropagation neural network.

[0010] Furthermore, the pre-training method for the intelligent interest recognition model includes: Initialize the training counter to iter=1 and initialize the maximum number of training iterations; The hyperparameters of the interest-based intelligent recognition model are encoded by particle swarm optimization to obtain a particle swarm; the particle swarm contains multiple different particles. The CRPSO algorithm is used to optimize the particles in the particle swarm, resulting in an optimized particle swarm. Determine if the value of iter is greater than or equal to the maximum number of training iterations. If it is, obtain the target particle based on the optimized particle swarm; otherwise, return to the step of optimizing the particle swarm. The target particle is decoded and used as the final hyperparameter of the interest-based intelligent recognition model, thus completing the pre-training of the interest-based intelligent recognition model.

[0011] Furthermore, based on the aforementioned eye-tracking interest score, the local area of ​​the painting or calligraphy currently being focused on by the user is determined and locked, including: Initialize the interest threshold and duration threshold; When the eye-tracking interest score for multiple consecutive time windows exceeds the interest threshold and the cumulative fixation field of view exceeds the duration threshold, the local image features corresponding to the current smooth fixation trajectory are locked as the local area of ​​the calligraphy or painting that the user is currently focusing on.

[0012] Furthermore, after determining and locking onto the specific area of ​​the painting or calligraphy that the user is currently focusing on, the process also includes: The specific appreciation category of the locked local area of ​​calligraphy and painting is output by performing intersection-union calculation with the pre-annotated semantic segmentation mask in the pre-constructed digital model of calligraphy and painting.

[0013] A second aspect of the present invention provides a calligraphy and painting appreciation locking device based on eye-tracking, comprising: An eye-tracking module is used to acquire user gaze data and determine the model gaze coordinates on a pre-built digital model of calligraphy and painting based on the user gaze data. The smoothing module is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time using a time window-based moving average algorithm, thereby eliminating gaze jitter noise and obtaining a smooth gaze trajectory. The feature extraction module is used to obtain gaze duration features, pupil dilation change features, and local image features based on the smooth gaze trajectory and user gaze data, so as to obtain multi-source heterogeneous feature data. The interest recognition module is used to analyze the multi-source heterogeneous feature data using a pre-trained intelligent interest recognition model to obtain the user's eye movement interest score for the current gaze area. The appreciation and locking module is used to determine and lock the local area of ​​the calligraphy and painting that the user is currently paying attention to based on the eye movement interest score, and to complete the appreciation and locking of calligraphy and painting based on eye movement capture.

[0014] Beneficial effects: This invention provides a method and apparatus for appreciating and locking calligraphy and painting based on eye-tracking. It acquires user gaze data and determines coordinates on a digital model of the calligraphy and painting. A time-window-based moving average algorithm is used to smooth the gaze trajectory, effectively filtering out eye-tracking noise. Then, gaze duration, pupil changes, and local image features are extracted to construct multi-source heterogeneous feature data. A pre-trained intelligent interest recognition model is used to analyze this multi-source heterogeneous feature data to obtain an eye-tracking interest score. Finally, based on this score, the local area of ​​the calligraphy or painting that the user is interested in is determined and locked. The fusion of multi-dimensional heterogeneous features improves the accuracy of interest recognition. Furthermore, a coupled resonant particle swarm optimization algorithm is used to optimize model parameters, and a pairwise interaction strategy between particles enhances the global optimization capability, significantly improving the model's convergence speed and locking accuracy, thus achieving a precise and intelligent interactive experience for calligraphy and painting appreciation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a method for appreciating and locking calligraphy and paintings based on eye-tracking, proposed in an embodiment of the present invention; Figure 2 This is a schematic diagram of a calligraphy and painting appreciation locking device based on eye-tracking capture, according to an embodiment of the present invention.

[0017] Among them, 201-eye tracking module, 202-smoothing module, 203-feature extraction module, 204-interest recognition module, and 205-appreciation locking module. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for appreciating and locking calligraphy and paintings based on eye-tracking, including: S101. Obtain user gaze data and determine model gaze coordinates on a pre-constructed digital model of calligraphy and painting based on the user gaze data; S102. A time window-based moving average algorithm is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time, eliminating gaze jitter noise and obtaining a smooth gaze trajectory. S103. Based on the smooth gaze trajectory and user gaze data, obtain gaze duration features, pupil dilation and sizing change features, and local image features to obtain multi-source heterogeneous feature data. S104. The multi-source heterogeneous feature data is analyzed using a pre-trained intelligent interest recognition model to obtain the user's eye movement interest score for the current gaze area. S105. Based on the eye-tracking interest score, determine and lock the local area of ​​the calligraphy or painting that the user is currently focusing on, and complete the calligraphy or painting appreciation and locking based on eye-tracking capture.

[0020] In some possible embodiments, acquiring user gaze data and determining model gaze coordinates on a pre-built digital model of calligraphy and painting based on the user gaze data includes: The user's gaze data is obtained by acquiring the user's infrared image, gaze duration characteristics, and pupil dilation variation characteristics. Based on the user's infrared image, the model gaze coordinates are determined on a pre-constructed digital model of calligraphy and painting.

[0021] Acquiring the user's gaze coordinates and pupil dilation / contraction variations is the core application of eye-tracking technology. This process primarily relies on specialized eye-tracking hardware and supporting data processing algorithms. Modern non-invasive eye trackers mainly employ pupil-corneal reflection technology. The system projects near-infrared light, imperceptible to the human eye, onto the user's eyes, and one or more high-speed infrared cameras capture the reflected images from the eyes, obtaining the user's infrared image. Two key features can be clearly identified in the infrared image: Center of the pupil: Light enters the pupil and is reflected from the retina, forming a relatively dark area; The corneal reflective spot (Pulchin spot): A bright spot formed by the reflection of light off the outer surface of the cornea. The position of this spot remains stable when the head is in a relatively fixed position. When a user's eyes move, the center of the pupil shifts, while the corneal reflector remains relatively stable. The system tracks the relative positional changes of these two features in real time using an algorithm and calculates the user's gaze direction (angle) using a pre-calibrated geometric model. Combined with the parameters of the display device, the gaze direction is ultimately mapped to the coordinates of the fixation point on the screen displaying the digital model of the painting or calligraphy. To more clearly distinguish the pupil (especially dark pupils), high-end eye trackers can also alternately use two infrared light sources at different positions to produce bright and dark pupil effects, enhancing the contrast between the pupil and iris through a differential algorithm.

[0022] Before the user starts a task or while in a relaxed state, the average pupil diameter under stable lighting is recorded as a baseline. The current pupil diameter minus the baseline pupil diameter can be used as the pupil dilation and contraction feature, or (current diameter - previous diameter) / sampling interval can be used as the pupil dilation and contraction feature to reflect the speed of pupil dilation or contraction.

[0023] When the system detects that a user has started viewing the current digital model of the painting or calligraphy, it records the start timestamp. When the model's gaze coordinates fall into a certain feature region (e.g., a pre-defined candidate area for seals through image segmentation), if the gaze moves out and then back in, the absolute duration of that region is accumulated. The absolute duration divided by (current time - start timestamp) is used as the gaze duration feature.

[0024] In some possible embodiments, a time-window-based moving average algorithm is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time, eliminating gaze jitter noise and obtaining a smooth gaze trajectory, including: Define a fixed-length time window T, and extract the time window [t] at time t. All model gaze coordinate sequences within [T,t]; The coordinate sequence is assigned a weighting coefficient based on time decay, with the weights decreasing as they approach the current time. tThe larger the coordinate weight, the greater the sum of all coordinate weights is to be; The weighting coefficients based on time decay can be: ; The number corresponding to the current time window i Each model gaze coordinate, i =1,2,...,NP, where NP is the total number of model gaze coordinates within the current time window. It is a natural constant. The attenuation coefficient is... For the first i The timestamps corresponding to the gaze coordinates of each model are designed such that the coordinates closer to time t have greater weights.

[0025] The weighted mean of the coordinates was calculated and used as the smoothed gaze coordinates at time t to filter out high-frequency eye movement microsaccades and physiological tremors. Smoothing is performed on the gaze coordinates of multiple consecutive models to obtain a smooth gaze trajectory.

[0026] It's worth noting that, because the algorithm is real-time, the time window is updated every few seconds. With this new time window, the smooth gaze coordinates can be recalculated, and the smooth gaze trajectory can be updated using these coordinates to obtain a fixed-dimensional smooth gaze trajectory. For example, the smooth gaze trajectory should include smooth gaze coordinates. When new smooth gaze coordinates arrive, the oldest smooth gaze coordinate is removed, and then the new smooth gaze coordinates are added to the smooth gaze trajectory sequentially, thus obtaining the new time window.

[0027] This invention employs a time-window-based moving average algorithm and introduces time-decay-based weighting coefficients to process the gaze trajectory. Compared to traditional simple mean filtering, this invention assigns greater weight to coordinates closer to the current moment, effectively filtering out inherent high-frequency physiological jitter and micro-jump visual noise, while avoiding the gaze lag effect caused by excessive smoothing. This results in a smooth gaze trajectory that more realistically and smoothly reflects the user's actual visual attention path, providing a high-quality data foundation for subsequent feature extraction.

[0028] In some possible embodiments, based on the smooth gaze trajectory and user gaze data, gaze duration features, pupil dilation variation features, and local image features are obtained to obtain multi-source heterogeneous feature data, including: Based on the user gaze data, determine the gaze duration characteristics and pupil dilation change characteristics corresponding to each smooth gaze coordinate in the smooth gaze trajectory; Based on the smooth gaze trajectory, on the pre-constructed digital model of calligraphy and painting, the smooth gaze coordinates are used as the center and local image features of a fixed size are cropped. For local image features, it is only necessary to focus on the area where the user's current gaze is located. Therefore, the last smooth gaze coordinate in the smooth gaze trajectory can be used as the center, and local image features of a fixed size (such as rectangles) can be cropped.

[0029] Smooth gaze trajectory, gaze duration features, pupil dilation variation features, and local image features are used together as multi-source heterogeneous feature data.

[0030] In some possible embodiments, a pre-trained intelligent interest recognition model is used to analyze the multi-source heterogeneous feature data to obtain the user's eye-tracking interest score for the current gaze region, including: The smooth gaze trajectory, gaze duration features, and pupil dilation variation features in the multi-source heterogeneous feature data are combined into a vector, and the temporal feature extraction module in the pre-trained interest degree intelligent recognition model is used to extract features from the vector to obtain temporal features. The visual feature extraction module in the pre-trained interest-based intelligent recognition model is used to extract local image features from the multi-source heterogeneous feature data to obtain visual features. The temporal features and the visual features are fused using the feature fusion module in a pre-trained interest-based intelligent recognition model to obtain fused features. The fused features are analyzed using the feature analysis module in the pre-trained interest-based intelligent recognition model to obtain the user's eye movement interest score for the current gaze region.

[0031] Temporal feature extraction module: It adopts a long short-term memory network (LSTM) and takes a temporal vector consisting of fixation duration, pupil change rate and trajectory coordinates as input to extract the temporal dependent features of user eye movement behavior.

[0032] Visual feature extraction module: It uses a convolutional neural network (CNN, such as ResNet-18) to extract deep semantic features of the calligraphy and painting images from local image patches as input.

[0033] Feature fusion and analysis: The feature vectors output by LSTM and CNN are concatenated and input into a fully connected layer (BP neural network), and an interest score between 0 and 1 is output.

[0034] In some possible embodiments, the temporal feature extraction module is configured as a long short-term memory network, the visual feature extraction module is configured as a convolutional neural network, the feature fusion module is used to concatenate visual features and fused features into a vector, and the feature analysis module is configured as a backpropagation neural network.

[0035] Existing technologies often focus only on the gaze point location, neglecting the user's psychological state and the image content. This invention constructs multi-source heterogeneous feature data including gaze duration features, pupil dilation variation features, and local image features. Among them, pupil dilation variation features can objectively reflect the user's cognitive load and emotional arousal (such as pupil dilation when viewing exquisite paintings), combined with local image features (texture, color, and other semantic information of calligraphy and paintings). When a user gazes at complex inscriptions, cognitive load increases, and pupils may dilate. At the same time, the convolutional neural network extracts dense text textures, and the model outputs a high interest score. If the user gazes at large blank areas without pupil change, the model outputs a low interest score. Invalid gazes are filtered out by extracting features from local image features.

[0036] By employing a dual-branch feature extraction and fusion approach combining LSTM and CNN, cross-modal analysis of user behavior and image content was achieved. This multi-dimensional feature fusion significantly improved the computational accuracy of eye-tracking interest scores and effectively reduced the false positive rate.

[0037] It is worth noting that the network described above is merely an example, and other models with the same data extraction capabilities can also be used to construct it. For instance, a multilayer perceptron can be used as the feature analysis module.

[0038] In some possible embodiments, the pre-training method of the interest-based intelligent recognition model includes: Initialize the training counter to iter=1 and initialize the maximum number of training iterations; The hyperparameters of the interest-based intelligent recognition model are encoded by particle swarm optimization to obtain a particle swarm; the particle swarm contains multiple different particles. Particle swarm initialization is a fairly standard technique, typically achieved by randomly initializing hyperparameters at their upper and lower bounds and encoding them into vectors. This will not be elaborated upon further here.

[0039] The CRPSO (Coupled Resonance Particle Swarm Optimization) algorithm is used to optimize the particles in the particle swarm, resulting in an optimized particle swarm. Optionally, sample fusion features and their corresponding artificial labels can be obtained during the training process. These artificial labels are human-annotated interest scores, which can be obtained by the experimenters collecting sample fusion features during the browsing process and manually scoring them.

[0040] The CRPSO algorithm is used to optimize the particles in the particle swarm, including: Social information learning for particles in a particle swarm is as follows: ; ; in, For the first iter The i-th particle in the training process, i=1,2,...,M, where M is the total number of particles in the particle swarm. For the particles after one update, for The update speed during the (iter+1)th training iteration. for The update speed during the iterth training process For inertial weights, The first learning factor can be set to 1.5, or it can be linearly reduced from 2.5 to 0.5. This is the second learning factor, which can be set to 2, or it can increase linearly from 0.5 to 2.5; The first random number between (0,1) The second random number between (0,1) for The corresponding historical best value, It is the optimal particle in the particle swarm. After the first update, the particles are paired up, and the first coupling resonance factor is obtained based on the paired particles: ; in, For the particle corresponding to the first d The first coupling resonance factor of the 3D hyperparameter, To the maximum number of training iterations, For one of the paired particles, the first particle d dimensional hyperparameters, For the second particle in a pair of paired particles, the first... d dimensional hyperparameters, The Euclidean distance between each pair of paired particles is denoted as . This is a minimal constant term, and it is set to 0.001; The second coupling resonance factor is obtained from the pairwise pairing of particles: ; in, For the particle corresponding to the first d The second coupling resonance factor of the 3D hyperparameter, It is a natural constant; Based on the first coupling resonance factor and the second coupling resonance factor, the coupling resonance information is obtained as follows: ; in, For the particle corresponding to the firstd Coupled resonance information of 3D hyperparameters, The third random number between (0,1); Based on the coupling resonance information, the paired particles are updated as follows: ; ; in, For the updated version , For the updated version , The fitness of one of the paired particles. The fitness of the other particle in a pairwise pair is set as the reciprocal of the sum of the loss function value and the minimum constant term. Determine if the value of iter is greater than or equal to the maximum number of training iterations. If it is, obtain the target particle based on the optimized particle swarm; otherwise, return to the step of optimizing the particle swarm. The target particle is decoded and used as the final hyperparameter of the interest-based intelligent recognition model, thus completing the pre-training of the interest-based intelligent recognition model.

[0041] In the pre-training stage of the intelligent interest recognition model, this invention introduces the Coupled Resonance Particle Swarm Optimization (CRPSO) algorithm for hyperparameter optimization. Compared to traditional particle swarm optimization, CRPSO introduces a coupled resonance strategy with pairwise information interaction, simulating the resonance effect between particles in a physical system.

[0042] Through pairwise coupling interactions between particles, the algorithm can exchange information more fully during the search process, avoiding the problem of traditional algorithms easily getting trapped in local optima, thus finding the globally optimal model hyperparameters (such as the number of neural network layers, nodes, learning rate, etc.). The resonance mechanism enhances the propagation efficiency of high-quality solutions, resulting in faster model convergence and higher training efficiency.

[0043] This technology directly improves the generalization ability and prediction accuracy of the interest-based intelligent recognition model, enabling the calligraphy and painting appreciation and locking system to maintain a high level of locking performance when facing different users and different styles of calligraphy and paintings.

[0044] In some possible embodiments, based on the eye-tracking interest score, the local area of ​​the painting or calligraphy currently being focused on by the user is determined and locked, including: Initialize the interest threshold and duration threshold; When the eye-tracking interest score for multiple consecutive time windows exceeds the interest threshold and the cumulative fixation field of view exceeds the duration threshold, the local image features corresponding to the current smooth fixation trajectory are locked as the local area of ​​the calligraphy or painting that the user is currently focusing on.

[0045] Based on eye-tracking interest scores, the system identifies and locks onto the local area of ​​the painting or calligraphy that the user is currently focusing on. An interest threshold (e.g., 0.8) and a duration threshold (e.g., 2 seconds) are set. When the system detects that a region's score continuously exceeds the interest threshold and the duration exceeds the duration threshold, it determines that the region is the user's focus area, automatically locks onto it, and displays relevant appreciation information through highlighting, pop-ups, or other methods.

[0046] In some possible embodiments, after determining and locking the local area of ​​the calligraphy or painting that the user is currently focusing on, the method further includes: The system performs an intersection-union (IoU) calculation between the locked local area of ​​the calligraphy or painting and the pre-annotated semantic segmentation mask in the pre-constructed digital model of the calligraphy or painting, and outputs the specific appreciation category of the locked area. The category of the mask with the highest matching degree can be used as the final locking result (e.g., output "Locked: Shen Zhou's inscription in the lower right corner"). Based on this, the system can trigger voice narration or display a high-definition 3D texture of the local area.

[0047] During the locking phase, this invention employs dual judgment conditions: an interest threshold and a duration threshold. Locking is only triggered when the interest score consistently exceeds the threshold and the cumulative gaze duration reaches the target. This mechanism effectively filters out false trigger signals generated by unconscious eye scanning, ensuring that the locking result truly represents the user's deep appreciation intent, and greatly improving the smoothness and accuracy of the interactive experience.

[0048] like Figure 2 As shown, based on the same inventive concept, this embodiment of the invention also provides a calligraphy and painting appreciation locking device based on eye-tracking capture, comprising: The eye-tracking module 201 is used to acquire user gaze data and determine the model gaze coordinates on a pre-constructed digital model of calligraphy and painting based on the user gaze data. The smoothing module 202 is used to smooth the gaze trajectory formed by the model gaze coordinates over continuous time using a time window-based moving average algorithm to eliminate gaze jitter noise and obtain a smooth gaze trajectory. The feature extraction module 203 is used to obtain gaze duration features, pupil dilation change features and local image features based on the smooth gaze trajectory and user gaze data, so as to obtain multi-source heterogeneous feature data. Interest recognition module 204 is used to analyze the multi-source heterogeneous feature data using a pre-trained intelligent interest recognition model to obtain the user's eye movement interest score for the current gaze area; The appreciation locking module 205 is used to determine and lock the local area of ​​the calligraphy and painting that the user is currently paying attention to based on the eye movement interest score, and complete the appreciation locking of calligraphy and painting based on eye movement capture.

[0049] The calligraphy and painting appreciation locking device based on eye-tracking provided in this embodiment of the invention can execute the above-mentioned method and technical solution. Its principle and beneficial effects are similar, and will not be described again here.

[0050] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps in the method disclosed in this invention.

[0051] This invention also provides a computer program product that, when run on an electronic device, causes a processor to execute the steps in the method disclosed in this invention.

[0052] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0053] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0057] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0058] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An eye movement capture-based calligraphy and painting appreciation locking method, characterized in that, include: Acquire user gaze data, and determine model gaze coordinates on a pre-constructed digital model of calligraphy and painting based on the user gaze data; A time window-based moving average algorithm is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time, eliminating gaze jitter noise and obtaining a smooth gaze trajectory. Based on the smooth gaze trajectory and user gaze data, gaze duration features, pupil dilation variation features, and local image features are obtained to obtain multi-source heterogeneous feature data; A pre-trained intelligent interest recognition model is used to analyze the multi-source heterogeneous feature data to obtain the user's eye movement interest score for the current gaze region; Based on the eye-tracking interest score, the local area of ​​the painting or calligraphy that the user is currently focusing on is determined and locked, thus completing the art appreciation and locking based on eye-tracking capture.

2. The calligraphy and painting appreciation locking method based on eye movement capture according to claim 1, characterized in that, Acquiring user gaze data and determining model gaze coordinates on a pre-constructed digital model of calligraphy and painting based on the user gaze data includes: The user's gaze data is obtained by acquiring the user's infrared image, gaze duration characteristics, and pupil dilation variation characteristics. Based on the user's infrared image, the model gaze coordinates are determined on a pre-constructed digital model of calligraphy and painting.

3. The calligraphy and painting appreciation locking method based on eye movement capture according to claim 1, characterized in that, A time-window-based moving average algorithm is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time, eliminating gaze jitter noise and obtaining a smooth gaze trajectory, including: Define a fixed-length time window T, and extract the time window [t] at time t. All model gaze coordinate sequences within [T,t]; The coordinate sequence is assigned a weighting coefficient based on time decay, with the weights decreasing as they approach the current time. t The larger the coordinate weight, the greater the sum of all coordinate weights is to be; The weighted mean of the coordinates was calculated and used as the smoothed gaze coordinates at time t to filter out high-frequency eye movement microsaccades and physiological tremors. Smoothing is performed on the gaze coordinates of multiple consecutive models to obtain a smooth gaze trajectory.

4. The method for appreciating and locking calligraphy and paintings based on eye-tracking capture according to claim 3, characterized in that, Based on the smooth gaze trajectory and user gaze data, gaze duration features, pupil dilation variation features, and local image features are obtained to generate multi-source heterogeneous feature data, including: Based on the user gaze data, determine the gaze duration characteristics and pupil dilation change characteristics corresponding to each smooth gaze coordinate in the smooth gaze trajectory; Based on the smooth gaze trajectory, on the pre-constructed digital model of calligraphy and painting, the smooth gaze coordinates are used as the center and local image features of a fixed size are cropped. Smooth gaze trajectory, gaze duration features, pupil dilation variation features, and local image features are used together as multi-source heterogeneous feature data.

5. The method for appreciating and locking calligraphy and paintings based on eye-tracking capture according to claim 4, characterized in that, A pre-trained intelligent interest recognition model is used to analyze the multi-source heterogeneous feature data to obtain the user's eye movement interest score for the current gaze region, including: The smooth gaze trajectory, gaze duration features, and pupil dilation variation features in the multi-source heterogeneous feature data are combined into a vector, and the temporal feature extraction module in the pre-trained interest degree intelligent recognition model is used to extract features from the vector to obtain temporal features. The visual feature extraction module in the pre-trained interest-based intelligent recognition model is used to extract local image features from the multi-source heterogeneous feature data to obtain visual features. The temporal features and the visual features are fused using the feature fusion module in a pre-trained interest-based intelligent recognition model to obtain fused features. The fused features are analyzed using the feature analysis module in the pre-trained interest-based intelligent recognition model to obtain the user's eye movement interest score for the current gaze region.

6. The method for appreciating and locking calligraphy and paintings based on eye-tracking capture according to claim 5, characterized in that, The temporal feature extraction module is configured as a long short-term memory network, the visual feature extraction module is configured as a convolutional neural network, the feature fusion module is used to concatenate visual features and fused features into a vector, and the feature analysis module is configured as a backpropagation neural network.

7. The method for appreciating and locking calligraphy and paintings based on eye-tracking capture according to claim 6, characterized in that, The pre-training method for the interest-based intelligent recognition model includes: Initialize the training counter to iter=1 and initialize the maximum number of training iterations; The hyperparameters of the interest-based intelligent recognition model are encoded by particle swarm optimization to obtain a particle swarm; the particle swarm contains multiple different particles. The CRPSO algorithm is used to optimize the particles in the particle swarm, resulting in an optimized particle swarm. Determine if the value of iter is greater than or equal to the maximum number of training iterations. If it is, obtain the target particle based on the optimized particle swarm; otherwise, return to the step of optimizing the particle swarm. The target particle is decoded and used as the final hyperparameter of the interest-based intelligent recognition model, thus completing the pre-training of the interest-based intelligent recognition model.

8. The method for appreciating and locking calligraphy and paintings based on eye-tracking capture according to claim 1, characterized in that, Based on the eye-tracking interest score, the local area of ​​the painting or calligraphy that the user is currently focusing on is determined and locked, including: Initialize the interest threshold and duration threshold; When the eye-tracking interest score for multiple consecutive time windows exceeds the interest threshold and the cumulative fixation field of view exceeds the duration threshold, the local image features corresponding to the current smooth fixation trajectory are locked as the local area of ​​the calligraphy or painting that the user is currently focusing on.

9. The method for appreciating and locking calligraphy and paintings based on eye-tracking capture according to claim 8, characterized in that, After determining and locking onto the specific area of ​​the painting or calligraphy that the user is currently focusing on, the process also includes: The specific appreciation category of the locked local area of ​​calligraphy and painting is output by performing intersection-union calculation with the pre-annotated semantic segmentation mask in the pre-constructed digital model of calligraphy and painting.

10. A calligraphy and painting appreciation locking device based on eye-tracking, characterized in that, include: An eye-tracking module is used to acquire user gaze data and determine the model gaze coordinates on a pre-built digital model of calligraphy and painting based on the user gaze data. The smoothing module is used to smooth the gaze trajectory formed by the model's gaze coordinates over continuous time using a time window-based moving average algorithm, thereby eliminating gaze jitter noise and obtaining a smooth gaze trajectory. The feature extraction module is used to obtain gaze duration features, pupil dilation change features, and local image features based on the smooth gaze trajectory and user gaze data, so as to obtain multi-source heterogeneous feature data. The interest recognition module is used to analyze the multi-source heterogeneous feature data using a pre-trained intelligent interest recognition model to obtain the user's eye movement interest score for the current gaze area. The appreciation and locking module is used to determine and lock the local area of ​​the calligraphy and painting that the user is currently paying attention to based on the eye movement interest score, and to complete the appreciation and locking of calligraphy and painting based on eye movement capture.

Images