Sign language key region data set construction method and system based on eye movement tracking

Abstract

The invention discloses a sign language key area data set construction method and system based on eye movement tracking, and the method comprises the steps: dividing a deaf person cognition subgroup through a three-dimensional classification matrix of deafness time * sign language learning age * communication habits, and generating a multi-dimensional sign language video adaptive to different subgroups; collecting eye movement data of each sub-group, labeling first-level physical regions such as hands and faces, and completing second-level cognitive weight labeling by combining sub-group features; and adjusting an action boundary threshold through a dynamic time warping algorithm, and outputting a data set fused by the physical annotation and the cognitive weight. The system comprises a cognitive subgroup division module, a video generation module, an eye movement acquisition module, a two-stage labeling module, a data preprocessing module, a dynamic time warping module and a data set output module. The problem that a traditional data set neglects cognitive differences of deaf people is solved, and the adaptability of a sign language recognition model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a method and system for constructing a sign language key area data set based on eye tracking. Background Art

[0002] Sign language, as the first language of the deaf community, is the core vehicle for information exchange and cultural heritage. With the rapid development of information technology, artificial intelligence-driven natural sign language recognition (SLR) and translation technologies have shown great potential, and are expected to bridge the communication gap between the deaf and hearing communities and promote the construction of a barrier-free social environment. The core challenge of sign language recognition technology lies in its high visual complexity: sign language vocabulary collaboratively expresses semantics through multimodal information such as dynamic changes in gestures, hand shapes, movement trajectories, facial expressions (such as changes in mouth shape), body posture, and even gaze direction. Its spatial and temporal dimensions contain rich grammatical and semantic structures. Therefore, the cornerstone of developing high-performance sign language recognition models, especially models that can accurately understand natural, continuous sign language, is to build a large-scale, high-quality, and finely annotated sign language dataset.

[0003] Traditional methods for constructing sign language datasets rely primarily on manually annotating key points or labeling the entire gesture / body with bounding boxes. While these methods have played a role in model training, they have fundamental limitations in capturing the semantically critical areas and cognitive focus that are crucial for sign language understanding. Studies have shown that when deaf people understand sign language, their visual attention distribution exhibits specific cognitive patterns, with their gaze often concentrating on specific areas that can most effectively convey semantic information, rather than being evenly distributed or simply following predefined points / boxes. This attention distribution information, based on the visual cognitive characteristics of the deaf population itself, is crucial for guiding the model to learn the essential characteristics of sign language and improving the model's ability to focus on key semantic elements and its robustness. Understanding how deaf people view sign language, revealing the priorities and patterns of their visual attention, and incorporating this scientific understanding into the dataset construction process are key breakthroughs in improving the performance of sign language recognition technology.

[0004] Therefore, it is necessary to improve the existing method and system for constructing sign language key area dataset based on eye tracking to solve the above problems. Summary of the Invention

[0005] The present invention overcomes the shortcomings of the existing technology and provides a method and system for constructing a sign language key area dataset based on eye tracking, aiming to solve the problem of data annotation homogeneity caused by ignoring the cognitive differences within the deaf community in the existing technology.

[0006] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for constructing a sign language key area dataset based on eye tracking, comprising: S1, deaf cognition data is acquired, and based on a three-dimensional classification matrix, cognition subgroups are divided, and multi-dimensional sign language videos are generated for deaf people in different groups; wherein the three-dimensional classification matrix is composed of three classification dimensions of deaf time, sign language acquisition age and communication habit; S2, eye tracking data of deaf people in different cognition subgroups is collected, and primary physical region annotation is performed; S3, based on the characteristics of the cognition subgroup, the eye tracking data is annotated with secondary cognitive weight; S4, the annotated eye tracking data is used as a training sample, the action boundary threshold is adjusted according to the cognition subgroup by a dynamic time warping algorithm, and a data set fused with physical annotation and cognitive weight is output.

[0007] In a preferred embodiment of the present application, in step S1, the deaf time is divided into prelingual deafness and postlingual deafness with 3 years as the critical period; the sign language acquisition age is divided into native and non-native with 6 years as the node; and the communication habit is divided into pure sign language dependence and sign language-oral bilingualism according to the proportion of sign language use.

[0008] In a preferred embodiment of the present application, in step S1, different categories of the three dimensions are combined to obtain 8 theoretical combination results; combined with the actual situation of deaf cognition development, invalid combinations are removed, and 6 effective cognition subgroups are output as: Prelingual deafness x native x pure sign language dependence; Prelingual deafness x native x sign language-oral bilingualism; Prelingual deafness x non-native x pure sign language dependence; Prelingual deafness x non-native x sign language-oral bilingualism; Postlingual deafness x non-native x pure sign language dependence; Postlingual deafness x non-native x sign language-oral bilingualism.

[0009] In a preferred embodiment of the present application, in step S2, the primary physical region annotation includes: Hand region annotation: taking wrist as starting point and fingertip as ending point, covering finger flexion and palm turning action; Face region annotation: covering eyebrow movement from hairline to lower jawline, eye state and mouth shape change; Gesture movement trajectory region annotation: polygon frame is used to annotate the up-down, left-right and front-back movement path of gesture in space; Mouth shape change region annotation: focusing on lip spreading, tooth exposure and tongue movement related to pronunciation.

[0010] In a preferred embodiment of the present application, the first-level physical region annotation is stored in JSON format, and includes the fields of cognitive subgroup ID, region ID, region type, frame sequence number, boundary coordinates, appearance timestamp, and disappearance timestamp; wherein the boundary coordinates include the upper-left corner and lower-right corner coordinates for a rectangular frame, and the vertex coordinates for a polygon frame; the coordinate values are accurate to the pixel level, and the timestamp is accurate to the millisecond level.

[0011] In a preferred embodiment of the present application, the second-level cognitive weight calculation of step S3 includes: The cumulative fixation total duration and fixation total number of each cognitive subgroup sample in the four physical regions are extracted, including the hand, face, gesture trajectory, and lip shape change; The cumulative fixation duration proportion and fixation number proportion of each region for a single sample are calculated; The average value of the region proportions is calculated, and a weight reference value is generated after normalization processing; The dynamic weight value is given in combination with the cognitive subgroup characteristics, and the result with weighted annotation is output.

[0012] In a preferred embodiment of the present application, the dynamic time warping algorithm of step S4 includes action boundary threshold adjustment: The average dynamic weight value of the target cognitive subgroup in the specific physical region is calculated; Based on the basic threshold, weight adjustment coefficient, and average dynamic weight value, the action boundary threshold of the subgroup is adjusted in proportion; The inter-frame distance matrix is constructed: if the boundary offset of the two frames of action regions exceeds the adjusted threshold, it is determined that the action boundary is discontinuous, and the inter-frame distance is set to infinity; otherwise, the Euclidean distance of the physical region key points between the two frames is calculated.

[0013] In a preferred embodiment of the present application, in step S1, different cognitive subgroups are processed differently in video content design according to their characteristics: The pre-lingual deaf group's video is embedded with high syntax complexity video segments, including role conversion, spatial verbs, and negative expression markers; The post-lingual deaf group's video is inserted with an easy-to-confuse lip shape contrast sequence, including a lip shape difference segment; At the same time, all groups uniformly include a basic vocabulary photo segment as a control variable.

[0014] In a preferred embodiment of the present application, when the data set is output in step S4: A three-dimensional index is established to associate the video ID, frame sequence number, and cognitive subgroup ID; Three layers of structured data are packaged into an HDF5 file: The basic layer includes standardized eye movement data; The marking layer comprises a physical region boundary coordinate sequence and a cognitive weight dynamic value sequence. The fusion layer comprises an adjusted action boundary threshold value, an optimal alignment path and a normalized feature matrix.

[0015] The application provides a sign language key region dataset construction system based on eye tracking, comprising modules: A cognitive subgroup division module is configured to obtain deaf cognitive data and divide cognitive subgroups based on a three-dimensional classification matrix. A multidimensional video generation module is configured to receive subgroup information output by the cognitive subgroup division module and generate multidimensional sign language videos adapted to different subgroups. An eye movement data acquisition module is configured to receive video files output by the multidimensional video generation module and acquire eye movement data of samples in each cognitive subgroup when watching the videos through an eye tracker. A first-level physical region marking module is configured to receive video files output by the eye movement data acquisition module and sample information output by the cognitive subgroup division module, mark hands, faces, gesture motion trajectories and mouth shape change regions in the videos by experts, verify consistency through intersection over union, and output marking data. A second-level cognitive weight marking module is configured to receive eye tracking original data output by the eye movement data acquisition module, marking data output by the first-level physical region marking module and subgroup features output by the cognitive subgroup division module, calculate weight reference values of each region and assign dynamic weights, and output marking results containing dynamic weight values. A data preprocessing module is configured to receive marking results output by the second-level cognitive weight marking module and perform data preprocessing. A dynamic time warping module is configured to receive standardized feature data output by the data preprocessing module and subgroup information output by the cognitive subgroup division module, adjust action boundary threshold values through a dynamic time warping algorithm, output an optimal alignment path and a fused feature matrix. A dataset output module is configured to receive the fused feature matrix output by the dynamic time warping module, encapsulate a dataset fused with physical marking and cognitive weights, and output a HDF5 format dataset file and a quality report.

[0016] The application solves the defects in the background art and has the following beneficial effects: (1) The application provides a sign language key area dataset construction method and system based on cognitive differences, deaf people are divided into 6 cognitive subgroups through a three-dimensional classification matrix, and different subgroups are designed with different visual cognitive characteristics; eye tracking data of each subgroup when watching videos is collected, and first-level physical region annotation is performed; then, based on the proportion of gaze duration and frequency, the weight reference value of each region is calculated, combined with the subgroup characteristics to give dynamic weight, and secondary cognitive weight annotation is formed; finally, using dynamic time warping algorithm, the average dynamic weight value of the subgroup in a specific region is used to adaptively adjust the action boundary threshold, realize time alignment optimization, and finally output the structured dataset integrating physical boundary coordinates and cognitive weight.

[0017] (2) The application divides 6 effective cognitive subgroups by constructing a three-dimensional classification matrix of "deaf time x sign language acquisition age x communication habit", fully considers the cognitive diversity of the deaf population, and no longer regards them as a homogeneous group; this subdivision method can accurately locate the inherent differences in visual attention of different subgroups, such as the dependence of prelingual deaf people on complex syntax and facial expressions and the attention difference of postlingual deaf people to lip reading information, which directly makes subsequent data collection more targeted. The existing technology ignores this cognitive difference, making the dataset difficult to adapt to the actual needs of different deaf populations, while the dataset under this feature can better adapt to the cognitive habits of various deaf people, further improving the adaptability and accuracy of the sign language recognition model trained based on the dataset in different groups.

[0018] (3) The application adopts a double annotation system combining first-level physical region annotation and second-level cognitive weight annotation, wherein the first-level annotation clearly defines the spatial boundaries of core physical regions such as hands, faces, gesture trajectories, and mouth changes, and the second-level annotation calculates the dynamic weight of each region based on the cognitive subgroup characteristics, quantifying the importance of different regions in the cognitive of a specific subgroup; the prelingual deaf mother tongue group has high weight for the face region, while the postlingual deaf group has high weight for the mouth change region; this system clearly defines the spatial carrier of sign language information and reflects the attention priority of different subgroups for these regions, realizing the fusion of physical space information and cognitive preference; this double annotation system makes the dataset contain both spatial details and dynamic cognitive weight, further enabling the trained model to more accurately capture the semantic emphasis in sign language expression.

[0019] (4) In the dynamic time warping algorithm, the action boundary threshold is adjusted according to the cognitive subgroup, the average dynamic weight of the target subgroup in a specific physical region is used to adjust the threshold in proportion to the basic threshold and the weight adjustment coefficient, and the continuity of the action boundary is considered when constructing the inter-frame distance matrix. The time alignment of the action sequence can adapt to the cognitive habits of different subgroups, such as relaxing the trajectory boundary threshold for subgroups that pay attention to gesture trajectories and tightening the mouth shape region threshold for subgroups that pay attention to mouth shapes, directly improving the accuracy and dynamics of action timing matching.

[0020] (5) The present application designs different video contents for different cognitive subgroups, such as embedding high syntax complexity segments for prelingual deaf groups and inserting easily confused mouth shape contrast sequences for postlingual deaf groups, while retaining uniform basic vocabulary segments as control variables. This design can accurately trigger the typical visual attention patterns of each subgroup, making the collected eye movement data more reflect their real cognitive characteristics and directly improving the representativeness of the data. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings; Figure 1 is a flowchart of the preferred embodiment of the present application; Figure 2 is a flowchart of step S1 of the preferred embodiment of the present application; Figure 3 is a cognitive subgroup division and video generation logic flowchart of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0023] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0024] SUMMARY OF THE APPLICATION The current sign language eye movement dataset construction generally adopts homogenization processing: deaf people are regarded as a single group to collect eye movement data, the hand region is taken as a unified attention target to generate an average fixation hotspot map, and model training data is generated by calculating the overall eye movement hotspot map; the core defect of this mode is the systematic neglect of cognitive differences: the existing technology does not distinguish the fundamental differences between prelingual deaf and postlingual deaf groups, resulting in the misplacement of the labeled region and the actual demand.

[0025] Prelingual deaf people rely on facial micro-expression and spatial relationship of gestures, while postlingual deaf people pay more attention to lip-reading and temporal coherence of gestures. In addition, lacking of knowledge of deaf linguistics, people mistakenly believe that "hearing loss = cognitive homogeneity", ignoring the remodeling effect of language acquisition age on visual processing strategy. The differences in processing strategy are not included in data grouping. In fact, native-level deaf people can efficiently capture the rhythmic units of gestures, while non-native people need to analyze the details of actions frame by frame.

[0026] The root of the dilemma of conventional technology lies in the double effects of cross-disciplinary cognitive fragmentation and simplified thinking. From the perspective of disciplinary barriers, computer vision teams often lack deaf linguistics and cognitive psychology background, and do not realize the remodeling effect of critical period of language acquisition on visual processing system. Prelingual deaf people form a visual dominant cognition centered on sign language during the critical period, while postlingual deaf people retain the inertia of oral thinking. This difference is not included in the experimental design, leading to the simplified path of selecting physical area static labeling for technical solutions.

[0027] The core logic of the present application breaks through the conventional one, which is to take the cognitive diversity of deaf people as the underlying basis for data construction, rather than a simplified interference variable. A three-dimensional classification matrix of "deafness time x sign language acquisition age x communication habit" is established to subdivide cognitive subgroups. Secondly, a dual labeling system of physical area and cognitive weight is constructed to solve the above problems.

[0028] Exemplary method: As shown in Figure 1 , a sign language key area dataset construction method based on eye tracking, comprising the steps of: S1, obtaining deaf cognitive data, dividing cognitive subgroups based on a three-dimensional classification matrix, and generating multi-dimensional sign language videos for deaf people in different groups; S2, collecting eye tracking data of deaf people in different cognitive subgroups, and performing primary physical area labeling; S3, based on the characteristics of cognitive subgroups, performing secondary cognitive weight labeling on the eye tracking data; S4, taking the labeled eye tracking data as training samples, adjusting the action boundary threshold according to the cognitive subgroups through dynamic time warping algorithm, and outputting the dataset fused with physical labeling and cognitive weight.

[0029] The existing technology regards deaf people as a homogeneous group, ignoring their cognitive differences, resulting in poor generalization ability of the dataset. The present application can accurately capture the visual attention features of different deaf groups by dividing subgroups through a three-dimensional classification matrix.

[0030] As shown in Figure 2 , Figure 3 , in step S1, deaf cognitive data is the core basis for dividing cognitive subgroups, and data is collected through questionnaire survey, interview and language ability test. The collected data is cleaned to eliminate invalid samples.

[0031] Among them, the deaf cognitive data refers to the basic information reflecting the language cognitive characteristics of the deaf, including: Deafness time, judging whether the deaf is prelingual deaf or postlingual deaf, taking the language critical period as the boundary; Sign language acquisition age, judging the proficiency and cognitive depth of the deaf in sign language, and distinguishing between native and non-native; Communication habits reflect the degree of dependence of the deaf on different language forms in daily communication, and distinguish between pure sign language dependence and sign language-oral bilingualism.

[0032] Specifically, the language critical period takes 3 years as the boundary to judge prelingual deaf and postlingual deaf.

[0033] If the deaf is deaf at birth or ≤3 years old, it belongs to prelingual deaf; such deaf people have not formed systematic cognition of oral language before the complete loss of hearing function, and language learning depends on visual channels from the beginning, and sign language is more likely to become their first language.

[0034] And if the deaf is deaf after 3 years old, and has mastered a certain degree of oral language before deafness, it belongs to postlingual deaf; this kind of deaf people have established an oral language thinking system through hearing before deafness, and even if they learn sign language later, they will retain cognitive inertia for oral language.

[0035] Sign language acquisition age takes 6 years as the key node to distinguish between native and non-native.

[0036] If the deaf starts to learn natural sign language at ≤6 years old, it belongs to native; at this stage, the language development critical period, the deaf can naturally form a language cognitive mode centered on sign language, and have an instinctive understanding of its rhythm, grammar and non-language features. And if the deaf starts to learn sign language after 6 years old, it belongs to non-native; at this time, the language cognitive mode has basically been shaped, and is mostly based on oral language, learning sign language is more as a communication tool, and it is difficult to achieve the proficiency of native language, and the capture and understanding of details need deliberate practice.

[0037] Communication habits take the proportion of sign language use as the main basis to distinguish between pure sign language dependence and sign language-oral bilingualism.

[0038] If the deaf uses sign language in daily communication usually ≥90%, and less combines with lip reading or oral language, it belongs to pure sign language dependence; such deaf people rely more on the visual features of sign language to transmit information, and have weaker understanding ability of oral language. And if the deaf uses sign language and combines with lip reading or oral language in daily communication, the proportion of sign language use is usually 30%-70%, it belongs to sign language-oral bilingualism; such deaf people have a certain oral language basis before deafness, or have been exposed to sign language and oral language at the same time during growth, and can understand and transmit information through multiple channels.

[0039] In step S1, the cleaned deaf cognition data is substituted into the three-dimensional classification matrix.

[0040] The three-dimensional classification matrix is a tool for classifying deaf cognition subgroups, which is composed of three independent classification dimensions of deafness time, sign language acquisition age and communication habit. Through the combination analysis of the three dimensions, the precise subdivision of deaf cognition characteristics is realized.

[0041] In this application, the three-dimensional classification matrix is the core framework for realizing the subdivision of cognitive diversity, and the specific division standards of its three dimensions have been specified. Based on this, the classification formula can be obtained: cognitive subgroup = deafness time category x sign language acquisition age category x communication habit category.

[0042] Among them, the deafness time category includes prelingual deafness and postlingual deafness; the sign language acquisition age category includes native level and non-native; the communication habit category includes pure sign language dependence and sign language-oral bilingualism.

[0043] Combining different categories of the three dimensions can obtain 8 theoretical combination results: However, combined with the actual situation of deaf cognition development, part of the combination does not exist due to logical contradiction.

[0044] The two combinations that do not exist are "postlingual deafness x native level x pure sign language dependence" and "postlingual deafness x native level x sign language-oral bilingualism".

[0045] From the association of deafness time and sign language acquisition age, postlingual deafness means deafness after the age of 3, at which time the deaf has established an oral thinking system to some extent. Native-level sign language acquisition requires exposure to natural sign language before the age of 6 to naturally form a language cognition mode centered on sign language. In the case of deafness after the age of 3, the deaf mainly learns oral language before deafness, and it is difficult for them to develop sign language to a native level within such a short time span (3-6 years old). Most postlingual deaf people start learning sign language later, usually in their adolescence or adulthood, and it is difficult for them to instinctively master the rhythm, grammar and non-verbal features of sign language as deaf people who have been exposed to sign language since an early age.

[0046] Further analysis from the perspective of communication habits, the post-lingual deaf will retain cognitive inertia for spoken language due to the early spoken language foundation, even if they learn sign language later. If there are two combinations of "post-lingual deaf x mother tongue level x pure sign language dependence" or "post-lingual deaf x mother tongue level x sign language-spoken language bilingualism", it means that in a short time after deafness, they not only improve sign language to the mother tongue level, but also meet the communication habit characteristics of pure sign language dependence or sign language-spoken language bilingualism, which is contrary to the law of deaf language cognitive development. Post-lingual deaf people tend to combine spoken language or lip reading in communication due to the early spoken language foundation, rather than relying solely on sign language, and the time cost and cognitive development process of reaching the mother tongue level of sign language do not support the emergence of these two combinations. Therefore, based on the actual law of deaf language cognitive development, these two combinations do not exist in reality.

[0047] Finally, six effective cognitive subgroups are formed, as shown in the following table: Table 1 Effective cognitive subgroups

[0048] Through this three-dimensional classification matrix, deaf people can be scientifically and systematically divided into different cognitive subgroups, laying a foundation for subsequent eye tracking data collection and data set construction for different subgroups, ensuring that the data set accurately reflects the cognitive characteristics of different deaf populations, thereby improving the adaptability and accuracy of sign language recognition models.

[0049] After dividing into six effective cognitive subgroups, multi-dimensional sign language videos need to be generated for deaf people in different groups; single sign language videos cannot cover the diverse communication needs in real-world scenarios, which may result in low recognition accuracy of the model in complex scenarios. Multi-dimensional videos can simulate sign language expressions in different environments, significantly improving the scene adaptability of the data set and enabling the trained sign language recognition model to be applied in real-world scenarios.

[0050] Multi-dimensional sign language videos are generated using a combination of real shooting and 3D engine synthesis. In the real shooting segment, at least three professional sign language performers are required, including two deaf professional sign language users and one hearing sign language interpreter, to ensure the standardization and naturalness of sign language expression. In the 3D engine synthesis segment, a sign language digital human model is constructed using professional 3D engines such as UnrealEngine, and the model must have consistent body movement accuracy and facial expression details as real sign language performers. Based on the video content of real shooting, different viewing angles (front, 45° side angle, overhead), lighting conditions (strong light, weak light, backlight), and background interference (static background, dynamic crowd) are simulated in the 3D engine to generate synthetic data corresponding to real videos.

[0051] According to the characteristics of different cognitive subgroups, different video content designs are processed differently: The video of the pre-lingual deaf group is embedded with high syntax complexity video segments, including: role conversion, spatial verbs, and negative expression markers, to meet the high dependence of the group on complex sign language syntax and facial expressions; The video of the post-lingual deaf group is inserted with confusing lip shape contrast sequences, such as the Chinese "cat / hat" and English "pat / bat" lip shape difference segments, to meet the attention needs of the group to lip reading information; At the same time, all groups uniformly include basic vocabulary photo segments as a control variable to ensure the comparability of data from different subgroups at the basic level.

[0052] Through differentiated content design, the generated multi-dimensional sign language video can more accurately trigger the typical visual attention patterns of different cognitive subgroups, and the collected eye movement data can better reflect the cognitive characteristics of each group, providing a more targeted data basis for subsequent labeling and model training.

[0053] In step S2, eye movement tracking data of the deaf person watching the multi-dimensional sign language video is collected by an eye tracker; The eye movement tracking data refers to the eye movement information of the deaf person watching the sign language video recorded by the eye tracker, including gaze point coordinates, gaze duration, gaze frequency, and saccade trajectory. In this application, eye movement tracking data is the core basis for analyzing the visual attention characteristics of the deaf, and can reflect the attention preferences of different cognitive subgroups for various regions in the video.

[0054] Primary physical region labeling refers to labeling key regions in the video that have physical entity properties. These regions are the core carriers of information in sign language expression. In this application, the primary physical region includes the hand region, the face region, the gesture motion trajectory region, and the lip shape change region.

[0055] The hand region covers finger movements and palm gestures; the face region includes eyebrows, eyes, and mouth; the gesture motion trajectory region refers to the movement path of gestures in space; and the lip shape change region refers to the opening and closing of the mouth and the change in shape.

[0056] Before collecting data: Select an experimental space with lighting between 300-500 lux and no glare interference, equipped with adjustable chairs and displays to ensure that the viewing angle and vertical distance of the deaf samples to the screen are maintained at 60-80 cm when watching videos, reducing eye movement data deviations caused by viewing angles.

[0057] Use a high-precision eye tracker with a sampling frequency ≥120Hz, such as Tobii Pro Fusion, and perform nine-point calibration on each sample before the experiment to ensure that the gaze point error is ≤0.5° visual angle. Calibration must be passed before data collection can begin.

[0058] The multi-dimensional sign language videos generated in step S1 are presented in a random order, and a 3-second still image is set as a preparation signal before each video is played, and a 2-second blank image is set as a buffer after the video is played, to avoid visual fatigue caused by continuous viewing of the samples.

[0059] In step S2, the samples of the 6 cognitive subgroups are arranged in independent experimental units respectively, and each sample watches the same set of multi-dimensional sign language videos independently. Each sample needs to complete 3 rounds of video watching, and the duration of each round of watching is controlled within 20 minutes, and the interval between rounds is 5-10 minutes to ensure data stability.

[0060] The eye tracker synchronously records all eye movement parameters during the sample watching process, including: the fixation point coordinates of each frame of picture, the duration of each fixation event, the cumulative fixation time ratio in each physical area, and the video timestamp is associated to realize the precise synchronization of eye movement data and video content.

[0061] The prior art does not collect eye movement data for different cognitive subgroups, resulting in that the labeled physical regions cannot reflect the group differences. The present application collects data and performs primary physical region labeling in subgroups, because the visual attention patterns of different cognitive subgroups are significantly different, prelingual deaf people pay more attention to faces, and postlingual deaf people pay more attention to mouths, and subgroup collection can accurately capture these differences; and the primary physical region labeling provides a basic spatial framework for subsequent cognitive weight analysis.

[0062] In step S2, primary physical region labeling is performed, the boundaries and ranges of the key physical regions in the video pictures, such as the hand region, the face region, the gesture motion trajectory region, and the mouth shape change region, are determined, rectangular frames or polygon frames are labeled frame by frame using a labeling tool, and the appearance and disappearance times of the regions are synchronously recorded. After being independently labeled by several experts, the consistency is verified by calculating the intersection over union, and the regions that do not meet the standard are re-labeled by consultation to ensure the accuracy and consistency of the labeling, and to provide a reliable basis for subsequent analysis.

[0063] The boundaries and ranges of the physical regions are determined to ensure the accuracy and consistency of the labeling.

[0064] Hand region: taking wrist as starting point and fingertip as ending point, covering the whole hand; covering the whole hand, specifically including finger flexion, extension, opening and closing, etc. motion, palm turning, tilting and other postures, and whether the hand is in static or dynamic state, it is within the labeling range; Face region: from hairline to lower jawline, specifically including eyebrow raising, frowning and other actions, eye opening, closing, gazing direction and other states, and mouth opening, closing, corner of the mouth lifting, drooping and other morphological changes; Gesture motion trajectory region: records the path coverage of the gesture moving in space, specifically according to the motion direction, it is divided into up-down, left-right, and front-back direction trajectory, and needs to complete the whole motion process from the starting to the ending position of the gesture. Mouth shape change region: focuses on the opening and closing and shape changes of the mouth due to pronunciation, specifically including the roundness of the lips, the exposure of the teeth, and the visible movement of the tongue, etc., which overlaps with the mouth part of the face region, but focuses more on the shape features related to pronunciation.

[0065] Use professional video annotation tools, such as Label Studio, to annotate the video frame by frame.

[0066] For relatively fixed regions such as hand region, face region, and mouth shape change region, use rectangular boxes for annotation to ensure that the box selection range completely covers the region boundary; for dynamic regions such as gesture motion trajectory region, use polygon boxes for annotation to adapt to the irregular shape of the trajectory.

[0067] The annotation accuracy is within ±3 pixels to ensure the accuracy of the region boundary. The annotation of the same region in different frames should maintain coherence, especially for the gesture motion trajectory region, which needs to ensure that the trajectories of the previous and subsequent frames are naturally connected and completely reflect the motion process of the gesture.

[0068] During the annotation process, the appearance time and disappearance time of the region need to be recorded synchronously and accurately correspond to the video timestamp to facilitate subsequent association analysis of eye movement data and annotated regions.

[0069] From the data end, the annotated data needs to be stored in JSON format, and each annotated data contains the fields of cognitive subgroup ID, region ID, region type, frame number, boundary coordinates, appearance timestamp, and disappearance timestamp; among them, the boundary coordinates include the left upper corner and right lower corner coordinates for rectangular boxes and the vertex coordinates for polygon boxes; the coordinate values are accurate to the pixel level, and the timestamp is accurate to the millisecond level.

[0070] In step S2, experts are professional personnel with knowledge of sign language linguistics and annotation experience, who independently complete the first-level physical region annotation of the same batch of videos.

[0071] Here, the expert needs to meet the following qualification requirements to ensure the professionalism and consistency of the annotation: Has more than 5 years of experience in sign language linguistics research or practice, and is familiar with the grammar rules and expression characteristics of at least 2 or more sign language systems; Has more than 3 years of experience in video annotation, has participated in at least 3 sign language related dataset construction projects, and can accurately understand the technical specifications of physical region annotation; Passes the sign language ability level test and can accurately identify the subtle movements and facial expression changes in sign language expression.

[0072] After annotation, the consistency of the annotation results is verified by calculating the intersection over union (IoU) of the annotated regions, requiring IoU ≥ 0.85. The calculation method of the intersection over union is: the ratio of the intersection area of two annotated regions to the union area.

[0073] Let the region annotated by expert A be A and the same region annotated by expert B be B, then the intersection over union calculation formula is , wherein represents the area of the overlapping part of the A and B annotated regions, represents the total area covered by the A and B annotated regions.

[0074] For annotated regions with IoU lower than 0.85, the annotation team jointly discusses and analyzes the reasons for the differences and re-annotates until the annotation consistency of all regions meets the requirements to ensure the objectivity and reliability of the annotation results.

[0075] In step S2, eye movement tracking data is collected by sub-cognitive groups and first-level physical region annotation is performed, realizing accurate capture and structured recording of visual attention characteristics of different deaf populations, and building a basic framework for second-level cognitive weight annotation in step S3.

[0076] In step S3, based on the characteristics of cognitive subgroups, second-level cognitive weight annotation is performed on the eye movement tracking data; Wherein, the second-level cognitive weight annotation refers to assigning dynamic weight values to each region in the first-level physical region annotation based on the visual attention characteristics of different cognitive subgroups, so as to quantify the importance of different regions in the information of the specific subgroup cognitive process. In this application, the second-level cognitive weight annotation is the key link between the physical region and the cognitive characteristics, which can make up for the limitation of the first-level physical region annotation that only reflects the spatial attributes and ignores the cognitive differences.

[0077] The visual attention patterns of different cognitive subgroups differ significantly, specifically as follows: Prelingual deaf x mother tongue level x pure sign language dependent subgroup: the highest proportion of attention is paid to the facial region, especially the expression changes of eyebrows and eyes, and the hand gesture trajectory region, because this subgroup relies on expression markers and spatial verb expressions in sign language grammar; Prelingual deaf x mother tongue level x sign language-oral language bilingual subgroup: while maintaining high attention to the facial region, the attention to the lip shape change region is slightly increased; Prelingual deaf x non-native x pure sign language dependent subgroup: more attention is paid to the details of finger movements and palm postures in the hand region, because non-native speakers need to analyze the movements frame by frame; Prelingual deaf x non-native x sign language-oral language bilingual subgroup: the attention to the hand region and the lip shape change region is balanced; ​Postlingually deaf x non-native x pure sign language dependent sub-group: although dependent on sign language, but due to the retention of oral language thinking habits, the attention to the temporal continuity of the gesture trajectory area is higher; Postlingually deaf x non-native x sign language-oral language bilingual sub-group: the lip-reading information in the mouth shape change area has the highest weight, while the hand area is also considered. The above weight reference value is based on the eye movement data collected in step S2 of each sub-group, that is, by calculating the average of the cumulative gaze duration proportion and the gaze frequency proportion of a certain area, and determining after normalization processing.

[0078] The weight reference value of each cognitive sub-group is calculated based on the eye movement data collected in step S2, and the specific process is as follows: From the eye tracking data of each cognitive sub-group, extract the cumulative gaze duration and gaze frequency of the hand area, face area, gesture trajectory area, and mouth shape change area when all samples watch the multi-dimensional sign language video; Calculate the cumulative gaze duration proportion and gaze frequency proportion of a certain area of a single sample; Specifically, the cumulative gaze duration proportion of a single sample on a certain area is Wherein, represents the cumulative gaze duration proportion of the ith sample in the jth cognitive sub-group on the kth area, represents the cumulative gaze duration of the ith sample in the jth cognitive sub-group in the kth area; represents the total gaze duration of the ith sample in the jth cognitive sub-group; The gaze frequency proportion of a single sample on a certain area is Wherein, represents the gaze frequency proportion of the ith sample in the jth cognitive sub-group on the kth area, represents the gaze frequency of the ith sample in the jth cognitive sub-group in the kth area, represents the total gaze frequency of the ith sample in the jth cognitive sub-group.

[0079] Then, calculate the average of the above proportions, and the weight reference value is , is the average of the cumulative gaze duration proportion and the gaze frequency proportion of a single sample on a certain area after normalization; is set according to the characteristics of the sub-group as the corresponding dynamic weight.

[0080] From the data end, the weight benchmark value calculation needs to build an intermediate data model, including four-level data structures of the original eye movement data table, the proportion calculation result table, the normalization result table, and the weight benchmark value table, which realizes cascading association through sample ID and region ID. The intermediate files generated during the calculation are stored in JSON format, and the configuration table contains the fields of cognitive subgroup ID, dynamic weight parameter, and effective time, supporting dynamic adjustment and version management.

[0081] In step S3, the eye movement tracking data is deeply fused with the cognitive subgroup characteristics through secondary cognitive weight labeling, forming a double-layer labeling structure of physical region + dynamic weight, laying a foundation for outputting a data set with spatial precision and cognitive adaptability.

[0082] In step S4, the labeled eye movement tracking data is used as a training sample to adjust the action boundary threshold value according to the cognitive subgroup through the dynamic time warping algorithm, and a data set with physical labeling and cognitive weight fusion is output.

[0083] The eye movement tracking data with secondary cognitive weight labeling completed in step S3 is standardized, specifically including: Data alignment: Based on the video timestamp, the boundary coordinate data of the first-level physical region labeling, the dynamic weight value of the second-level cognitive weight labeling, and the eye movement tracking data are aligned at the frame level to ensure accurate association of multi-dimensional data in the same frame; Adopting the 3σ principle to eliminate outliers in eye movement data, i.e., when the fixation duration and fixation times of a frame exceed ±3 times the standard deviation of the corresponding index mean of the cognitive subgroup, it is marked as an abnormal frame and removed; Key features are extracted from the preprocessed data, including: boundary coordinate sequence of each physical region, dynamic weight value sequence, and distribution density of fixation points in the region; among them, the distribution density is the number of fixation points per unit area; The feature values are normalized to the interval [0,1], and the normalization parameters used are the maximum and minimum values of each feature in the entire sample.

[0084] In step S4, the dynamic time warping model includes: The dynamic time warping algorithm is used to time-align the action sequences of different cognitive subgroups; specifically, the time window width = 5 frames, the distance measurement method is Euclidean distance, and the path constraint uses Itakura parallelogram constraint; For each cognitive subgroup x, the mean value of the dynamic weight value of all samples in the cognitive subgroup in each physical region is calculated ; Set the basic action boundary threshold value , which represents the maximum allowed offset reference value of the adjacent frame action region boundary; Adjust the threshold value for each subgroup according to the following formula: ,in, is the action boundary threshold of the x-th cognitive subgroup in a specific physical area, A coefficient that balances the impact of the dynamic weight value on the boundary threshold.

[0085] Based on the adjusted motion boundary threshold Calculate the m×n order distance matrix D of the two action sequences to be aligned. For the element in the ath row and bth column of the matrix , whose value is the Euclidean distance between the a-th frame and the b-th frame. If the boundary offset of the action area of ​​the two frames exceeds , then the value is assigned to infinity.

[0086] Calculate the optimal alignment path ;in, represents the minimum cumulative distance between the p-th frame of the first action sequence and the q-th frame of the second action sequence, reflecting the optimal alignment cost of the two frames and all previous frames; The Euclidean distance between the pth frame of the first action sequence and the qth frame of the second action sequence is the matching cost of the current frame; p and q are the frame indexes of the first action sequence and the second action sequence; where, , edge elements ,When one of the sequences is the 0th frame and the other sequence has entered the valid frame, the cumulative distance is infinite.

[0087] calculate When the dynamic weight value Normalization is performed as a weighting coefficient for distance calculation.

[0088] Output the optimal alignment path, minimum cumulative distance value, and path visualization results.

[0089] Step S4 is used to convert the preprocessed and fused multi-dimensional data into a standardized data set that can be directly used for model training; The dataset is a structured data set that integrates physical area annotations, cognitive weight information, eye movement characteristics, and dynamic time warping results; Base layer data: normalized raw eye tracking data; Annotation layer data: boundary coordinate sequence of first-level physical area annotation, dynamic weight value sequence of second-level cognitive weight annotation; Fusion layer data: action boundary threshold, time alignment path, and feature normalization results after dynamic time warping.

[0090] From the data end, the optimal alignment path data output by the dynamic time warping algorithm is associated with the first-level physical region annotation data, the second-level cognitive weight annotation data, and the preprocessed eye tracking data through video ID, frame sequence number, and cognitive subgroup ID to establish a three-dimensional association index.

[0091] The first-level physical region annotation data includes boundary coordinates and region types; the second-level cognitive weight annotation data includes dynamic weight values and calculation basis; and the preprocessed eye tracking data includes gaze point coordinates, gaze duration, and gaze frequency.

[0092] For each cognitive subgroup, a fusion matrix of cognitive subgroup ID x frame sequence number x physical region x feature dimension is constructed. The feature dimension includes region boundary coordinates, dynamic weight values, cumulative gaze duration proportion, gaze frequency proportion, alignment error value, and deviation pixel number from the optimal path, and the matrix element values are all normalized standard data range [0, 1].

[0093] Exemplary system: A sign language key region dataset construction system based on eye tracking, comprising: A cognitive subgroup division module is used to obtain deaf cognitive data and divide cognitive subgroups based on a three-dimensional classification matrix. Data is collected through questionnaire investigation, interview, and language ability test, and after cleaning, the division results of 6 effective cognitive subgroups and corresponding sample information are output.

[0094] A multi-dimensional video generation module is used to receive the subgroup information output by the cognitive subgroup division module and generate multi-dimensional sign language videos adapted to different subgroups. Real shooting and 3D engine synthesis are combined to output video files and video metadata containing differentiated content. The module transmits the video files to the eye movement data acquisition module.

[0095] An eye movement data acquisition module is used to receive the video files output by the multi-dimensional video generation module and collect eye movement data of each cognitive subgroup sample when watching the video through an eye tracker. Timestamped eye tracking raw data is output.

[0096] A first-level physical region annotation module is used to receive the video files output by the eye movement data acquisition module and the sample information output by the cognitive subgroup division module, and to annotate the hand, face, gesture motion trajectory, and mouth shape change region in the video by experts. After calculating the intersection over union ratio to verify consistency, the annotation data is output.

[0097] A second-level cognitive weight annotation module is used to receive the eye tracking raw data output by the eye movement data acquisition module, the annotation data output by the first-level physical region annotation module, and the subgroup features output by the cognitive subgroup division module, calculate the weight reference value of each region, and assign dynamic weights to output annotation results containing dynamic weight values.

[0098] Data preprocessing module: used for receiving the labeling results output by the secondary cognitive weight labeling module, performing data alignment, outlier rejection and feature extraction and normalization, and outputting standardized feature data.

[0099] Dynamic time warping module: used for receiving the standardized feature data output by the data preprocessing module and the subgroup information output by the cognitive subgroup division module, adjusting the action boundary threshold value through the dynamic time warping algorithm, and outputting the optimal alignment path and the fused feature matrix.

[0100] Dataset output module: used for receiving the fused feature matrix output by the dynamic time warping module, encapsulating the dataset fused with physical labeling and cognitive weight, and outputting the dataset file in HDF5 format and the quality report.

[0101] Data transmission relationship: The cognitive subgroup division module transmits subgroup information to the multi-dimensional video generation module, the eye movement data acquisition module, the secondary cognitive weight labeling module and the dynamic time warping module. The multi-dimensional video generation module transmits video files to the eye movement data acquisition module. The eye movement data acquisition module transmits eye movement data and video files to the primary physical region labeling module and the secondary cognitive weight labeling module. The primary physical region labeling module transmits physical region labeling data to the secondary cognitive weight labeling module. The secondary cognitive weight labeling module transmits labeling results with dynamic weight to the data preprocessing module. The data preprocessing module transmits standardized feature data to the dynamic time warping module, and the dynamic time warping module transmits the fused feature matrix to the dataset output module.

[0102] The above is the ideal embodiment of the present application, and through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the contents of the specification, and must be determined according to the scope of the claims.

Claims

1. A method for constructing a sign language key area dataset based on eye tracking, characterized in that: Including steps: S1. Acquire cognitive data of deaf individuals, divide cognitive subgroups based on a three-dimensional classification matrix, and generate multi-dimensional sign language videos for different groups of deaf individuals; wherein the three-dimensional classification matrix is ​​composed of three classification dimensions: duration of deafness, age of sign language acquisition, and communication habits; S2. Collect eye tracking data from deaf individuals in different cognitive subgroups and perform primary physical area annotation. S3. Based on the cognitive subgroup characteristics, the eye tracking data is labeled with secondary cognitive weights; S4. Use the annotated eye tracking data as training samples, adjust the action boundary threshold according to the cognitive subgroup through the dynamic time warping algorithm, and output a dataset that integrates physical annotations and cognitive weights.

2. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: In step S1, the deafness time is divided into pre-lingual deafness and post-lingual deafness with 3 years old as the critical period; the sign language acquisition age is divided into native level and non-native level with 6 years old as the node; the communication habit is divided into pure sign language dependence and sign-oral bilingualism with the proportion of sign language use.

3. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: In step S1, different categories of the three dimensions are combined to obtain eight theoretical combination results. Based on the actual cognitive development of deaf people, invalid combinations are removed and six valid cognitive subgroups are output: Prelingual deafness × native language level × pure sign language dependence; Prelingual deafness × native-level × sign-oral bilingualism; Prelingual deafness × non-native language × pure sign language dependence; Prelingual deafness × non-native language × sign-oral bilingualism; Postlingual deafness × non-native language × pure sign language dependence; Postlingual deafness × non-native language × sign-oral bilingualism.

4. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: In step S2, the first-level physical area labeling includes: Hand area annotation: starting from the wrist and ending at the fingertips, covering finger flexion and extension as well as palm flipping movements; Facial area annotation: covering eyebrow movements, eye states, and mouth shape changes from hairline to jawline; Gesture motion trajectory area annotation: Use polygonal boxes to mark the up and down, left and right, and front and back motion paths of gestures in space; Mouth shape area annotation: Focus on pronunciation-related forms such as lip rounding, tooth exposure, and tongue movement.

5. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: The first-level physical area annotation is stored in JSON format and contains fields such as cognitive subgroup ID, area ID, area type, frame number, boundary coordinates, appearance timestamp, and disappearance timestamp. The boundary coordinates include the coordinates of the upper left and lower right corners for rectangular boxes and the coordinates of each vertex for polygonal boxes. The coordinate values ​​are accurate to the pixel level, and the timestamps are accurate to the millisecond level.

6. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: The calculation of the secondary cognitive weight in step S3 includes: The total cumulative fixation duration and total number of fixations in four physical areas were extracted for each cognitive subgroup sample. The four physical areas included: hands, face, gesture movement trajectory, and lip shape changes. For a single sample, calculate the cumulative fixation duration and fixation frequency ratio of each area; Calculate the average value of each region's proportion and generate a weighted benchmark value through normalization; Dynamic weight values ​​are assigned based on the characteristics of cognitive subgroups, and the results are output with weight annotations.

7. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: The dynamic time warping algorithm in step S4 includes motion boundary threshold adjustment: Calculate the average dynamic weight value of the target cognitive subgroup in a specific physical area; Based on the basic threshold, weight adjustment coefficient and average dynamic weight value, the action boundary threshold of the subgroup is adjusted proportionally; Construct an inter-frame distance matrix: If the offset between the action region boundaries of two frames exceeds the adjusted threshold, the action boundary is determined to be discontinuous and the inter-frame distance is set to infinity; otherwise, the Euclidean distance between the key points of the physical region between the two frames is calculated.

8. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: In step S1, differentiated processing is performed on the video content design based on the characteristics of different cognitive subgroups: The videos of the prelingually deaf group were embedded with high-grammatical complexity video clips, including elements of role reversal, spatial verbs, and negative expression markers; The videos of the postlingually deaf group were inserted with easily confused lip-contrast sequences, including segments with lip shape differences; At the same time, all groups uniformly included basic vocabulary comparison segments as control variables.

9. The method for constructing a sign language key area dataset based on eye tracking according to claim 1, characterized in that: When step S4 outputs the data set: Establish a three-dimensional index to associate video ID, frame number and cognitive subgroup ID; Encapsulate three-layer structured data into HDF5 files: The base layer contains normalized eye movement data; The annotation layer contains a sequence of physical region boundary coordinates and a sequence of dynamic values ​​of cognitive weights; The fusion layer contains the adjusted action boundary threshold, optimal alignment path, and normalized feature matrix.

10. A system for constructing a sign language key area dataset based on eye tracking, based on the method for constructing a sign language key area dataset based on eye tracking according to any one of claims 1 to 9, characterized in that: Included modules: Cognitive subgrouping module, used to obtain cognitive data of deaf people and divide cognitive subgroups based on a three-dimensional classification matrix; A multi-dimensional video generation module is used to receive the subgroup information output by the cognitive subgroup division module and generate multi-dimensional sign language videos adapted to different subgroups; An eye movement data acquisition module is used to receive the video file output by the multi-dimensional video generation module and collect the eye movement data of each cognitive subgroup sample when watching the video through an eye tracker; The first-level physical region annotation module receives the video files output by the eye movement data acquisition module and the sample information output by the cognitive subgroup classification module. Experts annotate the hands, face, gesture movement trajectories, and lip shape change areas in the video, calculate the intersection-over-union ratio to verify consistency, and then output the annotated data. The secondary cognitive weight annotation module is used to receive the raw eye tracking data output by the eye movement data acquisition module, the annotated data output by the primary physical area annotation module, and the subgroup features output by the cognitive subgroup division module, calculate the weight baseline value of each area, assign dynamic weights, and output the annotation results including the dynamic weight values; The data preprocessing module is used to receive the labeling results output by the secondary cognitive weight labeling module and perform data preprocessing; The dynamic time warping module is used to receive the standardized feature data output by the data preprocessing module and the subgroup information output by the cognitive subgroup division module, adjust the action boundary threshold through the dynamic time warping algorithm, and output the optimal alignment path and the fused feature matrix; The dataset output module is used to receive the fusion feature matrix output by the dynamic time warping module, encapsulate it into a dataset fused with physical annotations and cognitive weights, and output the dataset file and quality report in HDF5 format.

Images