A sign language key region dataset construction method and system based on eye tracking

Abstract

The application discloses a sign language key region dataset construction method and system based on eye movement tracking, and comprises the following steps: deaf people are divided into cognitive subgroups through a three-dimensional classification matrix of deaf time, sign language acquisition age and communication habit, and multi-dimensional sign language videos suitable for different subgroups are generated; eye movement data of each subgroup is collected, and first-level physical region labeling such as hands and faces is performed; second-level cognitive weight labeling is completed in combination with subgroup characteristics; the action boundary threshold is adjusted through a dynamic time warping algorithm, and a dataset fused with physical labeling and cognitive weight is output. The system comprises cognitive subgroup division, video generation, eye movement collection, two-level labeling, data preprocessing, dynamic time warping and dataset output modules. The application solves the problem that traditional datasets ignore the cognitive differences of deaf people and improves the adaptability of sign language recognition models.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric digital data processing, and in particular to a sign language key region dataset construction method and system based on eye tracking. BACKGROUND

[0002] As the first language of the deaf community, sign language is the core carrier of information exchange and cultural heritage of the deaf community. With the rapid development of information technology, artificial intelligence-driven natural sign language recognition (SLR) and translation technology show great potential, which is expected to bridge the communication gap between the deaf and hearing communities and promote the construction of a barrier-free society. The core challenge of sign language recognition technology lies in its high visual complexity: sign language vocabulary is expressed through the dynamic changes of gestures, hand shapes, motion trajectories, facial expressions (such as mouth shape changes), body postures, and even gaze directions, which are multi-modal information that cooperatively express semantics, and both the spatial and temporal dimensions contain rich grammatical and semantic structures. Therefore, to develop a high-performance sign language recognition model, especially one that can accurately understand natural and continuous sign language, the cornerstone is to build a large-scale, high-quality, and finely annotated sign language dataset.

[0003] Traditional sign language dataset construction methods mainly rely on manual annotation of key points or bounding box annotation of the entire gesture / body. Although these methods have played a certain role in model training, they have fundamental limitations in capturing semantic key regions and cognitive focal points, which are crucial for sign language understanding. Studies have shown that when understanding sign language, the visual attention distribution of the deaf community has a specific cognitive pattern, and their gaze points tend to focus on specific areas that can most efficiently convey semantic information, rather than being evenly distributed or simply following predefined points / boxes. This attention distribution information based on the visual cognitive characteristics of the deaf community is crucial for guiding the model to learn the essential features of sign language and improving the model's ability to focus on key semantic elements and robustness. Understanding how the deaf watch sign language, revealing their visual attention priorities and patterns, and integrating this scientific knowledge into the dataset construction process are key breakthroughs in improving the performance of sign language recognition technology.

[0004] Therefore, it is necessary to improve the existing technology of sign language key region dataset construction method and system based on eye tracking to solve the above problems. SUMMARY

[0005] The present application overcomes the shortcomings of the prior art and provides a sign language key region dataset construction method and system based on eye tracking, aiming to solve the problem of data annotation homogeneity caused by ignoring the cognitive differences within the deaf community in the prior art.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a sign language key region dataset construction method based on eye tracking, comprising:

[0007] S1, deaf cognitive data is acquired, cognitive subgroups are divided based on a three-dimensional classification matrix, and multi-dimensional sign language videos are generated for deaf people in different groups; wherein the three-dimensional classification matrix is composed of three classification dimensions of deaf time, sign language acquisition age and communication habit;

[0008] S2, eye tracking data of deaf people in different cognitive subgroups is collected, and primary physical region annotation is performed;

[0009] S3, based on the characteristics of cognitive subgroups, the eye tracking data is annotated with secondary cognitive weight;

[0010] S4, the annotated eye tracking data is used as a training sample, the action boundary threshold is adjusted according to the cognitive subgroup by a dynamic time warping algorithm, and a data set fused with physical annotation and cognitive weight is output.

[0011] In a preferred embodiment of the present application, in step S1, the deaf time is divided into prelingual deafness and postlingual deafness with 3 years as the critical period; the sign language acquisition age is divided into native and non-native with 6 years as the node; and the communication habit is divided into pure sign language dependence and sign language-oral bilingualism according to the proportion of sign language use.

[0012] In a preferred embodiment of the present application, in step S1, different categories of the three dimensions are combined to obtain 8 theoretical combination results; combined with the actual situation of cognitive development of deaf people, invalid combinations are removed, and 6 effective cognitive subgroups are output:

[0013] Prelingual deafness x native x pure sign language dependence;

[0014] Prelingual deafness x native x sign language-oral bilingualism;

[0015] Prelingual deafness x non-native x pure sign language dependence;

[0016] Prelingual deafness x non-native x sign language-oral bilingualism;

[0017] Postlingual deafness x non-native x pure sign language dependence;

[0018] Postlingual deafness x non-native x sign language-oral bilingualism.

[0019] In a preferred embodiment of the present application, in step S2, the primary physical region annotation includes:

[0020] Hand region annotation: taking wrist as starting point and fingertip as ending point, covering finger flexion and palm turning action;

[0021] Face region annotation: covering eyebrow movement from hairline to chin line, eye state and mouth shape change;

[0022] Gesture motion trajectory region annotation: adopt polygon frame to annotate the up-down, left-right and front-back motion path of gestures in space;

[0023] Mouth shape change region annotation: focus on the pronunciation related forms of lip circle expansion, tooth exposure and tongue movement.

[0024] In a preferred embodiment of the present application, the first-level physical region annotation is stored in JSON format, including the fields of cognitive subgroup ID, region ID, region type, frame sequence number, boundary coordinates, appearance timestamp and disappearance timestamp; wherein the boundary coordinates include the left upper corner and right lower corner coordinates for a rectangular frame, and the vertex coordinates for a polygon frame; the coordinate values are accurate to the pixel level, and the timestamp is accurate to the millisecond level.

[0025] In a preferred embodiment of the present application, the second-level cognitive weight calculation of step S3 includes:

[0026] Extract the cumulative gaze total duration and gaze total times of each cognitive subgroup sample in the four physical regions, including hand, face, gesture motion trajectory and mouth shape change;

[0027] Calculate the cumulative gaze duration proportion and gaze times proportion of each region for a single sample;

[0028] Calculate the average value of the proportion of each region, and generate a weight reference value after normalization processing;

[0029] Combine the dynamic weight value given by the cognitive subgroup characteristics to output the result with weighted annotation.

[0030] In a preferred embodiment of the present application, the dynamic time warping algorithm of step S4 includes action boundary threshold adjustment:

[0031] Calculate the average dynamic weight value of the target cognitive subgroup in a specific physical region;

[0032] Based on the basic threshold, weight adjustment coefficient and average dynamic weight value, adjust the action boundary threshold of the subgroup in proportion;

[0033] Construct an inter-frame distance matrix: if the boundary offset of two frames of action regions exceeds the adjusted threshold, determine that the action boundary is discontinuous, and set the inter-frame distance to infinity; otherwise, calculate the Euclidean distance of the key points of the physical regions between the two frames.

[0034] In a preferred embodiment of the present application, in step S1, different cognitive subgroups are processed differently in video content design according to their characteristics:

[0035] High syntax complexity video segments are embedded in the video of prelingual deaf group, including elements such as role conversion, spatial verbs and negative expression markers;

[0036] The video of the post-lingual deaf group is inserted with a confusing mouth shape contrast sequence, including a lip shape difference segment;

[0037] Meanwhile, all groups uniformly include a basic vocabulary-to-photo segment as a control variable.

[0038] In a preferred embodiment of the present application, when the data set is output in step S4:

[0039] A three-dimensional index is established to associate the video ID, frame number and cognitive subgroup ID;

[0040] The three-layer structured data is packaged into an HDF5 file:

[0041] The basic layer contains standardized eye movement data;

[0042] The annotation layer contains a sequence of physical region boundary coordinates and a sequence of dynamic cognitive weight values;

[0043] The fusion layer contains the adjusted action boundary threshold, the optimal alignment path and the normalized feature matrix.

[0044] The present application provides a sign language key region data set construction system based on eye tracking, comprising modules:

[0045] A cognitive subgroup division module is used to obtain deaf cognitive data and divide cognitive subgroups based on a three-dimensional classification matrix;

[0046] A multi-dimensional video generation module is used to receive subgroup information output by the cognitive subgroup division module and generate multi-dimensional sign language videos adapted to different subgroups;

[0047] An eye movement data acquisition module is used to receive video files output by the multi-dimensional video generation module and acquire eye movement data of each cognitive subgroup sample when watching the video through an eye tracking instrument;

[0048] A primary physical region annotation module is used to receive video files output by the eye movement data acquisition module and sample information output by the cognitive subgroup division module, to annotate the hand, face, gesture trajectory and mouth shape change region in the video by experts, and to output annotation data after calculating the intersection over union ratio and verifying consistency;

[0049] A secondary cognitive weight annotation module is used to receive eye tracking raw data output by the eye movement data acquisition module, annotation data output by the primary physical region annotation module and subgroup features output by the cognitive subgroup division module, to calculate the weight reference value of each region and assign a dynamic weight, and to output annotation results containing dynamic weight values;

[0050] A data preprocessing module is used to receive annotation results output by the secondary cognitive weight annotation module and perform data preprocessing;

[0051] a dynamic time warping module, configured to receive the standardized feature data output by the data preprocessing module and the subgroup information output by the cognitive subgroup division module, adjust the action boundary threshold value through a dynamic time warping algorithm, output an optimal alignment path and a fused feature matrix;

[0052] a dataset output module, configured to receive the fused feature matrix output by the dynamic time warping module, encapsulate the dataset fused with the physical annotation and cognitive weight, and output a dataset file in HDF5 format and a quality report.

[0053] The present application solves the defects in the background art and has the following beneficial effects:

[0054] (1) The present application provides a sign language key region dataset construction method and system based on cognitive differences, which divides the deaf into 6 cognitive subgroups through a three-dimensional classification matrix, and designs differentiated sign language videos according to the visual cognitive characteristics of different subgroups; collects eye tracking data when each subgroup watches the videos, and performs primary physical region annotation; then calculates the weight benchmark value of each region based on the fixation duration and number ratio, combines the subgroup characteristics to give dynamic weights, and forms secondary cognitive weight annotation; finally, the dynamic time warping algorithm is used to adaptively adjust the action boundary threshold value according to the average dynamic weight value of the subgroup in a specific region, realize time alignment optimization, and finally output a structured dataset fused with physical boundary coordinates and cognitive weights.

[0055] (2) The present application divides 6 effective cognitive subgroups by constructing a three-dimensional classification matrix of "deafness time x sign language acquisition age x communication habit", fully considers the cognitive diversity of the deaf population, and no longer regards them as a homogeneous group; this subdivision method can accurately locate the inherent differences in visual attention of different subgroups, such as the dependence of prelingual deaf on complex grammar and facial expressions and the attention difference of postlingual deaf on lip-reading information, which directly makes the subsequent data collection more targeted. The existing technology ignores this cognitive difference, making it difficult for the dataset to adapt to the actual needs of different deaf populations, while the dataset under this feature can better fit the cognitive habits of various deaf people, further improving the adaptability and accuracy of the sign language recognition model trained based on this dataset in different groups.

[0056] (3) The present application adopts a double labeling system combining first-level physical region labeling and second-level cognitive weight labeling, wherein the first-level labeling clearly defines the spatial boundaries of core physical regions such as hands, faces, gesture motion trajectories, and mouth shape changes, and the second-level labeling calculates the dynamic weights of each region in combination with cognitive subgroup characteristics, quantifying the importance of different regions in specific subgroup cognition; the pre-lingual deaf mother tongue group has a high weight for the face region, while the post-lingual deaf group has a high weight for the mouth shape change region; this system not only clearly defines the spatial carriers of sign language information, but also reflects the attention priority of different subgroups for these regions, realizing the fusion of physical space information and cognitive preference; the double labeling system allows the data set to contain both spatial details and dynamic cognitive weights, further enabling the trained model to more accurately capture the semantic emphasis in sign language expression.

[0057] (4) In the dynamic time warping algorithm, the present application adjusts the motion boundary threshold value according to the cognitive subgroup, adjusts the threshold value in proportion according to the average dynamic weight of the target subgroup in a specific physical region, combines the basic threshold value and the weight adjustment coefficient, and considers the continuity of the motion boundary when constructing the inter-frame distance matrix. This makes the time alignment of the motion sequence adaptable to the cognitive habits of different subgroups, such as relaxing the trajectory boundary threshold value for subgroups that pay attention to gesture trajectories, and tightening the mouth shape region threshold value for subgroups that pay attention to mouth shapes, directly improving the accuracy and dynamics of motion timing matching.

[0058] (5) The present application designs differentiated video content for different cognitive subgroups, such as embedding high syntax complexity segments for pre-lingual deaf groups and inserting easily confused mouth shape contrast sequences for post-lingual deaf groups, while retaining uniform basic vocabulary segments as control variables. This design can accurately trigger the typical visual attention patterns of each subgroup, making the collected eye movement data more reflective of their true cognitive characteristics, directly improving the representativeness of the data. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, a brief introduction will be given below to the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor;

[0060] Figure 1 is a flowchart of the preferred embodiment of the present application;

[0061] Figure 2 is a flowchart of step S1 of the preferred embodiment of the present application;

[0062] Figure 3 is a logic flowchart of cognitive subgroup division and video generation of the preferred embodiment of the present application. DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0064] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other manners different from those described herein, and the scope of the present application is not limited to the specific embodiments disclosed below.

[0065] Summary of the application:

[0066] The current sign language eye movement data set construction generally adopts homogenization processing: deaf people are regarded as a single group to collect eye movement data, and average fixation hotspot maps are generated by taking the hand region as the unified attention target, and model training data are generated by calculating the overall eye movement hotspot map; the core defect of this mode is the systematic neglect of cognitive differences: the existing technology does not distinguish the fundamental differences between prelingual deaf and postlingual deaf groups, resulting in misalignment between labeled regions and actual needs.

[0067] Prelingual deaf people rely on facial micro-expressions and spatial relationships of gestures, while postlingual deaf people pay more attention to lip reading and temporal coherence of gestures; in addition, lacking knowledge of deaf linguistics, it is mistakenly believed that "hearing loss = cognitive homogeneity", and the remodeling effect of language acquisition age on visual processing strategy is ignored. The differences in this processing strategy are not included in the data grouping. In fact, native-level deaf people can efficiently capture gesture rhythmic units, and non-native speakers need to analyze action details frame by frame.

[0068] The root cause of the dilemma of conventional technology lies in the dual effects of cross-disciplinary cognitive fragmentation and simplified thinking. From the perspective of disciplinary barriers, computer vision teams often lack deaf linguistics and cognitive psychology backgrounds, and do not realize the remodeling effect of the critical period of language acquisition on the visual processing system. Prelingual deaf people form a visually dominant cognition centered on sign language during the critical period, while postlingual deaf people retain the inertia of spoken language thinking. This difference is not included in the experimental design, resulting in the simplified path of choosing physical region static labeling for the technical solution.

[0069] The core logic of the present application breaks through the conventional one, which is to take the cognitive diversity of deaf people as the underlying basis for data construction, rather than a simplified interference variable. A three-dimensional classification matrix of "deafness time x sign language acquisition age x communication habit" is established to subdivide cognitive subgroups; secondly, a dual labeling system of physical region and cognitive weight is constructed to solve the above problems.

[0070] Exemplary method:

[0071] AsFigure 1 As shown in the figure, a sign language key area dataset construction method based on eye tracking includes the following steps:

[0072] S1, deaf people's cognitive data is obtained, and cognitive subgroups are divided based on a three-dimensional classification matrix. Multi-dimensional sign language videos are generated for deaf people in different groups.

[0073] S2, collect eye tracking data of deaf people in different cognitive subgroups, and perform primary physical region annotation.

[0074] S3, based on the characteristics of cognitive subgroups, the eye tracking data is annotated with secondary cognitive weights.

[0075] S4, the annotated eye tracking data is used as a training sample, the action boundary threshold is adjusted according to the cognitive subgroup by dynamic time warping algorithm, and the dataset with physical annotation and cognitive weight fusion is output.

[0076] The prior art regards deaf people as a homogeneous group, ignoring their cognitive differences, resulting in poor generalization ability of the dataset. The present application can accurately capture the visual attention features of different deaf people by dividing subgroups using a three-dimensional classification matrix.

[0077] As shown in the figure, Figure 2 , Figure 3 In step S1, deaf people's cognitive data is the core basis for dividing cognitive subgroups. Data is collected through questionnaire surveys, interviews, and language proficiency tests. The collected data is cleaned to remove invalid samples.

[0078] Among them, the deaf people's cognitive data refers to the basic information reflecting the language cognitive characteristics of deaf people, including:

[0079] Deafness time, to determine whether the deaf person is prelingual deaf or postlingual deaf, with the critical period of language as the boundary;

[0080] Sign language acquisition age, to determine the proficiency and cognitive depth of deaf people's sign language, and to distinguish between native speakers and non-native speakers;

[0081] Communication habits reflect the degree of dependence of deaf people on different language forms in daily communication, and distinguish between pure sign language dependence and sign language-oral language bilingualism.

[0082] Specifically, the critical period of language is used as a boundary to determine prelingual deafness and postlingual deafness.

[0083] If the deaf person loses hearing at birth or ≤3 years old, it is prelingual deafness; such deaf people have not formed systematic cognition of oral language before the complete loss of auditory function, and language learning relies on visual channels from the beginning, and sign language is more likely to become their first language.

[0084] If the deaf person loses hearing at > 3 years old and has mastered a certain degree of spoken language before losing hearing, it belongs to post-lingual deafness; this kind of deaf person has established a spoken language thinking system through hearing before losing hearing, and even if they learn sign language later, they will still have cognitive inertia for spoken language.

[0085] The age of acquiring sign language is a key node of 6 years old to distinguish between native and non-native.

[0086] If the deaf person starts to learn natural sign language at ≤ 6 years old, it belongs to native; at this stage, the language development is in the critical period, and the deaf person can naturally form a language cognitive mode centered on sign language, and has an instinctive understanding of its rhythm, grammar and non-verbal features, just like hearing children acquiring spoken language.

[0087] If the deaf person starts to learn sign language at > 6 years old, it belongs to non-native; at this time, the language cognitive mode has basically been shaped, and is mostly based on spoken language, and learning sign language is more as a communication tool, and it is difficult to achieve the proficiency of native, and it needs deliberate practice to capture and understand the details.

[0088] The communication habit is mainly based on the proportion of sign language use to distinguish between pure sign language dependence and sign language-spoken language bilingualism.

[0089] If the deaf person usually uses sign language ≥ 90% in daily communication, and less combines with lip reading or spoken language, it belongs to pure sign language dependence; this kind of deaf person relies more on the visual features of sign language to transmit information, and has weaker understanding ability of spoken language.

[0090] If the deaf person uses sign language and combines with lip reading or spoken language in daily communication, and the proportion of sign language use is usually 30%-70%, it belongs to sign language-spoken language bilingualism; this kind of deaf person has a certain basis of spoken language before losing hearing, or has been exposed to sign language and spoken language at the same time during growth, and can understand and transmit information through multiple channels.

[0091] In step S1, the cleaned deaf cognitive data is substituted into the three-dimensional classification matrix.

[0092] The three-dimensional classification matrix is a tool for system classification of deaf cognitive subgroups, which is composed of three independent classification dimensions of deafness time, sign language acquisition age and communication habit, and through the combination analysis of the three dimensions, the accurate subdivision of the cognitive characteristics of the deaf is realized.

[0093] In this application, the three-dimensional classification matrix is the core framework for realizing the subdivision of cognitive diversity, and the specific division standards of its three dimensions have been specified, based on which the classification formula can be obtained: cognitive subgroup = deafness time category x sign language acquisition age category x communication habit category.

[0094] Among them, the deafness time category includes prelingual deafness and postlingual deafness; the sign language acquisition age category includes native and non-native; the communication habit category includes pure sign language dependence and sign language-oral language bilingualism.

[0095] Combining different categories of the three dimensions, eight theoretically combined results can be obtained:

[0096] However, combined with the actual situation of the cognitive development of deaf people, part of the combination does not exist due to logical contradiction.

[0097] The two combinations that do not exist are "postlingual deafness x native x pure sign language dependence" and "postlingual deafness x native x sign language-oral language bilingualism".

[0098] From the perspective of the relationship between deafness time and sign language acquisition age, postlingual deafness means deafness after the age of 3, at which time the deaf person has established an oral language thinking system to some extent. Native sign language acquisition requires exposure to natural sign language before the age of 6 to naturally form a language cognitive mode centered on sign language. In the case of deafness after the age of 3, the deaf person mainly learns oral language before deafness, and it is difficult for them to develop sign language to a native level within such a short time span (3-6 years old). Most postlingual deaf people start learning sign language later, usually in their teens or adulthood, making it difficult for them to instinctively master the rhythm, grammar, and non-verbal features of sign language as deaf people who have been exposed to sign language from an early age.

[0099] Further analysis from the perspective of communication habits, postlingual deaf people have an oral language foundation in the early stage, and even if they learn sign language later, they will retain cognitive inertia for oral language. If the two combinations "postlingual deafness x native x pure sign language dependence" or "postlingual deafness x native x sign language-oral language bilingualism" occur, it means that they have not only raised sign language to a native level within a short period of time after deafness, but also meet the communication habit characteristics of pure sign language dependence or sign language-oral language bilingualism, which is contrary to the law of deaf language cognitive development. Postlingual deaf people tend to combine oral language or lip reading in communication rather than relying solely on sign language due to their early oral language foundation, and the time cost and cognitive development process of reaching a native level of sign language do not support the occurrence of these two combinations.

[0100] Therefore, based on the actual law of deaf language cognitive development, these two combinations do not exist in reality.

[0101] Finally, six effective cognitive subgroups are formed, as shown in the following table:

[0102] Table 1 Effective cognitive subgroups

[0103]

[0104] Through the three-dimensional classification matrix, the deaf can be scientifically and systematically divided into different cognitive subgroups, laying a foundation for subsequent eye tracking data collection and data set construction for different subgroups, ensuring that the data set accurately reflects the cognitive characteristics of different deaf populations, thereby improving the adaptability and accuracy of the sign language recognition model.

[0105] After dividing into 6 effective cognitive subgroups, multi-dimensional sign language videos need to be generated for deaf people in different groups; single sign language videos cannot cover the diverse communication needs in real-world scenarios, which may result in lower recognition accuracy of the model in complex scenarios, while multi-dimensional videos can simulate sign language expression in different environments, significantly improving the scene adaptability of the data set, and enabling the trained sign language recognition model to be applied in real-world scenarios.

[0106] Multi-dimensional sign language videos are generated using a combination of real shooting and 3D engine synthesis; in the real shooting segment, at least 3 professional sign language performers are required, including at least 2 deaf professional sign language users and 1 hearing sign language interpreter, to ensure the standardization and naturalness of sign language expression. In the 3D engine synthesis segment, a professional 3D engine such as UnrealEngine is used to build a sign language digital human model, which must have consistent body movement accuracy and facial expression details as real sign language performers. Based on the video content of real shooting, different viewing angles (front, 45° side angle, overhead), lighting conditions (strong light, weak light, backlight), and background interference (static background, dynamic crowd) are simulated in the 3D engine to generate synthetic data corresponding to real videos.

[0107] According to the characteristics of different cognitive subgroups, differentiated processing is carried out on video content design:

[0108] For the prelingual deaf group, high syntax complexity video segments are embedded in the video, including: role switching, spatial verbs, and negative expression markers, to meet the high dependence of this group on complex sign language syntax and facial expressions;

[0109] For the postlingual deaf group, easy-to-confuse mouth shape contrast sequences are inserted into the video, such as the lip shape difference segments of Chinese "cat / hat" and English "pat / bat", to meet the attention needs of this group to lip reading information;

[0110] At the same time, all groups uniformly include basic vocabulary photo segments as a control variable to ensure the comparability of different subgroups' data at the basic level.

[0111] Through differentiated content design, the generated multi-dimensional sign language videos can more accurately trigger the typical visual attention patterns of different cognitive subgroups, making the collected eye tracking data more reflective of the cognitive characteristics of each group, providing a more targeted data foundation for subsequent annotation and model training.

[0112] In step S2, the eye tracking data of the deaf person watching the multi-dimensional sign language video is collected by an eye tracker;

[0113] The eye tracking data refers to the eye movement information of the deaf person watching the sign language video recorded by the eye tracker, including the gaze point coordinates, gaze duration, gaze frequency, and saccade trajectory. In this application, the eye tracking data is the core basis for analyzing the visual attention characteristics of the deaf person and can reflect the attention preferences of different cognitive subgroups for various regions in the video.

[0114] The first-level physical region labeling refers to labeling the key regions with physical entity attributes in the video image. These regions are the core carriers of information in sign language expression. In this application, the first-level physical region includes the hand region, face region, gesture motion trajectory region, and mouth shape change region.

[0115] The hand region covers finger movements and palm gestures; the face region includes eyebrows, eyes, and mouth; the gesture motion trajectory region refers to the movement path of the gesture in space; and the mouth shape change region refers to the opening and closing and shape changes of the mouth.

[0116] Before collecting data:

[0117] Select an experimental space with light intensity of 300-500 lux and no reflection interference, equipped with adjustable chairs and displays, to ensure that the viewing angle and screen vertical distance of the deaf samples watching the video are maintained at 60-80 cm, reducing the eye movement data deviation caused by viewing angle.

[0118] Use a high-precision eye tracker with a sampling frequency of ≥120Hz, such as Tobii Pro Fusion, and perform nine-point calibration on each sample before the experiment to ensure that the gaze point error is ≤0.5° visual angle. Calibration must be passed before data collection can begin.

[0119] Present the multi-dimensional sign language videos generated in step S1 in random order, set a 3-second still image as a preparation signal before each video plays, and set a 2-second blank image as a buffer after the video ends to avoid visual fatigue caused by continuous viewing by the sample.

[0120] In step S2, the deaf samples of the 6 cognitive subgroups are arranged in independent experimental units, and each sample watches the same set of multi-dimensional sign language videos. Each sample needs to complete 3 rounds of video watching, with each round lasting within 20 minutes, and a 5-10 minute break between rounds to ensure data stability.

[0121] The eye tracker synchronously records all eye movement parameters during the sample's viewing process, including the gaze point coordinates of each frame, the duration of individual gaze events, the cumulative gaze duration proportion in each physical region, and the video timestamp, achieving precise synchronization of eye movement data and video content.

[0122] The prior art does not collect eye movement data for different cognitive subgroups, resulting in annotated physical regions that cannot reflect population differences. The present application collects data and performs primary physical region annotation by subgroup, because the visual attention patterns of different cognitive subgroups differ significantly, with prelingual deaf native speakers paying more attention to faces and postlingual deaf non-native speakers paying more attention to mouths, and subgroup collection can accurately capture these differences; and primary physical region annotation provides a basic spatial framework for subsequent cognitive weight analysis.

[0123] In step S2, primary physical region annotation is performed to clearly define the boundaries and ranges of key physical regions in the video frame, such as the hand region, face region, gesture motion trajectory region, and mouth shape change region. A professional video annotation tool, such as Label Studio, is used to annotate each frame with a rectangular or polygonal frame, and the appearance and disappearance times of the regions are recorded simultaneously. Several experts independently annotate the regions, and the consistency is verified by calculating the intersection-over-union ratio. If the consistency is not met, the regions are re-annotated by consultation to ensure the accuracy and consistency of the annotation, providing a reliable basis for subsequent analysis.

[0124] The boundaries and ranges of each physical region are clearly defined to ensure the accuracy and consistency of the annotation.

[0125] Hand region: starting from the wrist and ending at the fingertips, covering the entire hand; covering the entire hand, including finger flexion, extension, and other movements, palm turning, tilting, and other postures, regardless of whether the hand is in a static or dynamic state.

[0126] Face region: from the hairline to the jawline, including eyebrow movements such as raising and frowning, eye states such as opening and closing, gaze direction, and mouth shape changes such as opening and closing, mouth corner lifting and drooping.

[0127] Gesture motion trajectory region: records the path covered by the hand gesture in space, and is divided into up-down, left-right, and front-back direction trajectories according to the motion direction, and needs to present the entire motion process from the starting to the ending position.

[0128] Mouth shape change region: focuses on the opening and closing of the mouth due to pronunciation, including lip spreading, tooth exposure, and tongue movement, and overlaps with the mouth part of the face region, but focuses more on the shape features related to pronunciation.

[0129] A professional video annotation tool, such as Label Studio, is used to annotate each frame of the video.

[0130] For relatively fixed areas such as hand regions, face regions, and mouth shape change regions, rectangular boxes are used for labeling to ensure that the box selection range completely covers the area boundary. For dynamic areas such as hand gesture trajectory regions, polygon boxes are used for labeling to adapt to the irregular shape of the trajectory.

[0131] The labeling accuracy is within ±3 pixels to ensure the accuracy of the area boundary. The labeling of the same area in different frames should be kept consistent, especially for the hand gesture trajectory region, which needs to ensure that the trajectories of the previous and subsequent frames are naturally connected and completely reflect the movement process of the hand gesture.

[0132] During the labeling process, the appearance time and disappearance time of the region need to be recorded synchronously and accurately correspond to the video timestamp to facilitate subsequent association analysis of eye movement data and labeled regions.

[0133] From the data end, the labeled data needs to be stored in JSON format, and each labeled data contains the fields of cognitive subgroup ID, region ID, region type, frame number, boundary coordinates, appearance timestamp, and disappearance timestamp. Among them, the boundary coordinates include the top-left corner and bottom-right corner coordinates for rectangular boxes and the vertex coordinates for polygon boxes. The coordinate values are accurate to the pixel level, and the timestamps are accurate to the millisecond level.

[0134] In step S2, the experts are professional personnel with knowledge of sign language linguistics and labeling experience, who independently complete the first-level physical region labeling of the same batch of videos.

[0135] The experts here need to meet the following qualification requirements to ensure the professionalism and consistency of the labeling:

[0136] Have more than 5 years of experience in sign language linguistics research or practice, and be familiar with the grammar rules and expression characteristics of at least 2 or more sign language systems;

[0137] Have more than 3 years of experience in video labeling, have participated in at least 3 sign language related dataset construction projects, and can accurately understand the technical specifications of physical region labeling;

[0138] Pass the sign language ability level test and can accurately identify subtle movements and facial expression changes in sign language expressions.

[0139] After labeling is completed, the consistency of the labeling results is verified by calculating the intersection over union (IoU) of the labeled regions, with the requirement that IoU≥0.85. The calculation method of the intersection over union is the ratio of the intersection area of two labeled regions to the union area.

[0140] Let the region labeled by expert A be A and the same region labeled by expert B be B, then the intersection over union calculation formula is where represents the area of the overlapping part of the two labeled regions A and B, Total area covered by two labeled regions A, B.

[0141] For the labeled regions with IoU lower than 0.85, the labeling team jointly discusses and analyzes the difference reasons and re-labels until the labeling consistency of all regions reaches the requirement to ensure the objectivity and reliability of the labeling results.

[0142] In step S2, the eye tracking data is collected by the sub-cognition group and the first-level physical region labeling is performed, so as to accurately capture and structure the visual attention characteristics of different deaf populations, and build a basic framework for the second-level cognitive weight labeling in step S3.

[0143] In step S3, the second-level cognitive weight labeling is performed on the eye tracking data based on the characteristics of the cognitive subgroups.

[0144] The second-level cognitive weight labeling refers to assigning dynamic weight values to each region in the first-level physical region labeling based on the visual attention characteristics of different cognitive subgroups, so as to quantify the importance of different regions in the information processing of specific subgroups. In this application, the second-level cognitive weight labeling is the key link between physical regions and cognitive characteristics, which can make up for the limitation of the first-level physical region labeling that only reflects spatial attributes and ignores cognitive differences.

[0145] The visual attention patterns of different cognitive subgroups are significantly different, which are specifically manifested as follows:

[0146] Prelingual deaf x native level x pure sign language dependent subgroup: the highest proportion of attention is paid to the facial region, especially the expression changes of eyebrows and eyes, and the hand gesture trajectory region, because this subgroup relies on the expression markers in the sign language grammar and the spatial verb expression;

[0147] Prelingual deaf x native level x sign language-spoken language bilingual subgroup: while maintaining high attention to the facial region, the attention to the lip movement region is slightly improved;

[0148] Prelingual deaf x non-native x pure sign language dependent subgroup: more attention is paid to the details of finger movements and palm postures in the hand region, because non-native speakers need to analyze the movements frame by frame;

[0149] Prelingual deaf x non-native x sign language-spoken language bilingual subgroup: the attention to the hand region and the lip movement region is balanced;

[0150] Postlingual deaf x non-native x pure sign language dependent subgroup: although relying on sign language, due to the retention of spoken language thinking habit, the attention to the temporal coherence of the hand gesture trajectory region is higher;

[0151] Postlingual deaf x non-native x sign language-spoken language bilingual subgroup: the weight of lip reading information in the lip movement region is the highest, while the hand region is also considered.

[0152] The weight reference value is based on the eye movement data collected in step S2 for each subgroup, that is, by calculating the average of the cumulative gaze duration proportion and the gaze frequency proportion of a certain region, and determining the normalized value.

[0153] The weight reference value of each cognitive subgroup is calculated based on the eye movement data collected in step S2, and the specific process is as follows:

[0154] From the eye tracking data of each cognitive subgroup, the cumulative gaze duration and gaze frequency of the hand region, face region, gesture motion trajectory region, and mouth shape change region when all samples watch the multi-dimensional sign language video are extracted.

[0155] The cumulative gaze duration proportion and gaze frequency proportion of a single sample on a certain region are calculated.

[0156] Specifically, the cumulative gaze duration proportion of a single sample on a certain region is wherein, represents the cumulative gaze duration proportion of the ith sample in the jth cognitive subgroup on the kth region, represents the cumulative gaze duration of the ith sample in the jth cognitive subgroup on the kth region; represents the total gaze duration of the ith sample in the jth cognitive subgroup;

[0157] The gaze frequency proportion of a single sample on a certain region is wherein, represents the gaze frequency proportion of the ith sample in the jth cognitive subgroup on the kth region, represents the gaze frequency of the ith sample in the jth cognitive subgroup on the kth region, represents the total gaze frequency of the ith sample in the jth cognitive subgroup.

[0158] Then, the average of the above proportions is calculated, and the weight reference value is , is the average of the cumulative gaze duration proportion and the gaze frequency proportion of a single sample on a certain region after normalization; is the corresponding dynamic weight, which is set according to the subgroup characteristics.

[0159] From the data end, the weight reference value calculation needs to build an intermediate data model, including four levels of data structure of the original eye movement data table, the proportion calculation result table, the normalization result table, and the weight reference value table, which are cascadedly associated through sample ID and region ID. The intermediate files generated in the calculation process are stored in JSON format, and the configuration table includes the fields of cognitive subgroup ID, dynamic weight parameter, and effective time, supporting dynamic adjustment and version management.

[0160] In step S3, the eye tracking data is deeply fused with cognitive sub-group characteristics through secondary cognitive weight labeling, forming a double-layer labeling structure of physical region + dynamic weight, laying a foundation for outputting a data set with spatial precision and cognitive adaptability.

[0161] In step S4, the labeled eye tracking data is used as a training sample to adjust the action boundary threshold value according to the cognitive sub-group through the dynamic time warping algorithm, and a data set with fused physical labeling and cognitive weight is output.

[0162] The eye tracking data with secondary cognitive weight labeling completed in step S3 is standardized, specifically including:

[0163] Data alignment: based on the video timestamp, the boundary coordinate data of the first-level physical region labeling, the dynamic weight value of the second-level cognitive weight labeling, and the eye tracking data are aligned at the frame level to ensure accurate association of multi-dimensional data in the same frame;

[0164] Abnormal values in the eye tracking data are removed using the 3σ principle, that is, when the fixation duration and fixation times of a frame exceed ±3 times the standard deviation of the corresponding index of the cognitive sub-group, it is marked as an abnormal frame and removed;

[0165] Key features are extracted from the preprocessed data, including: boundary coordinate sequence of each physical region, dynamic weight value sequence, and distribution density of fixation points in the region; wherein the distribution density is the number of fixation points per unit area;

[0166] The feature values are normalized to the interval [0, 1], and the normalization parameters used are the maximum and minimum values of each feature in the entire sample.

[0167] In step S4, the dynamic time warping model includes:

[0168] The dynamic time warping algorithm is used to time-align the action sequences of different cognitive sub-groups; specifically, the time window width is 5 frames, the distance measurement method is Euclidean distance, and the path constraint uses Itakura parallelogram constraint;

[0169] For each cognitive sub-group x, the mean value of the dynamic weight value of all samples in the cognitive sub-group in each physical region is calculated ;

[0170] The basic action boundary threshold value is set to , which represents the maximum allowed offset reference value of the adjacent frame action region boundary;

[0171] The threshold value is adjusted for each sub-group according to the following formula: , wherein is the action boundary threshold value of the xth cognitive sub-group in a specific physical region, A coefficient for balancing the influence degree of the dynamic weight value on the boundary threshold.

[0172] Based on the adjusted action boundary threshold Calculate the m x n order distance matrix D of the two action sequences to be aligned, and the element in the ath row and bth column of the matrix is The value is the Euclidean distance between the ath frame and the bth frame, and if the action region boundary offset of the two frames exceeds , the value is infinite.

[0173] Calculate the optimal alignment path ; wherein, represents the minimum cumulative distance between the pth frame of the first action sequence and the qth frame of the second action sequence, reflecting the optimal alignment cost of the two frames and all previous frames; The Euclidean distance between the pth frame of the first action sequence and the qth frame of the second action sequence is the matching cost of the current frame; p and q are the frame indexes of the first action sequence and the second action sequence; wherein, , the edge element When one of the sequences is the 0th frame and the other sequence has entered the valid frame, the cumulative distance is infinite.

[0174] When calculating , the dynamic weight value is normalized as a weighting coefficient for distance calculation.

[0175] Output the optimal alignment path, the minimum cumulative distance value, and the path visualization result.

[0176] Step S4 is used to convert the preprocessed and fused multi-dimensional data into a standardized data set that can be directly used for model training;

[0177] The data set integrates the structured data set of physical region annotation, cognitive weight information, eye movement features, and dynamic time warping results;

[0178] Basic layer data: original eye tracking data after standardization processing;

[0179] Annotation layer data: boundary coordinate sequence of first-level physical region annotation, dynamic weight value sequence of second-level cognitive weight annotation;

[0180] Fusion layer data: action boundary threshold after dynamic time warping, time alignment path, and feature normalization result.

[0181] From the data end, the optimal alignment path data output by the dynamic time warping algorithm is associated with the first-level physical region annotation data, the second-level cognitive weight annotation data, and the preprocessed eye tracking data through video ID, frame number, and cognitive subgroup ID to establish a three-dimensional association index.

[0182] wherein the first-level physical region annotation data contains boundary coordinates, region type; the second-level cognitive weight annotation data contains dynamic weight value, calculation basis; the pre-processed eye tracking data contains gaze point coordinates, gaze duration, gaze frequency.

[0183] For each cognitive subgroup, a fusion matrix of cognitive subgroup ID x frame sequence number x physical region x feature dimension is constructed. The feature dimension includes: region boundary coordinates, dynamic weight value, cumulative gaze duration proportion, gaze frequency proportion, alignment error value and deviation pixel number from the optimal path, and the matrix element value is normalized to the standardized data range [0, 1].

[0184] Exemplary system:

[0185] A sign language key region dataset construction system based on eye tracking, comprising:

[0186] A cognitive subgroup division module: used for acquiring deaf cognitive data and dividing cognitive subgroups based on a three-dimensional classification matrix. Data is collected through questionnaire survey, interview and language ability test, and after cleaning, the division results of 6 effective cognitive subgroups and the corresponding sample information are output.

[0187] A multi-dimensional video generation module: used for receiving the subgroup information output by the cognitive subgroup division module and generating multi-dimensional sign language videos adapted to different subgroups. Real shooting and 3D engine synthesis are combined to output video files and video metadata containing differentiated content. The module transmits the video files to the eye movement data acquisition module.

[0188] An eye movement data acquisition module: used for receiving the video files output by the multi-dimensional video generation module and collecting eye movement data of each cognitive subgroup sample when watching the video through an eye tracker. The output is time-stamped eye tracking raw data.

[0189] A first-level physical region annotation module: used for receiving the video files output by the eye movement data acquisition module and the sample information output by the cognitive subgroup division module, and annotating the hand, face, gesture motion trajectory and mouth shape change region in the video by experts. After calculating the intersection and union ratio to verify consistency, the annotation data is output.

[0190] A second-level cognitive weight annotation module: used for receiving the eye tracking raw data output by the eye movement data acquisition module, the annotation data output by the first-level physical region annotation module and the subgroup features output by the cognitive subgroup division module, calculating the weight reference value of each region and assigning dynamic weight, and outputting the annotation results containing dynamic weight value.

[0191] The data preprocessing module is configured to receive the labeling result output by the secondary cognitive weight labeling module, perform data alignment, abnormal value elimination and feature extraction and normalization, and output standardized feature data.

[0192] The dynamic time warping module is configured to receive the standardized feature data output by the data preprocessing module and the subgroup information output by the cognitive subgroup division module, adjust the action boundary threshold value through a dynamic time warping algorithm, and output an optimal alignment path and a fused feature matrix.

[0193] The data set output module is configured to receive the fused feature matrix output by the dynamic time warping module, encapsulate the fused data set of the physical labeling and the cognitive weight, and output an HDF5 format data set file and a quality report.

[0194] Data transmission relationship:

[0195] The cognitive subgroup division module transmits subgroup information to the multi-dimensional video generation module, the eye movement data acquisition module, the secondary cognitive weight labeling module and the dynamic time warping module.

[0196] The multi-dimensional video generation module transmits a video file to the eye movement data acquisition module.

[0197] The eye movement data acquisition module transmits eye movement data and a video file to the primary physical region labeling module and the secondary cognitive weight labeling module.

[0198] The primary physical region labeling module transmits physical region labeling data to the secondary cognitive weight labeling module.

[0199] The secondary cognitive weight labeling module transmits a labeling result with a dynamic weight to the data preprocessing module.

[0200] The data preprocessing module transmits standardized feature data to the dynamic time warping module, and the dynamic time warping module transmits a fused feature matrix to the data set output module.

[0201] The above is the ideal embodiment of the present application, and through the above description, relevant personnel can make various changes and modifications without deviating from the technical idea of the present application. The technical scope of the present application is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims

1. A method for constructing a sign language key region dataset based on eye tracking, characterized in that, The method comprises the steps of: S1, acquiring deaf cognition data, dividing cognition subgroups based on a three-dimensional classification matrix, and generating multi-dimensional sign language videos for deaf people in different groups; wherein the three-dimensional classification matrix is composed of three classification dimensions of deaf time, sign language acquisition age and communication habit; S2, collecting eye tracking data of deaf people in different cognition subgroups, and performing primary physical region annotation; S3, based on the characteristics of the cognition subgroups, performing secondary cognition weight annotation on the eye tracking data; S4, taking the annotated eye tracking data as training samples, adjusting the action boundary threshold value according to the cognition subgroups by using a dynamic time warping algorithm, and outputting a data set fused with physical annotation and cognition weight; In step S2, the primary physical region annotation comprises: Hand region annotation: taking wrist as starting point and fingertip as end point, covering finger flexion and palm turning action; Face region annotation: covering eyebrow action from hairline to lower jaw line, eye state and mouth shape change; Gesture trajectory region annotation: using polygon frame to annotate the up-down, left-right and front-back movement path of gesture in space; Oral shape change region annotation: focusing on the pronunciation related shape of lip spreading, tooth exposure and tongue movement; The secondary cognition weight calculation of step S3 comprises: Extracting the cumulative gaze total time and gaze total times of each cognition subgroup sample in four physical regions, including hand, face, gesture trajectory and oral shape change; Calculating the cumulative gaze time proportion and gaze times proportion of each region for a single sample; Calculating the average value of the proportion of each region, and generating a weight reference value after normalization processing; Assigning a dynamic weight value combined with the characteristics of the cognition subgroups, and outputting the result with weight annotation.

2. The method of claim 1, wherein the method further comprises: In step S1, the deaf time is divided into prelingual deafness and postlingual deafness with 3 years as the critical period; the sign language acquisition age is divided into native and non-native with 6 years as the node; and the communication habit is divided into pure sign language dependence and sign language-oral language bilingualism with the proportion of sign language use.

3. The method of claim 1, wherein the method further comprises: determining a plurality of key regions of the hand gesture based on the eye tracking data. In step S1, different categories of the three dimensions are combined to obtain 8 theoretical combination results; combined with the actual situation of deaf cognition development, invalid combinations are removed, and 6 effective cognition subgroups are output as: Prelingual deafness x native x pure sign language dependence; Prelingual deafness x native x sign language-oral language bilingualism; Prelingual deafness x non-native x pure sign language dependence; Prelingual deafness x non-native x sign language-oral language bilingualism; Postlingual deafness x non-native x pure sign language dependence; Postlingual deafness x non-native x sign language-oral language bilingualism.

4. The method of claim 1, wherein the method further comprises: The primary physical region annotation is stored in JSON format, including the fields of cognition subgroup ID, region ID, region type, frame sequence number, boundary coordinates, appearance timestamp and disappearance timestamp; wherein the boundary coordinates include the left upper corner and right lower corner coordinates of the rectangular frame, and the vertex coordinates of the polygon frame; the coordinate value is accurate to the pixel level, and the timestamp is accurate to the millisecond level.

5. The method of claim 1, wherein the method further comprises: The dynamic time warping algorithm of step S4 includes action boundary threshold value adjustment: Calculate the average dynamic weight value of the target cognition subgroup in a specific physical region; Based on the basic threshold value, the weight adjustment coefficient and the average dynamic weight value, adjust the action boundary threshold value of the subgroup in proportion; Constructing inter-frame distance matrix: if the two-frame action region boundary offset exceeds the adjusted threshold, it is determined that the action boundary is discontinuous, and the inter-frame distance is set to infinity; otherwise, the Euclidean distance of the physical region key points between the two frames is calculated.

6. The method of claim 1, wherein: In step S1, different cognitive subgroups are processed differently in video content design according to their characteristics: For pre-lingual deaf groups, high syntax complexity video segments are embedded in the video, including: role conversion, spatial verbs, and negative expression markers; For post-lingual deaf groups, easy-to-confuse lip shape comparison sequences are inserted into the video, including lip shape difference segments; At the same time, all groups uniformly include basic vocabulary photo segments as control variables.

7. The method of claim 1, wherein the method further comprises: When outputting the data set in step S4: Establish a three-dimensional index to associate video ID, frame number and cognitive subgroup ID; Packaging three layers of structured data into an HDF5 file: The basic layer contains standardized eye movement data; The labeled layer contains physical region boundary coordinate sequences and cognitive weight dynamic value sequences; The fusion layer contains the adjusted action boundary threshold, the optimal alignment path, and the normalized feature matrix.

8. A sign language key region dataset construction system based on eye tracking, based on the sign language key region dataset construction method based on eye tracking in any one of claims 1-7, characterized in that, Modules include: Cognitive subgroup division module for obtaining deaf cognitive data and dividing cognitive subgroups based on a three-dimensional classification matrix; Multi-dimensional video generation module for receiving subgroup information output by the cognitive subgroup division module and generating multi-dimensional sign language videos adapted to different subgroups; Eye movement data acquisition module for receiving video files output by the multi-dimensional video generation module and collecting eye movement data of each cognitive subgroup sample when watching the video through an eye tracker; First-level physical region annotation module for receiving video files output by the eye movement data acquisition module and sample information output by the cognitive subgroup division module, annotating hand, face, gesture trajectory and lip shape change regions in the video by experts, verifying consistency by calculating the intersection over union ratio, and outputting annotation data; Second-level cognitive weight annotation module for receiving eye movement tracking raw data output by the eye movement data acquisition module, annotation data output by the first-level physical region annotation module, and subgroup features output by the cognitive subgroup division module, calculating the weight reference value of each region and assigning dynamic weights, and outputting annotation results containing dynamic weight values; Data preprocessing module for receiving annotation results output by the second-level cognitive weight annotation module and performing data preprocessing; Dynamic time warping module for receiving standardized feature data output by the data preprocessing module and subgroup information output by the cognitive subgroup division module, adjusting the action boundary threshold through the dynamic time warping algorithm, outputting the optimal alignment path and the fused feature matrix; Dataset output module for receiving the fused feature matrix output by the dynamic time warping module, packaging it into a data set that combines physical annotation and cognitive weight, and outputting an HDF5 format data set file and a quality report.

Images