A method and device for calculating sign language similarity and a medium

Abstract

The present application relates to the technical field of sign language processing, and particularly relates to a sign language similarity calculation method and device and medium, the method comprising: obtaining a target sign language video of a sign language user; determining a hand joint sequence of the sign language user based on the target sign language video; extracting spatial scale features and time scale features based on the hand joint sequence; determining spatial differentiation values and time differentiation values by using a dynamic time warping algorithm based on the spatial scale features, the time scale features and standard features of a standard sign language; determining final spatial differences and final time differences based on the spatial differentiation values and the time differentiation values; and obtaining a sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial differences and the final time differences, thereby improving the accuracy of sign language similarity determination through decoupling and accurate quantification of space-time features.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of sign language processing, and in particular to a sign language similarity calculation method, device and medium. BACKGROUND

[0002] Sign language is an important communication tool for the deaf and mute population, and its standardization and accuracy directly affect the efficiency of information transmission. Moreover, sign language expression involves multiple dimensions such as hand shape, motion trajectory, spatial position, and time sequence, and the characteristics are complex. At present, the standardization determination of sign language motion, data set quality control, and sign language recognition mainly rely on manual completion, and there are problems such as low efficiency, strong subjectivity, and difficulty in large-scale application.

[0003] In related technologies, the similarity calculation method uses Euclidean distance, cosine similarity, etc., which is difficult to characterize the spatiotemporal characteristics of sign language motion. Although the traditional dynamic time warping (DTW) algorithm can process time series data, it has poor processing capability for complex spatiotemporal data. The method based on deep learning relies on large-scale labeled data, and the model is complex and has limited generalization ability.

[0004] Therefore, how to provide a sign language similarity calculation method that can decouple the spatiotemporal characteristics of sign language and improve the calculation accuracy is a technical problem to be solved at present. SUMMARY

[0005] In view of the above problems, the present application provides a sign language similarity calculation method, device and medium which can overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, the present application provides a sign language similarity calculation method, comprising:

[0007] obtaining a target sign language video of a sign language user;

[0008] determining a hand joint sequence of the sign language user based on the target sign language video;

[0009] extracting spatial scale features and time scale features based on the hand joint sequence;

[0010] determining spatial differentiation values and time differentiation values by using a dynamic time warping algorithm based on the spatial scale features, the time scale features, and standard features of a standard sign language;

[0011] determining final spatial differences and final time differences based on the spatial differentiation values and the time differentiation values;

[0012] obtaining a sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial differences and the final time differences.

[0013] Preferably, the hand joint points include left-hand finger joint points, right-hand finger joint points, left-arm joint points, right-arm joint points, left-wrist joint points and right-wrist joint points.

[0014] Preferably, based on the hand joint point sequence, spatial scale features and time scale features are extracted, including:

[0015] Based on the left-wrist joint point sequence and the right-wrist joint point sequence, redundant frames are filtered to obtain remaining frames;

[0016] Based on the remaining frames, a first motion trajectory feature of the left-wrist joint point and a second motion trajectory feature of the right-wrist joint point are determined;

[0017] The first motion trajectory feature and the second motion trajectory feature are taken as spatial scale features.

[0018] Preferably, based on the hand joint point sequence, spatial scale features and time scale features are extracted, including:

[0019] Based on the left-hand finger joint point sequence, the right-hand finger joint point sequence, the left-arm joint point sequence and the right-arm joint point sequence, a graph structure is constructed, the graph structure including edges between each joint point and any adjacent joint point;

[0020] Based on the graph structure, a direction vector of each edge is calculated;

[0021] Based on the direction vector, a feature matrix of consecutive frames of the target sign language video is obtained;

[0022] Based on the feature matrix, an average pooling result of every preset number of frames is determined;

[0023] Based on the feature matrix and the average pooling result, a time scale feature is obtained.

[0024] Preferably, based on the graph structure, a direction vector of each edge is calculated, including:

[0025] Based on the graph structure, three-dimensional coordinates of any two adjacent joint points are determined;

[0026] Based on the three-dimensional coordinates of the any two adjacent joint points, a direction vector of the edge between the any two adjacent joint points is calculated:

[0027] ;

[0028] wherein, and are any two adjacent joint points, are three-dimensional coordinates of , and are three-dimensional coordinates of three-dimensional coordinates of the arbitrary two adjacent joints, is a direction vector of an edge between the arbitrary two adjacent joints.

[0029] Preferably, determining the spatial differentiation value and the temporal differentiation value based on the spatial scale feature, the temporal scale feature and the standard feature of the standard sign language comprises:

[0030] determining a first difference in the spatial scale between two trajectory points based on the spatial scale feature and the standard feature of the standard sign language by using the Euclidean distance calculation, the two trajectory points being a wrist joint in the target sign language video and a same wrist joint in the corresponding standard sign language;

[0031] obtaining a spatial differentiation matrix based on the first difference;

[0032] calculating an included angle between direction vectors of edges at the same position in the corresponding two frames based on the temporal scale feature and the standard feature of the standard sign language, the two frames being one frame in the target sign language video and a corresponding frame in the standard sign language;

[0033] obtaining a discrete value based on the included angle;

[0034] obtaining a second difference in the temporal scale based on the discrete value by using the Manhattan distance;

[0035] obtaining a temporal differentiation matrix based on the second difference;

[0036] obtaining the spatial differentiation value and the temporal differentiation value respectively based on the spatial differentiation matrix and the temporal differentiation matrix by using the dynamic time warping algorithm or the Earth mover's distance calculation.

[0037] Preferably, determining the final spatial difference and the final temporal difference based on the spatial differentiation value and the temporal differentiation value comprises:

[0038] normalizing the spatial differentiation value and the temporal differentiation value respectively to obtain a normalized spatial difference and a normalized temporal difference;

[0039] determining the final spatial difference and the final temporal difference based on the normalized spatial difference and the normalized temporal difference.

[0040] Preferably, obtaining the sign language similarity between the target sign language of the sign language user and the standard sign language based on the final temporal difference and the final spatial difference is calculated according to the following formula:

[0041] ;

[0042] wherein, is the final temporal difference, is the final spatial difference, a spatial difference weight coefficient, a sign language similarity.

[0043] In a second aspect, the present application provides a sign language similarity calculation device, comprising:

[0044] an acquisition module, configured to acquire a target sign language video of a sign language user;

[0045] a first determination module, configured to determine a hand joint sequence of the sign language user based on the target sign language video;

[0046] an extraction module, configured to extract a spatial scale feature and a time scale feature based on the hand joint sequence;

[0047] a second determination module, configured to determine a spatial difference value and a time difference value based on the spatial scale feature, the time scale feature and a standard feature of a standard sign language by using a dynamic time warping algorithm;

[0048] a third determination module, configured to determine a final spatial difference and a final time difference based on the spatial difference value and the time difference value;

[0049] an obtaining module, configured to obtain a sign language similarity between the target sign language of the sign language user and the standard sign language based on the final time difference and the final spatial difference.

[0050] In a third aspect, the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to realize the method steps in the first aspect.

[0051] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program executable by a processor to realize the method in the first aspect.

[0052] The one or more technical solutions in the embodiments of the present application have at least the following technical effects or advantages:

[0053] The application provides a sign language similarity calculation method, which comprises the following steps: obtaining a target sign language video of a sign language user; determining a hand joint sequence of the sign language user based on the target sign language video; extracting spatial scale features and time scale features based on the hand joint sequence; determining spatial differentiation values and time differentiation values by using a dynamic time warping algorithm based on the spatial scale features, the time scale features and standard features of a standard sign language; determining final spatial differences and final time differences based on the spatial differentiation values and the time differentiation values; and obtaining the sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial differences and the final time differences, thereby improving the accuracy of sign language similarity determination through decoupling and accurate quantification of space-time features. BRIEF DESCRIPTION OF DRAWINGS

[0054] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application. Moreover, the same reference numerals in different figures represent the same or similar components. In the drawings:

[0055] Figure 1 A step flow diagram of the sign language similarity calculation method in the embodiment of the application is shown;

[0056] Figure 2 A schematic diagram of a hand joint in the embodiment of the application is shown;

[0057] Figure 3 A structural schematic diagram of a sign language similarity calculation device in the embodiment of the application is shown;

[0058] Figure 4 A computer device schematic diagram for implementing the sign language similarity calculation method in the embodiment of the application is shown. DETAILED DESCRIPTION

[0059] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided so that the present application can be more thoroughly understood and the scope of the present application can be accurately conveyed to those skilled in the art.

[0060] Embodiment one:

[0061] The embodiment of the present application provides a sign language similarity calculation method, as shown in Figure 1 The method comprises the following steps:

[0062] S101, obtaining a target sign language video of a sign language user;

[0063] S102, determine the hand joint sequence of the sign language user based on the target sign language video;

[0064] S103, extract the spatial scale feature and the time scale feature based on the hand joint sequence;

[0065] S104, determine the spatial differentiation value and the time differentiation value based on the spatial scale feature, the time scale feature, and the standard feature of the standard sign language by using the dynamic time warping algorithm;

[0066] S105, determine the final spatial difference and the final time difference based on the spatial differentiation value and the time differentiation value;

[0067] S106, obtain the sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial difference and the final time difference.

[0068] By calculating the sign language similarity between the target sign language of the sign language user and the standard sign language, on the one hand, the sign language standardization of the sign language user can be evaluated, and on the other hand, the sign language meaning can be effectively identified.

[0069] The calculation process of the sign language similarity will be described in detail as follows:

[0070] S101, obtain the target sign language video of the sign language user.

[0071] Then, S102, determine the hand joint sequence of the sign language user based on the target sign language video.

[0072] In a specific embodiment, as shown in Figure 2 The hand joint includes left hand finger joint, right hand finger joint, left arm joint, right arm joint, left wrist joint, and right wrist joint.

[0073] The left hand finger joint and the right hand finger joint each include finger joints, for example, the hand includes carpal bones, metacarpal bones, and phalangeal bones, and the joints are located between adjacent bones.

[0074] The positions of the hand joints in the continuous time frames are sequenced to obtain the hand joint sequence.

[0075] Next, S103 is executed to extract the spatial scale feature and the time scale feature based on the hand joint sequence.

[0076] Specifically, the spatial scale feature is extracted in the following manner:

[0077] Based on the left wrist joint sequence and the right wrist joint sequence, the redundant frames are filtered to obtain the remaining frames;

[0078] Based on the remaining frames, determine the first motion trajectory feature of the left wrist joint and the second motion trajectory feature of the right wrist joint.

[0079] The first motion trajectory feature and the second motion trajectory feature are taken as spatial scale features.

[0080] In a specific embodiment, the left wrist and the right wrist respectively represent the left hand and the right hand to simplify the calculation.

[0081] With the first frame wrist coordinate initial reference point, the Euclidean distance between the wrist coordinate in the subsequent frame and the initial reference point is calculated, and it is determined whether the Euclidean distance is less than a preset value. If yes, the corresponding frame is determined as a redundant frame and is deleted. If no, the corresponding frame is retained, and the corresponding wrist coordinate is taken as a new reference point, which is repeatedly processed until all frames are processed.

[0082] After the above processing, the first motion trajectory feature of the left wrist joint and the second motion trajectory feature of the right wrist joint are determined. The first motion trajectory feature and the second motion trajectory feature are taken as spatial scale features.

[0083] The time scale feature is extracted in the following manner:

[0084] Based on the left hand finger joint sequence, the right hand finger joint sequence, the left arm joint sequence, and the right arm joint sequence, a graph structure is constructed, which includes edges between each joint and any adjacent joint.

[0085] Based on the graph structure, the direction vector of each edge is calculated.

[0086] Based on the direction vector, a feature matrix of the continuous frames of the target sign language video is obtained.

[0087] Based on the feature matrix, the average pooling result of every preset number of frames is determined.

[0088] Based on the feature matrix and the average pooling result, the time scale feature is obtained.

[0089] First, the graph structure is obtained through graph construction. The edges of the graph structure are defined as the connection between adjacent joints, including the connection between finger joints, the connection between finger joints and palm joints, and the connection between palm joints and arm joints.

[0090] Next, the direction vector of each edge is calculated.

[0091] Specifically, based on the graph structure, the three-dimensional coordinates of any two joint nodes are determined. Based on the three-dimensional coordinates of any two adjacent joints, the direction vector of the edge between the two adjacent joints is calculated.

[0092] ;

[0093] wherein, and is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a direction vector of an edge between the arbitrary two adjacent joints.

[0094] According to the method, the direction vector of each edge corresponding to the graph structure is obtained, thereby obtaining the feature matrix of the continuous frames, which is specifically exhibited in the form of a matrix. The direction vector is used to represent the dynamic change of the hand shape in the time sequence as the core feature of the time dimension.

[0095] In order to obtain the macroscopic difference in the time scale, based on the feature matrix, the average pooling result of each preset number of frames is determined; based on the feature matrix and the average pooling result, the time scale feature is obtained.

[0096] Specifically, when determining the average pooling result of each preset number of frames, a first average pooling result can be obtained every two frames, or a second average pooling result can be obtained every four frames, wherein the first average pooling result obtained every two frames and the second average pooling result obtained every four frames are rough results, and the feature matrix of the continuous frames is a more detailed result. The rough result and the more detailed result can be averaged again, thereby obtaining the time scale feature.

[0097] Next, S104 is performed, based on the spatial scale feature, the time scale feature and the standard feature of the standard sign language, a dynamic time warping algorithm is used to determine the spatial differentiation value and the time differentiation value.

[0098] Based on the spatial scale feature and the standard feature of the standard sign language, a Euclidean distance calculation is used to determine the first difference in the spatial scale between two trajectory points, the two trajectory points being the wrist joint in the target sign language video and the same wrist joint in the corresponding standard sign language;

[0099] Based on the first difference, a spatial differentiation matrix is obtained.

[0100] Based on the time scale feature and the standard feature of the standard sign language, the included angle between the direction vectors of the edges at the same position in the corresponding two frames is calculated, the two frames being one frame in the target sign language video and one frame in the corresponding standard sign language;

[0101] Based on the included angle, a discrete numerical value is obtained;

[0102] Based on the discrete numerical value, a Manhattan distance is used to obtain the second difference in the time scale.

[0103] Based on the second difference, a time difference matrix is obtained.

[0104] Based on the space difference matrix and the time difference matrix, a dynamic time warping algorithm (DTW) or earth mover's distance (EMD) calculation is used to obtain a space difference value and a time difference value, respectively.

[0105] In the calculation of the space difference value, the space scale feature and the space feature of the standardized sign language are calculated by the Euclidean distance to obtain the space difference matrix. Based on the space difference matrix, the dynamic time warping algorithm calculation or the earth mover's distance calculation is used to obtain the space difference value.

[0106] The calculation formula is:

[0107]

[0108] wherein, is the coordinate value of the i-th trajectory point in the space scale feature, is the coordinate value of the i-th trajectory point in the space scale feature of the standard sign language, is the space difference matrix. In the calculation of the time difference value, the dynamic time warping algorithm or the earth mover's distance calculation is used to compare the time scale feature with the time feature of the standard sign language. Specifically, the included angle between the direction vectors of the edges at the same position in the corresponding two frames is obtained, and the included angle is mapped to the corresponding interval, which is the equal interval obtained by dividing the entire included angle interval according to a predetermined number. The discrete numerical value corresponding to the included angle is obtained, the Manhattan distance is summed to obtain the second difference of the two frames in the time scale, and based on the second difference, the time difference matrix is obtained.

[0109] The specific calculation formula is as follows:

[0110]

[0111] Taking the hand joint as an example, the included angle of the 9 direction vectors of the hand is calculated, is the discrete numerical value of the i-th frame in the time scale feature and the i-th frame in the time feature of the standard sign language,

[0112] is the time difference matrix. Next, S105 is executed, based on the space difference value and the time difference value, to determine the final space difference and the final time difference.

[0113]

[0114] ​​​​Specifically, the spatial differentiation value and the time differentiation value are normalized respectively to obtain normalized spatial differentiation and normalized time differentiation.

[0115] Based on the normalized spatial differentiation and the normalized time differentiation, a final time differentiation and a final spatial differentiation are determined.

[0116] Normalization processing is adopted to eliminate sequence length image.

[0117] Specifically, the normalization processing calculation formula is as follows:

[0118]

[0119] wherein, is the spatial differentiation value or the time differentiation value, is the minimum spatial differentiation value or the minimum time differentiation value under the same scale, is the average spatial differentiation value or the average time differentiation value under the same scale.

[0120] After obtaining the normalized spatial differentiation and the normalized time differentiation, the maximum value in all normalized spatial differentiations is taken as the final spatial differentiation. Specifically, , is the normalized spatial differentiation of the right wrist, is the normalized spatial differentiation of the left wrist, is the final spatial differentiation.

[0121] The final time differentiation is determined as follows:

[0122] After taking the average of the normalized time differentiations of each module under different average pooling, the maximum value of each normalized differentiation of the left and right hands is taken as the final time differentiation, wherein each module refers to the left hand finger joint node as a module, the right hand finger joint node as a module, the left arm joint node as a module, and the right arm joint node as a module. Specifically, wherein, is the normalized time differentiation of the right hand finger joint node, is the normalized time differentiation of the right arm joint node, is the normalized time differentiation of the left hand finger joint node, is the normalized time differentiation of the left arm joint node, is the final time differentiation.

[0123] Finally, S106 is executed to obtain the sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial differentiation and the final time differentiation.

[0124] Specifically, the calculation is performed according to the following formula:

[0125] ;

[0126] wherein, is a final time difference, is a final spatial difference, is a spatial difference weight coefficient, is a sign language similarity.

[0127] After obtaining the similarity of the target sign language and the standard sign language according to the above calculation formula, the similarity is compared with a set threshold value, which is set according to a normative judgment, and when the similarity is lower than the set threshold value, it is determined that the target sign language is not normative, and otherwise it is normative.

[0128] After obtaining the similarity of the target sign language and the standard sign language according to the above calculation formula, the similarity of each type of target sign language and the standard sign language is compared, and the type of the target sign language with the highest similarity is taken as the sign language recognition result, so as to achieve the purpose of recognizing the sign language.

[0129] The one or more technical solutions in the embodiments of the present application have at least the following technical effects or advantages:

[0130] The present application provides a sign language similarity calculation method, comprising: obtaining a target sign language video of a sign language user; determining a hand joint node sequence of the sign language user based on the target sign language video; extracting spatial scale features and time scale features based on the hand joint node sequence; determining spatial difference values and time difference values based on the spatial scale features, the time scale features, and standard features of a standard sign language using a dynamic time warping algorithm; determining final spatial differences and final time differences based on the spatial difference values and the time difference values; obtaining a sign language similarity of the target sign language of the sign language user and the standard sign language based on the final spatial differences and the final time differences, and further improving the accuracy of sign language similarity determination through decoupling and accurate quantization of space-time features.

[0131] Embodiment two:

[0132] Based on the same inventive concept, the present application also provides a sign language similarity calculation device, as shown in Figure 3 , comprising:

[0133] The acquisition module 301 is configured to acquire a target sign language video of a sign language user.

[0134] The first determination module 302 is configured to determine a hand joint node sequence of the sign language user based on the target sign language video.

[0135] The extraction module 303 is configured to extract spatial scale features and time scale features based on the hand joint node sequence.

[0136] The second determining module 304 is configured to determine a spatial difference value and a time difference value based on the spatial scale feature, the time scale feature, and the standard feature of the standard sign language.

[0137] The third determining module 305 is configured to determine a final spatial difference and a final time difference based on the spatial difference value and the time difference value.

[0138] The obtaining module 306 is configured to obtain a sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial difference and the final time difference.

[0139] In an optional implementation, the hand joint nodes include left-hand finger joint nodes, right-hand finger joint nodes, left-arm joint nodes, right-arm joint nodes, left-wrist joint nodes, and right-wrist joint nodes.

[0140] In an optional implementation, the extracting module 303 is configured to:

[0141] filter redundant frames based on the left-wrist joint node sequence and the right-wrist joint node sequence to obtain remaining frames;

[0142] determine a first motion trajectory feature of the left-wrist joint nodes and a second motion trajectory feature of the right-wrist joint nodes based on the remaining frames;

[0143] take the first motion trajectory feature and the second motion trajectory feature as the spatial scale feature.

[0144] In an optional implementation, the extracting module 303 is configured to:

[0145] construct a graph structure based on the left-hand finger joint node sequence, the right-hand finger joint node sequence, the left-arm joint node sequence, and the right-arm joint node sequence, the graph structure including edges between each joint node and any adjacent joint node;

[0146] calculate a direction vector of each edge based on the graph structure;

[0147] obtain a feature matrix of consecutive frames of the target sign language video based on the direction vector;

[0148] determine an average pooling result of each preset number of frames based on the feature matrix;

[0149] obtain a time scale feature based on the feature matrix and the average pooling result.

[0150] In an optional implementation, the extracting module 303 is configured to:

[0151] determine three-dimensional coordinates of any two adjacent joint nodes based on the graph structure;

[0152] based on the three-dimensional coordinates of the arbitrary two adjacent joints, a direction vector of an edge between the arbitrary two adjacent joints is calculated:

[0153] ;

[0154] wherein, and are arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a three-dimensional coordinate of the arbitrary two adjacent joints, is a direction vector of an edge between the arbitrary two adjacent joints.

[0155] In an optional implementation, the second determining module 304 is configured to:

[0156] based on the spatial scale feature and the standard feature of the standard sign language, a first difference in spatial scale between two trajectory points is determined by using the Euclidean distance calculation, the two trajectory points being a wrist joint in the target sign language video and a same wrist joint in the corresponding standard sign language;

[0157] based on the first difference, a spatial differentiation matrix is obtained;

[0158] based on the time scale feature and the standard feature of the standard sign language, an included angle between direction vectors of edges at the same position in the corresponding two frames is calculated;

[0159] based on the included angle, a discrete numerical value is obtained;

[0160] based on the discrete numerical value, a second difference in time scale between the two frames is obtained by using the Manhattan distance;

[0161] based on the second difference, a time differentiation matrix is obtained;

[0162] based on the spatial differentiation matrix and the time differentiation matrix, a spatial differentiation value and a time differentiation value are obtained by using the dynamic time warping algorithm or the earth mover's distance calculation.

[0163] In an optional implementation, the third determining module 305 is configured to:

[0164] the spatial differentiation value and the time differentiation value are normalized respectively to obtain a normalized spatial difference and a normalized time difference;

[0165] based on the normalized spatial difference and the normalized time difference, a final time difference and a final spatial difference are determined.

[0166] In an optional implementation, the obtaining module 306 is configured to:

[0167] Based on the final time difference and the final spatial difference, the sign language similarity between the target sign language and the standard sign language of the sign language user is obtained, calculated according to the following formula:

[0168] ;

[0169] in, For the final time difference, For the final spatial difference, Spatial difference weighting coefficient, For sign language similarity.

[0170] Example 3:

[0171] Based on the same inventive concept, embodiments of the present invention provide a computer device, such as... Figure 4 As shown, it includes a memory 404, a processor 402, and a computer program stored in the memory 404 and executable on the processor 402. When the processor 402 executes the program, it implements the steps of the above-described sign language similarity calculation method.

[0172] Among them, Figure 4 In this document, a bus architecture (represented by bus 400) is used. Bus 400 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 402 and memory represented by memory 404. Bus 400 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 406 provides an interface between bus 400 and receiver 401 and transmitter 403. Receiver 401 and transmitter 403 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 402 is responsible for managing bus 400 and general processing, while memory 404 can be used to store data used by processor 402 during operation.

[0173] Example 4:

[0174] Based on the same inventive concept, embodiments of the present invention provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described sign language similarity calculation method.

[0175] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present application is not intended to be limited to any particular programming language. It will be appreciated that there are many programming languages that can be used to implement the teachings herein, and any specific language can be chosen for use in this application.

[0176] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0177] Similarly, it is to be understood that the embodiments of the application can be altered, combined, or modified and constitute matters of application within the scope of the inventive concepts disclosed in this specification. Accordingly, the scope of the application should be determined by the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0178] It will be understood by those within the art that, in keeping with the scope of the application, certain steps within the embodiments of the methods can be performed in a different order, and / or certain components can be used in combination, without departing from the application. Additionally, those skilled in the art will recognize that the application can be practiced with more than one device. Therefore, although the application has been described in detail with reference to particular embodiments, it is not intended to be limited to the specific form set forth herein, but only to the spirit and the scope of the application and equivalents thereof, and further to practice the application in various ways. Numerous other changes can be made that will be readily apparent to those of ordinary skill in the art. The application is not limited to the details given herein, but can be practiced with the range of equivalents. Therefore, the scope of the application is to be interpreted only in conjunction with the appended claims and their legal equivalents.

[0179] Furthermore, those skilled in the art will recognize that, while certain embodiments described herein include certain features that are not included in other embodiments, combinations of features of the different embodiments are to be construed as being within the scope of the application and forming different embodiments. For example, in the DETAILED DESCRIPTION, any of the claimed embodiments can be used in any combination.

[0180] Various component embodiments of the application can be implemented in hardware, or as software modules running in one or more processors, or in combinations thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all of the functionality of some or all of the components of the sign language similarity computing apparatus and computer device according to embodiments of the application. The application can also be implemented as a program (e.g., computer program and computer program product) for performing part or all of the methods described herein. Such program(s) can be stored on a computer readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier medium, or in any other form.

[0181] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for performing a certain function. The functions of the separate means can be carried out by one specific means performing the functions. Similarly, a device recited as meaning for performing a certain function can be interpreted to mean a specific means performing the function. The words 'first','second' and 'third' etc. do not imply any order but are used for naming purposes only.

Claims

1. A method of calculating similarity of sign language, characterized by, The method comprises the following steps: acquiring a target sign language video of a sign language user; determining a hand joint sequence of the sign language user based on the target sign language video; extracting spatial scale features and time scale features based on the hand joint sequence; determining spatial difference values and time difference values based on the spatial scale features, the time scale features and standard features of a standard sign language by using a dynamic time warping algorithm, comprising: determining a first difference in the spatial scale between two track points by using a Euclidean distance calculation based on the spatial scale features and the standard features of the standard sign language, the two track points being a wrist joint in the target sign language video and a same wrist joint in the corresponding standard sign language; obtaining a spatial difference matrix based on the first difference; calculating an included angle between directional vectors of edges at the same position in corresponding two frames based on the time scale features and the standard features of the standard sign language, the two frames being one frame in the target sign language video and one frame in the corresponding standard sign language; obtaining discrete numerical values based on the included angle; obtaining a second difference in the time scale by using Manhattan distance based on the discrete numerical values; obtaining a time difference matrix based on the second difference; obtaining the spatial difference values and the time difference values respectively based on the spatial difference matrix and the time difference matrix by using the dynamic time warping algorithm; determining a final spatial difference and a final time difference based on the spatial difference values and the time difference values; obtaining a sign language similarity between the target sign language of the sign language user and the standard sign language based on the final spatial difference and the final time difference.

2. The method of claim 1, wherein, The hand joints comprise left-hand finger joints, right-hand finger joints, left-arm joints, right-arm joints, left-wrist joints and right-wrist joints.

3. The method of claim 2, wherein, The method for extracting the spatial scale features and the time scale features based on the hand joint sequence comprises: filtering redundant frames based on the left-wrist joint sequence and the right-wrist joint sequence to obtain remaining frames; determining a first motion track feature of the left-wrist joint and a second motion track feature of the right-wrist joint based on the remaining frames; taking the first motion track feature and the second motion track feature as the spatial scale features.

4. The method of claim 1, wherein, The method for extracting the spatial scale features and the time scale features based on the hand joint sequence comprises: constructing a graph structure based on the left-hand finger joint sequence, the right-hand finger joint sequence, the left-arm joint sequence and the right-arm joint sequence, the graph structure comprising edges between each joint and any adjacent joint; calculating directional vectors of each edge based on the graph structure; obtaining a feature matrix of consecutive frames of the target sign language video based on the directional vectors; determining an average pooling result of each preset number of frames based on the feature matrix; obtaining the time scale features based on the feature matrix and the average pooling result.

5. The method of claim 4, wherein, The method for calculating the directional vectors of each edge based on the graph structure comprises: determining three-dimensional coordinates of any two adjacent joints based on the graph structure; calculating a directional vector of an edge between the any two adjacent joints based on the three-dimensional coordinates of the any two adjacent joints: wherein and are three-dimensional coordinates of any two adjacent joints, and are three-dimensional coordinates of and, and is a direction vector of an edge between the any two adjacent joints.

6. The method of claim 1, wherein, Based on the spatial differentiation value and the time differentiation value, determine a final spatial difference and a final time difference, including: Respectively normalize the spatial differentiation value and the time differentiation value to obtain a normalized spatial difference and a normalized time difference; Based on the normalized spatial difference and the normalized time difference, determine the final spatial difference and the final time difference.

7. The method of claim 1, wherein, Based on the final time difference and the final spatial difference, obtain a sign language similarity of the target sign language of the sign language user and the standard sign language, calculated according to the following formula: wherein is the final time difference, is the final spatial difference, is a spatial difference weight coefficient, and is the sign language similarity.

8. A device for calculating similarity of sign language, characterized by, Including: An acquisition module, configured to acquire a target sign language video of a sign language user; A first determination module, configured to determine a hand joint sequence of the sign language user based on the target sign language video; An extraction module, configured to extract a spatial scale feature and a time scale feature based on the hand joint sequence; A second determination module, configured to determine a spatial differentiation value and a time differentiation value based on the spatial scale feature, the time scale feature, and a standard feature of a standard sign language using a dynamic time warping algorithm, and configured to: Based on the spatial scale feature and the standard feature of the standard sign language, determine a first difference in the spatial scale between two trajectory points using Euclidean distance calculation, wherein the two trajectory points are a wrist joint in the target sign language video and a same wrist joint in the corresponding standard sign language; Based on the first difference, obtain a spatial differentiation matrix; Based on the time scale feature and the standard feature of the standard sign language, calculate an included angle between direction vectors of edges at the same position in corresponding two frames, wherein the two frames are one frame in the target sign language video and one frame corresponding to the standard sign language; Based on the included angle, obtain a discrete numerical value; Based on the discrete numerical value, obtain a second difference in the time scale using Manhattan distance; Based on the second difference, obtain a time differentiation matrix; Based on the spatial differentiation matrix and the time differentiation matrix, respectively obtain the spatial differentiation value and the time differentiation value using the dynamic time warping algorithm; A third determination module, configured to determine a final spatial difference and a final time difference based on the spatial differentiation value and the time differentiation value; A obtaining module, configured to obtain a sign language similarity of the target sign language of the sign language user and the standard sign language based on the final time difference and the final spatial difference.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method in any one of claims 1-7.

Images