A natural human-computer interaction sign language recognition method

Abstract

This invention discloses a sign language recognition method for natural human-computer interaction, relating to the field of sign language recognition technology. The method includes: obtaining associated gestures; deconstructing real-time gestures into real-time key edges and real-time nodes, and deconstructing associated gestures into sample key edges and sample nodes; forming a real-time matrix of real-time gestures and a sample matrix of associated gestures; obtaining a real-time correction matrix; forming difference features between the real-time correction matrix and the sample matrix; selecting a target sample matrix; and using the sign language meaning of the standard gesture corresponding to the associated gesture in the target sample matrix as the real-time semantics of the real-time gesture; and forming continuous sign language content. By obtaining associated gestures, forming a real-time matrix of real-time gestures, forming a sample matrix of associated gestures, forming difference features, and performing fault-tolerant analysis on the difference features, a model that meets the usage requirements is formed, enabling the model to eliminate recognition errors and ensure recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of sign language recognition technology, specifically to a sign language recognition method for natural human-computer interaction. Background Technology

[0002] Sign language recognition plays a crucial role in human-computer interaction applications, such as education, gaming, and industrial production. As an important auxiliary tool for information transmission, sign language recognition research methods can be divided into sensor-based methods and vision-based methods. Wearable sensor-based methods, due to their inconvenience, are only suitable for certain specific scenarios, while vision-based methods, due to their superior convenience and real-time performance, are continuously being explored in various application scenarios. It is worth noting that the accuracy of this method still needs further improvement.

[0003] In natural human-computer interaction environments, changes in camera perspective and hand spatial posture are the main reasons for the low accuracy and poor robustness of sign language recognition systems. Existing technologies are insufficient in handling errors caused by camera perspective and hand spatial posture, resulting in room for improvement in recognition accuracy. Summary of the Invention

[0004] To address the aforementioned technical problems, a sign language recognition method for natural human-computer interaction is provided. This technical solution solves the problems mentioned in the background section.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A sign language recognition method for natural human-computer interaction includes:

[0007] Obtain at least one standard gesture used in sign language expression and the corresponding sign language meaning of the standard gesture;

[0008] At least one person is assigned to collect samples. The person collecting samples demonstrates hand gestures according to standard hand gestures. The demonstrated gestures are used as the corresponding gestures of the standard hand gestures.

[0009] Rotate similar gestures to obtain at least one associated gesture, and match the associated gesture to the standard gesture corresponding to the similar gesture.

[0010] Obtain at least one real-time gesture of human-computer interaction, deconstruct the real-time gesture into at least one real-time key edge and real-time node, and deconstruct the associated gesture into at least one sample key edge and sample node;

[0011] Based on real-time key edges and real-time nodes, a real-time matrix of real-time gestures is formed, and based on sample key edges and sample nodes, a sample matrix of associated gestures is formed.

[0012] The real-time matrix is ​​transformed to obtain the real-time correction matrix;

[0013] Form at least one difference feature between the real-time correction matrix and the sample matrix;

[0014] Fault tolerance analysis is performed on the differential features to obtain the target sample matrix. The sign language meaning of the standard gesture corresponding to the associated gesture of the target sample matrix is ​​used as the real-time semantics of the real-time gesture.

[0015] The real-time semantics of at least one real-time gesture are concatenated to form continuous sign language content.

[0016] Preferably, the process of forming at least one sample collection person includes the following steps:

[0017] Obtain the first range of values ​​for the palm size of the actual population, and obtain the second range of values ​​for the degree of palm obesity of the actual population. The degree of palm obesity is the ratio of palm thickness to palm length.

[0018] Divide the first value range into equal intervals to obtain at least one first test point; divide the second value range into equal intervals to obtain at least one second test point.

[0019] Randomly combine the first test point and the second test point to obtain at least one test group;

[0020] Individuals whose palm size is equal to the value of the first test point in the test group and whose palm fatness is equal to the value of the second test point in the test group will be used as the sample collection personnel.

[0021] Preferably, the step of rotating similar gestures to obtain at least one associated gesture includes the following steps:

[0022] At least one recognition point is uniformly selected within the range of 0 to 360 degrees. The gesture obtained by rotating the same type of gesture according to the value of the recognition point is used as the associated gesture. The rotation axis is the line connecting the center of gravity of the same type of gesture and the bottom midpoint of the same type of gesture.

[0023] Preferably, the step of deconstructing real-time gestures into at least one real-time key edge and real-time node, and deconstructing associated gestures into at least one sample key edge and sample node, includes the following steps:

[0024] A reference hand is acquired in advance, and the position of the joint in the reference hand is determined as the reference position. At least one reference texture line is identified at the reference position.

[0025] The maximum distance between the midpoints of two reference lines at the same reference position is taken as the first distance, and the minimum distance between the two reference positions is taken as the second distance.

[0026] The first distance is divided by the length of the reference hand to obtain the first ratio, and the second distance is divided by the length of the reference hand to obtain the second ratio. The average of the first ratio and the second ratio is taken as the critical ratio.

[0027] Using pixel gradient calculation, texture recognition is performed on real-time gestures to obtain at least one real-time texture line.

[0028] The real-time ratio of the two real-time texture lines is obtained by dividing the distance between the midpoints of the two real-time texture lines by the length of the real-time gesture.

[0029] When the real-time ratio is less than the critical ratio, the two real-time texture lines are judged to be of the same type; otherwise, no processing is performed.

[0030] The location of the real-time texture line of the same type is taken as the real-time node. The real-time node divides the real-time gesture into at least one real-time part. When the number of real-time nodes connected to the real-time part is one or two, the real-time part is taken as the real-time key edge; otherwise, the real-time part is taken as the real-time palm.

[0031] Using pixel gradient calculation, texture recognition is performed on associated gestures to obtain at least one sample texture line.

[0032] The distance between the midpoints of two sample pattern lines divided by the length of the associated gesture gives the sample ratio of the two sample pattern lines.

[0033] When the sample ratio is less than the critical ratio, the two sample patterns are judged to be of the same type; otherwise, no processing is performed.

[0034] The location of the sample pattern route of the same type is taken as the sample node. The sample node divides the associated gesture into at least one sample part. When the number of sample nodes connected to the sample part is one or two, the sample part is taken as the sample key edge; otherwise, the sample part is taken as the sample palm.

[0035] Preferably, the step of forming a real-time matrix of real-time gestures based on real-time key edges and real-time nodes includes the following steps:

[0036] At least one real-time node connected to the real-time hand is designated as the target real-time node, and at least one real-time node not connected to the real-time hand is designated as a non-target real-time node.

[0037] The real-time gesture is segmented at the target real-time node to obtain a first real-time local area and a second real-time local area, wherein the area of ​​the first real-time local area is smaller than the area of ​​the second real-time local area.

[0038] The non-target real-time nodes located in the first real-time local area are sorted in ascending order of their distance from the corresponding target real-time node to form a real-time feature sequence of the target real-time node.

[0039] According to the arrangement order of the target real-time nodes on the real-time palm, the target real-time nodes are arranged horizontally, and the real-time feature sequences are arranged directly below the corresponding target real-time nodes. All target real-time nodes and non-target real-time nodes form a real-time arrangement array.

[0040] The angle between the real-time key edge at the target real-time node and the real-time hand is used as the real-time parameter of the target real-time node, and the angle between the two real-time key edges at the non-target real-time node is used as the real-time parameter of the non-target real-time node.

[0041] Fill in the real-time parameters according to the arrangement of the real-time array, and fill in the missing parts with 0 to obtain the real-time matrix.

[0042] Preferably, forming the sample matrix of associated gestures based on sample key edges and sample nodes includes the following steps:

[0043] At least one sample node connected to the sample hand is designated as the target sample node, and at least one sample node not connected to the sample hand is designated as a non-target sample node.

[0044] The associated gesture is segmented at the target sample node to obtain a first sample local area and a second sample local area, wherein the area of ​​the first sample local area is smaller than the area of ​​the second sample local area.

[0045] The non-target sample nodes located in the local area of ​​the first sample are sorted in ascending order of their distance from the corresponding target sample node to form a sample feature sequence of the target sample nodes;

[0046] According to the arrangement order of the target sample nodes on the sample palm, the target sample nodes are arranged horizontally, and the sample feature sequences are arranged directly below the corresponding target sample nodes. All target sample nodes and non-target sample nodes form a sample arrangement array.

[0047] The angle between the key edge of the sample at the target sample node and the sample hand is used as the sample parameter of the target sample node, and the angle between the two key edges of the sample at the non-target sample node is used as the sample parameter of the non-target sample node.

[0048] Fill in the sample parameters according to the arrangement of the sample array, and fill in the missing parts with 0 to obtain the sample matrix.

[0049] Preferably, the transformation of the real-time matrix to obtain the real-time corrected matrix includes the following steps:

[0050] During human-computer interaction, instructions to make the target gesture are issued, and the actual gestures of the person performing the human-computer interaction are obtained.

[0051] The actual gestures are used as real-time gestures to obtain a real-time matrix of actual gestures, and the target gestures are used as associated gestures to obtain a sample matrix of target gestures.

[0052] The perturbation matrix is ​​obtained by subtracting the sample matrix of the target gesture from the real-time matrix of the actual gesture.

[0053] The real-time matrix is ​​superimposed with the perturbation matrix to obtain the real-time correction matrix.

[0054] Preferably, the formation of at least one difference feature between the real-time correction matrix and the sample matrix includes the following steps:

[0055] The difference matrix is ​​obtained by subtracting the sample matrix from the real-time correction matrix. The non-zero elements in the difference matrix are used as difference features.

[0056] Preferably, the step of performing fault-tolerant analysis on the differential features and filtering to obtain the target sample matrix includes the following steps:

[0057] A fault-tolerant analysis model is formed by combining the GAT model and the GCN model as templates.

[0058] Input the difference matrix into the fault tolerance analysis model to obtain the difference coefficients of the difference matrix. The sample matrix corresponding to the difference matrix with the smallest difference coefficient is taken as the target sample matrix.

[0059] Preferably, the process of combining the GAT model and the GCN model as templates to form a fault-tolerant analysis model includes the following steps:

[0060] Obtain the upper limit of computing power of the devices used for human-computer interaction, and form at least one permutation model. The permutation model is formed by combining the GAT model and the GCN model. The total computing power used by the GAT model and the GCN model in the permutation model is less than the upper limit of computing power.

[0061] Input the perturbation matrix into the GAT model to obtain the first matrix, and input the perturbation matrix into the GCN model to obtain the second matrix;

[0062] The first weight matrix is ​​obtained by subtracting the first matrix from the perturbation matrix, and the second weight matrix is ​​obtained by subtracting the second matrix from the perturbation matrix.

[0063] The difference effect matrix of the permutation model satisfies Among them, A ij B is the element in the i-th row and j-th column of the difference effect matrix. ij Let C be the element in the i-th row and j-th column of the perturbation matrix. ij D is the element in the i-th row and j-th column of the first weight matrix. ijis the element in the i-th row and j-th column of the second weight matrix, n is the number of times the GAT model appears in the permutation model, and m is the number of times the GCN model appears in the permutation model;

[0064] The absolute values ​​of the elements in the difference effect matrix are summed to obtain the difference degree of the permutation model. The permutation model with the smallest difference degree is used as the preliminary model.

[0065] Input the perturbation matrix into the preliminary model to obtain the preliminary matrix. Then, sum the absolute values ​​of the elements in the preliminary matrix to obtain the preliminary value.

[0066] The preliminary model with the smallest preliminary value is selected as the target model.

[0067] Construct a fault-tolerant analysis model using the target model: Input the difference matrix into the target model to obtain the difference matrix, and sum the absolute values ​​of the elements in the difference matrix to obtain the difference coefficients.

[0068] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0069] By obtaining associated gestures, forming a real-time matrix of real-time gestures, forming a sample matrix of associated gestures, forming differential features, and performing fault-tolerant analysis on the differential features, associated gestures of standard gestures from multiple perspectives are pre-constructed during recognition. This allows for a certain degree of correction of errors caused by changes in camera perspective and hand spatial posture during recognition. Simultaneously, fault-tolerant analysis is used during recognition, and different combinations are made based on the different emphases of the GAT model and GCN model. Based on the combination results, the model that best meets the usage requirements is selected, enabling the used model to further eliminate recognition errors and ensure recognition accuracy. Attached Figure Description

[0070] Figure 1 This is a flowchart illustrating the sign language recognition method for natural human-computer interaction according to the present invention.

[0071] Figure 2 This is a schematic diagram illustrating the process of forming at least one sample collection personnel according to the present invention;

[0072] Figure 3 This is a schematic diagram of the process of deconstructing real-time gestures into at least one real-time key edge and real-time node, and deconstructing associated gestures into at least one sample key edge and sample node according to the present invention.

[0073] Figure 4 This is a flowchart illustrating the process of forming a real-time matrix of real-time gestures based on real-time key edges and real-time nodes according to the present invention.

[0074] Figure 5 This is a schematic diagram of the process of forming a sample matrix of associated gestures based on sample key edges and sample nodes according to the present invention.

[0075] Figure 6 This is a schematic diagram of the process of transforming a real-time matrix to obtain a real-time corrected matrix according to the present invention.

[0076] Figure 7 This is a schematic diagram of the process of performing fault-tolerant analysis on the differential features and filtering to obtain the target sample matrix according to the present invention.

[0077] Figure 8 This is a schematic diagram of the process of forming a fault-tolerant analysis model by combining the GAT model and the GCN model as templates according to the present invention. Detailed Implementation

[0078] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0079] Reference Figure 1 As shown, a sign language recognition method for natural human-computer interaction includes:

[0080] Obtain at least one standard gesture used in sign language expression and the corresponding sign language meaning of the standard gesture;

[0081] At least one person is assigned to collect samples. The person collecting samples demonstrates hand gestures according to standard hand gestures. The demonstrated gestures are used as the corresponding gestures of the standard hand gestures.

[0082] Rotate similar gestures to obtain at least one associated gesture, and match the associated gesture to the standard gesture corresponding to the similar gesture.

[0083] Obtain at least one real-time gesture of human-computer interaction, deconstruct the real-time gesture into at least one real-time key edge and real-time node, and deconstruct the associated gesture into at least one sample key edge and sample node;

[0084] Based on real-time key edges and real-time nodes, a real-time matrix of real-time gestures is formed, and based on sample key edges and sample nodes, a sample matrix of associated gestures is formed.

[0085] The real-time matrix is ​​transformed to obtain the real-time correction matrix;

[0086] Form at least one difference feature between the real-time correction matrix and the sample matrix;

[0087] Fault tolerance analysis is performed on the differential features to obtain the target sample matrix. The sign language meaning of the standard gesture corresponding to the associated gesture of the target sample matrix is ​​used as the real-time semantics of the real-time gesture.

[0088] The real-time semantics of at least one real-time gesture are concatenated to form continuous sign language content.

[0089] This solution primarily aims to improve the accuracy of sign language recognition. During sign language recognition, the varying proportions of hand size among individuals can affect the accuracy. Furthermore, even with the same gesture, individual habits can lead to subtle differences between the displayed gesture and the standard gesture. Additionally, the variable hand position during interaction results in different perspectives in the acquired images, further contributing to recognition errors. Therefore, a series of steps are implemented to address these issues.

[0090] Reference Figure 2 As shown, forming at least one sample collection person includes the following steps:

[0091] Obtain the first range of values ​​for the palm size of the actual population, and obtain the second range of values ​​for the degree of palm obesity of the actual population. The degree of palm obesity is the ratio of palm thickness to palm length.

[0092] Divide the first value range into equal intervals to obtain at least one first test point; divide the second value range into equal intervals to obtain at least one second test point.

[0093] Randomly combine the first test point and the second test point to obtain at least one test group;

[0094] Individuals whose palm size is equal to the value of the first test point in the test group and whose palm fatness is equal to the value of the second test point in the test group will be used as the sample collection personnel.

[0095] Because people's hands vary in size, there will be slight differences when they perform the same gesture. To avoid the influence of these differences, a sample of personnel is created. The hands of all the sample personnel can approximate the hand conditions of almost all people. Therefore, the same gestures obtained using the sample personnel can represent the gestures formed by all people.

[0096] Rotating similar gestures to obtain at least one associated gesture involves the following steps:

[0097] At least one recognition point is uniformly selected within the range of 0 to 360 degrees. The gesture obtained by rotating the same type of gesture according to the value of the recognition point is used as the associated gesture. The rotation axis is the line connecting the center of gravity of the same type of gesture and the bottom midpoint of the same type of gesture.

[0098] Due to the perspective, each similar gesture will appear differently when the image is captured from different directions. Therefore, it is necessary to rotate similar gestures to obtain at least one associated gesture. Then, based on the recognition results of the associated gesture, the corresponding similar gesture can be found. Since similar gestures correspond to standard gestures, the meaning of sign language can be determined.

[0099] Reference Figure 3 As shown, deconstructing real-time gestures into at least one real-time key edge and a real-time node, and deconstructing associated gestures into at least one sample key edge and a sample node, includes the following steps:

[0100] A reference hand is acquired in advance, and the position of the joint in the reference hand is determined as the reference position. At least one reference texture line is identified at the reference position.

[0101] The maximum distance between the midpoints of two reference lines at the same reference position is taken as the first distance, and the minimum distance between the two reference positions is taken as the second distance.

[0102] The first distance is divided by the length of the reference hand to obtain the first ratio, and the second distance is divided by the length of the reference hand to obtain the second ratio. The average of the first ratio and the second ratio is taken as the critical ratio.

[0103] Using pixel gradient calculation, texture recognition is performed on real-time gestures to obtain at least one real-time texture line.

[0104] The real-time ratio of the two real-time texture lines is obtained by dividing the distance between the midpoints of the two real-time texture lines by the length of the real-time gesture.

[0105] When the real-time ratio is less than the critical ratio, the two real-time texture lines are judged to be of the same type; otherwise, no processing is performed.

[0106] The location of the real-time texture line of the same type is taken as the real-time node. The real-time node divides the real-time gesture into at least one real-time part. When the number of real-time nodes connected to the real-time part is one or two, the real-time part is taken as the real-time key edge; otherwise, the real-time part is taken as the real-time palm.

[0107] Using pixel gradient calculation, texture recognition is performed on associated gestures to obtain at least one sample texture line.

[0108] The distance between the midpoints of two sample pattern lines divided by the length of the associated gesture gives the sample ratio of the two sample pattern lines.

[0109] When the sample ratio is less than the critical ratio, the two sample patterns are judged to be of the same type; otherwise, no processing is performed.

[0110] The location of the sample pattern route of the same type is taken as the sample node. The sample node divides the associated gesture into at least one sample part. When the number of sample nodes connected to the sample part is one or two, the sample part is taken as the sample key edge; otherwise, the sample part is taken as the sample palm.

[0111] The deconstruction process of real-time gestures and associated gestures is consistent. Only real-time gestures are analyzed. Real-time gestures mainly depend on the bending of the fingers. Changes in the palm can be ignored. Changes in the fingers are characterized by the bending of the finger joints. When the bending angle at each joint is determined, the gesture is determined. This is because there is a joint connected to the palm. Since the bending angle at this joint is determined, the bending angle at the joint on the finger where that joint is located is also determined. Therefore, the shape of the finger is determined.

[0112] At this point, only the joints need to be identified. It is easy to see that there are uniform and obvious lines at the joints of the hand. Therefore, the location where the lines gather is the location of the joint. When the back of the hand faces the camera, the lines on the back of the hand are used for joint recognition. When the front of the hand faces the camera, the lines on the front of the hand are used for joint recognition. Since there are multiple lines during recognition, they need to be clustered. Since the spacing between lines at different joints is much greater than the spacing between lines at the same joint, the lines can be clustered based on this. However, since different hands are different sizes, distance cannot be used for judgment. But the proportions of different hands are similar, so the ratio of spacing to hand size is used for judgment, and then the joints can be identified, namely real-time nodes and sample nodes.

[0113] It is easy to know that the palm is connected to five real-time nodes, while the real-time key edge is the part of the finger that is divided by the joints. Therefore, it is connected to two real-time nodes or to one real-time node, and the tip of the finger is connected to one real-time node.

[0114] In the texture of the hand, the pixel values ​​of pixels change significantly, resulting in large changes in their gradients. This gradient can be used for texture recognition.

[0115] Reference Figure 4 As shown, forming a real-time matrix for real-time gestures based on real-time key edges and real-time nodes includes the following steps:

[0116] At least one real-time node connected to the real-time hand is designated as the target real-time node, and at least one real-time node not connected to the real-time hand is designated as a non-target real-time node.

[0117] The real-time gesture is segmented at the target real-time node to obtain a first real-time local area and a second real-time local area, wherein the area of ​​the first real-time local area is smaller than the area of ​​the second real-time local area.

[0118] The non-target real-time nodes located in the first real-time local area are sorted in ascending order of their distance from the corresponding target real-time node to form a real-time feature sequence of the target real-time node.

[0119] According to the arrangement order of the target real-time nodes on the real-time palm, the target real-time nodes are arranged horizontally, and the real-time feature sequences are arranged directly below the corresponding target real-time nodes. All target real-time nodes and non-target real-time nodes form a real-time arrangement array.

[0120] The angle between the real-time key edge at the target real-time node and the real-time hand is used as the real-time parameter of the target real-time node, and the angle between the two real-time key edges at the non-target real-time node is used as the real-time parameter of the non-target real-time node.

[0121] Fill in the real-time parameters according to the arrangement of the real-time array, and fill in the missing parts with 0 to obtain the real-time matrix.

[0122] The formation process of the sample matrix and the real-time matrix is ​​similar. Only the real-time matrix is ​​analyzed. According to the distribution of real-time nodes on the real-time gesture, the real-time matrix is ​​generated. First, the first row of the real-time matrix is ​​determined. That is, according to the arrangement order of the target real-time nodes on the real-time palm, the angle between the real-time key edge of the target real-time node and the real-time palm is filled in sequentially. Each column of the real-time matrix is ​​then filled in sequentially with the angle between the two real-time key edges of the non-target real-time nodes of the fingers where the target real-time node is located. Thus, the real-time matrix can be obtained. The real-time matrix is ​​a 3*5 matrix. However, since the thumb only has two joints, there will be gaps in this column of the real-time matrix. The gaps are filled with 0.

[0123] Reference Figure 5 As shown, forming a sample matrix of associated gestures based on sample key edges and sample nodes includes the following steps:

[0124] At least one sample node connected to the sample hand is designated as the target sample node, and at least one sample node not connected to the sample hand is designated as a non-target sample node.

[0125] The associated gesture is segmented at the target sample node to obtain a first sample local area and a second sample local area, wherein the area of ​​the first sample local area is smaller than the area of ​​the second sample local area.

[0126] The non-target sample nodes located in the local area of ​​the first sample are sorted in ascending order of their distance from the corresponding target sample node to form a sample feature sequence of the target sample nodes;

[0127] According to the arrangement order of the target sample nodes on the sample palm, the target sample nodes are arranged horizontally, and the sample feature sequences are arranged directly below the corresponding target sample nodes. All target sample nodes and non-target sample nodes form a sample arrangement array.

[0128] The angle between the key edge of the sample at the target sample node and the sample hand is used as the sample parameter of the target sample node, and the angle between the two key edges of the sample at the non-target sample node is used as the sample parameter of the non-target sample node.

[0129] Fill in the sample parameters according to the arrangement of the sample array, and fill in the missing parts with 0 to obtain the sample matrix.

[0130] Reference Figure 6 As shown, transforming the real-time matrix to obtain the real-time corrected matrix includes the following steps:

[0131] During human-computer interaction, instructions to make the target gesture are issued, and the actual gestures of the person performing the human-computer interaction are obtained.

[0132] The actual gestures are used as real-time gestures to obtain a real-time matrix of actual gestures, and the target gestures are used as associated gestures to obtain a sample matrix of target gestures.

[0133] The perturbation matrix is ​​obtained by subtracting the sample matrix of the target gesture from the real-time matrix of the actual gesture.

[0134] The real-time matrix is ​​superimposed with the perturbation matrix to obtain the real-time correction matrix.

[0135] Due to the various reasons analyzed above, the gestures of the person themselves will differ from the standard gestures. This difference will affect the recognition. Therefore, it is necessary to eliminate the difference. The perturbation matrix is ​​the representation matrix of this difference. Therefore, all real-time matrices will have a certain gap with the actual gestures. By superimposing the real-time matrix with the perturbation matrix, this gap can be reduced. Here, the real-time matrix superimposed with the perturbation matrix refers to the real-time matrix of the actual gesture and all other real-time matrices.

[0136] Forming at least one difference feature between the real-time correction matrix and the sample matrix includes the following steps:

[0137] The difference matrix is ​​obtained by subtracting the sample matrix from the real-time correction matrix. The non-zero elements in the difference matrix are used as difference features.

[0138] Reference Figure 7 As shown, the steps for performing fault tolerance analysis on the differential features and selecting the target sample matrix include:

[0139] A fault-tolerant analysis model is formed by combining the GAT model and the GCN model as templates.

[0140] Input the difference matrix into the fault tolerance analysis model to obtain the difference coefficients of the difference matrix. The sample matrix corresponding to the difference matrix with the smallest difference coefficient is taken as the target sample matrix.

[0141] Reference Figure 8 As shown, the process of forming a fault-tolerant analysis model by combining the GAT model and the GCN model as templates includes the following steps:

[0142] Obtain the upper limit of computing power of the devices used for human-computer interaction, and form at least one permutation model. The permutation model is formed by combining the GAT model and the GCN model. The total computing power used by the GAT model and the GCN model in the permutation model is less than the upper limit of computing power.

[0143] Input the perturbation matrix into the GAT model to obtain the first matrix, and input the perturbation matrix into the GCN model to obtain the second matrix;

[0144] The first weight matrix is ​​obtained by subtracting the first matrix from the perturbation matrix, and the second weight matrix is ​​obtained by subtracting the second matrix from the perturbation matrix.

[0145] The difference effect matrix of the permutation model satisfies Among them, A ij B is the element in the i-th row and j-th column of the difference effect matrix. ij Let C be the element in the i-th row and j-th column of the perturbation matrix. ij D is the element in the i-th row and j-th column of the first weight matrix. ij is the element in the i-th row and j-th column of the second weight matrix, n is the number of times the GAT model appears in the permutation model, and m is the number of times the GCN model appears in the permutation model;

[0146] The absolute values ​​of the elements in the difference effect matrix are summed to obtain the difference degree of the permutation model. The permutation model with the smallest difference degree is used as the preliminary model.

[0147] Input the perturbation matrix into the preliminary model to obtain the preliminary matrix. Then, sum the absolute values ​​of the elements in the preliminary matrix to obtain the preliminary value.

[0148] The preliminary model with the smallest preliminary value is selected as the target model.

[0149] Construct a fault-tolerant analysis model using the target model: Input the difference matrix into the target model to obtain the difference matrix, and sum the absolute values ​​of the elements in the difference matrix to obtain the difference coefficients.

[0150] During the analysis, the model used needs to have the smallest possible difference from the sample matrix of associated gestures, indicating a better effect on error elimination. Therefore, using the results generated for screening yields more accurate recognition results. When constructing the fault-tolerant analysis model, a combination of the GAT and GCN models is used. Within the limits of computing power, multiple combinations of the GAT and GCN models are performed, with each model appearing at least once. The effect varies depending on the arrangement. By estimating the error reduction effect of each combination, the combination with the smallest error is selected as the final model. However, based on the difference calculation process, multiple preliminary models exist because, when the number of n and m is determined, the difference in the arrangement models resulting from different arrangements of the GAT and GCN models is consistent. Due to their small number, they can be screened based on their actual error effects to obtain the target model and construct the fault-tolerant analysis model.

[0151] Furthermore, this solution also proposes a storage medium on which a computer-readable program is stored, which, when invoked, executes the aforementioned sign language recognition method for natural human-computer interaction.

[0152] It is understandable that the storage medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a DVD; or a semiconductor medium, such as a solid-state drive (SSD).

[0153] In summary, the advantages of this invention are as follows: by obtaining associated gestures, forming a real-time matrix of real-time gestures, forming a sample matrix of associated gestures, forming differential features, and performing fault-tolerant analysis on the differential features, associated gestures of standard gestures from multiple perspectives are pre-constructed during recognition. This allows for a certain degree of correction of errors caused by changes in camera perspective and hand spatial posture during recognition. Furthermore, fault-tolerant analysis is used during recognition, with different combinations of GAT and GCN models based on their respective emphases. Based on the combination results, a model that best meets the usage requirements is selected, further eliminating recognition errors and ensuring recognition accuracy.

[0154] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A sign language recognition method for natural human-computer interaction, characterized in that, include: Obtain at least one standard gesture used in sign language expression and the corresponding sign language meaning of the standard gesture; At least one person is assigned to collect samples. The person collecting samples demonstrates hand gestures according to standard hand gestures. The demonstrated gestures are used as the corresponding gestures of the standard hand gestures. Rotate similar gestures to obtain at least one associated gesture, and match the associated gesture to the standard gesture corresponding to the similar gesture. Obtain at least one real-time gesture of human-computer interaction, deconstruct the real-time gesture into at least one real-time key edge and real-time node, and deconstruct the associated gesture into at least one sample key edge and sample node; Based on real-time key edges and real-time nodes, a real-time matrix of real-time gestures is formed, and based on sample key edges and sample nodes, a sample matrix of associated gestures is formed. The real-time matrix is ​​transformed to obtain the real-time correction matrix; Form at least one difference feature between the real-time correction matrix and the sample matrix; Fault tolerance analysis is performed on the differential features to obtain the target sample matrix. The sign language meaning of the standard gesture corresponding to the associated gesture of the target sample matrix is ​​used as the real-time semantics of the real-time gesture. The real-time semantics of at least one real-time gesture are concatenated to form continuous sign language content; The process of rotating similar gestures to obtain at least one associated gesture includes the following steps: At least one recognition point is uniformly selected in the range of 0 to 360 degrees. The gesture obtained by rotating the same type of gesture according to the value of the recognition point is used as the associated gesture. The rotation axis is the line connecting the center of gravity of the same type of gesture and the bottom midpoint of the same type of gesture. The step of deconstructing real-time gestures into at least one real-time key edge and real-time node, and deconstructing associated gestures into at least one sample key edge and sample node, includes the following steps: A reference hand is acquired in advance, and the position of the joint in the reference hand is determined as the reference position. At least one reference texture line is identified at the reference position. The maximum distance between the midpoints of two reference lines at the same reference position is taken as the first distance, and the minimum distance between the two reference positions is taken as the second distance. The first distance is divided by the length of the reference hand to obtain the first ratio, and the second distance is divided by the length of the reference hand to obtain the second ratio. The average of the first ratio and the second ratio is taken as the critical ratio. Using pixel gradient calculation, texture recognition is performed on real-time gestures to obtain at least one real-time texture line. The real-time ratio of the two real-time texture lines is obtained by dividing the distance between the midpoints of the two real-time texture lines by the length of the real-time gesture. When the real-time ratio is less than the critical ratio, the two real-time texture lines are judged to be of the same type; otherwise, no processing is performed. The location of the real-time texture line of the same type is taken as the real-time node. The real-time node divides the real-time gesture into at least one real-time part. When the number of real-time nodes connected to the real-time part is one or two, the real-time part is taken as the real-time key edge; otherwise, the real-time part is taken as the real-time palm. Using pixel gradient calculation, texture recognition is performed on associated gestures to obtain at least one sample texture line. The distance between the midpoints of two sample pattern lines divided by the length of the associated gesture gives the sample ratio of the two sample pattern lines. When the sample ratio is less than the critical ratio, the two sample patterns are judged to be of the same type; otherwise, no processing is performed. The location of the sample pattern route of the same type is taken as the sample node. The sample node divides the associated gesture into at least one sample part. When the number of sample nodes connected to the sample part is one or two, the sample part is taken as the sample key edge; otherwise, the sample part is taken as the sample palm.

2. The sign language recognition method for natural human-computer interaction according to claim 1, characterized in that, The process of forming at least one sample collection personnel includes the following steps: Obtain the first range of values ​​for the palm size of the actual population, and obtain the second range of values ​​for the degree of palm obesity of the actual population. The degree of palm obesity is the ratio of palm thickness to palm length. Divide the first value range into equal intervals to obtain at least one first test point; divide the second value range into equal intervals to obtain at least one second test point. Randomly combine the first test point and the second test point to obtain at least one test group; Individuals whose palm size is equal to the value of the first test point in the test group and whose palm fatness is equal to the value of the second test point in the test group will be used as the sample collection personnel.

3. The sign language recognition method for natural human-computer interaction according to claim 2, characterized in that, The process of forming a real-time matrix for real-time gestures based on real-time key edges and real-time nodes includes the following steps: At least one real-time node connected to the real-time hand is designated as the target real-time node, and at least one real-time node not connected to the real-time hand is designated as a non-target real-time node. The real-time gesture is segmented at the target real-time node to obtain a first real-time local area and a second real-time local area, wherein the area of ​​the first real-time local area is smaller than the area of ​​the second real-time local area. The non-target real-time nodes located in the first real-time local area are sorted in ascending order of their distance from the corresponding target real-time node to form a real-time feature sequence of the target real-time node. According to the arrangement order of the target real-time nodes on the real-time palm, the target real-time nodes are arranged horizontally, and the real-time feature sequences are arranged directly below the corresponding target real-time nodes. All target real-time nodes and non-target real-time nodes form a real-time arrangement array. The angle between the real-time key edge at the target real-time node and the real-time hand is used as the real-time parameter of the target real-time node, and the angle between the two real-time key edges at the non-target real-time node is used as the real-time parameter of the non-target real-time node. Fill in the real-time parameters according to the arrangement of the real-time array, and fill in the missing parts with 0 to obtain the real-time matrix.

4. The sign language recognition method for natural human-computer interaction according to claim 3, characterized in that, The process of forming a sample matrix of associated gestures based on sample key edges and sample nodes includes the following steps: At least one sample node connected to the sample hand is designated as the target sample node, and at least one sample node not connected to the sample hand is designated as a non-target sample node. The associated gesture is segmented at the target sample node to obtain a first sample local area and a second sample local area, wherein the area of ​​the first sample local area is smaller than the area of ​​the second sample local area. The non-target sample nodes located in the local area of ​​the first sample are sorted in ascending order of their distance from the corresponding target sample node to form a sample feature sequence of the target sample nodes; According to the arrangement order of the target sample nodes on the sample palm, the target sample nodes are arranged horizontally, and the sample feature sequences are arranged directly below the corresponding target sample nodes. All target sample nodes and non-target sample nodes form a sample arrangement array. The angle between the key edge of the sample at the target sample node and the sample palm is used as the sample parameter of the target sample node, and the angle between the two key edges of the sample at the non-target sample node is used as the sample parameter of the non-target sample node. Fill in the sample parameters according to the arrangement of the sample array, and fill in the missing parts with 0 to obtain the sample matrix.

5. The sign language recognition method for natural human-computer interaction according to claim 4, characterized in that, The transformation of the real-time matrix to obtain the real-time corrected matrix includes the following steps: During human-computer interaction, instructions to make the target gesture are issued, and the actual gestures of the person performing the human-computer interaction are obtained. The actual gestures are used as real-time gestures to obtain a real-time matrix of actual gestures, and the target gestures are used as associated gestures to obtain a sample matrix of target gestures. The perturbation matrix is ​​obtained by subtracting the sample matrix of the target gesture from the real-time matrix of the actual gesture. The real-time matrix is ​​superimposed with the perturbation matrix to obtain the real-time correction matrix.

6. The sign language recognition method for natural human-computer interaction according to claim 5, characterized in that, The formation of at least one difference feature between the real-time correction matrix and the sample matrix includes the following steps: The difference matrix is ​​obtained by subtracting the sample matrix from the real-time correction matrix. The non-zero elements in the difference matrix are used as difference features.

7. The sign language recognition method for natural human-computer interaction according to claim 6, characterized in that, The step of performing fault-tolerant analysis on the differential features and selecting the target sample matrix includes the following steps: A fault-tolerant analysis model is formed by combining the GAT model and the GCN model as templates. Input the difference matrix into the fault tolerance analysis model to obtain the difference coefficients of the difference matrix. The sample matrix corresponding to the difference matrix with the smallest difference coefficient is taken as the target sample matrix.

8. The sign language recognition method for natural human-computer interaction according to claim 7, characterized in that, The process of combining the GAT model and the GCN model as templates to form a fault-tolerant analysis model includes the following steps: Obtain the upper limit of computing power of the devices used for human-computer interaction, and form at least one permutation model. The permutation model is formed by combining the GAT model and the GCN model. The total computing power used by the GAT model and the GCN model in the permutation model is less than the upper limit of computing power. Input the perturbation matrix into the GAT model to obtain the first matrix, and input the perturbation matrix into the GCN model to obtain the second matrix; The first weight matrix is ​​obtained by subtracting the first matrix from the perturbation matrix, and the second weight matrix is ​​obtained by subtracting the second matrix from the perturbation matrix. The difference effect matrix of the permutation model satisfies ,in, Let be the element in the i-th row and j-th column of the difference effect matrix. Let be the element in the i-th row and j-th column of the perturbation matrix. Let be the element in the i-th row and j-th column of the first weight matrix. is the element in the i-th row and j-th column of the second weight matrix, n is the number of times the GAT model appears in the permutation model, and m is the number of times the GCN model appears in the permutation model; The absolute values ​​of the elements in the difference effect matrix are summed to obtain the difference degree of the permutation model. The permutation model with the smallest difference degree is used as the preliminary model. Input the perturbation matrix into the preliminary model to obtain the preliminary matrix. Then, sum the absolute values ​​of the elements in the preliminary matrix to obtain the preliminary value. The preliminary model with the smallest preliminary value is selected as the target model. Construct a fault-tolerant analysis model using the target model: Input the difference matrix into the target model to obtain the difference matrix, and sum the absolute values ​​of the elements in the difference matrix to obtain the difference coefficients.

Images