Method, device, equipment and storage medium for recognizing medical terms and sign language

Abstract

The present application provides a method, device, equipment and storage medium for recognizing medical terms and sign language, and relates to the field of data processing technology. The method constructs an initial recognition model, extracts the features of sign language data and the features of medical terms respectively, and fuses the features of medical terms and the features of sign language data to obtain fused features, so that the initial recognition model can obtain more comprehensive feature information based on the fused features. Furthermore, the target recognition model trained according to the initial recognition model can obtain more accurate recognition results, improve the robustness of the target recognition model, and improve the accuracy of the conversion between sign language and medical terms. The target recognition model can not only convert medical terms into sign language, but also convert the sign language of hearing-impaired people into medical terms, solve the communication problems between hearing-impaired people and medical staff, and improve the quality and efficiency of medical services.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for recognizing medical terms and sign language. Background Art

[0002] The traditional medical service model relies on spoken language. Doctors need patients to provide a relatively detailed description of their symptoms to assist in decision-making. Regardless of whether they are congenital or acquired hearing-impaired, patients generally do not have good oral expression skills, and some congenital patients even lack the learning of natural language and lack understanding of natural language. Most hearing-impaired patients usually need relatives or professional sign language care, and caregivers to provide two-way translation when they see a doctor. There are problems such as difficulty in describing the intermediate translation and large errors in the description. In addition, the medical terminology system is huge and complex, which is a huge challenge for hearing-impaired patients to understand. Therefore, hearing-impaired patients are usually limited in obtaining medical services due to the inability to go to the doctor alone, real-time and cost, and it is difficult for them to obtain appropriate medical services.

[0003] With the development of information technology, using information technology to solve the problem of real-time communication for the hearing impaired has become one of the focuses of social attention. Traditional technical means mainly use internationally developed sign language libraries and simple natural language model processing, focusing on the use of software and hardware but not deeply involving various medical-related specific scenarios, causing difficulties for both doctors and patients to understand, which cannot meet the complex, changeable and highly professional medical service scenarios. Summary of the invention

[0004] In view of the shortcomings of the related technologies mentioned above, the present application provides a method, device, equipment and storage medium for recognizing medical terms and sign language to solve the above technical problems.

[0005] The present application provides a method for recognizing medical terms and sign language, comprising:

[0006] Acquire a training data set, wherein the training data set includes a plurality of training samples, wherein the training samples include sample sign language data and sample medical terminology data, wherein the sample sign language data and the sample medical terminology data of each training sample have the same language expression meaning, and the sample sign language data is image data;

[0007] Constructing an initial recognition model, the initial recognition model includes a feature extraction layer, a feature fusion layer, and an output layer; the feature extraction layer is used to extract features from sample medical terminology data to obtain medical terminology features, and to extract features from sample sign language data to obtain sign language features; the feature fusion layer is used to fuse the medical terminology features and the sign language features to obtain fusion features; and the output layer is used to decode and output the fusion features to obtain recognition results;

[0008] Training the initial recognition model to obtain a target recognition model;

[0009] Acquiring data to be identified, where the data to be identified is medical terminology data or sign language data;

[0010] The data to be recognized is recognized based on the target recognition model to obtain a recognition result; when the data to be recognized is medical terminology data, the recognition result is a sign language phrase; when the data to be recognized is sign language data, the recognition result is a medical terminology phrase.

[0011] In one embodiment of the present application, feature extraction is performed on sample medical terminology data to obtain medical terminology features, including:

[0012] Performing word segmentation processing on the sample medical term data to obtain a word segmentation sequence;

[0013] Performing word vector mapping on the word segmentation sequence to obtain a word segmentation vector for each word in the word segmentation sequence;

[0014] Determine the position code of each word in the word sequence;

[0015] According to the word segmentation vector and position code of each word, the initial feature vector is obtained;

[0016] Based on the multi-head attention mechanism, determining the multi-head attention output of the initial feature vector;

[0017] Performing feature extraction on the multi-head attention output to obtain initial features;

[0018] The initial features are normalized and residually connected to obtain the medical term features.

[0019] In one embodiment of the present application, feature extraction is performed on sample sign language data to obtain sign language features, including:

[0020] Capturing time series information of the sign language data in both forward and reverse directions simultaneously to obtain time series features;

[0021] Extracting spatial features of the sign language data to obtain image spatial features;

[0022] After convolution processing is performed on the image spatial features, a weighted image feature is obtained through an activation function;

[0023] The time series features and the weighted image features are fused to obtain sign language features.

[0024] In one embodiment of the present application, the feature fusion layer performs feature fusion on the medical term feature and the sign language feature to obtain a fusion feature, including:

[0025] After weighting the medical term features, the term features to be processed are obtained through an activation function, and after weighting the sign language features, the sign language features to be processed are obtained through an activation function;

[0026] Fusing the term feature to be processed and the sign language feature to be processed to obtain an initial feature;

[0027] Performing secondary fusion on the initial features to obtain intermediate features;

[0028] The intermediate features are fused three times to obtain the fused features.

[0029] In one embodiment of the present application, the fusion feature is decoded and outputted by the output layer to obtain a recognition result, including:

[0030] Performing nonlinear transformation on the fused features to obtain nonlinear features;

[0031] The nonlinear feature is linearly transformed to obtain the recognition result, wherein the recognition result includes a predicted phrase and a probability distribution of each output word in the predicted phrase, wherein the predicted phrase includes the sign language phrase or the medical term phrase.

[0032] To achieve the above objectives and other related objectives, the present application provides a device for recognizing medical terms and sign language, including:

[0033] A training data acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of training samples, wherein the training samples include sample sign language data and sample medical terminology data, wherein the sample sign language data and the sample medical terminology data of each training sample have the same language expression meaning, and the sample sign language data is image data;

[0034] A model building module is used to build an initial recognition model, wherein the initial recognition model includes a feature extraction layer, a feature fusion layer, and an output layer; the feature extraction layer is used to perform feature extraction on sample medical terminology data to obtain medical terminology features, and perform feature extraction on sample sign language data to obtain sign language features; the feature fusion layer is used to perform feature fusion on the medical terminology features and the sign language features to obtain fusion features; and the output layer is used to decode and output the fusion features to obtain recognition results;

[0035] A model training module, used to train the initial recognition model to obtain a target recognition model;

[0036] An identification data acquisition module, used to acquire data to be identified, wherein the data to be identified is medical terminology data or sign language data;

[0037] A recognition result determination module is used to recognize the data to be recognized based on the target recognition model to obtain a recognition result; when the data to be recognized is medical terminology data, the recognition result is a sign language phrase; when the data to be recognized is sign language data, the recognition result is a medical terminology phrase.

[0038] In one embodiment of the present application, the model building module includes a medical term feature extraction module, and the medical term feature extraction module is used to:

[0039] Performing word segmentation processing on the sample medical term data to obtain a word segmentation sequence;

[0040] Performing word vector mapping on the word segmentation sequence to obtain a word segmentation vector for each word in the word segmentation sequence;

[0041] Determine the position code of each word in the word sequence;

[0042] According to the word segmentation vector and position code of each word segmentation, a medical term feature vector is obtained;

[0043] Determining the output of each attention head in the medical term feature vector based on a multi-head attention mechanism;

[0044] Concatenate the output of each attention head to get the multi-head attention output;

[0045] Performing feature extraction on the multi-head attention output to obtain initial features;

[0046] The initial features are normalized and residually connected to obtain the medical term features.

[0047] In one embodiment of the present application, the model building module further includes a sign language feature extraction module, and the sign language feature extraction module is used to:

[0048] Capturing time series information of the sign language data in both forward and reverse directions simultaneously to obtain time series features;

[0049] Extracting spatial features of the sign language data to obtain image spatial features;

[0050] After convolution processing is performed on the image spatial features, a weighted image feature is obtained through an activation function;

[0051] The time series features and the weighted image features are fused to obtain sign language features.

[0052] To achieve the above-mentioned purpose and other related purposes, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements one or more of the aforementioned methods for recognizing medical terms and sign language.

[0053] To achieve the above objectives and other related objectives, the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer executes one or more of the aforementioned methods for recognizing medical terms and sign language.

[0054] As described above, the present application provides a method, device, equipment and storage medium for recognizing medical terms and sign language, which has the following beneficial effects:

[0055] A method for recognizing medical terms and sign language in the present application, the method constructs an initial recognition model, extracts the features of sign language data and the features of medical terms respectively, and fuses the features of medical terms and the features of sign language data to obtain fused features, so that the initial recognition model can obtain more comprehensive feature information based on the fused features. Furthermore, the target recognition model trained according to the initial recognition model can obtain more accurate recognition results, improve the robustness of the target recognition model, and improve the accuracy of the conversion between sign language and medical terms. The target recognition model can not only convert medical terms into sign language, but also convert the sign language of hearing-impaired people into medical terms, solve the communication problems between hearing-impaired people and medical staff, and improve the quality and efficiency of medical services.

[0056] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0058] Figure 1 is a flow chart of a method for recognizing medical terms and sign language according to an exemplary embodiment of the present application;

[0059] Figure 2 It is a structural block diagram of a device for recognizing medical terms and sign language shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0061] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0062] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0063] See also Figure 1 , Figure 1 FIG. 1 is a flowchart of a method for recognizing medical terms and sign language according to an exemplary embodiment of the present application. Figure 1 It can be seen that the recognition method of the medical terminology and sign language may include:

[0064] Step S110, obtaining a training data set.

[0065] The training data set includes multiple training samples, and the training samples include sample sign language data and sample medical terminology data. The sample sign language data and the sample medical terminology data of each training sample have the same language expression meaning, and the sample sign language data is image data.

[0066] In one embodiment of the present application, sample sign language data and sample medical terminology data may be first obtained. After the sample sign language data and the sample medical terminology data are obtained, data preprocessing may be performed on the sample sign language data and the sample medical terminology data. Data preprocessing may be text data preprocessing on the sample medical terminology data, image data preprocessing on the sample sign language data, and video data preprocessing on the sample sign language data. After preprocessing, the sample sign language data corresponding to each sample medical terminology data may be annotated to form a training sample, and the sample medical terminology data corresponding to each sign language data may be annotated to form a training sample, and the set of all training samples may be determined as a training data set. The training samples in the training data set may be divided into a training set, a validation set, and a test set to evaluate the generalization ability of the model.

[0067] Sample medical terminology data may include common medical terms, diagnostic terms, treatment terms, drug names, surgical terms, examination terms, test terms, specific medical field terms, professional medical vocabulary, medical phrases, medical expressions, outpatient scene dialogues, emergency scene dialogues, inpatient scene dialogues, medical term contexts, and medical term contexts. Sample sign language data may include basic sign language data, sign language action data, and sign language expression rules, wherein basic sign language data may include sign language vocabulary, sign language phrases, gesture videos, and gesture images, sign language action data may include hand shape, hand position, and gesture intensity, and sign language expression rules may include sign language grammar rules and sign language context.

[0068] For example, the sample medical terminology data is [You have acute respiratory distress syndrome, sudden gasping for breath due to exertion, blood supply to the brain is insufficient, and lungs are uncomfortable]. The translated sign language corresponding to the sample medical terminology data can be determined as the sample sign language data, and the sample medical terminology data and the corresponding sample sign language data can be determined as a training sample.

[0069] Step S120, constructing an initial recognition model.

[0070] Among them, the initial recognition model includes a feature extraction layer, a feature fusion layer, and an output layer; the feature extraction layer is used to extract features from sample medical terminology data to obtain medical terminology features, and to extract features from sample sign language data to obtain sign language features; the feature fusion layer is used to fuse medical terminology features and sign language features to obtain fusion features; the output layer is used to decode and output the fusion features to obtain recognition results.

[0071] In one embodiment of the present application, an initial recognition model including a feature extraction layer, a feature fusion layer, and an output layer may be constructed. The initial recognition model may also include an input layer located before the feature extraction layer. The input layer may convert the input into a type that the initial recognition model and the target recognition model can recognize (e.g., a fixed dimension of the input data).

[0072] Exemplarily, the feature extraction layer may include three loop layers, namely, loop one, loop two, and loop three. The input data can be feature extracted through loop one, loop two, and loop three, and the data features can be gradually refined to further extract deep-level data time dependencies, spatial features, and semantic contextual relationships, thereby improving the quality and sophistication of data representation, and is the basis for the feature fusion layer to better perform multimodal feature fusion. Multi-layer feature extraction can effectively improve the efficiency and processing power of the model, thereby meeting the conversion needs of complex and large medical terminology systems and sign language systems.

[0073] In one embodiment of the present application, in the feature fusion layer, the medical terminology data features and the sign language data features can be weightedly fused through the self-attention mechanism through multimodal feature fusion to generate multimodal data features, solve the problems of missing and inconsistency of information between modalities, improve the accuracy of bilingual conversion data, and enable the model to simultaneously utilize the semantic information of the language and the temporal and spatial information of the sign language to ensure the complete conversion and transmission of information and reduce information loss.

[0074] In one embodiment of the present application, the output layer can decode and output the multimodal feature fusion data to obtain a recognition result, which may include a predicted phrase and a probability distribution of each output word in the predicted phrase, and the predicted phrase includes a sign language phrase or a medical term phrase.

[0075] In one possible implementation, the process of extracting features from sample medical terminology data to obtain medical terminology features may include: performing word segmentation processing on the sample medical terminology data to obtain a word segmentation sequence; performing word vector mapping on the word segmentation sequence to obtain a word segmentation vector for each word in the word segmentation sequence; determining a position code for each word in the word segmentation sequence; obtaining a medical terminology feature vector based on the word segmentation vector and the position code for each word segment; determining a multi-head attention output of the medical terminology feature vector based on a multi-head attention mechanism; performing feature extraction on the multi-head attention output to obtain an initial feature; and normalizing and residually connecting the initial feature to obtain the medical terminology feature.

[0076] For example, the sample medical term data can be segmented first, and the segmented sequence can be output. , Indicates the number of participles, is a positive integer greater than 0, Indicates participles, and For example, the input sample medical terminology data may be "You have acute respiratory distress syndrome, sudden gasping for breath due to exertion, insufficient blood supply to the brain, and discomfort in the lungs." The sample medical terminology data may be segmented, and the resulting segmentation sequence may be ["you", "this is", "acute", "respiratory", "distress", "syndrome", ","", "exertional", "sudden", "gasping for breath", ","", "blood", "supply", "brain", ","", "lungs", "discomfort"].

[0077] To map the word vector for each word segment, you can use The method captures the semantic relationship between words and encodes these relationships into the form of mathematical vectors to obtain word segmentation vectors:

[0078] (Formula 1)

[0079] in, Indicates The word vector corresponding to each word.

[0080] It should be noted that by mapping each word segment to a word vector and obtaining the word segment vector, basic semantic features can be initially extracted to capture the intrinsic correlation between the word segments.

[0081] For example, "acute" can be mapped into a word vector, .

[0082] Determine the position encoding of each word in the word sequence:

[0083] (Formula 2)

[0084] (Formula 3)

[0085] (Formula 4)

[0086] in, Indicates the first participle, represents odd dimensions, represents an even dimension, represents the total dimension of the positional encoding, Indicates The positional encoding of the word segment, Indicates The positional encoding of the even-numbered dimensions of the word segments, Indicates The positional encoding of the odd-numbered dimensions of the word segments.

[0087] It should be noted that by determining the position code of each segmentation in the segmentation sequence, the segmentation position structure and context order can be adjusted to clarify the context relationship.

[0088] Integrate the word vectors and position encodings of all word segmentations to obtain the initial feature vector:

[0089] (Formula 5)

[0090] in, represents the initial eigenvector.

[0091] Based on the multi-head self-attention mechanism, the multi-head attention output of the initial feature vector is determined. (hereinafter referred to as M) through different linear transformation matrices to obtain different , , The matrix is ​​then used to obtain the output of each attention head, extract the context-related representation of the sample medical terminology data, and thus sort out the contextual relationship of medical terms.

[0092] Determine the multi-head attention output of the initial feature vector, first determine the output of any attention head:

[0093] (Formula 6)

[0094] (Formula 7)

[0095] (Formula 8)

[0096] in, , , represents the weight matrix (also known as the linear transformation matrix), is the query vector matrix, is the key vector matrix, is a matrix of value vectors.

[0097] calculate , The similarity between:

[0098] (Formula 9)

[0099] in, express The dimension is used to scale the dot product result to avoid the gradient vanishing problem caused by dividing by a large number. express The transposed matrix of .

[0100] Again Perform weighted summation to get the output of any attention head:

[0101] (Formula 10)

[0102] Concatenate the output of each attention head to get the multi-head attention output:

[0103] (Formula 11)

[0104] in, Indicates The output of an attention head is is the output transformation matrix, Represents the number of attention heads.

[0105] , Represents the dimension of each attention head output; represents the output dimension of the model, is a real number.

[0106] The multi-head attention mechanism can capture different contextual information such as symptom description and acuteness. It can capture deep and complex semantic relationships, further capture contextual dependencies, and ensure the semantic independence of other parts.

[0107] Perform feature extraction on the multi-head attention output to obtain the initial features: Apply the feedforward network to the context representation of each word segmentation and apply two fully connected layers and ReLU activation function to obtain the initial features. The initial features have a higher level of feature representation:

[0108] (Formula 12)

[0109] in, is the initial feature, and are the weight matrices of the two fully connected layers respectively, and is the bias term, Represents multi-head attention output.

[0110] It should be noted that feature extraction of multi-head attention output can form deep semantic features and logical structures to identify deeper implicit relationships.

[0111] Continue to normalize and residual connect the initial features to stabilize the training process and improve the model's expressiveness to obtain the medical term features:

[0112] (Formula 13)

[0113] in, Represents medical term features, Representation layer normalization, Represents the result of the feedforward operation on the input x.

[0114] In one possible implementation, the process of extracting features from sample sign language data to obtain sign language features may include simultaneously capturing time series information of the sign language data in both forward and reverse directions to obtain time series features; extracting spatial features of the sign language data to obtain image spatial features; performing convolution processing on the image spatial features and then performing an activation function to obtain weighted image features; and fusing the time series features and the weighted image features to obtain sign language features.

[0115] For example, feature extraction is performed on the sign language data. , the sign language data It can include multiple sign language images, each of which is 1 image / frame. For example, sign language data It can include ["I", "fast", "breathe", "up", "no", "head", "squint", "blink"], multiple sign language images, and each sign language segmentation is 1 image / frame.

[0116] , Represents the number of sign language images, n is a positive integer greater than 0, Indicates Sign language images.

[0117] Use LSTM network to process sign language data and extract time series features:

[0118] (Formula 14)

[0119] in, Indicates Time series features of sign language images, Represents the forward LSTM at time step The hidden state of Represents the backward LSTM at time step Hidden state.

[0120] Convolutional neural network can be used to extract sign language spatial features and obtain image spatial features:

[0121] (Formula 15)

[0122] in, Indicates Image space features of sign language images.

[0123] After convolution processing of the image spatial features, the weighted image features are obtained through the activation function:

[0124] (Formula 16)

[0125] in, Indicates The weighted image features of sign language images, Represents convolution processing.

[0126] The time series features and weighted image features are combined to obtain the sign language features:

[0127] (Formula 17)

[0128] in, Indicates sign language features.

[0129] In a possible implementation, the process of fusing medical term features and sign language features through a feature fusion layer to obtain fused features may include: performing weighted processing on the medical term features and obtaining the features of the term to be processed through an activation function, and performing weighted processing on the sign language features and obtaining the features of the sign language to be processed through an activation function; fusing the features of the term to be processed and the features of the sign language to be processed once to obtain initial features; fusing the initial features twice to obtain intermediate features; and fusing the intermediate features three times to obtain fused features.

[0130] Exemplarily, after weighting the medical term features, the term features to be processed are obtained through an activation function:

[0131] (Formula 18)

[0132] in, express function, The weight matrix representing the features of medical terms, represents the bias term, Represents the term features to be processed.

[0133] After weighted processing of the sign language features, the sign language features to be processed are obtained through the activation function:

[0134] (Formula 19)

[0135] in, express function, The weight matrix representing the sign language features, represents the bias term, Represents the sign language features to be processed.

[0136] The features of the term to be processed and the features of the sign language to be processed are fused to obtain the initial features:

[0137] (Formula 20)

[0138] in, Represents the initial features.

[0139] The initial features are fused twice to obtain the intermediate features:

[0140] (Formula 21)

[0141] in, express function, represents the weight matrix of the initial features, represents the bias term, Represents an intermediate feature.

[0142] The intermediate features are fused three times to obtain the fused features:

[0143] (Formula 22)

[0144] in, express function, represents the weight matrix of the intermediate features, represents the bias term, Represents fusion features.

[0145] After multiple feature fusion and weighted processing, the final multi-modal fusion feature is generated For model training, multiple fusions of medical terminology features and sign language features can reduce feature information loss and improve the generalization performance of the initial recognition model.

[0146] It should be noted that in the embodiment of the present application, the fusion feature integrates the information of the three modalities of semantic features, time series features, and spatial features, and the sample data required to obtain the three modal data in the embodiment of the present application is more and of higher dimension. Since the hospital scene is very complex, the method provided in the embodiment of the present application can be used to blend and complement the features of medical terminology data and sign language data to form new features, effectively improving the accuracy and naturalness of the two-way conversion, so that the target recognition model has better adaptability and expression ability when processing two types of data (medical terminology data and sign language data) and applications, and better robustness. Because in the embodiment of the present application, the medical terminology data and sign language data are deeply mined to form usable features, they can be fused to form multi-modal fusion features.

[0147] In a possible implementation, the process of decoding and outputting the fused features through the output layer to obtain the recognition result may include: performing a nonlinear transformation on the fused features to obtain nonlinear features; performing a linear transformation on the nonlinear features to obtain the recognition result, and the recognition result includes a predicted phrase and a probability distribution of each output word in the predicted phrase, and the predicted phrase includes a sign language phrase or a medical term phrase.

[0148] For example, in the conversion layer, the multi-modal fusion features Decode and output: Use nonlinear transformation to generate nonlinear features:

[0149] (Formula 23)

[0150] in, represents the weight matrix of fusion features, represents the bias term, Represents nonlinear characteristics.

[0151] Perform linear transformation on nonlinear features to obtain recognition results:

[0152] (Formula 24)

[0153] in, is the weight matrix of nonlinear features, is the bias term, Indicates the recognition result.

[0154] Step S130, training the initial recognition model to obtain a target recognition model.

[0155] In one embodiment of the present application, the initial recognition model can be trained to obtain a target recognition model. A loss function can be used to define the optimization goal, and the goal of training the initial recognition model is to minimize the loss function, which can be achieved through optimization algorithms such as gradient descent.

[0156] Step S140, obtaining data to be identified.

[0157] The data to be identified are medical terminology data or sign language data.

[0158] In one embodiment of the present application, data to be identified may be obtained. The medical terminology data may be text data, and the sign language data may be image data. The medical terminology data may be the content of a conversation between a hearing person and a hearing-impaired person, and the sign language data may be the content of a conversation between a hearing-impaired person and a hearing person.

[0159] For example, in a medical service scenario, when a hearing-impaired person and a hearing person communicate, communication barriers may occur because the two parties cannot understand each other's words. The hearing person can input conversation data through the translation terminal, and the conversation data can be text data or voice data. After receiving the conversation data, the translation terminal can convert the conversation data into text-type medical terminology data, and the converted medical terminology data is the data to be identified.

[0160] The hearing-impaired person can input continuous sign language images through the translation terminal. After acquiring the continuous sign language images, the terminal can determine the continuous sign language images as sign language data, which is the data to be recognized.

[0161] It should be noted that the translation terminal can be a mobile phone, tablet, etc.

[0162] Step S150, identifying the data to be identified based on the target identification model to obtain a identification result.

[0163] Among them, when the data to be recognized is medical terminology data, the recognition result is a sign language phrase; when the data to be recognized is sign language data, the recognition result is a medical terminology phrase.

[0164] In one embodiment of the present application, the data to be recognized can be recognized based on the target recognition model to obtain a recognition result. The data to be recognized can be input into the target recognition model, and the features of the data to be recognized can be extracted using a feature extraction layer. Since the initial recognition model corresponding to the target recognition model can extract medical terminology features and sign language features, the target recognition model can obtain the features of the data to be recognized for the data to be recognized input into the target recognition model, regardless of whether the data to be recognized is medical terminology data or sign language data. A corresponding recognition result is obtained based on the features of the data to be recognized.

[0165] It should be noted that when the data to be recognized is a medical term, the recognition result is a sign language phrase and the probability distribution corresponding to each word in the sign language phrase; when the data to be recognized is a sign language data, the recognition result is a medical term phrase and the probability distribution corresponding to each word in the medical term phrase.

[0166] Exemplarily, the input medical terminology data is ["you", "this is", "acute", "breathing", "distress", "syndrome", ",","exertional", "sudden", "panting", ",","blood", "insufficient supply", "brain", ",","lungs", "uncomfortable"]. The recognition result can be {"you":"0.98"; "mouth":"0.96"; "nose":"0.98"; "cover":"0.79"; "lungs":"0.842"; "back and forth":"0.6"; "movement":"0.61"; "more":"0.77"; "blood":"0.811"; "less":"0.811"}. The output result can be parsed and processed by the software and hardware terminal to form a continuous sign language image / video. The parsing method and parsing result can be arbitrary. For example, the software and hardware terminal can use natural language processing technology and computer vision to form a continuous sign language image / video on the recognition result.

[0167] Exemplarily, the input is a sign language image, and the meaning represented by the sign language image can be ["I", "fast", "breathe", "up", "no", "head", "squinting", "blinking", "shaking head"]. The recognition result can be {"I": "0.99"; "breathe": "0.83"; "fast": "0.92"; "panting": "0.799"; "unable": "0.98"; "head": "0.98"; "chaotic": "0.96"}. The output result can be used for software and hardware terminal parsing and processing to output natural language that conforms to the medical service scenario, such as "I breathe faster, I can't breathe, and my head is dizzy". The parsing method and result can be arbitrary. For example, the software and hardware terminal can use natural language processing technology to process the recognition result to obtain natural language that conforms to the medical service scenario.

[0168] The initial recognition model can perform real-time, efficient and accurate semantic, contextual, scenario and rule data mining and analysis on medical terminology data and sign language data, which is closer to the real medical service scenario. The target recognition model obtained by model training the initial recognition model can extract features from the input data to be recognized, which can be medical terminology data or sign language data. Since the medical terminology features and sign language features are integrated when the initial recognition model is trained, when the target recognition model extracts features from the data to be recognized, it can extract complete feature information from the data to be recognized, thereby improving the accuracy of recognizing the data to be recognized.

[0169] In summary, the solution of the embodiment of the present application constructs an initial recognition model, extracts the features of sign language data and the features of medical terms respectively, and fuses the features of medical terms and the features of sign language data to obtain fused features, so that the initial recognition model can obtain more comprehensive feature information based on the fused features. Furthermore, the target recognition model trained according to the initial recognition model can obtain more accurate recognition results, improve the robustness of the target recognition model, and improve the accuracy of the conversion between sign language and medical terms. The target recognition model can not only convert medical terms into sign language, but also convert the sign language of hearing-impaired people into medical terms, solve the communication problems between hearing-impaired people and medical staff, and improve the quality and efficiency of medical services.

[0170] Figure 2 FIG. 1 is a block diagram of a device for recognizing medical terms and sign language, shown in an exemplary embodiment of the present application. Figure 2 As shown, the exemplary medical terminology and sign language recognition device 200 includes:

[0171] The training data acquisition module 210 is used to acquire a training data set, which includes multiple training samples. The training samples include sample sign language data and sample medical terminology data. The sample sign language data and sample medical terminology data of each training sample have the same language expression meaning, and the sample sign language data is image data.

[0172] The model building module 220 is used to build an initial recognition model, which includes a feature extraction layer, a feature fusion layer, and an output layer; the feature extraction layer is used to perform feature extraction on sample medical terminology data to obtain medical terminology features, and to perform feature extraction on sample sign language data to obtain sign language features; the feature fusion layer is used to perform feature fusion on medical terminology features and sign language features to obtain fusion features; and the output layer is used to decode and output the fusion features to obtain recognition results.

[0173] The model training module 230 is used to train the initial recognition model to obtain a target recognition model.

[0174] The recognition data acquisition module 240 is used to acquire the data to be recognized, where the data to be recognized is medical terminology data or sign language data.

[0175] The recognition result determination module 250 is used to recognize the data to be recognized based on the target recognition model to obtain a recognition result; when the data to be recognized is medical terminology data, the recognition result is a sign language phrase; when the data to be recognized is sign language data, the recognition result is a medical terminology phrase.

[0176] In one embodiment of the present application, the model building module includes a medical term feature extraction module, and the medical term feature extraction module is used to:

[0177] Perform word segmentation on sample medical term data to obtain word segmentation sequences;

[0178] Perform word vector mapping on the word segmentation sequence to obtain the word segmentation vector of each word in the word segmentation sequence;

[0179] Determine the position code of each word in the word sequence;

[0180] According to the word segmentation vector and position code of each word segmentation, a medical term feature vector is obtained;

[0181] Based on the multi-head attention mechanism, determine the output of each attention head in the medical term feature vector;

[0182] Concatenate the output of each attention head to get the multi-head attention output;

[0183] Perform feature extraction on the multi-head attention output to obtain initial features;

[0184] The initial features are normalized and residually connected to obtain medical term features.

[0185] In one embodiment of the present application, the model building module further includes a sign language feature extraction module, and the sign language feature extraction module is used to:

[0186] Capture time series information of sign language data in both forward and reverse directions to obtain time series features;

[0187] Extract the spatial features of the sign language data to obtain the image spatial features;

[0188] After convolution processing of the image spatial features, weighted image features are obtained through activation function;

[0189] The time series features and weighted image features are fused to obtain sign language features.

[0190] In one embodiment of the present application, the model building module further includes a feature fusion module, and the feature fusion module is used to:

[0191] After weighting the medical term features, the term features to be processed are obtained through an activation function, and after weighting the sign language features, the sign language features to be processed are obtained through an activation function;

[0192] The term features to be processed and the sign language features to be processed are fused to obtain the initial features;

[0193] Perform secondary fusion on the initial features to obtain intermediate features;

[0194] The intermediate features are fused three times to obtain the fused features.

[0195] In one embodiment of the present application, the model building module further includes an output module, and the output module is used to:

[0196] Perform nonlinear transformation on the fused features to obtain nonlinear features;

[0197] The nonlinear features are linearly transformed to obtain a recognition result, which includes a predicted phrase and a probability distribution of each output word in the predicted phrase, and the predicted phrase includes a sign language phrase or a medical term phrase.

[0198] It should be noted that the medical terminology and sign language recognition device provided in the above embodiment and the medical terminology and sign language recognition method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In actual application, the medical terminology and sign language recognition device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0199] An embodiment of the present application further provides a computer device, including: a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the medical terminology and sign language recognition method provided in the above-mentioned embodiments is implemented.

[0200] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of a computer, enables the computer to execute the medical terminology and sign language recognition methods provided in the above-mentioned embodiments. The computer-readable storage medium may be included in the electronic device described in the above-mentioned embodiments, or may exist independently without being assembled into the electronic device.

[0201] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the medical terminology and sign language recognition method provided in the above-mentioned various embodiments.

[0202] In the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. "Including" and "comprising" mentioned throughout the specification and claims are open-ended terms and should be interpreted as "including but not limited to".

[0203] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for recognizing medical terms and sign language, characterized in that: include: Acquire a training data set, wherein the training data set includes a plurality of training samples, wherein the training samples include sample sign language data and sample medical terminology data, wherein the sample sign language data and the sample medical terminology data of each training sample have the same language expression meaning, and the sample sign language data is image data; Constructing an initial recognition model, the initial recognition model includes a feature extraction layer, a feature fusion layer, and an output layer; the feature extraction layer is used to extract features from sample medical terminology data to obtain medical terminology features, and to extract features from sample sign language data to obtain sign language features; the feature fusion layer is used to fuse the medical terminology features and the sign language features to obtain fusion features; and the output layer is used to decode and output the fusion features to obtain recognition results; Training the initial recognition model to obtain a target recognition model; Acquiring data to be identified, where the data to be identified is medical terminology data or sign language data; Recognize the data to be recognized based on the target recognition model to obtain a recognition result; when the data to be recognized is medical terminology data, the recognition result is a sign language phrase; when the data to be recognized is sign language data, the recognition result is a medical terminology phrase; Extracting features from sample medical terminology data to obtain medical terminology features, including: performing word segmentation processing on the sample medical terminology data to obtain a word segmentation sequence; performing word vector mapping on the word segmentation sequence to obtain a word segmentation vector of each word in the word segmentation sequence; determining a position code of each word in the word segmentation sequence; obtaining an initial feature vector according to the word segmentation vector and the position code of each word segmentation; determining a multi-head attention output of the initial feature vector based on a multi-head attention mechanism; performing feature extraction on the multi-head attention output to obtain an initial feature; normalizing and residually connecting the initial feature to obtain the medical terminology feature; Extracting features from the sample sign language data to obtain sign language features, including: capturing time series information from the sign language data in both forward and reverse directions to obtain time series features; extracting spatial features from the sign language data to obtain image spatial features; performing convolution processing on the image spatial features and then performing an activation function to obtain weighted image features; fusing the time series features with the weighted image features to obtain sign language features; The feature fusion layer is used to fuse the medical term feature and the sign language feature to obtain a fused feature, including: performing weighted processing on the medical term feature and then performing an activation function to obtain a term feature to be processed, and performing weighted processing on the sign language feature and then performing an activation function to obtain a sign language feature to be processed; fusing the term feature to be processed and the sign language feature to be processed once to obtain an initial feature; fusing the initial feature twice to obtain an intermediate feature; and fusing the intermediate feature three times to obtain the fused feature; The fused features are decoded and outputted through the output layer to obtain a recognition result, including: performing a nonlinear transformation on the fused features to obtain a nonlinear feature; performing a linear transformation on the nonlinear feature to obtain the recognition result, the recognition result including a predicted phrase and a probability distribution of each output word in the predicted phrase, the predicted phrase including the sign language phrase or the medical term phrase.

2. A device for recognizing medical terms and sign language, characterized in that: For implementing the method for recognizing medical terms and sign language as claimed in claim 1, the device for recognizing medical terms and sign language comprises: A training data acquisition module, used to acquire a training data set, wherein the training data set includes a plurality of training samples, wherein the training samples include sample sign language data and sample medical terminology data, wherein the sample sign language data and the sample medical terminology data of each training sample have the same language expression meaning, and the sample sign language data is image data; A model building module is used to build an initial recognition model, wherein the initial recognition model includes a feature extraction layer, a feature fusion layer, and an output layer; the feature extraction layer is used to perform feature extraction on sample medical terminology data to obtain medical terminology features, and perform feature extraction on sample sign language data to obtain sign language features; the feature fusion layer is used to perform feature fusion on the medical terminology features and the sign language features to obtain fusion features; and the output layer is used to decode and output the fusion features to obtain recognition results; A model training module, used to train the initial recognition model to obtain a target recognition model; An identification data acquisition module, used to acquire data to be identified, wherein the data to be identified is medical terminology data or sign language data; A recognition result determination module is used to recognize the data to be recognized based on the target recognition model to obtain a recognition result; when the data to be recognized is medical terminology data, the recognition result is a sign language phrase; when the data to be recognized is sign language data, the recognition result is a medical terminology phrase.

3. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method for recognizing medical terms and sign language according to claim 1 is implemented.

4. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method for recognizing medical terms and sign language as claimed in claim 1.

Images