Methods, apparatus, equipment and media for constructing medical auxiliary subtitle generation models

Abstract

This invention relates to the field of digital medical technology, and more particularly to a method, apparatus, device, and medium for constructing a medical-aided subtitle generation model. The method uses an encoder to extract audio features from hearing-impairment test audio, fuses the audio features with noise information, decodes the fusion result using a decoder to obtain initial subtitles, uses a naturalness discriminator to predict the naturalness of the initial subtitles to obtain a first predicted value, uses a semantic discriminator to predict the semantic relevance between the initial subtitles and the hearing-impairment test audio to obtain a second predicted value, calculates the similarity between the initial subtitles and the tagged subtitles, and updates the encoder and decoder based on the first predicted value, the second predicted value, and the similarity. The updated encoder and decoder form a subtitle generation model. Through multi-dimensional supervision, the simulation level of neural network subtitle generation is improved. By adding noise, the generated subtitles acquire diverse characteristics, improving the audio information experience for hearing-impaired patients.

Description

Technical Field

[0001] This invention relates to the field of digital medical technology, and in particular to a method, apparatus, equipment, and medium for constructing a medical auxiliary subtitle generation model. Background Technology

[0002] Subtitle generation is a cross-modal translation task, typically generating corresponding natural language text subtitles based on audio information. Unlike sound recognition tasks, subtitle generation focuses more on the semantic relationships between audio information and the background information of the audio, such as the speaker and the environment in which the audio is produced. Existing automatic subtitle generation models usually generate fixed subtitles determined by common words and simple grammar based on given audio information. With the rise of subtitle generation tasks, it can assist hearing-impaired patients in conducting remote video consultations, hearing impairment level tests, etc., by converting video or audio information into subtitle text for hearing-impaired patients to read.

[0003] However, because different subtitle annotators have varying degrees of focus on background information and semantic connections when describing audio information, the same audio segment can have diverse subtitles. Automatic subtitle generation models aim to simulate the annotation behavior of human subtitle annotators. Existing automatic subtitle generation models, due to their relatively fixed subtitle generation (i.e., poor diversity), produce subtitles with poor realism, making it easy for viewers to distinguish between manually annotated and generated subtitles. Therefore, improving the diversity of subtitle generation and thus enhancing its realism has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, apparatus, device and medium for constructing a medical auxiliary subtitle generation model to solve the problem of poor diversity in subtitle generation.

[0005] In a first aspect, embodiments of the present invention provide a method for constructing a medical auxiliary subtitle generation model, the method comprising:

[0006] Acquire hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to decode the features of the fusion result to obtain initial generated subtitles;

[0007] The probability that the initially generated subtitles belong to human annotations is predicted using a trained naturalness discriminator to obtain a first prediction value;

[0008] The semantic discriminator is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value;

[0009] Obtain the tagged subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the tagged subtitles, and update the parameters of the encoder and the decoder based on the first predicted value, the second predicted value and the similarity, to obtain the updated encoder and the updated decoder.

[0010] An updated subtitle generation model is formed by the updated encoder and the updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0011] Secondly, embodiments of the present invention provide a construction apparatus for a medical auxiliary subtitle generation model, the construction apparatus comprising:

[0012] The feature extraction module is used to acquire hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to perform feature decoding on the fusion result to obtain initial generated subtitles;

[0013] The first prediction module is used to predict the probability that the initially generated subtitles belong to human annotations using a trained naturalness discriminator, and obtain a first prediction value.

[0014] The second prediction module is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio using a trained semantic discriminator, and obtain a second prediction value.

[0015] The parameter update module is used to obtain the tagged subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the tagged subtitles, and update the parameters of the encoder and the decoder according to the first predicted value, the second predicted value and the similarity, so as to obtain the updated encoder and the updated decoder.

[0016] The model building module is used to form an updated subtitle generation model from the updated encoder and the updated decoder. The updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0017] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the construction method as described in the first aspect.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the construction method described in the first aspect.

[0019] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0020] The process involves acquiring hearing impairment test audio, extracting features from the audio using an encoder, fusing these features with randomly generated noise, and decoding the fused results to obtain initial subtitles. A trained naturalness discriminator predicts the probability that the initial subtitles are manually labeled, yielding a first prediction value. A trained semantic discriminator predicts the semantic relevance between the initial subtitles and the hearing impairment test audio, yielding a second prediction value. Labeled subtitles of the hearing impairment test audio are then acquired, and the similarity between the initial subtitles and the labeled subtitles is calculated. Based on the first and second prediction values ​​and the similarity, the encoder and decoder parameters are updated, resulting in updated encoders and decoders. These updated encoders and decoders form an updated subtitle generation model. Joint supervision based on naturalness, semantic relevance, and similarity to the labeled subtitles ensures the simulation level of the neural network in generating subtitles. Adding noise diversifies the generated subtitles, thus improving the diversity of generated subtitles while maintaining simulation level, thereby enhancing the experience of hearing-impaired patients accessing audio information in text form. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of an application environment for a method of constructing a medical auxiliary subtitle generation model provided in Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart illustrating a method for constructing a medical auxiliary subtitle generation model according to Embodiment 1 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of a medical auxiliary subtitle generation model construction device provided in Embodiment 2 of the present invention;

[0025] Figure 4This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0036] The present invention provides a method for constructing a medical auxiliary subtitle generation model, which can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] The aforementioned client and server can be deployed in medical settings for tasks such as intelligent diagnosis and treatment and remote consultation for hearing-impaired patients.

[0038] See Figure 2 This is a flowchart illustrating a method for constructing a medical auxiliary subtitle generation model according to Embodiment 1 of the present invention. The above construction method can be applied to... Figure 1The server-side component connects to the client via a corresponding computer device to retrieve hearing impairment test audio and label subtitles. These audio and subtitles can be input into the client by hearing impairment test personnel or selected from the client's stored content. Test personnel can be doctors, nurses, or hearing-impaired patients. Figure 2 As shown, the construction method may include the following steps:

[0039] Step S201: Obtain hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to decode the features of the fusion result to obtain the initial generated subtitles.

[0040] Among them, the hearing impairment test audio can refer to the training samples used to build a subtitle generation model to assist hearing-impaired patients in obtaining audio information. The hearing impairment test audio can be audio information carried in the video or separate audio information. The encoder can be used to extract features from the input audio information. The audio features can be used to characterize the feature information of the input audio information. The noise information can be randomly generated noise that conforms to a Gaussian distribution, i.e., white noise. The fusion result can refer to the audio features after adding noise. The decoder can be used to decode the audio features after adding noise into text information. The decoded text information is the initial generated subtitle.

[0041] Specifically, in this embodiment, the encoder employs multiple cascaded convolutional neural network structures. Each convolutional neural network structure may include at least one convolutional block, and a single convolutional block may include at least one convolutional layer. The output of each convolutional layer needs to be processed sequentially through a batch normalization layer and an activation function layer. The activation function used in the activation function layer may be the ReLU activation function. The output of each convolutional block needs to be processed through a pooling layer, which may be a max pooling layer. In particular, the output of the last convolutional block needs to be processed through a global average pooling layer. The output of the global average pooling layer can be processed using a multilayer perceptron to obtain audio features. The multilayer perceptron may include at least two linear layers, and an activation function layer exists between any two adjacent linear layers.

[0042] To reduce the need for a large number of audio samples from hearing impairment tests for parameter updates, the encoder can be pre-trained. Pre-training can be done in an unsupervised manner, such as by adding an additional decoder and co-training the encoder using a reconstruction task, where the input samples can be used as labels for training. Alternatively, a supervised manner can be used, such as by adding an additional classifier and co-training the encoder using a simple downstream task, such as an audio speaker gender classification task or an audio speaker age group classification task. Using a simple downstream task can effectively reduce the labeling cost of pre-training, while also enabling the pre-trained encoder to have a certain feature extraction capability.

[0043] The above steps involve acquiring hearing impairment test audio, using an encoder to extract features from the hearing impairment test audio to obtain audio features, fusing the audio features with randomly generated noise information to obtain a fusion result, and using a decoder to decode the features of the fusion result to obtain the initial generated subtitles. By adding noise, the initial generated subtitles have diverse characteristics, providing rich initial generated subtitles for subsequent model parameter updates, so as to determine the update direction based on the initial generated subtitles, thereby improving the performance of the updated model.

[0044] Step S202: Use the trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated, and obtain the first prediction value.

[0045] Among them, the trained naturalness discriminator can be used to identify whether the input subtitles are manually annotated, and the first prediction value can characterize the probability that the input subtitles are manually annotated.

[0046] Specifically, the naturalness discriminator can be viewed as a binary classification task, with one category being that the input subtitles are human-annotated and the other category being that the input subtitles are not human-annotated. The trained naturalness discriminator can be obtained by training a training set that mixes human-annotated subtitles and machine-generated subtitles.

[0047] Optionally, a trained naturalness discriminator is used to predict the probability that the initially generated subtitles are manually annotated, resulting in a first predicted value, including:

[0048] The initially generated subtitles are segmented into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles. All subtitle terms are then concatenated into a term sequence according to their positional order.

[0049] Using a trained gated recurrent unit, the probability of a word sequence belonging to a manually labeled term is predicted, and the first prediction value is obtained.

[0050] Here, subtitle terms can refer to terms in the initially generated subtitles, and their positions can be represented by numbers. For example, all subtitle terms in the initially generated subtitles are assigned numbers from 0 to N in order from left to right, where N+1 is the number of terms. The term sequence can include N+1 subtitle terms arranged in order of position. The trained gated recurrent unit can be used to extract term features from the term sequence.

[0051] Specifically, following the positional order, the input sequence number is initialized to 0. The subtitle term corresponding to the input sequence number is input into the trained gated recurrent unit to obtain the corresponding term feature. Using this term feature as the input feature, the input sequence number is incremented by 1. The input feature and the subtitle term corresponding to the input sequence number are input into the trained gated recurrent unit to obtain the corresponding term feature. Using this term feature as the input feature, the input sequence number is incremented by 1. The process of inputting the input feature and the subtitle term corresponding to the input sequence number into the trained gated recurrent unit to obtain the corresponding term feature is repeated. The input sequence number is incremented by 1 again. This process continues until the input sequence number is N+1, at which point the iteration stops, and the final output term feature is obtained. A multilayer perceptron is used to process the final output term feature, and the processing result is mapped using a sigmoid function to obtain the first predicted value.

[0052] In this embodiment, feature extraction of the initial generated subtitles is achieved through a gated loop unit, which can fully extract the contextual information of the initial generated subtitles, improve the accuracy of naturalness identification, and thus improve the subtitle generation capability of the encoder and decoder that are updated with the naturalness identification result as a reference, thereby improving the simulation level of subtitle generation.

[0053] The above-mentioned step of using a trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated, and obtaining the first predicted value, uses the first predicted value as supervision information for the subsequent model parameter update process, thereby ensuring that the subtitle generation model can generate subtitles that meet the naturalness requirement and improving the simulation level of subtitle generation.

[0054] Step S203: Use the trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value.

[0055] The trained semantic discriminator can be used to identify whether the semantics of the initially generated subtitles and the hearing-impaired test audio are similar, and the second prediction value can characterize the degree of semantic consistency between the initially generated subtitles and the hearing-impaired test audio.

[0056] Optionally, a trained semantic discriminator is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value, including:

[0057] Using an audio encoder with fixed parameters, semantic features of the hearing impairment test audio are extracted to obtain audio semantic features;

[0058] Using the trained gated recurrent unit, the semantic features of the initially generated subtitles are extracted to obtain the subtitle semantic features;

[0059] The similarity between audio semantic features and subtitle semantic features is calculated to obtain the similarity calculation result, which is then used as the second prediction value.

[0060] The audio encoder can be used to extract semantic features from the hearing impairment test audio. The audio semantic features can be used to represent the semantic information of the hearing impairment test audio. The trained gated recurrent unit can be used to extract semantic features from the initially generated subtitles. The subtitle semantic features can be used to represent the semantic information of the initially generated subtitles. The similarity calculation results can represent the similarity between the semantics of the initially generated subtitles and the hearing impairment test audio.

[0061] Specifically, the audio encoder can be implemented using an audio neural network. It should be noted that the audio encoder is pre-trained with fixed parameters, and the audio semantic features and subtitle semantic features have the same size.

[0062] Audio semantic features and subtitle semantic features are embedded into a shared embedding space through a multilayer perceptron, respectively, to obtain audio embeddings of corresponding audio semantic features and subtitle embeddings of corresponding subtitle semantic features. The similarity between the audio embeddings and subtitle embeddings is calculated. The similarity can be calculated using Euclidean distance, cosine similarity, etc. In this embodiment, cosine similarity is used for similarity calculation. The obtained similarity calculation result takes a value range of [-1, 1]. The closer the similarity calculation result is to 1, the more similar the audio embedding and subtitle embedding are. The closer the similarity calculation result is to -1, the less similar the audio embedding and subtitle embedding are. In this embodiment, the ReLU function is used to map the similarity calculation result to the range of [0, 1], so that the similarity calculation result can be regarded as a probability, that is, the probability that the initial generated subtitles and the hearing impairment test audio are semantically consistent. The similarity calculation result between the audio embedding and the subtitle embedding is used as the similarity calculation result between the audio semantic features and the subtitle semantic features.

[0063] In this embodiment, the feature extraction of the initial generated subtitles is achieved through a gated loop unit, which can fully extract the contextual information of the initial generated subtitles and improve the extraction accuracy of subtitle semantic features. The similarity between audio semantic features and subtitle semantic features is used as the semantic identification result, thereby improving the subtitle generation capability of the encoder and decoder that are updated with the semantic identification result as a reference, thus improving the semantic accuracy of subtitle generation.

[0064] The above steps, which use a trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value, use the first predicted value as supervision information for the subsequent model parameter update process. This ensures that the subtitle generation model can generate subtitles that satisfy the audio semantics, thereby improving the semantic accuracy of subtitle generation.

[0065] Step S204: Obtain the tagged subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the tagged subtitles, and update the parameters of the encoder and decoder based on the first predicted value, the second predicted value and the similarity, to obtain the updated encoder and the updated decoder.

[0066] Among them, the tag caption can refer to the reference caption annotated by humans, and the similarity can be used to characterize the degree of similarity between the initially generated caption and the tag caption.

[0067] Optionally, the similarity between the initially generated captions and the tagged captions is calculated, including:

[0068] The initially generated subtitles are segmented into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles. All subtitle terms are then concatenated in order of position to form the first term sequence.

[0069] The tag caption is segmented into terms to obtain at least two tag terms and their positions in the tag caption. All tag terms are then concatenated in order of position to form the second term sequence.

[0070] Calculate the similarity between the first term sequence and the second term sequence, and use the result as the similarity between the initially generated subtitles and the tagged subtitles.

[0071] The first term sequence may include all subtitle terms arranged by position, and the second term sequence may include all tag terms arranged by position.

[0072] Specifically, the similarity between the first term sequence and the second term sequence can be calculated using cosine similarity. In this case, the main focus is on whether the terms used in the first term sequence and the second term sequence are similar, without involving the semantic information of the subtitle text.

[0073] In this embodiment, the similarity of terms is measured so that the initial generated subtitles can be evaluated based on a fixed objective reference. This improves the subtitle generation capability of the encoder and decoder, which are updated with similarity as a reference, thereby further improving the simulation level of subtitle generation.

[0074] Optionally, based on the first predicted value, the second predicted value, and the similarity, the encoder and decoder parameters are updated to obtain an updated encoder and an updated decoder, including:

[0075] The first update loss is calculated based on the first and second predicted values. The encoder and decoder are then updated according to the first update loss until the first update loss converges, resulting in a preliminarily updated encoder and decoder.

[0076] Based on the similarity, the second update loss is calculated. Based on the second update loss, the parameters of the initially updated encoder and decoder are updated until the second update loss converges, resulting in the updated encoder and decoder.

[0077] The first update loss can be used to guide the encoder and decoder to update parameters, and the second update loss can be used to guide the encoder and decoder that have been initially updated to update parameters.

[0078] Specifically, similarity provides feedback to the model update in the form of regular labels, enabling the model to learn certain label information based on the feedback of the first and second predicted values. However, since the labels are usually fixed, a two-stage training method is adopted. That is, the first stage of training is performed based on the first and second predicted values, and after the first stage of training is completed, the second stage of training is performed based on the similarity, so that the similarity does not affect the feedback of the first and second predicted values.

[0079] In this embodiment, a two-stage training method is used to ensure that the training processes do not interfere with each other, taking into account naturalness, semantic relevance and tag information, thereby improving the simulation level of subtitle generation.

[0080] The above steps involve obtaining tagged subtitles from the audio of the hearing impairment test, calculating the similarity between the initial generated subtitles and the tagged subtitles, updating the parameters of the encoder and decoder based on the first predicted value, the second predicted value, and the similarity, and obtaining the updated encoder and decoder. This process uses naturalness, semantic relevance, and similarity with the tagged subtitles for joint supervision to ensure the simulation level of the neural network in generating subtitles.

[0081] Step S205: An updated subtitle generation model is formed from the updated encoder and the updated decoder.

[0082] Among them, the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0083] Optionally, the construction method also includes:

[0084] The updated encoder is used to extract features from the hearing impairment test audio to obtain updated audio features. The updated audio features are then fused with randomly generated noise information to obtain an updated fusion result. The updated decoder is used to decode the features of the updated fusion result to obtain updated subtitles.

[0085] The trained naturalness discriminator is used to predict the probability that the updated subtitles are manually annotated, and a third prediction value is obtained.

[0086] The trained semantic discriminator is used to predict the semantic relevance between the updated generated subtitles and the hearing impairment test audio, resulting in a fourth prediction value.

[0087] Based on the third predicted value, the fourth predicted value, and the similarity, the parameters of the trained naturalness discriminator and the trained semantic discriminator are updated to obtain the updated naturalness discriminator and the updated naturalness discriminator.

[0088] Using the updated naturalness discriminator as the trained naturalness discriminator, the updated semantic discriminator as the trained semantic discriminator, the updated encoder as the encoder, and the updated decoder as the decoder, the preset number of iterations is increased by one, and the initial value of the number of iterations is zero.

[0089] Return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets the preset value, then stop iterating, and obtain the final updated encoder and the final updated decoder. The final updated encoder and the final updated decoder form the final updated subtitle generation model.

[0090] Among them, the third predicted value can characterize the probability that the updated subtitles are manually annotated, and the fourth predicted value can characterize the probability that the updated generated subtitles and the hearing impairment test audio are semantically consistent.

[0091] Specifically, the preset value can be set to 3, and implementers can adjust this preset value according to the actual situation.

[0092] In this embodiment, the idea of ​​adversarial network training is adopted to train the subtitle generation model composed of encoder and decoder and the discrimination model composed of naturalness discriminator and semantic discriminator in turn. This can further improve the generation ability of the subtitle generation model and the discrimination ability of the discrimination model, thereby improving the simulation degree and fidelity of the generated subtitles.

[0093] Optionally, after increasing the preset number of iterations by one, it also includes:

[0094] Calculate the difference between the third predicted value and the first predicted value to obtain the first difference; calculate the difference between the fourth predicted value and the second predicted value to obtain the second difference.

[0095] When both the first difference and the second difference are less than the preset threshold, the preset condition is determined to be met.

[0096] Accordingly, the process returns to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets a preset value, at which point the iteration stops, including:

[0097] Return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the preset conditions are met or the number of iterations meets the preset value, then stop iterating.

[0098] Among them, the first difference can represent the difference between the third predicted value and the first predicted value, the second difference can represent the difference between the fourth predicted value and the second predicted value, the preset threshold can measure the changes in naturalness and semantic relevance, and the preset condition can be used as the condition for stopping the iteration.

[0099] Specifically, when both the first difference and the second difference are less than the preset threshold, it indicates that the generation ability of the subtitle generation model and the discrimination ability of the discrimination model are approximately balanced, and it is no longer possible to improve the model performance of one of them through adversarial training.

[0100] In this embodiment, by adding a preset condition that both the first difference and the second difference are less than a preset threshold, the iteration can be stopped in time when the adversarial training effect deteriorates, thereby improving the efficiency of model updates.

[0101] The above steps, which form an updated subtitle generation model from the updated encoder and decoder, and use the updated subtitle generation model to generate subtitles, can generate highly realistic subtitles while ensuring that the semantics conform to the audio information.

[0102] In this embodiment, naturalness, semantic relevance, and similarity to the tagged subtitles are jointly supervised to ensure the simulation level of the neural network in generating subtitles. By adding noise, the generated subtitles are made more diverse, thereby improving the diversity of generated subtitles while ensuring the simulation level, and thus improving the experience of hearing-impaired patients in obtaining audio information in text form.

[0103] Corresponding to the method for constructing a medical auxiliary subtitle generation model in the above embodiment, Figure 3This diagram illustrates a structural block diagram of a construction device for a medical auxiliary subtitle generation model provided in Embodiment 2 of the present invention. The device is applied to a server, and the corresponding computer device is connected to a client to obtain hearing impairment test audio and labeled subtitles from the client. The hearing impairment test audio and labeled subtitles can be input into the client by hearing impairment testers or selected from content stored on the client. Testers can refer to doctors, nurses, or hearing-impaired patients. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0104] See Figure 3 The construction device includes:

[0105] Feature extraction module 31 is used to acquire hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to decode the features of the fusion result to obtain the initial generated subtitles;

[0106] The first prediction module 32 is used to predict the probability that the initially generated subtitles belong to human annotations using a trained naturalness discriminator, and obtain the first prediction value.

[0107] The second prediction module 33 is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio using a trained semantic discriminator, and obtain a second prediction value.

[0108] The parameter update module 34 is used to obtain the tagged subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the tagged subtitles, and update the parameters of the encoder and decoder based on the first predicted value, the second predicted value and the similarity, so as to obtain the updated encoder and the updated decoder.

[0109] The model building module 35 is used to form an updated subtitle generation model from the updated encoder and the updated decoder. The updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0110] Optionally, the first prediction module 32 mentioned above includes:

[0111] The first term segmentation unit is used to segment the initially generated subtitles into terms, obtain at least two subtitle terms and their positions in the initially generated subtitles, and concatenate all subtitle terms into a term sequence according to their position order.

[0112] The artificial probability prediction unit is used to predict the probability that a word sequence belongs to a manually labeled term using a trained gated recurrent unit, and obtains the first prediction value.

[0113] Optionally, the second prediction module 33 mentioned above includes:

[0114] The audio feature extraction unit is used to extract semantic features from hearing impairment test audio using an audio encoder with fixed parameters, thereby obtaining audio semantic features.

[0115] The subtitle feature extraction unit is used to extract the semantic features of the initially generated subtitles using the trained gated recurrent unit, thus obtaining the subtitle semantic features;

[0116] The similarity calculation unit is used to calculate the similarity between audio semantic features and subtitle semantic features, obtain the similarity calculation result, and use the similarity calculation result as the second prediction value.

[0117] Optionally, the parameter update module 34 mentioned above includes:

[0118] The second term segmentation unit is used to segment the initially generated subtitles into terms, obtain at least two subtitle terms and their positions in the initially generated subtitles, and concatenate all subtitle terms in order of position to form the first term sequence.

[0119] The third term segmentation unit is used to segment the tag subtitles into terms, obtain at least two tag terms and their positions in the tag subtitles, and concatenate all tag terms in order of position to form the second term sequence.

[0120] The similarity determination unit is used to calculate the similarity between the first term sequence and the second term sequence, and the calculation result is used as the similarity between the initially generated subtitles and the tag subtitles.

[0121] Optionally, the parameter update module 34 mentioned above includes:

[0122] The first parameter update unit is used to calculate the first update loss based on the first prediction value and the second prediction value, and update the parameters of the encoder and decoder based on the first update loss until the first update loss converges, so as to obtain the initially updated encoder and the initially updated decoder.

[0123] The second parameter update unit is used to calculate the second update loss based on the similarity. Based on the second update loss, the parameters of the initially updated encoder and decoder are updated until the second update loss converges, resulting in the updated encoder and decoder.

[0124] Optionally, the above-mentioned construction apparatus further includes:

[0125] The updated subtitle generation module is used to extract features from the hearing impairment test audio using the updated encoder to obtain updated audio features. The updated audio features are then fused with randomly generated noise information to obtain an updated fusion result. Finally, the updated decoder is used to decode the updated fusion result to obtain updated subtitles.

[0126] The naturalness identification module is used to predict the probability that the updated generated subtitles belong to human annotations using a trained naturalness discriminator, and obtain a third prediction value.

[0127] The relevance identification module is used to predict the semantic relevance between the updated generated subtitles and the hearing impairment test audio using a trained semantic discriminator, and obtain a fourth prediction value.

[0128] The discriminator update module is used to update the parameters of the trained naturalness discriminator and the trained semantic discriminator based on the third prediction value, the fourth prediction value and the similarity, so as to obtain the updated naturalness discriminator and the updated naturalness discriminator.

[0129] The iteration term determination module is used to use the updated naturalness discriminator as the trained naturalness discriminator, the updated semantic discriminator as the trained semantic discriminator, the updated encoder as the encoder, and the updated decoder as the decoder, and to increment the preset iteration number by one, with the initial value of the iteration number being zero.

[0130] The iterative execution module is used to return to the steps of using the encoder to extract features from the acquired hearing impairment test audio and obtain audio features until the number of iterations meets the preset value, at which point the iteration stops and the final updated encoder and decoder are obtained. The final updated encoder and decoder form the final updated subtitle generation model.

[0131] Optionally, the above-mentioned construction apparatus further includes:

[0132] The difference calculation module is used to calculate the difference between the third predicted value and the first predicted value to obtain the first difference, and to calculate the difference between the fourth predicted value and the second predicted value to obtain the second difference;

[0133] The condition determination module is used to determine that the preset condition is met when both the first difference and the second difference are less than the preset threshold.

[0134] Accordingly, the above iterative execution module includes:

[0135] The condition judgment unit is used to return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the preset conditions are met or the number of iterations meets the preset value, and then the iteration stops.

[0136] It should be noted that the information interaction and execution process between the above modules and units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0137] Figure 4This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 (Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, wherein the processor executes the computer program to implement the steps in any of the above-described construction method embodiments.

[0138] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 4 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0139] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0140] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0141] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0142] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0143] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0146] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for constructing a medical auxiliary subtitle generation model, characterized in that, The construction method includes: Acquire hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to decode the features of the fusion result to obtain initial generated subtitles; The probability that the initially generated subtitles belong to human annotations is predicted using a trained naturalness discriminator to obtain a first prediction value; The semantic discriminator is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value; Obtain the tagged subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the tagged subtitles, and update the parameters of the encoder and the decoder based on the first predicted value, the second predicted value and the similarity, to obtain the updated encoder and the updated decoder. An updated subtitle generation model is formed by the updated encoder and the updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

2. The construction method according to claim 1, characterized in that, The step of using a trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated, and obtaining a first predicted value, includes: The initially generated subtitles are segmented into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles. All subtitle terms are then concatenated into a term sequence according to their positional order. Using a trained gated recurrent unit, the probability that the term sequence belongs to a manually labeled term is predicted to obtain the first predicted value.

3. The construction method according to claim 1, characterized in that, The step of using a trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value includes: Using an audio encoder with fixed parameters, the semantic features of the hearing impairment test audio are extracted to obtain audio semantic features; Using the trained gated recurrent unit, the semantic features of the initially generated subtitles are extracted to obtain the subtitle semantic features; The similarity between the audio semantic features and the subtitle semantic features is calculated to obtain the similarity calculation result, which is then used as the second predicted value.

4. The construction method according to claim 1, characterized in that, The calculation of the similarity between the initially generated caption and the tagged caption includes: The initially generated subtitles are segmented into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles. All subtitle terms are then concatenated in order of position to form the first term sequence. The tag caption is segmented into terms to obtain at least two tag terms and their positions in the tag caption. All tag terms are then concatenated in order of position to form a second term sequence. Calculate the similarity between the first term sequence and the second term sequence, and use the calculation result as the similarity between the initially generated subtitle and the tagged subtitle.

5. The construction method according to claim 1, characterized in that, The step of updating the parameters of the encoder and the decoder based on the first predicted value, the second predicted value, and the similarity to obtain an updated encoder and an updated decoder includes: Based on the first predicted value and the second predicted value, the first update loss is calculated. Based on the first update loss, the encoder and the decoder are updated with parameters until the first update loss converges, and the encoder and decoder are preliminarily updated. Based on the similarity, a second update loss is calculated. Based on the second update loss, the parameters of the initially updated encoder and the initially updated decoder are updated until the second update loss converges, thus obtaining the updated encoder and the updated decoder.

6. The construction method according to any one of claims 1 to 5, characterized in that, The construction method also includes: The updated encoder is used to extract features from the hearing impairment test audio to obtain updated audio features. The updated audio features are then fused with randomly generated noise information to obtain an updated fusion result. The updated decoder is used to decode the updated fusion result to obtain updated subtitles. The trained naturalness discriminator is used to predict the probability that the updated subtitles are manually annotated, and a third prediction value is obtained. The trained semantic discriminator is used to predict the semantic relevance between the updated subtitles and the hearing impairment test audio to obtain a fourth prediction value; Based on the third predicted value, the fourth predicted value, and the similarity, the parameters of the trained naturalness discriminator and the trained semantic discriminator are updated to obtain the updated naturalness discriminator and the updated naturalness discriminator. Using the updated naturalness discriminator as the trained naturalness discriminator, the updated semantic discriminator as the trained semantic discriminator, the updated encoder as the encoder, and the updated decoder as the decoder, the preset number of iterations is increased by one, and the initial value of the number of iterations is zero. Return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets the preset value, then stop iterating, and obtain the final updated encoder and the final updated decoder. The final updated encoder and the final updated decoder form the final updated subtitle generation model.

7. The construction method according to claim 6, characterized in that, After increasing the preset number of iterations by one, the method further includes: Calculate the difference between the third predicted value and the first predicted value to obtain the first difference; calculate the difference between the fourth predicted value and the second predicted value to obtain the second difference. When both the first difference and the second difference are less than a preset threshold, it is determined that the preset condition is met; Accordingly, the step of returning to the execution of the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets a preset value, and then stopping the iteration, includes: Return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the preset condition is met or the number of iterations meets the preset value, then stop iterating.

8. A device for constructing a medical auxiliary subtitle generation model, characterized in that, The construction apparatus includes: The feature extraction module is used to acquire hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to perform feature decoding on the fusion result to obtain initial generated subtitles; The first prediction module is used to predict the probability that the initially generated subtitles belong to human annotations using a trained naturalness discriminator, and obtain a first prediction value. The second prediction module is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio using a trained semantic discriminator, and obtain a second prediction value. The parameter update module is used to obtain the tagged subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the tagged subtitles, and update the parameters of the encoder and the decoder according to the first predicted value, the second predicted value and the similarity, so as to obtain the updated encoder and the updated decoder. The model building module is used to form an updated subtitle generation model from the updated encoder and the updated decoder. The updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the construction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the construction method as described in any one of claims 1 to 7.

Images