Construction method and device of subtitle generation model for medical assistance, equipment and medium

Abstract

The invention relates to the technical field of digital medical treatment, in particular to a construction method and device of a subtitle generation model for medical assistance, equipment and a medium. According to the method, an encoder is used for extracting audio features of hearing impairment test audio, the audio features and noise information are fused, a decoder is used for decoding a fusion result to obtain initial generated subtitles, a naturalness discriminator is used for predicting naturalness of the initial generated subtitles to obtain a first predicted value, and the first predicted value is used for generating a second predicted value. Using a semantic discriminator to predict the semantic correlation between the initially generated subtitles and the hearing impairment test audio to obtain a second predicted value, calculating the similarity between the initially generated subtitles and the label subtitles, updating the encoder and the decoder according to the first predicted value, the second predicted value and the similarity, forming a subtitle generation model by the updated encoder and decoder, and generating the subtitles according to the subtitle generation model. Through multi-dimensional supervision, the simulation degree of subtitle generation by the neural network is improved, the subtitle generation has the characteristic of diversification through a noise adding mode, and the audio information acquisition experience of the hearing-impaired patient is improved.

Description

Technical Field

[0001] The present invention relates to the field of digital medical technology, and in particular to a method, device, equipment and medium for constructing a subtitle generation model for medical assistance. Background Art

[0002] The subtitle generation task belongs to the cross-modal translation task, which usually generates corresponding natural language text subtitles based on audio information. Different from the sound recognition task, the subtitle generation task focuses more on the semantic association between audio information and the sound background information in the audio information, such as the sound object, the sound environment, etc. The existing automatic subtitle generation model usually generates fixed subtitles determined by common words and simple grammar based on the given audio information. With the rise of subtitle generation tasks, it can assist hearing-impaired patients in remote video consultations, hearing impairment tests, etc., and convert video or audio information into subtitle text for hearing-impaired patients to read.

[0003] However, different subtitle annotators may annotate the same audio information with different subtitles according to their different attention to the background information and context semantic association when describing the audio information. The automatic subtitle generation model aims to simulate the annotation behavior of subtitle annotators. The existing automatic subtitle generation model generates fixed subtitles, which means that the diversity of the generated subtitles is poor, resulting in poor simulation of the generated subtitles. In other words, it is easy for subtitle viewers to distinguish between manually annotated subtitles and generated subtitles. Therefore, how to improve the diversity of subtitle generation and thus improve the simulation of subtitle generation has become an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method, device, equipment and medium for constructing a subtitle generation model for medical assistance to solve the problem of poor diversity in subtitle generation.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing a medical auxiliary subtitle generation model, the method comprising: Acquire hearing impairment test audio, use an encoder to perform feature extraction on the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to perform feature decoding on the fusion result to obtain initial generated subtitles; Using a trained naturalness discriminator, predicting the probability that the initially generated subtitles are human-annotated, to obtain a first prediction value; Using the trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio, to obtain a second prediction value; Obtaining labeled subtitles of the hearing impairment test audio, calculating the similarity between the initially generated subtitles and the labeled subtitles, and updating parameters of the encoder and the decoder according to the first prediction value, the second prediction value, and the similarity to obtain an updated encoder and an updated decoder; An updated subtitle generation model is formed by the updated encoder and the updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0006] In a second aspect, an embodiment of the present invention provides a device for constructing a medical auxiliary subtitle generation model, the device comprising: A feature extraction module is used to obtain a hearing impairment test audio, use an encoder to perform feature extraction on the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to perform feature decoding on the fusion result to obtain initial generated subtitles; A first prediction module, configured to use a trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated, and obtain a first prediction value; A second prediction module is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio using a trained semantic discriminator to obtain a second prediction value; a parameter updating module, configured to obtain the labeled subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the labeled subtitles, and update the parameters of the encoder and the decoder according to the first predicted value, the second predicted value and the similarity to obtain an updated encoder and an updated decoder; The model building module is used to form an updated subtitle generation model using the updated encoder and the updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0007] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the construction method described in the first aspect when executing the computer program.

[0008] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the construction method described in the first aspect is implemented.

[0009] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: Acquire hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, use a decoder to decode the fusion result to obtain initial generated subtitles, use a trained natural discriminator to predict the probability that the initial generated subtitles are manually labeled to obtain a first prediction value, use a trained semantic discriminator to predict the semantic relevance between the initial generated subtitles and the hearing impairment test audio to obtain a second prediction value, obtain labeled subtitles of the hearing impairment test audio, calculate the similarity between the initial generated subtitles and the labeled subtitles, update the parameters of the encoder and the decoder according to the first prediction value, the second prediction value and the similarity to obtain an updated encoder and an updated decoder, form an updated subtitle generation model with the updated encoder and the updated decoder, perform joint supervision with naturalness, semantic relevance and similarity with labeled subtitles, ensure the simulation degree of the neural network in generating subtitles, make the generated subtitles have diverse characteristics by adding noise, thereby improving the diversity of the generated subtitles while ensuring the simulation degree of the generated subtitles, and thereby improving the experience of hearing-impaired patients in obtaining audio information in text form. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0011] Figure 1 This is a schematic diagram of an application environment of a method for constructing a medical auxiliary subtitle generation model provided in the first embodiment of the present invention; Figure 2 It is a flowchart of a method for constructing a medical auxiliary subtitle generation model provided in the first embodiment of the present invention; Figure 3 It is a structural schematic diagram of a device for constructing a medical auxiliary subtitle generation model provided by the second embodiment of the present invention; Figure 4 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0013] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0014] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0015] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0016] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0018] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0019] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0020] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0021] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0022] A method for constructing a medical auxiliary subtitle generation model provided in the first embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs) and other computer devices. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0023] The above-mentioned client and server can be deployed in medical scenarios to be applied to tasks such as intelligent diagnosis and treatment of hearing-impaired patients and remote consultation.

[0024] See also Figure 2 , is a flow chart of a method for constructing a medical auxiliary subtitle generation model provided in the first embodiment of the present invention. The above construction method can be applied to Figure 1The server in the embodiment of the present invention is connected to the client by a computer device corresponding to the server to obtain the hearing impairment test audio and label subtitles from the client. The hearing impairment test audio and label subtitles can be input into the client by the tester of the hearing impairment test or selected from the client storage content. The tester can refer to a doctor, a nurse, or a hearing-impaired patient. Figure 2 As shown, the construction method may include the following steps: Step S201, obtain hearing impairment test audio, use an encoder to extract features from the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, use a decoder to feature decode the fusion result to obtain initial generated subtitles.

[0025] Among them, the hearing impairment test audio may refer to a training sample used to construct a subtitle generation model for assisting hearing-impaired patients in obtaining audio information. The hearing impairment test audio may be audio information carried by a video or separate audio information. The encoder may be used to extract features from the input audio information. The audio features may be used to characterize feature information of the input audio information. The noise information may be randomly generated noise that conforms to a Gaussian distribution, that is, white noise. The fusion result may refer to the audio features after adding noise. The decoder may be used to decode the audio features after adding noise into text information. The decoded text information is the initially generated subtitles.

[0026] Specifically, in this embodiment, the encoder adopts multiple serially connected convolutional neural network structures, each convolutional neural network structure may include at least one convolutional block, a single convolutional block may include at least one convolutional layer, the output of each convolutional layer needs to be processed by a batch normalization layer and an activation function layer in sequence, the activation function adopted by the activation function layer may be a ReLU activation function, the output of each convolutional block needs to be processed by a pooling layer, the pooling layer may adopt a maximum pooling layer, in particular, the output of the last convolutional block needs to be processed by a global average pooling layer, the output of the global average pooling layer may be processed using a multilayer perceptron to obtain audio features, the multilayer perceptron may include at least two linear layers, and there is an activation function layer between any two adjacent linear layers.

[0027] In order to reduce the demand for the number of hearing impairment test audios for parameter updates, the encoder can be pre-trained in advance. An unsupervised approach can be used for pre-training, such as adding an additional decoder and jointly using the encoder for pre-training using a reconstruction task. In this case, training can be performed using the input samples as labels. A supervised approach can also be used, such as adding an additional classifier and jointly using the encoder for pre-training using a simple downstream task. The downstream task can be an audio speaker gender classification task, an audio speaker age group classification task, etc. The use of simple downstream tasks can effectively reduce the pre-training annotation cost, and at the same time enable the pre-trained encoder to have a certain feature extraction capability.

[0028] The above steps include obtaining hearing impairment test audio, using an encoder to extract features from the hearing impairment test audio to obtain audio features, fusing the audio features with randomly generated noise information to obtain a fusion result, and using a decoder to feature decode the fusion result to obtain initial subtitle generation. By adding noise, the initial generated subtitles have diversity, providing rich initial generated subtitles for subsequent model parameter updates, so as to determine the update direction according to the initial generated subtitles, thereby improving the performance of the updated model.

[0029] Step S202: Use the trained naturalness discriminator to predict the probability that the initially generated subtitles are human-annotated to obtain a first prediction value.

[0030] The trained naturalness discriminator can be used to identify whether the input subtitles are manually annotated, and the first prediction value can represent the possibility that the input subtitles are manually annotated.

[0031] Specifically, the naturalness discriminator can be regarded as a two-classification task, one classification category is that the input subtitles are manually annotated, and the other classification category is that the input subtitles are not manually annotated. The trained naturalness discriminator can be trained using a training set that is a mixture of manually annotated subtitles and machine-generated subtitles.

[0032] Optionally, the trained naturalness discriminator is used to predict the probability that the initially generated subtitles are human-annotated to obtain a first prediction value, including: The initially generated subtitles are segmented into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles, and all subtitle terms are spliced ​​into a term sequence in order of position; The trained gated recurrent unit is used to predict the probability that the word sequence belongs to human annotation to obtain a first prediction value.

[0033] Among them, the subtitle terms may refer to the terms in the initially generated subtitles, and the positions may be represented by serial numbers. For example, all subtitle terms in the initially generated subtitles are assigned serial numbers from 0 to N in order from left to right, N+1 is the number of terms, and the term sequence may include N+1 subtitle terms arranged according to positions. The trained gated recurrent unit may be used to extract term features of the term sequence.

[0034] Specifically, according to the position order, the input sequence number is initialized to 0, the subtitle term corresponding to the input sequence number is input into the trained gated recurrent unit to obtain the corresponding term feature, the term feature is used as the input feature, the input sequence number is increased by 1, the input feature and the subtitle term corresponding to the input sequence number are input into the trained gated recurrent unit to obtain the corresponding term feature, the term feature is used as the input feature, the input sequence number is increased by 1, and the step of inputting the subtitle term corresponding to the input feature and the input sequence number into the trained gated recurrent unit to obtain the corresponding term feature, the term feature is used as the input feature, and the input sequence number is increased by 1 is returned to execute, until the input sequence number is N+1, the iteration is stopped, and the final output term feature is obtained, the final output term feature is processed by a multilayer perceptron, and the processing result is mapped by a sigmoid function to obtain a first prediction value.

[0035] In this embodiment, feature extraction of the initially generated subtitles is achieved through a gated recurrent unit, which can fully extract the context information of the initially generated subtitles, improve the accuracy of naturalness identification, and further improve the subtitle generation capability of the encoder and decoder that are updated with reference to the naturalness identification results, thereby improving the simulation degree of subtitle generation.

[0036] The above-mentioned step of using the trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated to obtain a first prediction value uses the first prediction value as supervision information for the subsequent model parameter update process, thereby ensuring that the subtitle generation model can generate subtitles that meet naturalness, thereby improving the simulation degree of subtitle generation.

[0037] Step S203: Use the trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing-impaired test audio to obtain a second prediction value.

[0038] Among them, the trained semantic identifier can be used to identify whether the semantics of the initially generated subtitles and the hearing impairment test audio are similar, and the second prediction value can represent the degree of semantic consistency between the initially generated subtitles and the hearing impairment test audio.

[0039] Optionally, the semantic relevance between the initially generated subtitles and the hearing-impaired test audio is predicted using a trained semantic discriminator to obtain a second prediction value, including: Using an audio encoder with fixed parameters, the semantic features of the hearing impairment test audio are extracted to obtain audio semantic features; Use the trained gated recurrent unit to extract the semantic features of the initially generated subtitles to obtain the subtitle semantic features; The similarity between the audio semantic feature and the subtitle semantic feature is calculated to obtain a similarity calculation result, and the similarity calculation result is used as the second prediction value.

[0040] Among them, the audio encoder can be used to extract the semantic features of the hearing impairment test audio, the audio semantic features can be used to characterize the semantic information of the hearing impairment test audio, the trained gated recurrent unit can be used to extract the semantic features of the initial generated subtitles, the subtitle semantic features can be used to characterize the semantic information of the initial generated subtitles, and the similarity calculation results can characterize the similarity between the semantics of the initial generated subtitles and the hearing impairment test audio.

[0041] Specifically, the audio encoder can be implemented using an audio neural network. It should be noted that the audio encoder is trained and has fixed parameters, and the sizes of the audio semantic features and the subtitle semantic features are consistent.

[0042] The audio semantic features and the subtitle semantic features are respectively embedded into the shared embedding space through a multi-layer perceptron to obtain the audio embedding corresponding to the audio semantic features and the subtitle embedding corresponding to the subtitle semantic features, and the similarity between the audio embedding and the subtitle embedding is calculated. The similarity can be calculated using Euclidean distance, cosine similarity, etc. In this embodiment, cosine similarity is used to calculate the similarity, and the obtained similarity calculation result has a value range of [-1, 1]. The closer the similarity calculation result is to 1, the more similar the audio embedding and the subtitle embedding are. The closer the similarity calculation result is to -1, the less similar the audio embedding and the subtitle embedding are. In this embodiment, the ReLU function is used to map the similarity calculation result to the range of [0, 1], so that the similarity calculation result can be regarded as a probability, that is, the probability of semantic consistency between the initially generated subtitles and the hearing impairment test audio, and the similarity calculation result of the audio embedding and the subtitle embedding is used as the similarity calculation result between the audio semantic features and the subtitle semantic features.

[0043] In this embodiment, feature extraction of initially generated subtitles is implemented through a gated recurrent unit, which can fully extract context information of the initially generated subtitles, improve the accuracy of extracting semantic features of subtitles, and use the similarity between audio semantic features and subtitle semantic features as semantic identification results to improve the subtitle generation capability of the encoder and decoder that are updated with reference to the semantic identification results, thereby improving the semantic accuracy of subtitle generation.

[0044] The above-mentioned step of using the trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain the second prediction value uses the first prediction value as the supervision information for the subsequent model parameter update process, thereby ensuring that the subtitle generation model can generate subtitles that meet the audio semantics, thereby improving the semantic accuracy of subtitle generation.

[0045] Step S204, obtain the labeled subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the labeled subtitles, and update the parameters of the encoder and decoder according to the first prediction value, the second prediction value and the similarity to obtain an updated encoder and an updated decoder.

[0046] The labeled subtitles may refer to reference subtitles annotated manually, and the similarity may be used to represent the degree of similarity between the initially generated subtitles and the labeled subtitles.

[0047] Optionally, the similarity between the initially generated subtitles and the labeled subtitles is calculated, including: Performing word segmentation on the initially generated subtitles to obtain at least two subtitle word items and their positions in the initially generated subtitles, and splicing all the subtitle word items into a first word item sequence in order of position; The label subtitles are segmented into terms to obtain at least two label terms and their positions in the label subtitles, and all the label terms are spliced ​​into a second term sequence in order of position; The similarity between the first word sequence and the second word sequence is calculated, and the calculated result is used as the similarity between the initially generated subtitles and the labeled subtitles.

[0048] The first term sequence may include all subtitle terms arranged by position, and the second term sequence may include all tag terms arranged by position.

[0049] Specifically, the similarity between the first term sequence and the second term sequence can be calculated using cosine similarity, which mainly measures whether the terms used in the first term sequence and the second term sequence are similar, without involving semantic information of the subtitle text.

[0050] In this embodiment, the similarity of word usage is measured so that the initial generated subtitles are evaluated based on a fixed objective reference, thereby improving the subtitle generation capabilities of the encoder and decoder that are updated based on the similarity, thereby further improving the simulation level of subtitle generation.

[0051] Optionally, updating parameters of the encoder and the decoder according to the first prediction value, the second prediction value and the similarity to obtain an updated encoder and an updated decoder includes: A first update loss is calculated according to the first prediction value and the second prediction value, and parameters of the encoder and the decoder are updated according to the first update loss until the first update loss converges, thereby obtaining a preliminarily updated encoder and a preliminarily updated decoder; According to the similarity, a second updated loss is calculated, and according to the second updated loss, parameters of the preliminarily updated encoder and the preliminarily updated decoder are updated until the second updated loss converges to obtain an updated encoder and an updated decoder.

[0052] Among them, the first update loss can be used to guide the encoder and decoder to update parameters, and the second update loss can be used to guide the encoder and decoder that have been preliminarily updated to update parameters.

[0053] Specifically, the similarity provides feedback to the model update in the form of a regular label, so that based on the feedback of the first prediction value and the second prediction value, certain label information can be learned. However, since the labels are usually fixed, a two-stage training method is adopted. That is, the first stage training is performed based on the first prediction value and the second prediction value, and after the first stage training is completed, the second stage training is performed based on the similarity, so that the similarity will not affect the feedback of the first prediction value and the second prediction value.

[0054] In this embodiment, a two-stage training method is adopted so that the training processes will not interfere with each other, and naturalness, semantic relevance and label information are taken into consideration, thereby improving the simulation degree of subtitle generation.

[0055] The above steps of obtaining labeled subtitles of the hearing impairment test audio, calculating the similarity between the initially generated subtitles and the labeled subtitles, updating the parameters of the encoder and the decoder according to the first prediction value, the second prediction value and the similarity, and obtaining an updated encoder and an updated decoder, jointly supervised by naturalness, semantic relevance and similarity with the labeled subtitles, ensure the degree of simulation of the neural network in generating subtitles.

[0056] Step S205: An updated subtitle generation model is formed by the updated encoder and the updated decoder.

[0057] Among them, the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0058] Optionally, the build method also includes: Use the updated encoder to extract features from the hearing-impaired test audio to obtain updated audio features, fuse the updated audio features with randomly generated noise information to obtain updated fusion results, and use the updated decoder to feature decode the updated fusion results to obtain updated generated subtitles; Use the trained naturalness discriminator to predict the probability that the updated subtitles are manually annotated, and obtain a third prediction value; Using the trained semantic discriminator to predict the semantic relevance between the updated subtitles and the hearing-impaired test audio, obtaining a fourth prediction value; According to the third prediction value, the fourth prediction value and the similarity, the parameters of the trained naturalness discriminator and the trained semantic discriminator are updated to obtain an updated naturalness discriminator and an updated naturalness discriminator; Using the updated natural discriminator as the trained natural discriminator, using the updated semantic discriminator as the trained semantic discriminator, using the updated encoder as the encoder, using the updated decoder as the decoder, increasing the preset number of iterations by one, and the initial value of the number of iterations is zero; Return to executing the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets the preset value, stop the iteration, and obtain the final updated encoder and the final updated decoder. The final updated encoder and the final updated decoder form a final updated subtitle generation model.

[0059] Among them, the third prediction value can represent the possibility that the updated subtitles are manually marked, and the fourth prediction value can represent the possibility that the updated generated subtitles are semantically consistent with the hearing impairment test audio.

[0060] Specifically, the preset value may be set to 3, and the implementer may adjust the preset value according to actual conditions.

[0061] In this embodiment, the idea of ​​adversarial network training is adopted to train the subtitle generation model composed of an encoder and a decoder and the identification model composed of a naturalness discriminator and a semantic discriminator in sequence, so that the generation ability of the subtitle generation model and the identification ability of the identification model can be further improved, thereby improving the simulation degree and fidelity of the generated subtitles.

[0062] Optionally, after increasing the preset number of iterations by one, the method further includes: Calculate the difference between the third predicted value and the first predicted value to obtain a first difference, and calculate the difference between the fourth predicted value and the second predicted value to obtain a second difference; When both the first difference and the second difference are smaller than a preset threshold, it is determined that the preset condition is met; Accordingly, returning to the step of extracting features from the acquired hearing impairment test audio using the encoder to obtain audio features, until the number of iterations meets the preset value, the iteration is stopped, including: Return to executing the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until a preset condition is met or the number of iterations meets a preset value, and then stop iterating.

[0063] Among them, the first difference can represent the difference between the third prediction value and the first prediction value, the second difference can represent the difference between the fourth prediction value and the second prediction value, the preset threshold can measure the change in naturalness and semantic relevance, and the preset condition can be used as a condition for stopping iteration.

[0064] Specifically, when both the first difference and the second difference are smaller than the preset threshold, it indicates that the generation capability of the subtitle generation model and the identification capability of the identification model are approximately balanced, and the model effect of one of them cannot be improved through adversarial training.

[0065] In this embodiment, by adding the preset condition that both the first difference and the second difference are smaller than the preset threshold, the iteration can be stopped in time when the adversarial training effect deteriorates, thereby improving the efficiency of model updating.

[0066] The above-mentioned step of forming an updated subtitle generation model by using an updated encoder and an updated decoder, and performing subtitle generation by using the updated subtitle generation model, can generate subtitles with strong simulation as much as possible while ensuring that the semantics are consistent with the audio information.

[0067] In this embodiment, naturalness, semantic relevance and similarity with label subtitles are used for joint supervision to ensure the degree of simulation of subtitle generation by the neural network. The generated subtitles are made diverse by adding noise, thereby improving the diversity of generated subtitles while ensuring the degree of simulation of the generated subtitles, thereby improving the experience of hearing-impaired patients in obtaining audio information in text form.

[0068] Corresponding to the method for constructing a medical auxiliary subtitle generation model in the above embodiment, Figure 3 The structural block diagram of the construction device of the medical assistance subtitle generation model provided by the second embodiment of the present invention is shown. The construction device is applied to the server. The computer device corresponding to the server is connected to the client to obtain the hearing impairment test audio and label subtitles from the client. The hearing impairment test audio and label subtitles can be input into the client by the tester of the hearing impairment test or selected from the client storage content. The tester can refer to a doctor, a nurse and a hearing-impaired patient. For the convenience of explanation, only the part related to the embodiment of the present invention is shown.

[0069] See also Figure 3 , the construction device comprises: The feature extraction module 31 is used to obtain the hearing impairment test audio, use the encoder to extract the features of the hearing impairment test audio to obtain audio features, fuse the audio features with the randomly generated noise information to obtain a fusion result, and use the decoder to feature decode the fusion result to obtain initial generated subtitles; A first prediction module 32 is used to use the trained naturalness discriminator to predict the probability that the initially generated subtitles are human-annotated, and obtain a first prediction value; A second prediction module 33 is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio using the trained semantic discriminator to obtain a second prediction value; The parameter updating module 34 is used to obtain the labeled subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the labeled subtitles, and update the parameters of the encoder and the decoder according to the first prediction value, the second prediction value and the similarity to obtain an updated encoder and an updated decoder; The model building module 35 is used to form an updated subtitle generation model using an updated encoder and an updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

[0070] Optionally, the first prediction module 32 includes: A first term segmentation unit is used to segment the initially generated subtitles into terms, obtain at least two subtitle terms and their positions in the initially generated subtitles, and splice all subtitle terms into a term sequence in order of position; The artificial probability prediction unit is used to use the trained gated recurrent unit to predict the probability that the word sequence belongs to the artificial annotation to obtain a first prediction value.

[0071] Optionally, the second prediction module 33 includes: An audio feature extraction unit, used to extract semantic features of the hearing impairment test audio using an audio encoder with fixed parameters to obtain audio semantic features; A subtitle feature extraction unit, used to extract semantic features of initially generated subtitles using a trained gated recurrent unit to obtain subtitle semantic features; The similarity calculation unit is used to calculate the similarity between the audio semantic feature and the subtitle semantic feature, obtain a similarity calculation result, and use the similarity calculation result as the second prediction value.

[0072] Optionally, the parameter updating module 34 includes: A second term segmentation unit is used to segment the initially generated subtitles into terms, obtain at least two subtitle terms and their positions in the initially generated subtitles, and splice all the subtitle terms into a first term sequence in order of position; A third term segmentation unit is used to segment the label subtitle into terms to obtain at least two label terms and their positions in the label subtitle, and to splice all the label terms into a second term sequence in order of their positions; The similarity determination unit is used to calculate the similarity between the first word sequence and the second word sequence, and use the calculation result as the similarity between the initially generated subtitles and the label subtitles.

[0073] Optionally, the parameter updating module 34 includes: A first parameter updating unit is used to calculate a first update loss according to the first prediction value and the second prediction value, and update parameters of the encoder and the decoder according to the first update loss until the first update loss converges to obtain a preliminarily updated encoder and a preliminarily updated decoder; The second parameter updating unit is used to calculate a second updated loss according to the similarity, and update the parameters of the preliminarily updated encoder and the preliminarily updated decoder according to the second updated loss until the second updated loss converges to obtain an updated encoder and an updated decoder.

[0074] Optionally, the construction device further comprises: An updated subtitle generation module is used to extract features from the hearing impairment test audio using an updated encoder to obtain updated audio features, fuse the updated audio features with randomly generated noise information to obtain an updated fusion result, and use an updated decoder to feature decode the updated fusion result to obtain updated generated subtitles; A naturalness identification module, used to use the trained naturalness identifier to predict the probability that the updated subtitles are manually annotated, and obtain a third prediction value; A relevance identification module, used to predict the semantic relevance between the updated subtitles and the hearing impairment test audio using the trained semantic identifier to obtain a fourth prediction value; A discriminator updating module, used for updating parameters of the trained naturalness discriminator and the trained semantic discriminator according to the third prediction value, the fourth prediction value and the similarity, to obtain an updated naturalness discriminator and an updated naturalness discriminator; an iteration item determination module, for using the updated naturalness discriminator as the trained naturalness discriminator, the updated semantic discriminator as the trained semantic discriminator, the updated encoder as the encoder, and the updated decoder as the decoder, and increasing a preset number of iterations by one, wherein the initial value of the number of iterations is zero; The iterative execution module is used to return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets the preset value, stop the iteration, and obtain the final updated encoder and the final updated decoder. The final updated encoder and the final updated decoder form a final updated subtitle generation model.

[0075] Optionally, the construction device further comprises: A difference calculation module, used for calculating the difference between the third prediction value and the first prediction value to obtain the first difference, and calculating the difference between the fourth prediction value and the second prediction value to obtain the second difference; A condition determination module, configured to determine that a preset condition is satisfied when both the first difference and the second difference are less than a preset threshold; Accordingly, the above-mentioned iterative execution module includes: The conditional judgment unit is used to return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, and stop iterating until a preset condition is met or the number of iterations meets a preset value.

[0076] It should be noted that the information interaction, execution process and other contents between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0077] Figure 4 This is a schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned construction method embodiments are implemented.

[0078] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0079] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0080] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0081] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0082] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0083] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0084] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0085] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for constructing a subtitle generation model for medical assistance, characterized in that: The construction method comprises: Acquire hearing impairment test audio, use an encoder to perform feature extraction on the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to perform feature decoding on the fusion result to obtain initial generated subtitles; Using a trained naturalness discriminator, predicting the probability that the initially generated subtitles are human-annotated, to obtain a first prediction value; Using the trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio, to obtain a second prediction value; Obtaining labeled subtitles of the hearing impairment test audio, calculating the similarity between the initially generated subtitles and the labeled subtitles, and updating parameters of the encoder and the decoder according to the first prediction value, the second prediction value, and the similarity to obtain an updated encoder and an updated decoder; An updated subtitle generation model is formed by the updated encoder and the updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

2. The construction method according to claim 1, characterized in that: The method of using the trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated to obtain a first prediction value includes: Segmenting the initially generated subtitles into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles, and splicing all subtitle terms into a term sequence in order of position; The trained gated recurrent unit is used to predict the probability that the word sequence is manually labeled to obtain the first prediction value.

3. The construction method according to claim 1, characterized in that: The method of using the trained semantic discriminator to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio to obtain a second prediction value includes: Using an audio encoder with fixed parameters, extracting semantic features of the hearing impairment test audio to obtain audio semantic features; Using the trained gated recurrent unit, extracting semantic features of the initially generated subtitles to obtain subtitle semantic features; The similarity between the audio semantic feature and the subtitle semantic feature is calculated to obtain a similarity calculation result, and the similarity calculation result is used as the second prediction value.

4. The construction method according to claim 1, characterized in that: The calculating the similarity between the initially generated subtitles and the labeled subtitles includes: Segmenting the initially generated subtitles into terms to obtain at least two subtitle terms and their positions in the initially generated subtitles, and splicing all subtitle terms into a first term sequence in order of position; The label subtitle is segmented into terms to obtain at least two label terms and their positions in the label subtitle, and all the label terms are spliced ​​into a second term sequence in order of position; The similarity between the first word sequence and the second word sequence is calculated, and the calculated result is used as the similarity between the initially generated subtitles and the labeled subtitles.

5. The construction method according to claim 1, characterized in that: The updating of parameters of the encoder and the decoder according to the first prediction value, the second prediction value and the similarity to obtain an updated encoder and an updated decoder includes: A first update loss is calculated according to the first prediction value and the second prediction value, and parameters of the encoder and the decoder are updated according to the first update loss until the first update loss converges, thereby obtaining a preliminarily updated encoder and a preliminarily updated decoder; According to the similarity, a second update loss is calculated, and according to the second update loss, parameters of the initially updated encoder and the initially updated decoder are updated until the second update loss converges to obtain the updated encoder and the updated decoder.

6. The construction method according to any one of claims 1 to 5, characterized in that: The construction method also includes: Using the updated encoder to extract features from the hearing impairment test audio to obtain updated audio features, fusing the updated audio features with randomly generated noise information to obtain updated fusion results, and using the updated decoder to feature decode the updated fusion results to obtain updated generated subtitles; Using the trained naturalness discriminator, predicting the probability that the updated subtitles are manually annotated, to obtain a third prediction value; Using the trained semantic discriminator to predict the semantic relevance between the updated subtitles and the hearing impairment test audio, to obtain a fourth prediction value; According to the third prediction value, the fourth prediction value and the similarity, updating the parameters of the trained naturalness discriminator and the trained semantic discriminator to obtain an updated naturalness discriminator and an updated naturalness discriminator; Using the updated natural discriminator as the trained natural discriminator, using the updated semantic discriminator as the trained semantic discriminator, using the updated encoder as the encoder, using the updated decoder as the decoder, increasing the preset number of iterations by one, the initial value of the number of iterations being zero; Return to executing the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, until the number of iterations meets the preset value, stop iteration, and obtain a final updated encoder and a final updated decoder, and form a final updated subtitle generation model with the final updated encoder and the final updated decoder.

7. The construction method according to claim 6, characterized in that: After increasing the preset number of iterations by one, the method further includes: Calculating a difference between the third predicted value and the first predicted value to obtain a first difference, and calculating a difference between the fourth predicted value and the second predicted value to obtain a second difference; When both the first difference and the second difference are smaller than a preset threshold, determining that a preset condition is met; Accordingly, the step of returning to perform the step of extracting features from the acquired hearing impairment test audio using the encoder to obtain audio features, until the number of iterations meets a preset value, and stopping the iteration, includes: Return to the step of using the encoder to extract features from the acquired hearing impairment test audio to obtain audio features, and stop iterating until the preset condition is met or the number of iterations meets the preset value.

8. A device for constructing a medical auxiliary subtitle generation model, characterized in that: The construction device comprises: A feature extraction module is used to obtain a hearing impairment test audio, use an encoder to perform feature extraction on the hearing impairment test audio to obtain audio features, fuse the audio features with randomly generated noise information to obtain a fusion result, and use a decoder to perform feature decoding on the fusion result to obtain initial generated subtitles; A first prediction module, configured to use a trained naturalness discriminator to predict the probability that the initially generated subtitles are manually annotated, and obtain a first prediction value; A second prediction module is used to predict the semantic relevance between the initially generated subtitles and the hearing impairment test audio using a trained semantic discriminator to obtain a second prediction value; a parameter updating module, configured to obtain the labeled subtitles of the hearing impairment test audio, calculate the similarity between the initially generated subtitles and the labeled subtitles, and update the parameters of the encoder and the decoder according to the first predicted value, the second predicted value and the similarity to obtain an updated encoder and an updated decoder; The model building module is used to form an updated subtitle generation model using the updated encoder and the updated decoder, and the updated subtitle generation model is used to generate subtitles for real hearing-impaired audio.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the construction method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the construction method according to any one of claims 1 to 7 is implemented.

Images