Error correction method and device for converting medical voice into text, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a medical voice-to-text error correction method, device and equipment and a medium, and the method comprises the steps: constructing a multi-modal medical voice model, initializing the multi-modal medical voice model according to the parameters of a visual encoder and a text encoder of a visual language pre-training model, and carrying out the pre-training of the multi-modal medical voice model; receiving the paired medical image and the text report after error simulation to train a multi-modal medical voice model; constructing a medical voice-to-text error correction model according to the voice-to-text model and the trained multi-mode medical voice model, and receiving to-be-corrected voice and a medical image corresponding to the voice to train the medical voice-to-text error correction model; and performing medical voice-to-text error correction on the input medical voice by adopting the trained medical voice-to-text error correction model. And the medical voice-to-text conversion effect is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology, speech processing and medical health technology, and in particular to a medical speech-to-text error correction method, device, equipment and medium. Background Art

[0002] In the prior art, the Automatic Speech Recognition (ASR) system in the medical field can be applied to medical diagnosis, remote consultation and medical data recording, etc. Medical data includes personal health records, prescriptions, examination reports, medical images, etc. However, when converting speech to text, due to the complexity and professionalism of medical terminology, recognition errors often occur and the accuracy is limited, especially in the face of complex speech conditions such as noise and accents. Summary of the invention

[0003] The present invention provides a medical speech-to-text error correction method, device, computer equipment and medium to solve the technical problems of inaccurate speech recognition and poor recognition effect.

[0004] In a first aspect, a medical speech-to-text error correction method is provided, comprising:

[0005] Construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0006] A medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model;

[0007] The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

[0008] In a second aspect, a medical speech-to-text error correction device is provided, comprising:

[0009] A multimodal medical speech model construction module is used to construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0010] A medical speech-to-text error correction model construction module is used to construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0011] The error correction module is used to perform medical speech-to-text error correction on the input medical speech using the trained medical speech-to-text error correction model.

[0012] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned medical speech-to-text error correction method when executing the computer program.

[0013] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned medical speech-to-text error correction method are implemented.

[0014] In the scheme implemented by the above-mentioned medical speech-to-text error correction method, device, computer equipment and storage medium, a multimodal medical speech model can be constructed, the multimodal medical speech model can be initialized according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and the paired medical images and the text report after error simulation are received to train the multimodal medical speech model; a medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model; the trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech. In the present invention, for medical data records under medical and health services, especially in radiology In scenarios such as medicine that require a large amount of voice transcription, a multimodal medical voice model can be used to extract features from paired medical images and text reports and generate corresponding texts. The multimodal medical voice model is initialized according to the parameters of the visual language pre-training model to utilize the semantic association between the paired medical images and text reports to correct text errors. Furthermore, training the multimodal medical voice model with text reports after error simulation can improve the model's error correction capability. The medical voice-to-text error correction model obtains the corresponding text by performing voice content recognition on the input voice through the voice-to-text model, and cooperates with the multimodal medical voice model to realize medical voice-to-text error correction of the input medical voice, thereby improving the accuracy of medical voice-to-text conversion and the conversion effect of medical voice-to-text. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0016] Figure 1 2 is a schematic diagram of an application environment of a medical speech-to-text error correction method according to an embodiment of the present invention;

[0017] Figure 2 It is a flow chart of a medical speech-to-text error correction method in one embodiment of the present invention;

[0018] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0019] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0020] Figure 5 It is a structural schematic diagram of a medical speech-to-text error correction device in one embodiment of the present invention;

[0021] Figure 6 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0022] Figure 7 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The medical speech-to-text error correction method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. The server can receive input medical speech through the client, and the server can communicate with the database to obtain paired medical images and text reports, and obtain paired medical images and text reports after error simulation based on the paired medical images and text reports, train a multimodal medical speech model initialized by parameters of a visual encoder and a text encoder according to a visual language pre-training model, construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, process the paired medical images and text reports to obtain speech to be corrected and its corresponding medical image, train the medical speech-to-text error correction model, use the trained medical speech-to-text error correction model to perform medical speech-to-text error correction on the input medical speech, and feed the final corrected text back to the client. In the present invention, for the medical and health industry, Medical data records under the service, especially in the scene that needs a lot of voice transcription such as radiology, can use the multimodal medical voice model to extract the features of the paired medical images and text reports and generate the corresponding text, and the multimodal medical voice model is initialized according to the parameters of the visual language pre-training model to use the semantic relevance between the paired medical images and text reports to correct text errors, and the multimodal medical voice model is trained with the text report after the error simulation to improve the error correction ability of the model, and the medical voice-to-text error correction model obtains the corresponding text by voice content recognition of the input voice through the voice-to-text model, and cooperates with the multimodal medical voice model to realize the medical voice-to-text error correction of the input medical voice, improve the accuracy of medical voice-to-text, and improve the conversion effect of medical voice-to-text. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0025] See also Figure 2 As shown, Figure 2 A flowchart of a medical speech-to-text error correction method provided by an embodiment of the present invention includes the following steps:

[0026] S10: Construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model.

[0027] Among them, the multimodal medical speech model includes a visual encoder and a text encoder to extract corresponding image features and text features. The multimodal medical speech model also includes a text decoder to correct erroneous content in the text through the content contained in the image. The multimodal medical speech model is trained to strengthen the model to generate corrected speech transcription text through the semantic information of the image modality. The multimodal medical speech model is initialized according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and the visual encoder and the text encoder of the multimodal medical speech model share the parameters of the visual encoder and the text encoder of the visual language pre-training model. The text report after error simulation refers to the text report obtained after the text report corresponding to the medical image is error-modified to simulate the text report generated when the speech recognition error occurs. The multimodal medical speech model is trained using the text report after error simulation, which is beneficial to improving the error correction ability of the model. Among them, if Figure 3 As shown, in step S10, that is, constructing a multimodal medical speech model, initializing the multimodal medical speech model according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, receiving paired medical images and text reports after error simulation to train the multimodal medical speech model, includes the following steps:

[0028] S11: Build a multimodal medical speech model, initialize the visual encoder of the multimodal medical speech model according to the parameters of the visual encoder of the visual language pre-training model, and initialize the text encoder of the multimodal medical speech model according to the parameters of the text encoder of the visual language pre-training model.

[0029] S12: receiving input paired medical images and text reports after error simulation to a multimodal medical speech model.

[0030] S13: Using the visual encoder of the multimodal medical speech model to extract medical image embedding features from the received medical image.

[0031] S14: Use the text encoder of the multimodal medical speech model to extract text embedding features from the received error simulated text report.

[0032] S15: An alternating cross-attention mechanism is used to fuse medical image embedding features and text embedding features to obtain multimodal text representation.

[0033] S16: A text decoder using a multimodal medical speech model generates a corrected text report based on the multimodal text representation.

[0034] S17: Calculate the loss based on the corrected text report and the real text report before the error simulation, and optimize the multimodal medical speech model based on the calculated loss.

[0035] Among them, the optimized multimodal medical speech model can also use AdamW optimizer, warmup cosineannealing scheduler and other technologies to optimize model parameters. In step S17, that is, according to the corrected text report and the real text report before the error simulation, the loss is calculated. The loss calculated according to the corrected text report and the real text report before the error simulation in the multimodal medical speech model can be specifically:

[0036] The cross entropy loss function is used to calculate the loss of each word in the corrected text report and the real text report before the error simulation, so as to minimize the loss and improve the accuracy of the model.

[0037] S20: Construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model.

[0038] After training the multimodal medical speech model, in order to accurately correct the errors in speech transcription, a medical speech-to-text error correction model including a speech-to-text model and a multimodal medical speech model needs to be constructed, and the medical speech-to-text error correction model needs to be trained to apply the multimodal medical speech model to actual speech recognition, and the error correction ability and accuracy of the model need to be enhanced through training. The speech-to-text model is non-specific and does not require additional adjustments, so it has wide applicability.

[0039] In some embodiments of the invention, Figure 4 As shown, a specific solution for training a medical speech-to-text error correction model is provided. In S20, a medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model, which specifically includes the following steps S21-S25:

[0040] S21: Construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model.

[0041] S22: Receive the speech to be corrected and its corresponding medical image.

[0042] S23: Using a speech-to-text model to receive the speech to be corrected and recognize the speech content to obtain an initial transcription text.

[0043] S24: Using a multimodal medical speech model to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image.

[0044] Among them, step S24, that is, using the multimodal medical speech model to obtain a multimodal text representation according to the obtained initial transcription text and the corresponding medical image, specifically includes:

[0045] A visual encoder of a multimodal medical speech model is used to extract medical image embedding features from a medical image corresponding to the received initial transcription text;

[0046] A text encoder using a multimodal medical speech model extracts text embedding features from the received initial transcription text;

[0047] The alternating cross-attention mechanism is used to fuse the medical image embedding features and text embedding features to obtain multimodal text representation.

[0048] S25: A multimodal medical speech model is used to generate a corrected text report according to the obtained multimodal text representation to correct the initial transcription text and obtain a final corrected text.

[0049] Among them, step S25, that is, using the multimodal medical speech model to generate a corrected text report according to the obtained multimodal text representation to correct the initial transcription text to obtain the final corrected text, is specifically:

[0050] A text decoder using a multimodal medical speech model generates a corrected text report according to the obtained multimodal text representation, and corrects the initial transcription text according to the generated corrected text report to obtain a final corrected text.

[0051] By constructing a medical speech-to-text error correction model that includes a speech-to-text model and a multimodal medical speech model, and training the medical speech-to-text error correction model, the multimodal medical speech model can be applied to actual speech recognition, and the error correction ability and accuracy of the model can be enhanced through training.

[0052] S30: Using the trained medical speech-to-text error correction model to perform medical speech-to-text error correction on the input medical speech, so as to obtain the accurate text corresponding to the input medical speech and improve the accuracy and effect of the conversion.

[0053] Specifically, before step S10, that is, before building the multimodal medical speech model, the process further includes:

[0054] Receive paired medical images and text reports to train a visual language pre-trained model;

[0055] A visual encoder using a visual language pre-training model extracts medical image features from a received medical image; wherein the visual encoder can use a ViT architecture to extract image features.

[0056] A text encoder using a visual language pre-training model extracts text features from a received text report; wherein the text encoder may use a Transformer architecture to extract text features.

[0057] An alternating cross-attention mechanism is used to fuse medical image features and text features to obtain multimodal representation.

[0058] Among them, the trained visual language pre-training model can also use AdamW optimizer, warmup cosineannealing scheduler and other technologies to optimize model parameters. By training the visual language pre-training model, the parameters of the visual language pre-training model are optimized, and then the parameters of the text encoder and visual encoder of the visual language pre-training model are optimized, so as to optimize the parameters of the text encoder and visual encoder of the multimodal medical speech model that shares the parameters of the text encoder and visual encoder of the visual language pre-training model.

[0059] Specifically, after adopting the alternating cross attention mechanism to fuse the medical image features and the text features to obtain the multimodal representation, the method further includes:

[0060] Cross-modal contrastive learning and intra-modal contrastive learning are performed on the obtained multimodal representations to improve the model's ability to understand image and text semantics.

[0061] Among them, the modality may include images, voice and text. Multimodality refers to at least two modalities. Intra-modality contrastive learning refers to contrastive learning within the same modality, which is to calculate the similarity of different samples of the same modality to increase the distance between different samples and reduce the distance between the same samples. The samples can be medical images or text reports. The samples of image modality are medical images, and the samples of text modality are text reports. Cross-modality contrastive learning refers to multimodal contrastive learning, which is to calculate the similarity of multimodal paired data and increase the distance between different multimodal paired data. Images and corresponding texts, voices and corresponding texts are different multimodal paired data.

[0062] Specifically, before step S10, that is, before building the multimodal medical speech model, the process further includes:

[0063] Collect medical images and their corresponding text reports; the medical images and their corresponding text reports are collected by the server from the database. The medical images and their corresponding text reports can be chest X-rays and their corresponding text reports obtained from a public medical image database, or real medical images and doctor's oral voice records obtained from a hospital database. The medical image database can be MIMIC-CXR.

[0064] Randomly delete words, replace words and / or insert words in the text report corresponding to the medical image to obtain a text report after error simulation to simulate speech recognition errors and improve the error correction ability of the model.

[0065] Specifically, after collecting the medical images and the corresponding text reports, the method further includes:

[0066] A text-to-speech model is used to convert the text reports corresponding to the collected medical images into voice files, and the voice parameters are set; the voice parameters include speech-related information such as speaking speed and pitch. By setting the voice parameters, the obtained voice files are ensured to be close to the real voice, thereby improving the reliability of model training.

[0067] Gaussian white noise is added to the converted voice files to obtain multiple noisy voice files as the voices to be corrected. Gaussian white noise with different signal-to-noise ratios is added to the converted voice files to simulate noisy voice files with different degrees of noise interference, and different noisy voice files are input into the model training model as the voices to be corrected, so as to improve the robustness of the model in practical applications.

[0068] It can be seen that in the above scheme, for medical data records under medical and health services, especially in scenarios such as radiology departments that require a large amount of voice transcription, a multimodal medical voice model can be used to extract features from paired medical images and text reports and generate corresponding texts, and the multimodal medical voice model is initialized according to the parameters of the visual language pre-training model to utilize the semantic correlation between the paired medical images and text reports to correct text errors. Moreover, the multimodal medical voice model is trained using text reports after error simulation to improve the model's error correction ability. The medical voice-to-text error correction model obtains the corresponding text by performing voice content recognition on the input voice through the voice-to-text model, and cooperates with the multimodal medical voice model to realize medical voice-to-text error correction of the input medical voice, thereby improving the accuracy of medical voice-to-text conversion and the conversion effect of medical voice-to-text.

[0069] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0070] In one embodiment, a medical speech-to-text error correction device is provided, and the medical speech-to-text error correction device corresponds one-to-one to the medical speech-to-text error correction method in the above embodiment. Figure 5As shown, the medical speech-to-text error correction device includes a multimodal medical speech model construction module 101, a medical speech-to-text error correction model construction module 102 and an error correction module 103. The functional modules are described in detail as follows:

[0071] The multimodal medical speech model construction module 101 is used to construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0072] A medical speech-to-text error correction model building module 102 is used to build a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0073] The error correction module 103 is used to perform medical speech-to-text error correction on the input medical speech using the trained medical speech-to-text error correction model.

[0074] In one embodiment, the multimodal medical speech model building module 101 is specifically used for:

[0075] Construct a multimodal medical speech model, initialize the visual encoder of the multimodal medical speech model according to the parameters of the visual encoder of the visual language pre-training model, and initialize the text encoder of the multimodal medical speech model according to the parameters of the text encoder of the visual language pre-training model;

[0076] receiving input pairs of medical images and text reports after error simulation to a multimodal medical speech model;

[0077] A visual encoder of a multimodal medical speech model is used to extract medical image embedding features from a received medical image;

[0078] A text encoder using a multimodal medical speech model is used to extract text embedding features from the received error simulated text report;

[0079] The alternating cross-attention mechanism is used to fuse the medical image embedding features and text embedding features to obtain multimodal text representation;

[0080] A text decoder using a multimodal medical speech model generates a corrected text report based on the multimodal text representation;

[0081] The loss is calculated based on the corrected text report and the real text report before error simulation, and the multimodal medical speech model is optimized based on the calculated loss.

[0082] The loss calculated based on the corrected text report and the real text report before the error simulation is specifically:

[0083] The cross entropy loss function is used to calculate the loss of each word in the corrected text report and the real text report before error simulation.

[0084] In one embodiment, the multimodal medical speech model building module 101 is further used for:

[0085] Receive paired medical images and text reports to train a visual language pre-trained model;

[0086] A visual encoder using a visual language pre-training model is used to extract medical image features from the received medical image;

[0087] A text encoder using a visual language pre-trained model is used to extract text features from the received text report;

[0088] An alternating cross-attention mechanism is used to fuse medical image features and text features to obtain multimodal representation.

[0089] In one embodiment, the multimodal medical speech model building module 101 is further used for:

[0090] Collect medical images and their corresponding text reports;

[0091] Randomly delete words, replace words and / or insert words in the text report corresponding to the medical image to obtain a text report after error simulation.

[0092] In one embodiment, the multimodal medical speech model building module 101 is further used for:

[0093] Use the text-to-speech model to convert the text reports corresponding to the collected medical images into voice files and set the voice parameters;

[0094] Gaussian white noise is added to the converted speech files to obtain multiple noisy speech files as speech to be corrected.

[0095] In one embodiment, the medical speech-to-text error correction model building module 102 is specifically used to:

[0096] Building a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model;

[0097] receiving speech to be corrected and its corresponding medical image;

[0098] A speech-to-text model is used to receive the speech to be corrected and recognize the speech content to obtain an initial transcription text;

[0099] A multimodal medical speech model is used to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image;

[0100] A multimodal medical speech model is used to generate a corrected text report based on the obtained multimodal text representation to correct the initial transcription text and obtain the final corrected text.

[0101] The present invention provides a medical speech-to-text error correction device, which uses a multimodal medical speech model to extract features from paired medical images and text reports and generate corresponding texts, and the multimodal medical speech model is initialized according to the parameters of a visual language pre-training model to use the semantic association between the paired medical images and text reports to correct text errors. Moreover, the multimodal medical speech model is trained using the text report after error simulation to improve the error correction ability of the model. The medical speech-to-text error correction model uses the speech-to-text model to perform speech content recognition on the input speech to obtain the corresponding text, and cooperates with the multimodal medical speech model to realize medical speech-to-text error correction of the input medical speech, thereby improving the accuracy of medical speech-to-text conversion and improving the conversion effect of medical speech-to-text.

[0102] For the specific definition of the medical speech-to-text error correction device, please refer to the definition of the medical speech-to-text error correction method above, which will not be repeated here. The various modules in the above-mentioned medical speech-to-text error correction device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps of a medical speech-to-text error correction method service side.

[0104] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a medical speech-to-text error correction method.

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0106] Construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0107] A medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model;

[0108] The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

[0109] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0110] Construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0111] A medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model;

[0112] The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

[0113] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0116] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A medical speech-to-text error correction method, characterized in that: include: Construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model; A medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model; The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

2. The medical speech-to-text error correction method according to claim 1, characterized in that: The multimodal medical speech model is constructed, the multimodal medical speech model is initialized according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and the paired medical images and the text report after the error simulation are received to train the multimodal medical speech model, including: Construct a multimodal medical speech model, initialize the visual encoder of the multimodal medical speech model according to the parameters of the visual encoder of the visual language pre-training model, and initialize the text encoder of the multimodal medical speech model according to the parameters of the text encoder of the visual language pre-training model; receiving input pairs of medical images and text reports after error simulation to a multimodal medical speech model; A visual encoder of a multimodal medical speech model is used to extract medical image embedding features from a received medical image; A text encoder using a multimodal medical speech model is used to extract text embedding features from the received error simulated text report; The alternating cross-attention mechanism is used to fuse the medical image embedding features and text embedding features to obtain multimodal text representation; A text decoder using a multimodal medical speech model generates a corrected text report based on the multimodal text representation; The loss is calculated based on the corrected text report and the real text report before error simulation, and the multimodal medical speech model is optimized based on the calculated loss.

3. The medical speech-to-text error correction method according to claim 2, characterized in that: The loss calculated based on the corrected text report and the real text report before the error simulation is specifically: The cross entropy loss function is used to calculate the loss of each word in the corrected text report and the real text report before error simulation.

4. The medical speech-to-text error correction method according to claim 1, characterized in that: The medical speech-to-text error correction model is constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model, including: Building a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model; receiving speech to be corrected and its corresponding medical image; A speech-to-text model is used to receive the speech to be corrected and recognize the speech content to obtain an initial transcription text; A multimodal medical speech model is used to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image; A multimodal medical speech model is used to generate a corrected text report based on the obtained multimodal text representation to correct the initial transcription text and obtain the final corrected text.

5. The medical speech-to-text error correction method according to claim 1, characterized in that: Before constructing the multimodal medical speech model, the method further includes: Receive paired medical images and text reports to train a visual language pre-trained model; A visual encoder using a visual language pre-training model is used to extract medical image features from the received medical image; A text encoder using a visual language pre-trained model is used to extract text features from the received text report; An alternating cross-attention mechanism is used to fuse medical image features and text features to obtain multimodal representation.

6. The medical speech-to-text error correction method according to claim 1, characterized in that: Before constructing the multimodal medical speech model, the method further includes: Collect medical images and their corresponding text reports; Randomly delete words, replace words and / or insert words in the text report corresponding to the medical image to obtain a text report after error simulation.

7. The medical speech-to-text error correction method according to claim 6, characterized in that: After collecting the medical images and their corresponding text reports, the method further includes: Use the text-to-speech model to convert the text reports corresponding to the collected medical images into voice files and set the voice parameters; Gaussian white noise is added to the converted speech files to obtain multiple noisy speech files as speech to be corrected.

8. A medical speech-to-text error correction device, characterized in that: include: A multimodal medical speech model construction module is used to construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model; A medical speech-to-text error correction model construction module is used to construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model; The error correction module is used to perform medical speech-to-text error correction on the input medical speech using the trained medical speech-to-text error correction model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the medical speech-to-text error correction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the medical speech-to-text error correction method according to any one of claims 1 to 7 are implemented.

Images