Medical speech-to-text error correction method, device, equipment, and medium

Abstract

The present invention relates to the field of artificial intelligence technology and discloses a medical speech-to-text error correction method, apparatus, device, and medium. The method comprises: constructing a multimodal medical speech model, initializing the multimodal medical speech model based on parameters of a visual encoder and a text encoder of a visual language pre-training model, receiving paired medical images and text reports after error simulation to train the multimodal medical speech model; constructing a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, receiving speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model; and using the trained medical speech-to-text error correction model to correct medical speech-to-text errors. This improves the medical speech-to-text conversion effect.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology, speech processing and medical health technology, and in particular to a medical speech-to-text error correction method, device, equipment and medium. Background Art

[0002] Existing automatic speech recognition (ASR) systems in the medical field are used for medical diagnosis, remote consultations, and recording of medical data, including personal health records, prescriptions, examination reports, and medical images. However, due to the complexity and specialized nature of medical terminology, recognition errors often occur when converting speech to text, limiting accuracy. This is particularly true when dealing with complex speech conditions such as noise and accents. Summary of the Invention

[0003] The present invention provides a medical speech-to-text error correction method, device, computer equipment and medium to solve the technical problems of inaccurate speech recognition and poor recognition effect.

[0004] In a first aspect, a medical speech-to-text error correction method is provided, comprising:

[0005] Build a multimodal medical speech model, initialize the multimodal medical speech model based on the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0006] Building a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receiving the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0007] The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

[0008] In a second aspect, a medical speech-to-text error correction device is provided, comprising:

[0009] A multimodal medical speech model construction module is used to construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0010] A medical speech-to-text error correction model construction module is used to construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and to receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0011] The error correction module is used to perform medical speech-to-text error correction on the input medical speech using the trained medical speech-to-text error correction model.

[0012] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned medical speech-to-text error correction method when executing the computer program.

[0013] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned medical speech-to-text error correction method are implemented.

[0014] In the scheme implemented by the above-mentioned medical speech-to-text error correction method, device, computer equipment and storage medium, a multimodal medical speech model can be constructed, the multimodal medical speech model can be initialized according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and the paired medical images and the text report after error simulation are received to train the multimodal medical speech model; a medical speech-to-text error correction model can be constructed according to the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model; the trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech. In the present invention, for medical data records under medical and health services, especially in radiology In scenarios such as medicine that require a large amount of voice transcription, a multimodal medical voice model can be used to extract features from paired medical images and text reports and generate corresponding text. The multimodal medical voice model is initialized according to the parameters of the visual language pre-training model to utilize the semantic correlation between the paired medical images and text reports to correct text errors. Moreover, training the multimodal medical voice model with text reports after error simulation can improve the model's error correction ability. The medical voice-to-text error correction model obtains the corresponding text by performing voice content recognition on the input voice through the voice-to-text model, and cooperates with the multimodal medical voice model to realize medical voice-to-text error correction of the input medical voice, thereby improving the accuracy of medical voice-to-text and the conversion effect of medical voice-to-text. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0016] Figure 1 2. It is a schematic diagram of an application environment of a medical speech-to-text error correction method according to an embodiment of the present invention;

[0017] Figure 2 This is a flow chart of a medical speech-to-text error correction method according to an embodiment of the present invention;

[0018] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0019] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0020] Figure 5 This is a structural diagram of a medical speech-to-text error correction device according to an embodiment of the present invention;

[0021] Figure 6 is a structural diagram of a computer device in one embodiment of the present invention;

[0022] Figure 7 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] The medical speech-to-text error correction method provided by the embodiment of the present invention can be applied in the following fields: Figure 1In an application environment, the client communicates with the server through a network. The server can receive input medical speech through the client, and the server can communicate with the database to obtain paired medical images and text reports, and obtain paired medical images and error-simulated text reports based on the paired medical images and text reports, train a multimodal medical speech model initialized with parameters of the visual encoder and text encoder based on the visual language pre-training model, construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, process the paired medical images and text reports to obtain the speech to be corrected and its corresponding medical image, train the medical speech-to-text error correction model, use the trained medical speech-to-text error correction model to perform medical speech-to-text error correction on the input medical speech, and feed the final corrected text back to the client. In the present invention, for the medical and health industry In medical data recording services, particularly in scenarios such as radiology departments that require large amounts of speech transcription, a multimodal medical speech model can be used to extract features from paired medical images and text reports and generate corresponding text. The multimodal medical speech model is initialized based on the parameters of a visual language pre-trained model to exploit the semantic relevance between the paired medical images and text reports for text error correction. Furthermore, training the multimodal medical speech model with text reports after error simulation can improve the model's error correction capabilities. The medical speech-to-text error correction model uses the speech-to-text model to perform speech content recognition on the input speech to obtain the corresponding text. This, in conjunction with the multimodal medical speech model, corrects medical speech-to-text errors on the input medical speech, improving the accuracy of the medical speech-to-text conversion and the quality of the medical speech-to-text conversion. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0025] See also Figure 2 As shown, Figure 2 A flowchart of a medical speech-to-text error correction method provided by an embodiment of the present invention includes the following steps:

[0026] S10: Construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, receive paired medical images and text reports after error simulation to train the multimodal medical speech model.

[0027] Among them, the multimodal medical speech model includes a visual encoder and a text encoder to extract the corresponding image features and text features. The multimodal medical speech model also includes a text decoder to correct the erroneous content in the text through the content contained in the image. The multimodal medical speech model is trained to strengthen the model to generate corrected speech transcription text through the semantic information of the image modality. The multimodal medical speech model is initialized according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and the visual encoder and text encoder of the multimodal medical speech model share the parameters of the visual encoder and text encoder of the visual language pre-training model. The text report after error simulation refers to the text report obtained after the text report corresponding to the medical image is erroneously modified to simulate the text report generated when the speech recognition error occurs. The multimodal medical speech model is trained using the text report after error simulation, which is beneficial to improving the error correction ability of the model. Among them, if Figure 3 As shown, in step S10, i.e., constructing a multimodal medical speech model, initializing the multimodal medical speech model according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, receiving paired medical images and text reports after error simulation to train the multimodal medical speech model, includes the following steps:

[0028] S11: Build a multimodal medical speech model, initialize the visual encoder of the multimodal medical speech model according to the parameters of the visual encoder of the visual language pre-training model, and initialize the text encoder of the multimodal medical speech model according to the parameters of the text encoder of the visual language pre-training model.

[0029] S12: Receive input paired medical images and text reports after error simulation to the multimodal medical speech model.

[0030] S13: A visual encoder of a multimodal medical speech model is used to extract medical image embedding features from the received medical image.

[0031] S14: A text encoder based on a multimodal medical speech model is used to extract text embedding features from the received error simulated text report.

[0032] S15: Use alternating cross-attention mechanism to fuse medical image embedding features and text embedding features to obtain multimodal text representation.

[0033] S16: A text decoder using a multimodal medical speech model generates a corrected text report based on the multimodal text representation.

[0034] S17: Calculate the loss based on the corrected text report and the actual text report before the error simulation, and optimize the multimodal medical speech model based on the calculated loss.

[0035] Among them, the optimized multimodal medical speech model can also use AdamW optimizer, warmup cosineannealing scheduler and other technologies to optimize model parameters. In step S17, that is, according to the corrected text report and the real text report before the error simulation, the loss is calculated. The loss calculated according to the corrected text report and the real text report before the error simulation in the multimodal medical speech model can be specifically calculated as follows:

[0036] The cross entropy loss function is used to calculate the loss of each word in the corrected text report and the real text report before the error simulation. This can minimize the loss and improve the accuracy of the model.

[0037] S20: Construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model.

[0038] After training the multimodal medical speech model, to accurately correct errors in speech transcription, a medical speech-to-text error correction model must be constructed, encompassing both the speech-to-text model and the multimodal medical speech model. This model must then be trained to apply the multimodal medical speech model to actual speech recognition. Training strengthens the model's error correction capabilities and accuracy. The speech-to-text model is non-specific and requires no additional tuning, making it widely applicable.

[0039] In some embodiments of the invention, Figure 4 As shown, a specific solution for training a medical speech-to-text error correction model is provided. In S20, a medical speech-to-text error correction model is constructed based on the speech-to-text model and the trained multimodal medical speech model. The speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model. The solution specifically includes the following steps S21-S25:

[0040] S21: Construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model.

[0041] S22: Receive the speech to be corrected and its corresponding medical image.

[0042] S23: Using a speech-to-text model to receive the speech to be corrected and recognize the speech content to obtain an initial transcription text.

[0043] S24: A multimodal medical speech model is used to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image.

[0044] Among them, step S24, that is, using the multimodal medical speech model to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image, specifically includes:

[0045] A visual encoder based on a multimodal medical speech model is used to extract medical image embedding features from the medical image corresponding to the received initial transcription text;

[0046] A text encoder using a multimodal medical speech model extracts text embedding features from the received initial transcription text;

[0047] The alternating cross-attention mechanism is used to fuse medical image embedding features and text embedding features to obtain multimodal text representation.

[0048] S25: A multimodal medical speech model is used to generate a corrected text report based on the obtained multimodal text representation to correct the initial transcription text and obtain a final corrected text.

[0049] Among them, step S25, that is, using the multimodal medical speech model to generate a corrected text report based on the obtained multimodal text representation to correct the initial transcription text to obtain the final corrected text, specifically:

[0050] A text decoder using a multimodal medical speech model generates a corrected text report based on the obtained multimodal text representation, and corrects the initial transcription text based on the generated corrected text report to obtain a final corrected text.

[0051] By constructing a medical speech-to-text error correction model that includes a speech-to-text model and a multimodal medical speech model, and training the medical speech-to-text error correction model, the multimodal medical speech model can be applied to actual speech recognition, and the model's error correction ability and accuracy can be enhanced through training.

[0052] S30: Using the trained medical speech-to-text error correction model, the medical speech-to-text error correction is performed on the input medical speech to obtain the correct text corresponding to the input medical speech, thereby improving the accuracy and effect of the conversion.

[0053] Specifically, before step S10, that is, before building the multimodal medical speech model, the process further includes:

[0054] Receive paired medical images and text reports to train a visual language pre-trained model;

[0055] A visual encoder using a visual language pre-training model extracts medical image features from a received medical image; wherein the visual encoder can use a ViT architecture to extract image features.

[0056] A text encoder using a visual language pre-training model extracts text features from a received text report; wherein the text encoder may use a Transformer architecture to extract text features.

[0057] An alternating cross-attention mechanism is used to fuse medical image features and text features to obtain multimodal representation.

[0058] The trained visual-language pre-training model can also be optimized for model parameters using technologies such as the AdamW optimizer and the warmup cosineannealing scheduler. By training the visual-language pre-training model, the parameters of the visual-language pre-training model are optimized, and the parameters of the text encoder and visual encoder of the visual-language pre-training model are further optimized. This optimizes the parameters of the text encoder and visual encoder of the multimodal medical speech model that share the parameters of the text encoder and visual encoder of the visual-language pre-training model.

[0059] Specifically, after fusing medical image features and text features using the alternating cross attention mechanism to obtain a multimodal representation, the method further includes:

[0060] Cross-modal contrastive learning and intra-modal contrastive learning are performed on the obtained multimodal representations to improve the model's ability to understand image and text semantics.

[0061] Modalities can include images, speech, and text. Multimodality refers to at least two modalities. Intra-modality contrastive learning is contrastive learning within the same modality. It calculates the similarity of different samples of the same modality to increase the distance between different samples and decrease the distance between the same samples. Samples can be medical images or text reports. Samples of the image modality are medical images, and samples of the text modality are text reports. Cross-modal contrastive learning refers to multimodal contrastive learning. It calculates the similarity of multimodal paired data to increase the distance between different multimodal paired data. Images and corresponding texts, and speech and corresponding texts are different multimodal paired data.

[0062] Specifically, before step S10, that is, before building the multimodal medical speech model, the process further includes:

[0063] Collect medical images and their corresponding text reports; the medical images and their corresponding text reports are collected by the server from a database. The medical images and their corresponding text reports can be chest X-rays and their corresponding text reports obtained from a public medical image database, or real medical images and doctor's oral recordings obtained from a hospital database. The medical image database can be MIMIC-CXR.

[0064] Randomly delete words, replace words, and / or insert words in the text report corresponding to the medical image to obtain an error-simulated text report to simulate speech recognition errors and improve the model's error correction ability.

[0065] Specifically, after collecting the medical images and their corresponding text reports, the method further includes:

[0066] A text-to-speech model is used to convert the text reports corresponding to the collected medical images into voice files, and voice parameters are set; voice parameters include speech-related information such as speaking speed and pitch. By setting voice parameters, the obtained voice files are ensured to be close to real speech, thereby improving the reliability of model training.

[0067] Gaussian white noise is added to the converted speech files to obtain multiple noisy speech files as the speech to be corrected. Gaussian white noise with different signal-to-noise ratios is added to the converted speech files to simulate noisy speech files with varying degrees of noise interference. These different noisy speech files are input as the speech to be corrected into the model training model to improve the model's robustness in practical applications.

[0068] It can be seen that in the above scheme, for medical data records under medical and health services, especially in scenarios such as radiology departments that require a large amount of voice transcription, a multimodal medical voice model can be used to extract features from paired medical images and text reports and generate corresponding texts. The multimodal medical voice model is initialized according to the parameters of the visual language pre-training model to utilize the semantic correlation between the paired medical images and text reports to correct text errors. Moreover, the multimodal medical voice model is trained using text reports after error simulation to improve the model's error correction ability. The medical voice-to-text error correction model obtains the corresponding text by performing voice content recognition on the input voice through the voice-to-text model, and cooperates with the multimodal medical voice model to realize medical voice-to-text error correction of the input medical voice, thereby improving the accuracy of medical voice-to-text conversion and improving the conversion effect of medical voice-to-text.

[0069] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0070] In one embodiment, a medical speech-to-text error correction device is provided, which corresponds to the medical speech-to-text error correction method in the above embodiment. Figure 5As shown, the medical speech-to-text error correction device includes a multimodal medical speech model construction module 101, a medical speech-to-text error correction model construction module 102, and an error correction module 103. The functional modules are described in detail as follows:

[0071] A multimodal medical speech model construction module 101 is used to construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0072] A medical speech-to-text error correction model construction module 102 is configured to construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and to receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0073] The error correction module 103 is used to perform medical speech-to-text error correction on the input medical speech using the trained medical speech-to-text error correction model.

[0074] In one embodiment, the multimodal medical speech model construction module 101 is specifically configured to:

[0075] Build a multimodal medical speech model, initialize the visual encoder of the multimodal medical speech model according to the parameters of the visual encoder of the visual language pre-training model, and initialize the text encoder of the multimodal medical speech model according to the parameters of the text encoder of the visual language pre-training model;

[0076] receiving input pairs of medical images and text reports after error simulation to a multimodal medical speech model;

[0077] A visual encoder based on a multimodal medical speech model is used to extract medical image embedding features from received medical images;

[0078] A text encoder based on a multimodal medical speech model is used to extract text embedding features from the received text report after error simulation.

[0079] Adopting alternating cross-attention mechanism to fuse medical image embedding features and text embedding features to obtain multimodal text representation;

[0080] A text decoder using a multimodal medical speech model generates a corrected text report based on the multimodal text representation;

[0081] The loss is calculated based on the corrected text report and the real text report before error simulation, and the multimodal medical speech model is optimized based on the calculated loss.

[0082] The loss calculated based on the corrected text report and the real text report before the error simulation is specifically:

[0083] The cross entropy loss function is used to calculate the loss of each word in the corrected text report and the real text report before error simulation.

[0084] In one embodiment, the multimodal medical speech model building module 101 is further configured to:

[0085] Receive paired medical images and text reports to train a visual language pre-trained model;

[0086] A visual encoder based on a visual language pre-training model is used to extract medical image features from the received medical image;

[0087] A text encoder based on a visual language pre-training model is used to extract text features from the received text report;

[0088] An alternating cross-attention mechanism is used to fuse medical image features and text features to obtain multimodal representation.

[0089] In one embodiment, the multimodal medical speech model building module 101 is further configured to:

[0090] Collect medical images and their corresponding text reports;

[0091] Randomly delete words, replace words, and / or insert words into the text report corresponding to the medical image to obtain a text report after error simulation.

[0092] In one embodiment, the multimodal medical speech model building module 101 is further configured to:

[0093] Use the text-to-speech model to convert the text reports corresponding to the collected medical images into voice files and set the voice parameters;

[0094] Gaussian white noise is added to the converted speech files to obtain multiple noisy speech files as speech to be corrected.

[0095] In one embodiment, the medical speech-to-text error correction model building module 102 is specifically configured to:

[0096] Build a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model;

[0097] receiving speech to be corrected and its corresponding medical image;

[0098] A speech-to-text model is used to receive the speech to be corrected and recognize the speech content to obtain the initial transcription text;

[0099] A multimodal medical speech model is used to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image;

[0100] A multimodal medical speech model is used to generate a corrected text report based on the obtained multimodal text representation to correct the initial transcription text and obtain the final corrected text.

[0101] The present invention provides a medical speech-to-text error correction device, which uses a multimodal medical speech model to extract features from paired medical images and text reports and generate corresponding text. The multimodal medical speech model is initialized according to the parameters of a visual language pre-training model to utilize the semantic relevance between the paired medical images and text reports to correct text errors. Moreover, the multimodal medical speech model is trained using the text reports after error simulation to improve the error correction ability of the model. The medical speech-to-text error correction model uses the speech-to-text model to perform speech content recognition on the input speech to obtain the corresponding text, and cooperates with the multimodal medical speech model to realize medical speech-to-text error correction of the input medical speech, thereby improving the accuracy of medical speech-to-text conversion and improving the conversion effect of medical speech-to-text.

[0102] The specific limitations of the medical speech-to-text error correction device can be found in the limitations of the medical speech-to-text error correction method described above and will not be further elaborated here. Each module in the aforementioned medical speech-to-text error correction device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the aforementioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each of the aforementioned modules.

[0103] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a medical speech-to-text error correction method.

[0104] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a medical speech-to-text error correction method.

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0106] Build a multimodal medical speech model, initialize the multimodal medical speech model based on the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0107] Building a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receiving the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0108] The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0110] Build a multimodal medical speech model, initialize the multimodal medical speech model based on the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model;

[0111] Building a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receiving the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model;

[0112] The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

[0113] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0114] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0116] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A medical speech-to-text error correction method, characterized in that: include: Build a multimodal medical speech model, initialize the multimodal medical speech model based on the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model; Building a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and receiving the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model; The trained medical speech-to-text error correction model is used to perform medical speech-to-text error correction on the input medical speech.

2. The medical speech-to-text error correction method according to claim 1, wherein: The multimodal medical speech model is constructed, the multimodal medical speech model is initialized according to the parameters of the visual encoder and the text encoder of the visual language pre-training model, and paired medical images and text reports after error simulation are received to train the multimodal medical speech model, including: Build a multimodal medical speech model, initialize the visual encoder of the multimodal medical speech model according to the parameters of the visual encoder of the visual language pre-training model, and initialize the text encoder of the multimodal medical speech model according to the parameters of the text encoder of the visual language pre-training model; receiving input pairs of medical images and text reports after error simulation to a multimodal medical speech model; A visual encoder based on a multimodal medical speech model is used to extract medical image embedding features from received medical images; A text encoder based on a multimodal medical speech model is used to extract text embedding features from the received text report after error simulation. Adopting alternating cross-attention mechanism to fuse medical image embedding features and text embedding features to obtain multimodal text representation; A text decoder using a multimodal medical speech model generates a corrected text report based on the multimodal text representation; The loss is calculated based on the corrected text report and the real text report before error simulation, and the multimodal medical speech model is optimized based on the calculated loss.

3. The medical speech-to-text error correction method according to claim 2, wherein: The loss calculated based on the corrected text report and the real text report before the error simulation is specifically: The cross entropy loss function is used to calculate the loss of each word in the corrected text report and the real text report before error simulation.

4. The medical speech-to-text error correction method according to claim 1, wherein: The medical speech-to-text error correction model is constructed based on the speech-to-text model and the trained multimodal medical speech model, and the speech to be corrected and its corresponding medical image are received to train the medical speech-to-text error correction model, including: Build a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model; receiving speech to be corrected and its corresponding medical image; A speech-to-text model is used to receive the speech to be corrected and recognize the speech content to obtain the initial transcription text; A multimodal medical speech model is used to obtain a multimodal text representation based on the obtained initial transcription text and the corresponding medical image; A multimodal medical speech model is used to generate a corrected text report based on the obtained multimodal text representation to correct the initial transcription text and obtain the final corrected text.

5. The medical speech-to-text error correction method according to claim 1, wherein: Before constructing the multimodal medical speech model, the method further includes: Receive paired medical images and text reports to train a visual language pre-trained model; A visual encoder based on a visual language pre-training model is used to extract medical image features from the received medical image; A text encoder based on a visual language pre-training model is used to extract text features from the received text report; An alternating cross-attention mechanism is used to fuse medical image features and text features to obtain multimodal representation.

6. The medical speech-to-text error correction method according to claim 1, wherein: Before constructing the multimodal medical speech model, the method further includes: Collect medical images and their corresponding text reports; Randomly delete words, replace words, and / or insert words into the text report corresponding to the medical image to obtain a text report after error simulation.

7. The medical speech-to-text error correction method according to claim 6, wherein: After collecting the medical images and their corresponding text reports, the method further includes: Use the text-to-speech model to convert the text reports corresponding to the collected medical images into voice files and set the voice parameters; Gaussian white noise is added to the converted speech files to obtain multiple noisy speech files as speech to be corrected.

8. A medical speech-to-text error correction device, characterized in that: include: A multimodal medical speech model construction module is used to construct a multimodal medical speech model, initialize the multimodal medical speech model according to the parameters of the visual encoder and text encoder of the visual language pre-training model, and receive paired medical images and text reports after error simulation to train the multimodal medical speech model; A medical speech-to-text error correction model construction module is used to construct a medical speech-to-text error correction model based on the speech-to-text model and the trained multimodal medical speech model, and to receive the speech to be corrected and its corresponding medical image to train the medical speech-to-text error correction model; The error correction module is used to perform medical speech-to-text error correction on the input medical speech using the trained medical speech-to-text error correction model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the medical speech-to-text error correction method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the medical speech-to-text error correction method according to any one of claims 1 to 7 are implemented.

Images