Methods, devices, equipment and media for generating oral disease image reports

Abstract

This invention relates to the field of artificial intelligence technology, and provides a method, apparatus, device, and medium for generating oral disease image reports. On one hand, the structured content of each initial sample is extracted and split into multiple entity items. While keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples, thereby achieving text enhancement of the samples. On the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text-enhanced samples. The pre-trained Swin model is used as the view model to extract features from each image of potential malignant diseases of the oral mucosa and the corresponding expanded image, thereby achieving image enhancement of the samples. This helps to train a more powerful report generator, making the generated oral disease image reports more accurate, so as to assist medical personnel in quickly and accurately interpreting the images.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for generating oral disease image reports. Background Technology

[0002] In the early screening of Oral Potential Malignant Disorder (OPMD), the interpretation and assessment of intraoral photographs are of paramount importance. However, traditional deep learning models have limited capabilities when processing multimodal data, struggling to extract detailed information from images and generate structured reports. Furthermore, even multimodal research primarily focuses on global reasoning tasks (such as detection), lacking evaluation of the fine-grained image recognition capabilities of large-scale language models. Therefore, this field still faces numerous challenges.

[0003] First, fusing the modal differences between image input and text output is a fundamental challenge in the automatic generation of medical reports. Medical images contain complex information that must be accurately interpreted and converted into coherent text, requiring sophisticated algorithms to map visual patterns to medical terminology.

[0004] Secondly, medical images exhibit unique visual biases, with lesion areas typically occupying only a small portion of the image, resulting in high similarity between normal and abnormal images. This necessitates that automated medical report generation systems be more sensitive to subtle differences than general image captioning models.

[0005] Furthermore, noise in medical reports, such as time information, can confuse models, leading to inaccuracies or hallucinations.

[0006] Therefore, how to accurately generate image reports of oral diseases has become an urgent problem to be solved. Summary of the Invention

[0007] In view of the above, it is necessary to provide a method, apparatus, device and medium for generating oral disease image reports, in order to solve the problem of inaccurate generation of oral disease image reports.

[0008] A method for generating oral disease image reports, the method comprising:

[0009] Reports of potential malignant diseases of the oral mucosa were obtained as initial samples;

[0010] Extract the structured content from each initial sample and split the structured content into multiple entity items;

[0011] Obtain multiple synonymous entities corresponding to the same entity item in each initial sample;

[0012] While keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples;

[0013] Obtain the constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model;

[0014] The SimCLR model is used to expand the view of the text-enhanced sample to obtain expanded images of each potential malignant disease of the oral mucosa in the text-enhanced sample;

[0015] The pre-trained Swin Transformer model is used as the view model to extract features from each image of potential malignant oral mucosa disease and its corresponding extended image, resulting in a vector representation of each image.

[0016] The Transformer model is trained using the text-enhanced samples and the vector representation of each image as training data to obtain the report generator;

[0017] In response to the oral disease image report generation instruction based on the target oral mucosa potential malignant disease image, the target vector representation is obtained by extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model;

[0018] The target vector representation is input into the report generator to obtain a target oral disease image report.

[0019] According to a preferred embodiment of the present invention, the step of randomly recombining multiple synonymous entities and multiple entity items corresponding to each initial sample to obtain a text-enhanced sample includes:

[0020] For each initial sample, obtain each entity item in the initial sample, and obtain multiple synonymous entities included under each entity item;

[0021] During each random recombination, one of the synonymous entities included under each entity item is randomly selected to replace the original entity in the initial sample;

[0022] The text-enhanced sample is obtained by integrating all non-repeating samples obtained after random recombination.

[0023] According to a preferred embodiment of the present invention, before obtaining the constructed image enhancement model, the method further includes:

[0024] Construct an initial model using the SimCLR model and the view model;

[0025] Obtain initial training samples and train the initial model using the initial training samples;

[0026] During the training process of the initial model, the view of the initial training samples is expanded based on the SimCLR model to obtain expanded samples;

[0027] The similarity between each initial training sample and its corresponding expanded sample is calculated using the cosine similarity algorithm, and the loss function value of the initial model is also calculated.

[0028] Training stops when the similarity is higher than the configured similarity and the loss function value converges.

[0029] The currently obtained model is determined as the image enhancement model.

[0030] According to a preferred embodiment of the present invention, the step of using the SimCLR model to perform view expansion on the text-enhanced sample to obtain expanded images of each potential malignant disease image of the oral mucosa in the text-enhanced sample includes:

[0031] The text-enhanced sample is randomly cropped, and the cropped sample is resized to obtain the first image;

[0032] The colors of the text-enhanced samples are randomly perturbed to obtain a second image;

[0033] The text-enhanced sample is randomly rotated to obtain a third image;

[0034] The text-enhanced sample is flipped to obtain the fourth image;

[0035] The first image, the second image, the third image, and the fourth image are integrated to obtain the extended image.

[0036] According to a preferred embodiment of the present invention, after obtaining the report generator, the method further includes:

[0037] The accuracy of words and sentences in the reports generated by the report generator was assessed using a bilingual evaluation alternative BLEU4.

[0038] The quality of text summaries in reports generated by the report generator is detected using the longest common subsequence ROUGE-L.

[0039] The CIDEr detection system was used to evaluate the rate at which the report generator captured important information during the report generation process, based on consensus-based image descriptions.

[0040] When the accuracy is greater than or equal to the configured accuracy, the text summary quality reaches the configured quality, and the acquisition rate is greater than or equal to the configured acquisition rate, the report generator is determined to meet the report generation requirements.

[0041] According to a preferred embodiment of the present invention, the step of inputting the target vector representation into the report generator to obtain a target oral disease image report includes:

[0042] The encoder of the report generator receives the standard vector representation;

[0043] The target oral disease image report is generated using the decoder of the report generator.

[0044] According to a preferred embodiment of the present invention, after obtaining the target oral disease image report, the method further includes:

[0045] The target oral disease image report is sent to the triggerer of the oral disease image report generation instruction.

[0046] An oral disease image report generation device, the oral disease image report generation device comprising:

[0047] The acquisition unit is used to acquire reports of potential malignant diseases of the oral mucosa as initial samples.

[0048] A splitting unit is used to extract the structured content from each initial sample and split the structured content into multiple entity items;

[0049] The acquisition unit is also used to acquire multiple synonymous entities corresponding to the same entity item in each initial sample;

[0050] The recombination unit is used to randomly recombine multiple synonymous entities and multiple entity items corresponding to each initial sample while keeping the original diagnostic results of each initial sample unchanged, so as to obtain text-enhanced samples.

[0051] The acquisition unit is further configured to acquire the constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model;

[0052] An expansion unit is used to expand the view of the text-enhanced sample using the SimCLR model to obtain expanded images of each potential malignant disease image of the oral mucosa in the text-enhanced sample.

[0053] The extraction unit is used to extract features from each image of a potential malignant disease of the oral mucosa and its corresponding extended image using a pre-trained Swin Transformer model as the view model, so as to obtain a vector representation of each image.

[0054] The training unit is used to train the Transformer model using the text-enhanced samples and the vector representation of each image as training data to obtain the report generator;

[0055] The extraction unit is further configured to, in response to an oral disease image report generation instruction based on a potential malignant disease image of the target oral mucosa, use the Swing Transformer model to extract features from the potential malignant disease image of the target oral mucosa to obtain a target vector representation.

[0056] An input unit is used to input the target vector representation into the report generator to obtain a target oral disease image report.

[0057] A computer device, the computer device comprising:

[0058] Memory, storing at least one instruction;

[0059] The processor executes instructions stored in the memory to implement the oral disease image report generation method.

[0060] A computer-readable storage medium storing at least one instruction, which is executed by a processor in a computer device to implement the oral disease image report generation method.

[0061] As can be seen from the above technical solutions, on the one hand, the structured content in each initial sample is extracted and split into multiple entity items. Multiple synonymous entities corresponding to the same entity item in each initial sample are obtained. Under the premise of keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples, thereby achieving text enhancement of the samples. On the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text-enhanced samples. The pre-trained Swin Transformer model is used as the view model to extract features from each image of potential malignant diseases of the oral mucosa and the corresponding expanded image, thereby achieving image enhancement of the samples. This helps to train a more powerful report generator, making the generated oral disease image reports more accurate, so as to assist medical staff in quickly and accurately interpreting the images. Attached Figure Description

[0062] Figure 1 This is a flowchart of a preferred embodiment of the oral disease image report generation method of the present invention.

[0063] Figure 2 This is a functional block diagram of a preferred embodiment of the oral disease image report generation device of the present invention.

[0064] Figure 3 This is a schematic diagram of the structure of a computer device that implements a preferred embodiment of the oral disease image report generation method of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] like Figure 1 The diagram shown is a flowchart of a preferred embodiment of the oral disease image report generation method of the present invention. The order of the steps in this flowchart can be changed, and some steps can be omitted, depending on different needs.

[0067] The oral disease image report generation method is applied to one or more computer devices. The computer device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0068] The computer device can be any electronic product that can interact with the user, such as a personal computer, tablet computer, smartphone, personal digital assistant (PDA), interactive network television (IPTV), smart wearable device, etc.

[0069] The computer equipment may also include network equipment and / or user equipment. The network equipment includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0070] The server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0071] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] Foundational technologies in artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0073] The network in which the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, and virtual private network (VPN).

[0074] S10, Obtain reports of potential malignant diseases of the oral mucosa as initial samples.

[0075] In this embodiment, the oral potential malignant disorder (OPMD) report can be obtained from a trusted platform to ensure the availability and accuracy of the oral potential malignant disorder report.

[0076] S11, extract the structured content from each initial sample, and split the structured content into multiple entity items.

[0077] For example, when the initial sample mentions "a red patch of about 0.5cm × 0.5cm was found on the patient's tongue, with slight bleeding and clear borders, and the preliminary diagnosis is a potential malignant disease of the oral mucosa", the entity item may include: "location", "shape", "number", "size", "clinical manifestations", "lesion borders" and "diagnosis".

[0078] S12, obtain multiple synonymous entities corresponding to the same entity item in each initial sample.

[0079] Continuing with the example above, the entities under each entity item can be:

[0080] Location: Tongue;

[0081] Shape: 0.5cm × 0.5cm;

[0082] Quantity: One location;

[0083] Size: 0.5cm × 0.5cm;

[0084] Clinical manifestations: minor bleeding;

[0085] Lesion boundaries: well-defined;

[0086] Diagnosis: Potential malignant disease of the oral mucosa.

[0087] Among them, the synonymous entities of the entity "tongue" in the entity item "location" can include "buccal mucosa", etc.

[0088] S13. While keeping the original diagnostic results of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples.

[0089] In this embodiment, the step of randomly recombining multiple synonymous entities and multiple entity items corresponding to each initial sample to obtain text-enhanced samples includes:

[0090] For each initial sample, obtain each entity item in the initial sample, and obtain multiple synonymous entities included under each entity item;

[0091] During each random recombination, one of the synonymous entities included under each entity item is randomly selected to replace the original entity in the initial sample;

[0092] The text-enhanced sample is obtained by integrating all non-repeating samples obtained after random recombination.

[0093] For example, if an initial sample includes entity A and entity B, and the synonyms for entity A are X and Y, and the synonyms for entity B are M and N, then after random recombination, the following four reports can be obtained as the text enhancement samples:

[0094] Entity A: X, Entity B: M;

[0095] Entity A: X, Entity B: N;

[0096] Entity A: Y, Entity B: M;

[0097] Entity A: Y, Entity B: N.

[0098] Through the above embodiments, text enhancement of the report can be performed without changing the diagnostic results, thereby providing sufficient and reliable samples for subsequent model training.

[0099] S14, Obtain the constructed image enhancement model; wherein, the image enhancement model includes a SimCLR model and a view model.

[0100] The SimCLR model is used to enhance images.

[0101] The view model is used to extract important image features.

[0102] In this embodiment, before obtaining the constructed image enhancement model, the method further includes:

[0103] Construct an initial model using the SimCLR model and the view model;

[0104] Obtain initial training samples and train the initial model using the initial training samples;

[0105] During the training process of the initial model, the view of the initial training samples is expanded based on the SimCLR model to obtain expanded samples;

[0106] The similarity between each initial training sample and its corresponding expanded sample is calculated using the cosine similarity algorithm, and the loss function value of the initial model is also calculated.

[0107] Training stops when the similarity is higher than the configured similarity and the loss function value converges.

[0108] The currently obtained model is determined as the image enhancement model.

[0109] The configuration similarity and the loss function value can be configured comprehensively based on model performance and usage environment.

[0110] Through the above embodiments, the similarity between the extended image and the original image can be evaluated based on cosine similarity, and the effect of the model can be further limited by the loss function, so that the trained image enhancement model has better data processing performance.

[0111] S15, the SimCLR model is used to expand the view of the text-enhanced sample to obtain expanded images of each potential malignant disease image of the oral mucosa in the text-enhanced sample.

[0112] In this embodiment, the step of using the SimCLR model to expand the view of the text-enhanced sample to obtain expanded images of each potential malignant oral mucosal disease image in the text-enhanced sample includes:

[0113] The text-enhanced sample is randomly cropped, and the cropped sample is resized to obtain the first image;

[0114] The colors of the text-enhanced samples are randomly perturbed to obtain a second image;

[0115] The text-enhanced sample is randomly rotated to obtain a third image;

[0116] The text-enhanced sample is flipped to obtain the fourth image;

[0117] The first image, the second image, the third image, and the fourth image are integrated to obtain the extended image.

[0118] Through the above embodiments, image enhancement can be further performed on the basis of text enhancement, which significantly improves the generalization ability and data utilization, and effectively solves the problem of scarce oral disease image report samples.

[0119] S16, using the pre-trained Swin Transformer model as the view model, feature extraction is performed on each image of potential malignant oral mucosa disease and its corresponding extended image to obtain a vector representation of each image.

[0120] In this embodiment, using a pre-trained Swing Transformer model can improve the training efficiency of the model. Furthermore, the pre-trained Swing Transformer model already possesses relevant knowledge in the corresponding domain, which can also improve the accuracy of the model.

[0121] S17, the Transformer model is trained using the text-enhanced samples and the vector representation of each image as training data to obtain the report generator.

[0122] In this embodiment, in order to make a fair comparison with existing models, the effectiveness of the report generator also needs to be evaluated as part of the report generation process and report quality.

[0123] Specifically, after obtaining the report generator, the method further includes:

[0124] The accuracy of words and sentences in the reports generated by the report generator was tested using the Bilingual Evaluation Understudy 4 (BLEU4).

[0125] The quality of text summaries in reports generated by the report generator is detected using the longest common subsequence (ROUGE-L, Recall-Oriented Understudy for Gisting Evaluation).

[0126] The Consensus-based Image Description Evaluation (CIDEr) is used to detect the rate at which the report generator acquires important information during the report generation process.

[0127] When the accuracy is greater than or equal to the configured accuracy, the text summary quality reaches the configured quality, and the acquisition rate is greater than or equal to the configured acquisition rate, the report generator is determined to meet the report generation requirements.

[0128] The configuration accuracy, configuration quality, and configuration acquisition rate can be configured according to actual needs.

[0129] The above embodiments enable a comprehensive evaluation of the usability of the trained report generator by combining multiple dimensions of indicators.

[0130] In this embodiment, inputting the target vector representation into the report generator to obtain a target oral disease image report includes:

[0131] The encoder of the report generator receives the standard vector representation;

[0132] The target oral disease image report is generated using the decoder of the report generator.

[0133] Through the above embodiments, oral disease image reports can be automatically generated based on the trained report generator.

[0134] S18, in response to the oral disease image report generation instruction based on the target oral mucosa potential malignant disease image, the target vector representation is obtained by extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model.

[0135] In this embodiment, the oral disease image report generation instruction can be triggered by relevant medical staff to respond promptly to their image viewing needs.

[0136] S19, the target vector representation is input into the report generator to obtain a target oral disease image report.

[0137] In this embodiment, after obtaining the target oral disease image report, the method further includes:

[0138] The target oral disease image report is sent to the triggerer of the oral disease image report generation instruction.

[0139] For example, the trigger can be relevant medical staff. Sending the target oral disease image report to the medical staff can help them quickly interpret the images. The more accurate the target oral disease image report is, the better the medical staff can interpret the images in conjunction with the target oral disease image report.

[0140] As can be seen from the above technical solutions, on the one hand, the structured content in each initial sample is extracted and split into multiple entity items. Multiple synonymous entities corresponding to the same entity item in each initial sample are obtained. Under the premise of keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples, thereby achieving text enhancement of the samples. On the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text-enhanced samples. The pre-trained Swin Transformer model is used as the view model to extract features from each image of potential malignant diseases of the oral mucosa and the corresponding expanded image, thereby achieving image enhancement of the samples. This helps to train a more powerful report generator, making the generated oral disease image reports more accurate, so as to assist medical staff in quickly and accurately interpreting the images.

[0141] like Figure 2 The diagram shown is a functional block diagram of a preferred embodiment of the oral disease image report generation device of the present invention. The oral disease image report generation device 11 includes an acquisition unit 110, a splitting unit 111, a recombination unit 112, an expansion unit 113, an extraction unit 114, a training unit 115, and an input unit 116. The module / unit referred to in this invention refers to a series of computer program segments that can be executed by a processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0142] The acquisition unit 110 is used to acquire reports of potential malignant diseases of the oral mucosa as initial samples.

[0143] The splitting unit 111 is used to extract the structured content from each initial sample and split the structured content into multiple entity items;

[0144] The acquisition unit 110 is also used to acquire multiple synonymous entities corresponding to the same entity item in each initial sample;

[0145] The recombination unit 112 is used to randomly recombine multiple synonymous entities and multiple entity items corresponding to each initial sample while keeping the original diagnostic results of each initial sample unchanged, to obtain a text-enhanced sample.

[0146] The acquisition unit 110 is further configured to acquire the constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model;

[0147] The expansion unit 113 is used to expand the view of the text enhancement sample using the SimCLR model to obtain expanded images of each potential malignant disease image of the oral mucosa in the text enhancement sample.

[0148] The extraction unit 114 is used to extract features from each image of a potential malignant disease of the oral mucosa and its corresponding extended image using a pre-trained Swing Transformer model as the view model, so as to obtain a vector representation of each image.

[0149] The training unit 115 is used to train the Transformer model using the text-enhanced samples and the vector representation of each image as training data to obtain the report generator;

[0150] The extraction unit 114 is further configured to, in response to an oral disease image report generation instruction based on a potential malignant disease image of the target oral mucosa, use the Swing Transformer model to extract features from the potential malignant disease image of the target oral mucosa to obtain a target vector representation.

[0151] The input unit 116 is used to input the target vector representation into the report generator to obtain a target oral disease image report.

[0152] As can be seen from the above technical solutions, on the one hand, the structured content in each initial sample is extracted and split into multiple entity items. Multiple synonymous entities corresponding to the same entity item in each initial sample are obtained. Under the premise of keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples, thereby achieving text enhancement of the samples. On the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text-enhanced samples. The pre-trained Swin Transformer model is used as the view model to extract features from each image of potential malignant diseases of the oral mucosa and the corresponding expanded image, thereby achieving image enhancement of the samples. This helps to train a more powerful report generator, making the generated oral disease image reports more accurate, so as to assist medical staff in quickly and accurately interpreting the images.

[0153] like Figure 3 The diagram shown is a schematic representation of the structure of a computer device for implementing the oral disease image report generation method of the present invention.

[0154] The computer device 1 may include a memory 12, a processor 13, and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as an oral disease image report generation program.

[0155] Those skilled in the art will understand that the schematic diagram is merely an example of computer device 1 and does not constitute a limitation on computer device 1. Computer device 1 can be either a bus topology or a star topology. Computer device 1 may also include more or fewer other hardware or software than shown in the diagram, or different component arrangements. For example, computer device 1 may also include input / output devices, network access devices, etc.

[0156] It should be noted that the computer device 1 described is merely an example. Other existing or future electronic products that are adaptable to this invention should also be included within the scope of protection of this invention and are incorporated herein by reference.

[0157] The memory 12 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the computer device 1, such as a portable hard drive of the computer device 1. In other embodiments, the memory 12 can be an external storage device of the computer device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 1. Furthermore, the memory 12 can include both internal and external storage units of the computer device 1. The memory 12 can be used not only to store application software and various types of data installed on the computer device 1, such as the code of an oral disease image report generation program, but also to temporarily store data that has been output or will be output.

[0158] In some embodiments, the processor 13 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits packaged with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 13 is the control unit of the computer device 1, connecting various components of the computer device 1 via various interfaces and lines. It executes programs or modules stored in the memory 12 (e.g., executing an oral disease image report generation program) and calls data stored in the memory 12 to perform various functions of the computer device 1 and process data.

[0159] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes these applications to implement the steps in the various oral disease image report generation method embodiments described above, for example... Figure 1 The steps are shown.

[0160] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, which describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, a splitting unit 111, a reassembly unit 112, an expansion unit 113, an extraction unit 114, a training unit 115, and an input unit 116.

[0161] The integrated unit implemented as a software functional module described above can be stored in a computer-readable storage medium. This software functional module, stored in a storage medium, includes several instructions to cause a computer device (which may be a personal computer, computer equipment, or network device, etc.) or processor to execute portions of the oral disease image report generation method described in the various embodiments of the present invention.

[0162] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0163] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0164] Combination Figure 1 The memory 12 in the computer device 1 stores multiple instructions to implement a method for generating oral disease image reports, and the processor 13 can execute the multiple instructions to achieve the following:

[0165] Reports of potential malignant diseases of the oral mucosa were obtained as initial samples;

[0166] Extract the structured content from each initial sample and split the structured content into multiple entity items;

[0167] Obtain multiple synonymous entities corresponding to the same entity item in each initial sample;

[0168] While keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples;

[0169] Obtain the constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model;

[0170] The SimCLR model is used to expand the view of the text-enhanced sample to obtain expanded images of each potential malignant disease of the oral mucosa in the text-enhanced sample;

[0171] The pre-trained Swin Transformer model is used as the view model to extract features from each image of potential malignant oral mucosa disease and its corresponding extended image, resulting in a vector representation of each image.

[0172] The Transformer model is trained using the text-enhanced samples and the vector representation of each image as training data to obtain the report generator;

[0173] In response to the oral disease image report generation instruction based on the target oral mucosa potential malignant disease image, the target vector representation is obtained by extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model;

[0174] The target vector representation is input into the report generator to obtain a target oral disease image report.

[0175] Specifically, the processor 13's implementation method for the above instructions can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating image reports of oral diseases, characterized in that, The method for generating oral disease image reports includes: Reports of potential malignant diseases of the oral mucosa were obtained as initial samples; The structured content is extracted from each initial sample and then broken down into multiple entity items; wherein, the entity items include location, shape, quantity, size, clinical manifestation, lesion boundary, and diagnosis; Obtain multiple synonymous entities corresponding to the same entity item in each initial sample; While keeping the original diagnostic results of each initial sample unchanged, multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text-enhanced samples. This includes: for each initial sample, obtaining each entity item in the initial sample, and obtaining multiple synonymous entities included under each entity item; during each random recombination, randomly selecting any one synonymous entity from the multiple synonymous entities included under each entity item to replace the original entity in the initial sample; and integrating all non-repeating samples obtained after random recombination to obtain the text-enhanced sample. Obtain the constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model; The SimCLR model is used to expand the view of the text-enhanced sample to obtain expanded images of each potential malignant disease of the oral mucosa in the text-enhanced sample; The pre-trained Swin Transformer model is used as the view model to extract features from images of potential malignant diseases of the oral mucosa and their corresponding extended images, resulting in a vector representation of each image; wherein the pre-trained Swin Transformer model possesses relevant knowledge in the corresponding domain. The Transformer model is trained using the text-enhanced samples and the vector representation of each image as training data to obtain the report generator; In response to the oral disease image report generation instruction based on the target oral mucosa potential malignant disease image, the target vector representation is obtained by extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model; The target vector representation is input into the report generator to obtain a target oral disease image report; Specifically, after obtaining the report generator, the accuracy of words and sentences in the report generated by the report generator is detected using the bilingual evaluation alternative BLEU4; the quality of the text summary in the report generated by the report generator is detected using the longest common subsequence ROUGE-L; the acquisition rate of important information by the report generator during the report generation process is detected using the consensus image description evaluation CIDEr; when the accuracy is greater than or equal to the configured accuracy, the text summary quality reaches the configured quality, and the acquisition rate is greater than or equal to the configured acquisition rate, it is determined that the report generator meets the report generation requirements. Before acquiring the constructed image enhancement model, the process further includes: constructing an initial model using the SimCLR model and the view model; acquiring initial training samples and training the initial model using the initial training samples; during the training of the initial model, expanding the view of the initial training samples based on the SimCLR model to obtain expanded samples; calculating the similarity between each initial training sample and the corresponding expanded sample using a cosine similarity algorithm, and calculating the loss function value of the initial model; stopping training when the similarity is higher than the configured similarity and the loss function value reaches convergence; and determining the currently obtained model as the image enhancement model.

2. The method for generating oral disease image reports as described in claim 1, characterized in that, The process of using the SimCLR model to expand the view of the text-enhanced samples to obtain expanded images of each potential malignant oral mucosal disease in the text-enhanced samples includes: The text-enhanced sample is randomly cropped, and the cropped sample is resized to obtain the first image; The colors of the text-enhanced samples are randomly perturbed to obtain a second image; The text-enhanced sample is randomly rotated to obtain a third image; The text-enhanced sample is flipped to obtain the fourth image; The first image, the second image, the third image, and the fourth image are integrated to obtain the extended image.

3. The method for generating oral disease image reports as described in claim 1, characterized in that, The step of inputting the target vector representation into the report generator to obtain a target oral disease image report includes: The encoder of the report generator receives the target vector representation; The target oral disease image report is generated using the decoder of the report generator.

4. The method for generating oral disease image reports as described in claim 1, characterized in that, After obtaining the target oral disease image report, the method further includes: The target oral disease image report is sent to the triggerer of the oral disease image report generation instruction.

5. An oral disease image report generation device, characterized in that, Used to perform the oral disease image report generation method as described in any one of claims 1 to 4.

6. A computer device, characterized in that, The computer device includes: Memory, storing at least one instruction; and The processor executes instructions stored in the memory to implement the oral disease image report generation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, which is executed by a processor in a computer device to implement the oral disease image report generation method as described in any one of claims 1 to 4.

Images