Oral disease image report generation method and device, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and provides an oral disease image report generation method, device, equipment and medium, on one hand, structural content in each initial sample is extracted and split into a plurality of entity items, and on the premise that the original diagnosis result of each initial sample is kept unchanged, the original diagnosis result of each initial sample is obtained; performing random recombination on the plurality of synonymous entities and the plurality of entity items corresponding to each initial sample to obtain a text enhancement sample, and realizing text enhancement of the sample; on the other hand, a SimCLR model in the image enhancement model is adopted to carry out view expansion on the text enhancement sample, and a pre-trained Swin model is used as a view model to carry out feature extraction on each oral mucosa potential malignant disease image and a corresponding expansion image, so that the image enhancement of the sample is realized, a report generator with stronger training performance is assisted, and the image enhancement efficiency is improved. And the generated oral disease image report is more accurate, so that medical personnel can be assisted to quickly and accurately read the image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating an oral disease image report. Background Art

[0002] The importance of interpreting and judging intraoral photographs in early screening for oral potential malignant disorder (OPMD) is self-evident. However, traditional deep learning models are limited in their ability to process multimodal data, making it difficult to extract detailed information from images and generate structured reports. Furthermore, even multimodal research has primarily focused on global reasoning tasks (such as detection), lacking the ability to evaluate large-scale language models for fine-grained image recognition. Consequently, this field still faces numerous challenges.

[0003] First, fusing the modality differences between image input and text output is a fundamental challenge in automatic medical report generation. Medical images contain complex information that must be accurately interpreted and converted into coherent text, which requires complex algorithms to map visual patterns to medical terminology.

[0004] Secondly, medical images exhibit unique visual biases, with the lesion area usually occupying only a small part of the image, resulting in a high degree of similarity between normal and abnormal images. This requires the automatic medical report generation system to be more sensitive to subtle differences than general image caption generation models.

[0005] Furthermore, noise in medical reports, such as temporal information, may confuse the model, leading to inaccuracies or hallucinations.

[0006] Therefore, how to accurately generate oral disease image reports has become an urgent problem to be solved. Summary of the Invention

[0007] In view of the above, it is necessary to provide a method, device, equipment and medium for generating an oral disease image report, aiming to solve the problem of being unable to accurately generate an oral disease image report.

[0008] A method for generating an oral disease image report, the method comprising: Obtain reports of potential malignant diseases of the oral mucosa as an initial sample; Extracting structured content from each initial sample and splitting the structured content into multiple entity items; Obtain multiple synonymous entities corresponding to the same entity term in each initial sample; Under the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly reorganized to obtain text enhancement samples; Acquire a constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model; Performing view expansion on the text enhancement sample using the SimCLR model to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample; Using a pre-trained Swin Transformer model as the view model, feature extraction is performed on each oral mucosal potential malignant disease image and the corresponding extended image to obtain a vector representation of each image; Using the text enhancement samples and the vector representation of each image as training data, a Transformer model is trained to obtain a report generator; In response to an oral disease image report generation instruction based on a target oral mucosa potential malignant disease image, extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model to obtain a target vector representation; The target vector representation is input into the report generator to obtain a target oral disease image report.

[0009] According to a preferred embodiment of the present invention, randomly recombining the multiple synonymous entities and the multiple entity terms corresponding to each initial sample to obtain a text enhancement sample includes: For each initial sample, obtain each entity item in the initial sample, and obtain multiple synonymous entities included in each entity item; During each random reorganization, any one synonymous entity is randomly selected from the multiple synonymous entities included in each entity item to replace the original entity in the initial sample; All non-repeated samples obtained after random recombination are integrated to obtain the text enhancement sample.

[0010] According to a preferred embodiment of the present invention, before acquiring the constructed image enhancement model, the method further includes: Constructing an initial model using the SimCLR model and the view model; Obtaining initial training samples, and using the initial training samples to train the initial model; During the training of the initial model, performing view expansion on the initial training samples based on the SimCLR model to obtain expanded samples; Calculating the similarity between each initial training sample and the corresponding extended sample using a cosine similarity algorithm, and calculating the loss function value of the initial model; When the similarity is higher than the configuration similarity and the loss function value reaches convergence, stop training; The currently obtained model is determined as the image enhancement model.

[0011] According to a preferred embodiment of the present invention, the use of the SimCLR model to perform view expansion on the text enhancement sample to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample includes: Randomly cropping the text enhancement sample and restoring the size of the cropped sample to obtain a first image; Randomly perturbing the color of the text enhancement sample to obtain a second image; Randomly rotating the text enhancement sample to obtain a third image; Flipping the text enhancement sample to obtain a fourth image; The first image, the second image, the third image, and the fourth image are integrated to obtain the extended image.

[0012] According to a preferred embodiment of the present invention, after obtaining the report generator, the method further includes: Using the bilingual evaluation substitute BLEU4 to detect the accuracy of words and sentences in the report generated by the report generator; Using the longest common subsequence ROUGE-L to detect the quality of the text summary in the report generated by the report generator; The consensus image description evaluation (CIDEr) was used to detect the acquisition rate of important information by the report generator during the report generation process; When the accuracy rate is greater than or equal to the configured accuracy rate, the text summary quality reaches the configured quality, and the acquisition rate is greater than or equal to the configured acquisition rate, it is determined that the report generator meets the report generation requirements.

[0013] According to a preferred embodiment of the present invention, inputting the target vector representation into the report generator to obtain a target oral disease image report includes: receiving, with an encoder of the report generator, the scalar vector representation; The target oral disease image report is generated using a decoder of the report generator.

[0014] According to a preferred embodiment of the present invention, after obtaining the target oral disease image report, the method further includes: The target oral disease image report is sent to a trigger of the oral disease image report generation instruction.

[0015] An oral disease image report generating device, the oral disease image report generating device comprising: An acquisition unit, used to obtain reports of potential malignant diseases of the oral mucosa as initial samples; a splitting unit, configured to extract structured content from each initial sample and split the structured content into multiple entity items; The acquisition unit is further configured to acquire multiple synonymous entities corresponding to the same entity term in each initial sample; A recombining unit is used to randomly recombining multiple synonymous entities and multiple entity items corresponding to each initial sample while keeping the original diagnosis result of each initial sample unchanged to obtain a text enhancement sample; The acquisition unit is further configured to acquire a constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model; an expansion unit, configured to perform view expansion on the text enhancement sample using the SimCLR model to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample; an extraction unit, configured to extract features from each oral mucosal potential malignant disease image and the corresponding extended image using a pre-trained Swin Transformer model as the view model, to obtain a vector representation of each image; A training unit, configured to train a Transformer model using the text enhancement samples and the vector representation of each image as training data to obtain a report generator; The extraction unit is further configured to, in response to an oral disease image report generation instruction based on a target oral mucosa potential malignant disease image, perform feature extraction on the target oral mucosa potential malignant disease image using the Swin Transformer model to obtain a target vector representation; An input unit is used to input the target vector representation into the report generator to obtain a target oral disease image report.

[0016] A computer device, comprising: a memory storing at least one instruction; A processor executes instructions stored in the memory to implement the oral disease image report generation method.

[0017] A computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the oral disease image report generation method.

[0018] It can be seen from the above technical solutions that, on the one hand, the structured content in each initial sample is extracted, and the structured content is split into multiple entity items, and multiple synonymous entities corresponding to the same entity item in each initial sample are obtained. On the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text enhancement samples, thereby realizing text enhancement of the samples; on the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text enhancement samples, and the pre-trained Swin Transformer model is used as the view model to extract features of each oral mucosal potential malignant disease image and the corresponding extended image, thereby realizing image enhancement of the samples, thereby assisting in training a more powerful report generator, making the generated oral disease image report more accurate, and assisting medical staff in reading images quickly and accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a preferred embodiment of the method for generating an oral disease image report of the present invention.

[0020] Figure 2 It is a functional module diagram of a preferred embodiment of the oral disease image report generating device of the present invention.

[0021] Figure 3 It is a structural diagram of a computer device according to a preferred embodiment of the present invention for realizing a method for generating an oral disease image report. DETAILED DESCRIPTION

[0022] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the method for generating an oral disease image report according to the present invention. The order of the steps in the flow chart can be changed and some steps can be omitted according to different requirements.

[0024] The oral disease image report generation method is applied to one or more computer devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. Their hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0025] The computer device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), an interactive network television (IPTV), a smart wearable device, etc.

[0026] The computer device may also include a network device and / or a user device, wherein the network device includes, but is not limited to, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers.

[0027] The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0028] Among them, Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0029] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0030] The network where the computer device is located includes but is not limited to the Internet, wide area network, metropolitan area network, local area network, virtual private network (VPN), etc.

[0031] S10, obtain reports of potential malignant diseases of the oral mucosa as an initial sample.

[0032] In this embodiment, the oral mucosal potential malignant disease (OPMD) report can be obtained from a trusted platform to ensure the availability and accuracy of the oral mucosal potential malignant disease report.

[0033] S11, extracting structured content from each initial sample and splitting the structured content into multiple entity items.

[0034] For example, when the initial sample states that "a red patch measuring approximately 0.5 cm × 0.5 cm was found on the patient's tongue, with slight bleeding and clear boundaries, and was preliminarily diagnosed as a potential malignant disease of the oral mucosa", the entity items may include: "location", "shape", "quantity", "size", "clinical manifestations", "lesion boundaries" and "diagnosis".

[0035] S12, obtaining multiple synonymous entities corresponding to the same entity item in each initial sample.

[0036] Continuing with the above example, the entities under each entity can be: Location: tongue; Shape: 0.5cm×0.5cm; Quantity: one place; Size: 0.5cm×0.5cm; Clinical manifestations: slight bleeding; Lesion boundary: clear boundary; Diagnosis: Potential malignant disease of oral mucosa.

[0037] Among them, synonymous entities of the entity "tongue" of the entity item "position" may include "buccal mucosa" and the like.

[0038] S13, under the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly reorganized to obtain a text enhancement sample.

[0039] In this embodiment, randomly recombining multiple synonymous entities and multiple entity terms corresponding to each initial sample to obtain a text enhancement sample includes: For each initial sample, obtain each entity item in the initial sample, and obtain multiple synonymous entities included in each entity item; During each random reorganization, any one synonymous entity is randomly selected from the multiple synonymous entities included in each entity item to replace the original entity in the initial sample; All non-repeated samples obtained after random recombination are integrated to obtain the text enhancement sample.

[0040] For example, an initial sample includes entity item A and entity item B. Entity item A corresponds to synonymous entities X and Y, and entity item B corresponds to synonymous entities M and N. After random recombination, the following four reports can be obtained as the text enhancement samples: Entity item A: X, entity item B: M; Entity A: X, Entity B: N; Entity item A: Y, entity item B: M; Entity A: Y, Entity B: N.

[0041] Through the above embodiments, the report can be text-enhanced without changing the diagnosis results, thereby providing sufficient and reliable samples for subsequent model training.

[0042] S14, obtaining a constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model.

[0043] The SimCLR model is used to enhance images.

[0044] The view model is used to extract important image features.

[0045] In this embodiment, before obtaining the constructed image enhancement model, the method further includes: Constructing an initial model using the SimCLR model and the view model; Obtaining initial training samples, and using the initial training samples to train the initial model; During the training of the initial model, performing view expansion on the initial training samples based on the SimCLR model to obtain expanded samples; Calculating the similarity between each initial training sample and the corresponding extended sample using a cosine similarity algorithm, and calculating the loss function value of the initial model; When the similarity is higher than the configuration similarity and the loss function value reaches convergence, stop training; The currently obtained model is determined as the image enhancement model.

[0046] Among them, the configuration similarity and the loss function value can be comprehensively configured according to model performance and usage environment.

[0047] Through the above embodiments, the similarity between the expanded image and the original image can be evaluated based on cosine similarity, and the effect of the model can be further limited in combination with the loss function, so that the trained image enhancement model has better data processing effect.

[0048] S15, using the SimCLR model to perform view expansion on the text enhancement sample to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample.

[0049] In this embodiment, the SimCLR model is used to perform view expansion on the text enhancement sample to obtain the expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample, including: Randomly cropping the text enhancement sample and restoring the size of the cropped sample to obtain a first image; Randomly perturbing the color of the text enhancement sample to obtain a second image; Randomly rotating the text enhancement sample to obtain a third image; Flipping the text enhancement sample to obtain a fourth image; The first image, the second image, the third image, and the fourth image are integrated to obtain the extended image.

[0050] Through the above embodiments, image enhancement can be further performed on the basis of text enhancement, which significantly improves the generalization ability and data utilization, and effectively solves the problem of scarcity of oral disease image report samples.

[0051] S16, using a pre-trained Swin Transformer model as the view model to perform feature extraction on each oral mucosal potential malignant disease image and the corresponding extended image to obtain a vector representation of each image.

[0052] In this embodiment, the use of a pre-trained Swin Transformer model can improve the training efficiency of the model. Moreover, the pre-trained Swin Transformer model already has relevant knowledge in the corresponding field, thereby also improving the accuracy of the model.

[0053] S17, using the text enhancement samples and the vector representation of each image as training data to train a Transformer model to obtain a report generator.

[0054] In this embodiment, in order to make a fair comparison with the existing model, the effectiveness of the report generator needs to be evaluated as part of the report generation process and report quality.

[0055] Specifically, after obtaining the report generator, the method further includes: Using the Bilingual Evaluation Understudy 4 (BLEU4) to test the accuracy of words and sentences in the report generated by the report generator; Using the longest common subsequence ROUGE-L (Recall-Oriented Understudy for GistingEvaluation-Longest Common Subsequence) to detect the quality of the text summary in the report generated by the report generator; The consensus-based image description evaluation (CIDEr) is used to detect the acquisition rate of important information by the report generator during the report generation process. When the accuracy rate is greater than or equal to the configured accuracy rate, the text summary quality reaches the configured quality, and the acquisition rate is greater than or equal to the configured acquisition rate, it is determined that the report generator meets the report generation requirements.

[0056] Among them, the configuration accuracy, the configuration quality, and the configuration acquisition rate can be configured according to actual needs.

[0057] Through the above embodiments, the usability of the trained report generator can be comprehensively evaluated by combining multi-dimensional indicators.

[0058] In this embodiment, inputting the target vector representation into the report generator to obtain a target oral disease image report includes: receiving, with an encoder of the report generator, the scalar vector representation; The target oral disease image report is generated using a decoder of the report generator.

[0059] Through the above embodiments, an oral disease image report can be automatically generated based on the trained report generator.

[0060] S18 , in response to an oral disease image report generation instruction based on a target oral mucosa potential malignant disease image, extract features of the target oral mucosa potential malignant disease image using the Swin Transformer model to obtain a target vector representation.

[0061] In this embodiment, the oral disease image report generation instruction can be triggered by relevant medical staff to promptly respond to the medical staff's image reading needs.

[0062] S19, inputting the target vector representation into the report generator to obtain a target oral disease image report.

[0063] In this embodiment, after obtaining the target oral disease image report, the method further includes: The target oral disease image report is sent to a trigger of the oral disease image report generation instruction.

[0064] For example, the trigger may be a relevant medical staff member. Sending the target oral disease image report to the medical staff member can assist the medical staff member in quickly reading the image. The more accurate the target oral disease image report is, the better the medical staff member can read the image in combination with the target oral disease image report.

[0065] It can be seen from the above technical solutions that, on the one hand, the structured content in each initial sample is extracted, and the structured content is split into multiple entity items, and multiple synonymous entities corresponding to the same entity item in each initial sample are obtained. On the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text enhancement samples, thereby realizing text enhancement of the samples; on the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text enhancement samples, and the pre-trained Swin Transformer model is used as the view model to extract features of each oral mucosal potential malignant disease image and the corresponding extended image, thereby realizing image enhancement of the samples, thereby assisting in training a more powerful report generator, making the generated oral disease image report more accurate, and assisting medical staff in reading images quickly and accurately.

[0066] like Figure 2 , which is a functional module diagram of a preferred embodiment of the oral disease image report generating device of the present invention. The oral disease image report generating device 11 includes an acquisition unit 110, a splitting unit 111, a reassembly unit 112, an expansion unit 113, an extraction unit 114, a training unit 115, and an input unit 116. As used herein, a module / unit refers to a series of computer program segments that can be executed by a processor and perform fixed functions, and are stored in a memory. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0067] The acquisition unit 110 is used to acquire a report on a potential malignant disease of the oral mucosa as an initial sample; The splitting unit 111 is used to extract structured content from each initial sample and split the structured content into multiple entity items; The acquisition unit 110 is further configured to acquire multiple synonymous entities corresponding to the same entity term in each initial sample; The recombining unit 112 is used to randomly recombining multiple synonymous entities and multiple entity items corresponding to each initial sample while keeping the original diagnosis result of each initial sample unchanged, so as to obtain a text enhancement sample; The acquisition unit 110 is further configured to acquire a constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model; The expansion unit 113 is configured to perform view expansion on the text enhancement sample using the SimCLR model to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample; The extraction unit 114 is configured to extract features from each oral mucosal potential malignant disease image and the corresponding extended image using a pre-trained Swin Transformer model as the view model to obtain a vector representation of each image; The training unit 115 is configured to train a Transformer model using the text enhancement samples and the vector representation of each image as training data to obtain a report generator; The extraction unit 114 is further configured to, in response to an oral disease image report generation instruction based on a target oral mucosa potential malignant disease image, extract features of the target oral mucosa potential malignant disease image using the Swin Transformer model to obtain a target vector representation; The input unit 116 is used to input the target vector representation into the report generator to obtain a target oral disease image report.

[0068] It can be seen from the above technical solutions that, on the one hand, the structured content in each initial sample is extracted, and the structured content is split into multiple entity items, and multiple synonymous entities corresponding to the same entity item in each initial sample are obtained. On the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly recombined to obtain text enhancement samples, thereby realizing text enhancement of the samples; on the other hand, the SimCLR model in the image enhancement model is used to expand the view of the text enhancement samples, and the pre-trained Swin Transformer model is used as the view model to extract features of each oral mucosal potential malignant disease image and the corresponding extended image, thereby realizing image enhancement of the samples, thereby assisting in training a more powerful report generator, making the generated oral disease image report more accurate, and assisting medical staff in reading images quickly and accurately.

[0069] like Figure 3 FIG. 1 is a schematic diagram of the structure of a computer device for implementing a preferred embodiment of the method for generating an oral disease image report according to the present invention.

[0070] The computer device 1 may include a memory 12, a processor 13 and a bus (the arrow in the figure represents the bus), and may also include a computer program stored in the memory 12 and executable on the processor 13, such as an oral disease image report generation program.

[0071] Those skilled in the art will understand that the schematic diagram is merely an example of the computer device 1 and does not constitute a limitation on the computer device 1. The computer device 1 may have either a bus structure or a star structure. The computer device 1 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components. For example, the computer device 1 may also include input and output devices, network access devices, etc.

[0072] It should be noted that the computer device 1 is only an example. Other existing or future electronic products that are suitable for the present invention should also be included in the scope of protection of the present invention and included here by reference.

[0073] The memory 12 includes at least one type of readable storage medium, including flash memory, a removable hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the computer device 1, such as a removable hard disk of the computer device 1. In other embodiments, the memory 12 may also be an external storage device of the computer device 1, such as a plug-in removable hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the computer device 1. Furthermore, the memory 12 may include both an internal storage unit of the computer device 1 and an external storage device. The memory 12 can be used not only to store application software installed on the computer device 1 and various types of data, such as the code of the oral disease image report generation program, but also to temporarily store data that has been output or is about to be output.

[0074] In some embodiments, the processor 13 may be comprised of an integrated circuit, such as a single packaged integrated circuit or multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (control unit) of the computer device 1, connecting the various components of the computer device 1 using various interfaces and circuits. It executes programs or modules stored in the memory 12 (e.g., a program for generating oral disease image reports) and accesses data stored in the memory 12 to perform various functions and process data.

[0075] The processor 13 executes the operating system of the computer device 1 and various installed applications. The processor 13 executes the applications to implement the steps in the above-mentioned embodiments of the method for generating an oral disease image report, for example Figure 1 Steps shown.

[0076] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to implement the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device 1. For example, the computer program may be divided into an acquisition unit 110, a splitting unit 111, a reassembly unit 112, an expansion unit 113, an extraction unit 114, a training unit 115, and an input unit 116.

[0077] The integrated unit implemented as a software functional module can be stored in a computer-readable storage medium. The software functional module stored in the storage medium includes instructions for causing a computer device (which can be a personal computer, computer device, or network device, etc.) or a processor to execute portions of the oral disease image report generation method described in various embodiments of the present invention.

[0078] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0079] It will be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the computer device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0080] Combine Figure 1 The memory 12 in the computer device 1 stores a plurality of instructions to implement a method for generating an oral disease image report, and the processor 13 can execute the plurality of instructions to implement: Obtain reports of potential malignant diseases of the oral mucosa as an initial sample; Extracting structured content from each initial sample and splitting the structured content into multiple entity items; Obtain multiple synonymous entities corresponding to the same entity term in each initial sample; Under the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly reorganized to obtain text enhancement samples; Acquire a constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model; Performing view expansion on the text enhancement sample using the SimCLR model to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample; Using a pre-trained Swin Transformer model as the view model, feature extraction is performed on each oral mucosal potential malignant disease image and the corresponding extended image to obtain a vector representation of each image; Using the text enhancement samples and the vector representation of each image as training data, a Transformer model is trained to obtain a report generator; In response to an oral disease image report generation instruction based on a target oral mucosa potential malignant disease image, extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model to obtain a target vector representation; The target vector representation is input into the report generator to obtain a target oral disease image report.

[0081] Specifically, the specific implementation method of the processor 13 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating an oral disease image report, characterized in that: The oral disease image report generation method comprises: Obtain reports of potential malignant diseases of the oral mucosa as an initial sample; Extracting structured content from each initial sample and splitting the structured content into multiple entity items; Obtain multiple synonymous entities corresponding to the same entity term in each initial sample; Under the premise of keeping the original diagnosis result of each initial sample unchanged, the multiple synonymous entities and multiple entity items corresponding to each initial sample are randomly reorganized to obtain text enhancement samples; Acquire a constructed image enhancement model; wherein the image enhancement model includes a SimCLR model and a view model; Performing view expansion on the text enhancement sample using the SimCLR model to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample; Using a pre-trained Swin Transformer model as the view model, feature extraction is performed on each oral mucosal potential malignant disease image and the corresponding extended image to obtain a vector representation of each image; Using the text enhancement samples and the vector representation of each image as training data, a Transformer model is trained to obtain a report generator; In response to an oral disease image report generation instruction based on a target oral mucosa potential malignant disease image, extracting features from the target oral mucosa potential malignant disease image using the Swin Transformer model to obtain a target vector representation; The target vector representation is input into the report generator to obtain a target oral disease image report.

2. The method for generating an oral disease image report according to claim 1, wherein: The text enhancement sample obtained by randomly recombining multiple synonymous entities and multiple entity items corresponding to each initial sample includes: For each initial sample, obtain each entity item in the initial sample, and obtain multiple synonymous entities included in each entity item; During each random reorganization, any one synonymous entity is randomly selected from the multiple synonymous entities included in each entity item to replace the original entity in the initial sample; All non-repeated samples obtained after random recombination are integrated to obtain the text enhancement sample.

3. The method for generating an oral disease image report according to claim 1, wherein: Before acquiring the constructed image enhancement model, the method further includes: Constructing an initial model using the SimCLR model and the view model; Obtaining initial training samples, and using the initial training samples to train the initial model; During the training of the initial model, performing view expansion on the initial training samples based on the SimCLR model to obtain expanded samples; Calculating the similarity between each initial training sample and the corresponding extended sample using a cosine similarity algorithm, and calculating the loss function value of the initial model; When the similarity is higher than the configuration similarity and the loss function value reaches convergence, stop training; The currently obtained model is determined as the image enhancement model.

4. The method for generating an oral disease image report according to claim 1, wherein: The SimCLR model is used to perform view expansion on the text enhancement sample to obtain an expanded image of each potential malignant disease image of the oral mucosa in the text enhancement sample, including: Randomly cropping the text enhancement sample and restoring the size of the cropped sample to obtain a first image; Randomly perturbing the color of the text enhancement sample to obtain a second image; Randomly rotating the text enhancement sample to obtain a third image; Flipping the text enhancement sample to obtain a fourth image; The first image, the second image, the third image, and the fourth image are integrated to obtain the extended image.

5. The method for generating an oral disease image report according to claim 1, wherein: After obtaining the report generator, the method further includes: Using the bilingual evaluation substitute BLEU4 to detect the accuracy of words and sentences in the report generated by the report generator; Using the longest common subsequence ROUGE-L to detect the quality of the text summary in the report generated by the report generator; The consensus image description evaluation (CIDEr) was used to detect the acquisition rate of important information by the report generator during the report generation process; When the accuracy rate is greater than or equal to the configured accuracy rate, the text summary quality reaches the configured quality, and the acquisition rate is greater than or equal to the configured acquisition rate, it is determined that the report generator meets the report generation requirements.

6. The method for generating an oral disease image report according to claim 1, wherein: Inputting the target vector representation into the report generator to obtain a target oral disease image report includes: receiving, with an encoder of the report generator, the scalar vector representation; The target oral disease image report is generated using a decoder of the report generator.

7. The method for generating an oral disease image report according to claim 1, wherein: After obtaining the target oral disease image report, the method further includes: The target oral disease image report is sent to a trigger of the oral disease image report generation instruction.

8. An oral disease image report generating device, characterized in that: Used to execute the oral disease image report generation method as described in any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device comprises: a memory storing at least one instruction; and A processor executes instructions stored in the memory to implement the oral disease image report generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and the at least one instruction is executed by a processor in a computer device to implement the oral disease image report generation method according to any one of claims 1 to 7.

Images