Stomach infection diagnosis method and device, storage medium and program product

Abstract

The invention discloses a stomach infection diagnosis method and device, a storage medium and a program product. Comprising the steps that a stomach endoscopic image set of a to-be-diagnosed case is acquired, and the stomach endoscopic image set comprises stomach endoscopic images corresponding to at least two stomach parts; the stomach endoscopic image set and a part description text are input into the fine-adjusted part recognition model, target stomach parts corresponding to different stomach endoscopic images are determined, and the part description text is used for describing at least two stomach parts; based on expert diagnosis maps of different target stomach parts, screening a suspected infection image set from the stomach endoscopic image set; based on the suspected infection image set, the stomach infection condition of the to-be-diagnosed case is determined, and the stomach infection condition is one of infection, non-infection and once infection. By adopting the method provided by the embodiment of the invention, the accuracy of stomach infection condition diagnosis can be improved, and the complexity of the adopted model can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer medical technology, and in particular to a gastric infection diagnosis method, device, storage medium and program product. Background Art

[0002] Nowadays, gastric infection is a common type of gastric disease, which may include bacterial infection. For example, Helicobacter pylori (H pylori) is a Gram-negative bacterium that mainly colonizes the human stomach. It destroys the gastric mucosal barrier through various pathogenic mechanisms, triggering an inflammatory response, thereby promoting the occurrence and development of gastric diseases. Nowadays, the diagnosis of gastric infection can be performed by endoscopic image analysis.

[0003] In the related art, an image intelligent recognition model is used to determine whether there is an infection based on endoscopic images. In the process of training the image intelligent model, a large number of sample images with fine annotations are required as training samples so that the image intelligent recognition model can realize the diagnosis of the case, and the corresponding deep learning model is trained for the endoscopic images of different gastric parts to realize the diagnosis of different gastric parts.

[0004] However, the solutions provided by related technologies often only judge whether an infection is based on endoscopic images, which results in the need for a large number of finely labeled sample images in the training process. In addition, endoscopic images of different parts can only be diagnosed using corresponding models, which also increases the demand for training samples. In the real world, most cases do not have fine annotations, which results in poor training results and low accuracy of intelligent diagnosis. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device, storage medium and program product for diagnosing gastric infection, aiming to combine gastric endoscopic images with part description text to accurately determine the part to which the gastric endoscopic image belongs, which is conducive to further improving the accuracy of the diagnosis result.

[0006] To achieve the above objectives, on the one hand, the present application provides a method for diagnosing gastric infection, the method comprising:

[0007] Acquire a set of gastric endoscopic images of a case to be diagnosed, wherein the set of gastric endoscopic images includes gastric endoscopic images corresponding to at least two gastric parts;

[0008] Inputting the set of gastric endoscopic images and the part description text into the fine-tuned part recognition model to determine the target gastric parts corresponding to different gastric endoscopic images, wherein the part description text is used to describe at least two gastric parts;

[0009] Based on expert diagnosis atlases of different target stomach parts, screening out a suspected infection image set from the stomach endoscopic image set;

[0010] Based on the suspected infection image set, the stomach infection condition of the case to be diagnosed is determined, and the stomach infection condition is one of infection, non-infection and previous infection.

[0011] As a further improvement of the present application, the step of determining the target stomach location corresponding to different stomach endoscopic images includes:

[0012] Performing image encoding on the gastric endoscopic image by using an image encoder in the part recognition model to obtain a first endoscopic image vector;

[0013] Performing text encoding on the part description text by using a text encoder in the part recognition model to obtain a description text vector;

[0014] The target stomach part is determined based on the similarity between the first endoscopic image vector and the description text vectors corresponding to different parts.

[0015] As a further improvement of the present application, the step of determining the target stomach part based on the similarity between the endoscopic image vector and the description text vectors corresponding to different parts includes:

[0016] Calculating the cosine similarity between the endoscopic image vector and different description text vectors through the part recognition model;

[0017] When the cosine similarity between the endoscopic image vector and the first description text vector is higher than a first similarity threshold, the stomach part corresponding to the first description text vector is determined to be the target stomach part.

[0018] As a further improvement of the present application, the expert diagnosis atlas based on different target stomach parts, screening out a suspected infection image set from the stomach endoscopic image set, includes:

[0019] Performing image encoding on the gastric endoscopic image of the target gastric part by using the image encoder in the part recognition model to obtain a second endoscopic image vector;

[0020] By means of the image encoder, image encoding is performed on the expert diagnosis atlas of the target stomach part to obtain a diagnosis atlas vector;

[0021] determining a cosine similarity between the second endoscopic image vector and the diagnostic atlas vector;

[0022] When the cosine similarity corresponding to the second endoscopic image vector is higher than a second similarity threshold, determining that the gastric endoscopic image of the target stomach part is a suspected infection image;

[0023] Each gastric endoscopic image in the gastric endoscopic image set is traversed, and different suspected infection images form the suspected infection image set.

[0024] As a further improvement of the present application, one of the target stomach parts corresponds to at least two of the expert diagnosis atlases, and the determining of the cosine similarity between the second endoscopic image vector and the diagnostic atlas vector includes:

[0025] Flattening the diagnostic map vectors corresponding to at least two of the expert diagnostic maps to obtain a one-dimensional diagnostic map vector;

[0026] Determine the cosine similarity between the second endoscopic image vector and the one-dimensional diagnostic atlas vector.

[0027] As a further improvement of the present application, the step of determining the stomach infection condition of the case to be diagnosed based on the suspected infection image set includes:

[0028] By using the image encoder in the part recognition model, each suspected infection image in the suspected infection image set is image encoded to obtain a suspected infection image vector;

[0029] Determining the probability distribution of the case to be diagnosed in different infection situations based on the suspected infection image vector by using a diagnostic model;

[0030] Based on the probability distribution, the stomach infection condition of the case to be diagnosed is determined.

[0031] As a further improvement of the present application, the method further includes:

[0032] Performing image filtering on the suspected infection image set to obtain a filtered target suspected infection image set;

[0033] The step of determining the stomach infection condition of the case to be diagnosed based on the suspected infection image set includes:

[0034] Based on the filtered target suspected infection image set, the stomach infection condition of the case to be diagnosed is determined.

[0035] As a further improvement of the present application, the image filtering includes image deduplication filtering and image overexposure filtering;

[0036] The image filtering is performed on the suspected infection image set to obtain a filtered target suspected infection image set, including:

[0037] Calculating the cosine similarity between every two suspected infection images of the same target stomach part; in the case where the cosine similarity between the first suspected infection image and the second suspected infection image is higher than the third similarity threshold, determining that the first suspected infection image and the second suspected infection image are duplicate images; performing deduplication filtering on the duplicate images to obtain a set of intermediate suspected infection images after the first filtering;

[0038] Determine the grayscale histogram of each suspected infected image in the intermediate suspected infected image set; perform overexposure filtering on the suspected infected images in the suspected infected image set based on the grayscale histogram, filter out the overexposed images, and obtain the target suspected infected image set.

[0039] As a further improvement of the present application, the method further includes:

[0040] Acquire a training sample pair, wherein the training sample pair includes a sample gastric endoscopic image and a sample text, wherein the sample text is used to characterize a gastric part, a lesion label, and a text of microscopic findings corresponding to the sample gastric endoscopic image;

[0041] The pre-trained model is fine-tuned based on the training samples to obtain the fine-tuned part recognition model.

[0042] As a further improvement of the present application, the diagnostic model is obtained by training an untrained diagnostic model based on sample infection images and infection status labels.

[0043] On the other hand, the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps in the gastric infection diagnosis method provided in any of the above aspects.

[0044] On the other hand, the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the steps in the gastric infection diagnosis method provided in any of the above aspects.

[0045] On the other hand, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the gastric infection diagnosis method provided in any of the above aspects.

[0046] The solution provided in the embodiment of the present application has at least the following beneficial effects:

[0047] In an embodiment of the present application, the stomach part to which the stomach endoscopic image belongs is determined based on the gastric endoscopic image and the part description text, so that the part judgment is more accurate. Subsequently, based on the expert diagnosis atlas, the gastric endoscopic image of the target stomach part has been determined for further judgment, thereby further enhancing the accuracy of the judgment of the stomach part to which the gastric endoscopic model belongs. Compared with the diagnosis method that relies solely on endoscopic images in the related art, the diagnostic accuracy is improved. In addition, the solution provided in the present application fine-tunes the part recognition model, and does not require a large number of finely labeled sample images. When making a diagnosis, there is no need to use different models for images of different parts, which further reduces the demand for the number of samples, and reduces the complexity of the model structure, thereby improving the applicability of the model in actual scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flow chart of a method for diagnosing gastric infection provided by an exemplary embodiment of the present application is shown;

[0049] Figure 2 A flowchart of a process for determining target stomach locations corresponding to different stomach endoscopic images provided by an exemplary embodiment of the present application is shown;

[0050] Figure 3 A flowchart of fine-tuning a pre-trained model provided by an exemplary embodiment of the present application is shown;

[0051] Figure 4 A flow chart of a gastric infection diagnosis process provided by an exemplary embodiment of the present application is shown;

[0052] Figure 5 A flowchart showing a process of fine-tuning a diagnostic model provided by an exemplary embodiment of the present application is shown;

[0053] Figure 6 A schematic diagram of a process for diagnosing gastric infection provided by an exemplary embodiment of the present application is shown;

[0054] Figure 7 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional changes made by a person skilled in the art based on these embodiments are all within the scope of protection of the present invention.

[0056] It should be noted that the term "comprises" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0057] It should be noted that the solution provided in the embodiment of the present application is executed by an electronic device, which may be a personal computer, a laptop, a server, a digital video recorder, a smart mobile device, etc. In the following embodiment, a computer device is taken as an example to illustrate the denoising method of a video frame image.

[0058] Please refer to Figure 1 , which shows a flow chart of a method for diagnosing gastric infection provided by an exemplary embodiment of the present application. The method comprises the following steps:

[0059] Step 101, obtaining a set of gastric endoscopic images of a case to be diagnosed.

[0060] The gastric endoscopic image set includes gastric endoscopic images corresponding to at least two gastric parts.

[0061] When acquiring gastric endoscopic images, it is necessary to adopt a professional gastric endoscopic image acquisition method, such as electronic gastroscopy, gastrointestinal capsule endoscopy, etc. Optionally, when a continuous gastric endoscopic image video stream is acquired through a gastric examination device, multiple frames of gastric endoscopic images are acquired from the video stream by sampling to form a gastric endoscopic image set, and the quality and clarity of the sampled gastric endoscopic images need to be guaranteed. After acquiring the gastric endoscopic images, they are stored in a database in a certain format for subsequent processing.

[0062] Step 102: Input the set of gastric endoscopic images and the part description text into the fine-tuned part recognition model to determine the target gastric parts corresponding to different gastric endoscopic images.

[0063] The part description text is used to describe at least two stomach parts, and the at least two stomach parts may include the gastric body, the gastric antrum, etc.

[0064] The gastric endoscopic image collection includes multiple images captured from the stomach, and these images have characteristics of different stomach structures and morphologies.

[0065] The fine-tuned part recognition model learns the feature representation of different parts in the gastric endoscopic image during the fine-tuning stage. In this step, the part recognition model matches the features of the input gastric endoscopic image with the features described in the part description text, thereby determining the gastric part to which the gastric endoscopic image belongs. The gastric description text describes at least two gastric parts in concise and accurate language, such as "gastric body part" and "gastric antrum part", which is helpful for assisting the part recognition model to locate the part features in the endoscopic image.

[0066] Step 103, based on the expert diagnosis atlas of different target stomach parts, a set of suspected infection images is screened out from the set of stomach endoscopic images.

[0067] The expert diagnosis atlas is constructed based on the analysis of endoscopic images of a large number of clinical gastric infection cases, and the characteristics are summarized and summarized to construct atlases for different gastric parts. For example, for the expert H pylori (Helicobacter pylori) atlas, it includes capsule endoscopy gastric body and gastric antrum images under different infection states given by doctor experts.

[0068] Since there are examples of different infection states in the expert diagnosis atlas of the target stomach area, it is possible to screen based on the expert atlas, preliminarily screen out images of suspected infection in each stomach area, and then further diagnose its infection state through subsequent steps.

[0069] Furthermore, the expert diagnostic atlas can be regularly updated and maintained.

[0070] Step 104: Determine the stomach infection status of the case to be diagnosed based on the suspected infection image set.

[0071] The gastric infection status is one of infection, non-infection and previous infection. The previous infection is the sterilization state.

[0072] Optionally, the gastric infection situation can be determined based on the number, size, distribution and other information of the lesion areas in the suspected infection image in combination with pre-set diagnostic rules.

[0073] Optionally, a diagnostic model is used to determine the gastric infection status, and the diagnostic model obtains a diagnosis result of the infection status based on image features of gastric endoscopic images of different parts.

[0074] In summary, in the embodiment of the present application, the stomach part to which the stomach endoscopic image belongs is determined based on the gastric endoscopic image and the part description text, and the part judgment is more accurate. Subsequently, based on the expert diagnosis atlas, the gastric endoscopic image of the target stomach part has been determined for further judgment, thereby further enhancing the accuracy of the judgment of the stomach part to which the gastric endoscopic model belongs. Compared with the diagnosis method that relies solely on endoscopic images in the related art, the diagnostic accuracy is improved. In addition, the solution provided in the present application fine-tunes the part recognition model, and does not require a large number of finely labeled sample images. When making a diagnosis, there is no need to use different models for images of different parts, which further reduces the demand for the number of samples, and reduces the complexity of the model structure, thereby improving the applicability of the model in actual scenarios.

[0075] The process of determining the target stomach image corresponding to different stomach endoscopic images is actually a process of matching the two based on image features and text features. The process of determining the target stomach location based on an image recognition model will be described below through an illustrative embodiment.

[0076] Please refer to Figure 2 , which shows a flow chart of a process for determining target stomach locations corresponding to different stomach endoscopic images provided by an exemplary embodiment of the present application. The process includes the following steps:

[0077] Step 201 : perform image encoding on a gastric endoscopic image by using an image encoder in a part recognition model to obtain a first endoscopic image vector.

[0078] The image encoder is the main component in the part recognition model. Through image encoding, the visual data of gastric endoscopic images is converted into a vector representation that can be efficiently processed and analyzed by computers.

[0079] Optionally, the image encoder in the part recognition model may be built based on a convolutional neural network architecture.

[0080] Step 202: Encode the part description text by using the text encoder in the part recognition model to obtain a description text vector.

[0081] Similar to the image encoder, the text encoder in the part recognition model can convert the part description text in natural language into a vector form that is easy for computers to process and analyze, thereby ensuring that the text information can be effectively integrated with the image information in the subsequent process, thereby assisting the part recognition model to accurately determine the target stomach part corresponding to the gastric endoscopic image.

[0082] Step 203: Determine the target stomach part based on the similarity between the first endoscopic image vector and the description text vectors corresponding to different parts.

[0083] The first endoscopic image vector contains key visual features in the gastric endoscopic image, and the description text contains semantic features in the corresponding part description text. Comparing the similarity between the two is actually looking for the gastric part whose presentation features of the endoscopic image best match the text features of the description text, so as to accurately determine the target gastric part corresponding to the endoscopic image.

[0084] Optionally, the similarity between the first endoscopic image vector and the description text vector may be determined by using different vector similarity measurement methods, such as cosine similarity, the inverse of the Euclidean distance, and the Pearson correlation coefficient.

[0085] In a possible implementation, the computer device determines the target stomach part based on cosine similarity. First, the computer device calculates the cosine similarity between the endoscopic image vector and different description text vectors through a part recognition model.

[0086] For example, there are two description texts for the stomach, the antrum and the body, and the corresponding description text vectors are V 胃窦 and V 胃体 , and the first endoscopic image vector of one of the endoscopic images is V 图像 , then it is necessary to calculate the cosine similarity between the first endoscopic image vector and different description text vectors, and the two cosine similarities are S 胃窦 and S 胃体 .

[0087] Subsequently, when the cosine similarity between the endoscopic image vector and the first description text vector is higher than a first similarity threshold, the computer device determines that the stomach part corresponding to the first description text vector is the target stomach part.

[0088] For example, if the first similarity threshold is 0.96, then in S 胃窦 If S is greater than 0.96, the target stomach part corresponding to the endoscopic image is determined to be the gastric antrum. 胃体 If it is greater than 0.96, it is determined that the target stomach part corresponding to the endoscopic image is the gastric body.

[0089] The computer device performs the above process on all stomach images in the stomach endoscopic image set, so as to determine the target stomach parts corresponding to different stomach endoscopic images.

[0090] The site recognition model determines the target stomach site based on calculating similarity, making full use of information in both image and text dimensions to accurately determine the target stomach site corresponding to each gastric endoscopic image, laying the foundation for subsequent infection diagnosis.

[0091] Optionally, a CLIP (Contrastive Language-Image Pretraining) model can be used as a part recognition model. Before the part recognition model is applied to the scene of the embodiment of the present application for stomach part recognition, the computer device needs to fine-tune the pre-trained model so that it can perform specific learning for the stomach infection diagnosis scene, thereby realizing stomach part recognition. The process of fine-tuning the part recognition model is described below.

[0092] First, a training sample pair is obtained, wherein the training sample pair includes a sample gastric endoscopic image and a sample text.

[0093] The sample text is used to characterize the stomach part, lesion label and microscopic findings text corresponding to the sample gastric endoscopic image.

[0094] The training sample pairs are derived from actual gastric endoscopy processes, where the sample gastric endoscopy images include real gastric endoscopy images of different gastric parts obtained by multiple patients under different infection conditions. The descriptive text is the text information corresponding to these real gastric endoscopy images, specifically including the stomach parts corresponding to the sample gastric endoscopy images. And it includes lesion labels, such as "gastritis lesions", "gastric ulcers" and "bacterial infections", etc. In addition, the text of microscopic findings refers to the doctor's record of the gastric mucosal conditions, vascular textures, etc. observed in the image during clinical diagnosis.

[0095] Subsequently, the pre-trained model is fine-tuned based on the training samples to obtain a fine-tuned part recognition model.

[0096] Among them, the pre-trained model is a model that has been pre-trained through large-scale general image and text data, and has the ability to associate images and texts.

[0097] In the process of fine-tuning the pre-trained model based on the training samples, the computer device inputs the training samples into the pre-trained model. In the process of forward propagation of the model, the sample gastric endoscopic image is encoded by the image encoder to obtain the sample image feature vector, and the sample text is encoded by the text encoder to obtain the sample text feature vector. During the training process, the core idea is to maximize the similarity between matching samples (sample image features and sample text features) while minimizing the similarity between non-matching samples.

[0098] Optionally, the matching loss can be calculated using the loss function shown below. The loss function formula is:

[0099]

[0100] Where N is the batch size, and are the sample image features of the i-th sample gastric endoscopic image and the sample text feature vector of the i-th sample text. τ is the temperature parameter used to scale the cosine similarity.

[0101] in, Represents the cosine similarity between the sample image feature vector and the sample text feature vector. The formula for calculating cosine similarity is:

[0102]

[0103] Cosine similarity is the dot product of the sample image feature vector and the sample text feature vector divided by the product of the lengths of the two vectors.

[0104] In each training cycle, training iterations are performed, the loss function is calculated, and the loss gradient is back-propagated to update the parameters of the part recognition model.

[0105] Optionally, during multiple rounds of training, the computer device retains the model structure and parameters after each round of training iteration, verifies it based on the data in the verification sample pair, and selects the model with the smallest average loss among all the data in the verification sample pair as the optimal model as the part recognition model.

[0106] After obtaining the sample gastric endoscopic images and sample texts, the computer device divides them according to a certain ratio, and divides them into training sample pairs and verification sample pairs, and the verification sample pairs are used to test the performance of the fine-tuned part recognition model. The ratio can be 7:3 or 6:4. Optionally, the number of infected case samples and the number of uninfected case samples in the training sample pairs and the verification sample pairs should be roughly equal.

[0107] Please refer to Figure 3 , which shows a flow chart of fine-tuning a pre-trained model provided by an illustrative embodiment of the present application. First, prepare training sample pairs, that is, obtain paired data of sample gastric endoscopic images and sample texts, and use a certain proportion of the data as a training data set. Then, fine-tune the pre-trained model based on the training data set. Finally, a trained part recognition model is obtained.

[0108] In an embodiment of the present application, the pre-trained model is fine-tuned based on the training samples so that the pre-trained large model can be applicable to the gastric diagnosis scenario, and the part recognition model is applied, that is, in the process of determining the target stomach part based on the input gastric endoscopic image and the part description text using the part recognition model, the text and image are matched based on the similarity between the text features and the image features, that is, the target stomach part is determined. Compared with part recognition based only on images, the solution provided in this embodiment requires fewer samples for training, and the samples do not need to be accurately labeled, which has a better training effect, thereby improving the accuracy of part recognition.

[0109] In the embodiment of the present application, after determining the target site, it is necessary to perform expert atlas screening and site diagnosis. This process will directly affect the final diagnosis result. The process of gastric diagnosis will be described below through an illustrative embodiment.

[0110] Please refer to Figure 4 , which shows a flow chart of a gastric infection diagnosis process provided by an exemplary embodiment of the present application. The process includes the following steps:

[0111] Step 401 : performing image encoding on a gastric endoscopic image of a target gastric part by using an image encoder in a part recognition model to obtain a second endoscopic image vector.

[0112] Optionally, when 25+5 gastric endoscopic images of the case to be diagnosed are selected, the computer device encodes the gastric endoscopic images through an image encoder to obtain a feature vector with a scale of [1, 768]. Subsequently, the feature vectors are repeatedly concatenated to obtain a second endoscopic image vector with a scale of [1, 25*768] or [1, 5*768].

[0113] Step 402, image encoding is performed on the expert diagnostic atlas of the target stomach area through an image encoder to obtain a diagnostic atlas vector.

[0114] Optionally, in the case where there are gastric endoscopic images of two gastric parts, in the expert diagnostic atlas, the atlas data of the two parts (25 images, 5 images) are encoded by the computer device through an image encoder to obtain diagnostic atlas vectors with scales of [25, 768] and [5, 768].

[0115] Step 403, determining the cosine similarity between the second endoscopic image vector and the diagnostic atlas vector.

[0116] When determining the cosine similarity between the second endoscopic image vector and the diagnostic atlas vector, each gastric endoscopic image is compared for similarity with the fusion features of 25 expert diagnostic atlases and the fusion features of 5 expert diagnostic atlases.

[0117] Since the dimensions of the second endoscopic image vector and the expert diagnosis atlas vector are not the same after image encoding, the computer device can use a vector flattening operation to first process the diagnosis atlas vector and then perform a similarity comparison.

[0118] In a possible implementation, the computer device flattens the diagnostic atlas vectors corresponding to the at least two expert diagnostic atlases to obtain a one-dimensional diagnostic atlas vector. Subsequently, the cosine similarity between the second endoscopic image vector and the one-dimensional diagnostic atlas vector is determined.

[0119] Among them, the vector flattening operation refers to expanding the vector according to a certain rule or method for further calculation or analysis. For example, the diagnostic atlas vector belonging to a high-dimensional vector is flattened to obtain a one-dimensional vector for similarity calculation with the second endoscopic image vector.

[0120] For example, the diagnostic map vectors with scales of [25, 768] and [5, 768] are flattened respectively to obtain vectors with scales of [1, 25*768] and [1, 5*768].

[0121] Step 404 , when the cosine similarity corresponding to the second endoscopic image vector is higher than a second similarity threshold, the gastric endoscopic image of the target stomach part is determined to be a suspected infection image.

[0122] Optionally, the computer device calculates cosine similarity between the second endoscopic image vector and the diagnostic atlas vector according to the parts, and filters the corresponding endoscopic images of different target gastric parts, so as to further screen out gastric endoscopic images corresponding to different parts.

[0123] Illustratively, if the second similarity threshold is 0.96, then when the corresponding cosine similarity is higher than 0.96, the endoscopic image is determined to be a suspected infection image of the corresponding stomach area.

[0124] Step 405, traverse each gastric endoscopic image in the gastric endoscopic image set, and different suspected infection images constitute a suspected infection image set.

[0125] When the suspected infection image sets corresponding to the various parts are obtained, the suspected infection images are grouped according to the stomach parts into suspected infection image sets corresponding to different stomach parts.

[0126] In a possible implementation, the computer device performs image filtering on the suspected infection image set to obtain a filtered target suspected infection image set.

[0127] Optionally, the image filtering includes the following two filtering methods: image deduplication filtering and image overexposure filtering.

[0128] Among them, the purpose of image filtering is to remove images with high similarity in the set of suspected infection images. The process includes calculating the cosine similarity between every two suspected infection images of the same target stomach area; when the cosine similarity between the first suspected infection image and the second suspected infection image is higher than the third similarity threshold, determining that the first suspected infection image and the second suspected infection image are duplicate images; and performing deduplication filtering on the duplicate images to obtain an intermediate suspected infection image set after the first filtration.

[0129] Optionally, the third similarity threshold is set to 0.96.

[0130] Among them, the purpose of overexposure filtering is to remove overexposed images in the suspected infection image set to avoid affecting subsequent diagnosis.

[0131] The computer device first determines the grayscale histogram of each suspected infected image in the intermediate suspected infected image set; based on the grayscale histogram, the suspected infected images in the suspected infected image set are overexposed and filtered to remove the overexposed images, thereby obtaining the target suspected infected image set.

[0132] Optionally, when determining an overexposed image, the computer device determines the number of pixels in its grayscale histogram whose grayscale values ​​exceed the grayscale threshold. If the proportion of this number of pixels in the total number of pixels reaches a ratio threshold, it is considered that the suspected infected image may be an overexposed image and needs to be filtered out. The ratio threshold may be 1%.

[0133] Optionally, after performing image filtering on the suspected infection image set to obtain a filtered target suspected infection image set, the stomach infection condition of the case to be diagnosed is determined based on the filtered target suspected infection image set.

[0134] Step 406: perform image encoding on each suspected infection image in the suspected infection image set by using the image encoder in the part recognition model to obtain a suspected infection image vector.

[0135] In this step, the specific implementation method of image encoding of the suspected infection image can refer to the process of image encoding of the gastric endoscopic image in the above step, which will not be described in detail here.

[0136] Step 407, determining the probability distribution of the case to be diagnosed in different infection situations based on the suspected infection image vector by using the diagnosis model.

[0137] The computer device inputs the suspected infection image vector into the diagnostic model, and the output layer of the diagnostic model outputs the probability distribution of the case to be diagnosed in different infection conditions. For example, the output may be presented as a probability vector, such as [0.3, 0.5, 0.2], where different probabilities correspond to different indexes, which are manually set during the training process. The calculation of this probability distribution is based on the rules learned by the diagnostic model and the characteristics of the currently input suspected infection image vector. The model will weigh the degree of match between various image features and different infection conditions, and convert them into corresponding probability values ​​through the internal calculation mechanism.

[0138] Optionally, the diagnosis model uses a pre-trained MLP (Multilayer Perceptron). In the case of diagnosing stomach infection based on the model, the model needs to be fine-tuned.

[0139] Optionally, the diagnostic model is obtained by training an untrained diagnostic model based on sample infection images and infection status labels.

[0140] In clinical practice, there may be fewer sample infection images with infection status labels, so data augmentation can be used to expand the data.

[0141] First, the computer device scales the acquired sample infection image to a scale of 224*224. Then, the sample infection image and its corresponding infection status label are augmented by about 10 times, for example, by random angle rotation, horizontal and vertical flipping, and random distortion.

[0142] Optionally, after scaling the sample infected image, the computer device divides it based on a certain ratio to obtain a training data set and a verification data set. The data in the training data set is augmented, while the data in the verification data set does not need to be augmented. The division ratio can be 7:3 or 6:4.

[0143] Before training, the computer equipment encodes the sample infection images (including images A and B corresponding to two stomach parts) through an image encoder to obtain a feature vector with a scale of [A+B, S], where S is the feature vector length of the image encoder. Subsequently, a diagnostic model with an input size of [A+B, S] and an output size of [1, N] is established, where N is the number of infection status categories, specifically three categories: uninfected, infected, and previously infected. Previously infected means that the patient has been sterilized.

[0144] Each layer of the diagnostic model contains a fully connected layer, a Relu activation layer and a Dropout layer to calculate the output state of the intermediate nodes. The last layer of MLP is the Softmax layer, which takes the intermediate state value calculated by the previous layer as input and outputs a probability distribution of [1, N]. In the embodiment of the present application, it is a probability distribution of [1, 3]. The index at the final maximum probability is the classification result output.

[0145] Optionally, a cross entropy loss is calculated between the output of the Softmax layer and the class labels in the input. For example, if a multi-class cross entropy loss is used, the loss function is as follows:

[0146]

[0147] Where L CE represents the loss value, N represents the number of categories, i represents the index variable, and y i represents the value of the sample on label i, and P represents the probability.

[0148] The computer device retains its model parameter values ​​during multiple rounds of training, and finally selects the model with the highest average classification accuracy among all the data in the validation set as the optimal model, i.e., the diagnostic model.

[0149] Please refer to Figure 5 , which shows a flowchart of the process of training a diagnostic model provided by an illustrative embodiment of the present application. In this process, the computer device first prepares a sample data set, namely, a sample infection image and a corresponding infection status label, and divides the sample data set into a training set and a verification set, wherein the training set is expanded by data augmentation. Subsequently, the computer device encodes the sample infection image through the image encoder in the part recognition model to obtain the encoded data set, which is input into an untrained classification model for training, thereby obtaining a diagnostic model.

[0150] Step 408, determining the stomach infection condition of the case to be diagnosed based on the probability distribution.

[0151] Assuming that the set index 0 is uninfected, index 1 is infected, and index 2 is once infected, if the index is 0 at the maximum value of the probability distribution, it indicates that the case to be detected is not infected; if the index is 1 at the maximum value of the probability distribution, it indicates that the case to be detected is infected; if the index is 2 at the maximum value of the probability distribution, it indicates that the case to be detected has been infected and is now in a sterilization state.

[0152] In the embodiment of the present application, the computer device first further determines the gastric endoscopy image of the target gastric part based on the expert diagnosis atlas, thereby enhancing the accuracy of the judgment of the gastric part to which the gastric endoscopy model belongs. In addition, the computer device performs image filtering, which is conducive to narrowing the data range for diagnosis, reducing the calculation amount of the subsequent diagnosis model, and filtering out images of poor quality to improve the accuracy of diagnosis. In the end, only a simple diagnostic network is needed to diagnose the infection status. Under the premise of ensuring the accuracy of diagnosis, it has a simpler model structure than the prior art.

[0153] Please refer to Figure 6 , which shows a schematic diagram of a process for diagnosing gastric infection provided by an exemplary embodiment of the present application. First, the computer device inputs the part description text and the gastric endoscopic image of the case to be diagnosed into the part recognition model, encodes the part description text through the text encoder in the part recognition model, and encodes the gastric endoscopic image through the image encoder, so as to determine the target gastric parts corresponding to different gastric endoscopic images through text prompts. The two part description texts in the figure are "gastric body part image" and "gastric antrum part image" respectively. Subsequently, the computer device retrieves the gastric endoscopic images of the two parts in combination with the expert diagnosis atlas (corresponding to the expert atlas of Helicobacter pylori images in the figure), so as to obtain the suspected infection endoscopic image. Subsequently, the computer device performs image de-redundancy (i.e., image de-duplication filtering) according to the suspected infection endoscopic image, filters based on similarity, and removes overexposed images (i.e., image overexposure filtering), and uses a histogram for filtering, so as to obtain a target suspected infection image set. After the computer device performs image encoding on the target suspected infection image based on the image encoder in the part recognition model, the graphic encoding result is input into the diagnosis model to obtain the infection situation.

[0154] Please refer to Figure 7 , which shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of the present application, which shows a block diagram of the structure of an electronic device provided by an exemplary embodiment of the present application. The electronic device 700 in the present application may include one or more of the following components: a processor 710 and a memory 720.

[0155] Optionally, the processor 710 executes the steps of the gastric infection diagnosis method provided in any of the above embodiments by running or executing instructions, programs, code sets or instruction sets stored in the memory 720, and calling data stored in the memory 720.

[0156] In addition, the processor can also perform various functions of the device and process data. Optionally, the processor 710 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 710 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the touch display; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 710, but may be implemented separately through a chip.

[0157] The memory 720 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 720 includes a non-transitory computer-readable storage medium. The memory 720 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 720 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the device (such as audio data, a phone book), etc.

[0158] The device in the embodiment of the present application further includes a communication component 730 and a display component 740. The communication component 730 may be a Bluetooth component, a WiFi component, an NFC (Near Field Communication) component, etc., for communicating with an external device (server or other device) via a wired or wireless network; the display component 740 is used to display a graphical user interface and / or receive user interaction operations.

[0159] Optionally, the electronic device includes a computer system, which may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0160] In addition, those skilled in the art will appreciate that the structure of the device shown in the above drawings does not constitute a limitation on the device, and the device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a speaker, and a power supply, which will not be described in detail here.

[0161] An embodiment of the present application also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the storage medium stores at least one program code, and the program code is loaded and executed by a processor to implement the gastric infection diagnosis method of any of the above embodiments.

[0162] In the context of the present disclosure, a computer-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0163] The embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium. When the computer program is executed by a processor, the method for diagnosing gastric infection in any of the above embodiments is implemented.

Claims

1. A method for diagnosing gastric infection, characterized in that: The method comprises: Acquire a set of gastric endoscopic images of a case to be diagnosed, wherein the set of gastric endoscopic images includes gastric endoscopic images corresponding to at least two gastric parts; Inputting the set of gastric endoscopic images and the part description text into the fine-tuned part recognition model to determine the target gastric parts corresponding to different gastric endoscopic images, wherein the part description text is used to describe at least two gastric parts; Based on expert diagnosis atlases of different target gastric sites, screening a suspected infection image set from the gastric endoscopic image set; Based on the suspected infection image set, the stomach infection condition of the case to be diagnosed is determined, and the stomach infection condition is one of infection, non-infection and previous infection.

2. The method according to claim 1, characterized in that The step of determining the target stomach location corresponding to different stomach endoscopic images includes: Performing image encoding on the gastric endoscopic image by using an image encoder in the part recognition model to obtain a first endoscopic image vector; Performing text encoding on the part description text by using a text encoder in the part recognition model to obtain a description text vector; The target stomach part is determined based on the similarity between the first endoscopic image vector and the description text vectors corresponding to different parts.

3. The method according to claim 2, characterized in that The step of determining the target stomach part based on the similarity between the endoscopic image vector and the description text vectors corresponding to different parts includes: Calculating the cosine similarity between the endoscopic image vector and different description text vectors through the part recognition model; When the cosine similarity between the endoscopic image vector and the first description text vector is higher than a first similarity threshold, the stomach part corresponding to the first description text vector is determined to be the target stomach part.

4. The method according to claim 1, characterized in that The expert diagnosis atlas based on different target stomach parts, screening out a suspected infection image set from the stomach endoscopic image set, includes: Performing image encoding on the gastric endoscopic image of the target gastric part by using the image encoder in the part recognition model to obtain a second endoscopic image vector; By means of the image encoder, image encoding is performed on the expert diagnostic atlas of the target stomach area to obtain a diagnostic atlas vector; determining a cosine similarity between the second endoscopic image vector and the diagnostic atlas vector; When the cosine similarity corresponding to the second endoscopic image vector is higher than a second similarity threshold, determining that the gastric endoscopic image of the target stomach part is a suspected infection image; Each gastric endoscopic image in the gastric endoscopic image set is traversed, and different suspected infection images form the suspected infection image set.

5. The method according to claim 4, characterized in that One of the target stomach parts corresponds to at least two of the expert diagnosis atlases, and the determining of the cosine similarity between the second endoscopic image vector and the diagnosis atlas vector includes: Flattening the diagnostic map vectors corresponding to at least two of the expert diagnostic maps to obtain a one-dimensional diagnostic map vector; Determine the cosine similarity between the second endoscopic image vector and the one-dimensional diagnostic atlas vector.

6. The method according to claim 1, characterized in that The step of determining the stomach infection condition of the case to be diagnosed based on the suspected infection image set includes: By using the image encoder in the part recognition model, each suspected infection image in the suspected infection image set is image encoded to obtain a suspected infection image vector; Determining the probability distribution of the case to be diagnosed in different infection situations based on the suspected infection image vector by using a diagnostic model; Based on the probability distribution, the stomach infection condition of the case to be diagnosed is determined.

7. The method according to claim 1, characterized in that The method further comprises: Performing image filtering on the suspected infection image set to obtain a filtered target suspected infection image set; The step of determining the stomach infection condition of the case to be diagnosed based on the suspected infection image set includes: Based on the filtered target suspected infection image set, the stomach infection condition of the case to be diagnosed is determined.

8. The method according to claim 7, characterized in that The image filtering includes image deduplication filtering and image overexposure filtering; The image filtering is performed on the suspected infection image set to obtain a filtered target suspected infection image set, including: Calculating the cosine similarity between every two suspected infection images of the same target stomach part; in the case where the cosine similarity between the first suspected infection image and the second suspected infection image is higher than the third similarity threshold, determining that the first suspected infection image and the second suspected infection image are duplicate images; performing deduplication filtering on the duplicate images to obtain a set of intermediate suspected infection images after the first filtering; Determine the grayscale histogram of each suspected infected image in the intermediate suspected infected image set; perform overexposure filtering on the suspected infected images in the suspected infected image set based on the grayscale histogram, filter out the overexposed images, and obtain the target suspected infected image set.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire a training sample pair, wherein the training sample pair includes a sample gastric endoscopic image and a sample text, wherein the sample text is used to characterize a gastric part, a lesion label, and a text of microscopic findings corresponding to the sample gastric endoscopic image; The pre-trained model is fine-tuned based on the training samples to obtain the fine-tuned part recognition model.

10. The method according to claim 6, characterized in that The diagnostic model is obtained by training an untrained diagnostic model based on sample infection images and infection status labels.

11. An electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the gastric infection diagnosis method according to any one of claims 1-10.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the gastric infection diagnosis method according to any one of claims 1-10.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for diagnosing gastric infection according to any one of claims 1 to 10 is implemented.

Images