Anomaly detection method and device based on large language model

Abstract

The present application discloses an anomaly detection method and device based on a large language model, wherein a plurality of second endoscopic detection images and corresponding image description annotation texts are split into a training set and a test set according to a preset ratio; a preset neural network model is fine-tuned based on the training set to obtain a fine-tuned model; the fine-tuned model is tested based on the test set to obtain a model test result; when the model test result does not meet the preset conditions, a plurality of sixth endoscopic detection images and corresponding multiple image description annotation texts are added to the training set to obtain a new training set; a new model test result corresponding to the new fine-tuned model is determined based on the new training set and the test set until the new model test result meets the preset conditions; the new fine-tuned model is determined as a target anomaly detection model; the endoscopic detection image to be identified is input into the target anomaly detection model for anomaly detection to obtain anomaly detection description text.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an anomaly detection method and device based on a large language model. Background Art

[0002] At present, most of the traditional automated auxiliary detection technologies based on deep learning methods are based on convolutional neural network models for tasks such as image classification, target detection, and entity segmentation. For example, based on convolutional neural networks or machine learning models, the classification, detection, and segmentation tasks of image visual feature parameters are performed. However, these methods also have many limitations. For example, they strongly rely on the use of image visual features to classify, detect, and segment images, and cannot provide the corresponding abnormal detection description text for the image, and the interpretability is poor. Summary of the invention

[0003] The embodiments of the present application provide an anomaly detection method and device based on a large language model, which can provide an anomaly detection description text corresponding to an image to improve interpretability.

[0004] In a first aspect, the present application provides an anomaly detection method based on a large language model, comprising:

[0005] Acquire a first endoscopic detection image and a corresponding image description annotation text;

[0006] Determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on the plurality of first endoscopic detection images and corresponding image description annotation texts;

[0007] Splitting the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio, wherein the training set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts, and the test set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts;

[0008] Fine-tune the preset neural network model based on the training set to obtain a fine-tuned model;

[0009] Testing the fine-tuning model based on the test set to obtain a model test result;

[0010] When the model test result does not meet the preset condition, obtaining a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts;

[0011] Adding a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts into the training set to obtain a new training set;

[0012] Determine a new model test result corresponding to the new fine-tuning model based on the new training set and test set, until the new model test result meets the preset conditions;

[0013] Determine the new fine-tuned model whose new model test results meet the preset conditions as the target anomaly detection model;

[0014] When the endoscopic detection image to be identified is obtained, the endoscopic detection image to be identified is input into the target anomaly detection model for anomaly detection, and an anomaly detection description text corresponding to the endoscopic detection image to be identified is obtained.

[0015] In a second aspect, the present application provides an anomaly detection device based on a large language model, comprising:

[0016] A first acquisition module is used to acquire a first endoscopic detection image and a corresponding image description annotation text;

[0017] A first determination module, configured to determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of first endoscopic detection images and corresponding image description annotation texts;

[0018] A splitting module, used to split the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio, wherein the training set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts, and the test set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts;

[0019] A fine-tuning module, used to fine-tune the preset neural network model based on the training set to obtain a fine-tuning model;

[0020] A testing module, used to test the fine-tuning model based on the test set to obtain a model testing result;

[0021] A second acquisition module, used for acquiring a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts when the model test result does not meet the preset condition;

[0022] An adding module, used for adding a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts into the training set to obtain a new training set;

[0023] A second determination module is used to determine a new model test result corresponding to the new fine-tuning model based on the new training set and the test set, until the new model test result meets a preset condition;

[0024] A third determination module is used to determine a new fine-tuning model whose new model test results meet preset conditions as a target anomaly detection model;

[0025] The anomaly detection module is used to input the endoscopic detection image to be identified into the target anomaly detection model for anomaly detection when the endoscopic detection image to be identified is acquired, so as to obtain the anomaly detection description text corresponding to the endoscopic detection image to be identified.

[0026] In a third aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the anomaly detection method based on a large language model provided in the present application.

[0027] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the large language model-based anomaly detection method provided in the present application.

[0028] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implements the steps in the large language model-based anomaly detection method provided in the present application.

[0029] In the present application, compared with the related art, a first endoscopic detection image and a corresponding image description annotation text are obtained; based on the multiple first endoscopic detection images and the corresponding image description annotation texts, multiple second endoscopic detection images and the corresponding image description annotation texts are determined; the multiple second endoscopic detection images and the corresponding image description annotation texts are split into a training set and a test set according to a preset ratio, wherein the training set includes multiple second endoscopic detection images and the corresponding image description annotation texts, and the test set includes multiple second endoscopic detection images and the corresponding image description annotation texts; a preset neural network model is fine-tuned based on the training set to obtain a fine-tuned model; the fine-tuned model is tested based on the test set to obtain a model test result; when When the model test result does not meet the preset conditions, obtain multiple sixth endoscopic detection images and corresponding multiple image description annotation texts; add multiple sixth endoscopic detection images and corresponding multiple image description annotation texts into the training set to obtain a new training set; determine the new model test results corresponding to the new fine-tuning model based on the new training set and the test set until the new model test results meet the preset conditions; determine the new fine-tuning model whose new model test results meet the preset conditions as the target anomaly detection model; when the endoscopic detection image to be identified is obtained, input the endoscopic detection image to be identified into the target anomaly detection model for anomaly detection, and obtain the anomaly detection description text corresponding to the endoscopic detection image to be identified. This application can provide the anomaly detection description text corresponding to the image to improve interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 is a scenario diagram of an anomaly detection system based on a large language model provided in an embodiment of the present application;

[0032] Figure 2 It is a flowchart of an embodiment of an anomaly detection method based on a large language model provided in an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of an endoscopic detection image in one embodiment of an abnormality detection method based on a large language model provided in an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of an endoscopic detection report in one embodiment of the abnormality detection method based on a large language model provided in an embodiment of the present application;

[0035] Figure 5 It is a structural diagram of an anomaly detection device based on a large language model provided in an embodiment of the present application;

[0036] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0038] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] In order to improve the effect of anomaly detection based on a large language model, the embodiments of the present application provide an anomaly detection method based on a large language model, an anomaly detection device based on a large language model, an electronic device, a computer-readable storage medium, and a computer program product. Among them, the anomaly detection method based on a large language model can be executed by an anomaly detection device based on a large language model, or by an electronic device integrated with the anomaly detection device based on a large language model.

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0043] Please refer to Figure 1 , the present application also provides an anomaly detection system based on a large language model, such as Figure 1 As shown, the electronic device 100 of the anomaly detection system based on a large language model is integrated with the anomaly detection device based on a large language model provided by the present application.

[0044] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0045] In addition, if Figure 1 As shown, the large language model-based anomaly detection system may further include a memory 200 for storing original data, intermediate data, and result data.

[0046] It should be noted that Figure 1 The scenario diagram of the anomaly detection system based on a large language model shown is merely an example. The anomaly detection system based on a large language model and the scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the anomaly detection system based on a large language model and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0047] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0048] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of an anomaly detection method based on a large language model provided in an embodiment of the present application. Figure 2 As shown, the process of the anomaly detection method based on a large language model provided by this application is as follows:

[0049] 201. Obtain a first endoscopic detection image and a corresponding image description annotation text.

[0050] In the embodiment of the present application, the plurality of first endoscopic detection images may be endoscopic images obtained by endoscopy, images obtained by gastroscopy, or images obtained by colonoscopy, which may be set according to specific circumstances. The image description annotation text corresponding to the first endoscopic detection image is the text determined by the doctor based on the first endoscopic detection image to describe the first endoscopic detection image.

[0051] like Figure 3 As shown, Figure 3 A plurality of first endoscopic detection images are schematically shown.

[0052] like Figure 4 As shown, Figure 4 The diagram shows a plurality of first endoscopic detection images and corresponding image description annotation texts on the detection report. The shooting parts displayed on the plurality of first endoscopic detection images may be the esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, etc.

[0053] Furthermore, we built a unified annotation template based on professional knowledge, invited multiple experts to annotate the large model image-text pair data set, and conducted consistency evaluation after the annotation by multiple experts. We also conducted expert review on the labeled image-text pairs with poor consistency to form a preliminary image-text pair annotation data set. Based on actual training and clinical application needs, we used methods such as rotation, flipping, cropping, scaling, adjusting brightness and color to enhance the unbalanced data types, and finally formed a high-quality image-text pair data set, which included the first endoscopic detection image and the corresponding image description annotation text.

[0054] In a specific embodiment, the first endoscopic detection image and the corresponding image description annotation text may be input by the user. For example, the first endoscopic detection image is an image of a stomach angle, and the text in the image description text area is: "Stomach angle: arc-shaped, smooth".

[0055] 202. Determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on the plurality of first endoscopic detection images and corresponding image description annotation texts.

[0056] In a specific embodiment, a plurality of first endoscopic detection images and corresponding image description annotation texts are determined as a plurality of second endoscopic detection images and corresponding image description annotation texts.

[0057] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of first endoscopic detection images and corresponding image description annotation texts includes:

[0058] (1) Obtaining an image resolution of each first endoscopic detection image and a preset resolution threshold of each first endoscopic detection image.

[0059] The preset resolution threshold of each first endoscopic detection image may be the same threshold, which can be set according to specific circumstances. For example, the preset resolution threshold of each first endoscopic detection image is 5 million ppi (pixels per inch).

[0060] In a specific embodiment, the preset resolution thresholds of the first endoscopic detection images may not be the same threshold.

[0061] Specifically, a shooting part detection is performed on each first endoscopic detection image to obtain the shooting part of each first endoscopic detection image; a shooting part text of the first endoscopic detection image is determined based on the shooting part of the first endoscopic detection image, and different shooting part texts correspond to different preset resolution thresholds; a preset resolution threshold of each first endoscopic detection image is determined based on the shooting part text of each first endoscopic detection image. For example, the shooting part text may be esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, and duodenum. When the shooting part text is esophagus, the corresponding preset resolution threshold is 4 million ppi; when the shooting part text is cardia, the corresponding preset resolution threshold is 6 million ppi. Different shooting part texts correspond to different preset resolution thresholds to avoid screening out too many images.

[0062] (2) Determine the first endoscopic detection images whose image resolution is higher than the corresponding preset resolution threshold as a plurality of fourth endoscopic detection images.

[0063] Specifically, determine whether the image resolution of the first endoscopic detection image is higher than the preset resolution threshold of the first endoscopic detection image. If the image resolution of the first endoscopic detection image is higher than the preset resolution threshold of the first endoscopic detection image, the first endoscopic detection image is determined as the fourth endoscopic detection image. If the image resolution of the first endoscopic detection image is not higher than the preset resolution threshold of the first endoscopic detection image, the first endoscopic detection image is not determined as the fourth endoscopic detection image.

[0064] (3) Determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on the plurality of fourth endoscopic detection images and corresponding image description annotation texts.

[0065] In a specific embodiment, the plurality of fourth endoscopic detection images and the corresponding image description annotation texts are determined as the plurality of second endoscopic detection images and the corresponding image description annotation texts.

[0066] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of fourth endoscopic detection images and corresponding image description annotation texts includes:

[0067] (1) Performing image segmentation on the fourth endoscopic detection image to obtain an unclean segmented area and a lesion segmented area of ​​the fourth endoscopic detection image.

[0068] Pre-train an image segmentation model, such as a Segment Anything Model (SAM) model, a Unet model, a mask-rcnn model, etc. The image segmentation model is used to segment the unclean area, the lesion area, and the normal mucosal area of ​​the endoscopic detection image. The unclean segmentation area includes areas containing artifacts, flushing, biopsy, surgery, bubbles, and residues. The lesion segmentation area is the area where the lesion is detected.

[0069] (2) Determine an unclean area ratio of an unclean segmented region and a lesion area ratio of a lesion segmented region of the fourth endoscopic detection image.

[0070] Among them, the unclean area ratio of the unclean segmented area is the area ratio of the unclean segmented area to the fourth endoscopic detection image. The normal mucosal area ratio of the normal mucosal area is the area ratio of the normal mucosal area to the fourth endoscopic detection image. The lesion segmentation area ratio of the lesion segmentation area is the area ratio of the lesion segmentation area to the fourth endoscopic detection image. For example, the lesion area ratio of the lesion segmentation area is A, the unclean segmentation area ratio is C, and the normal mucosal area ratio is B, because A+B+C=1, B is also equal to =1-AC.

[0071] (3) Based on a preset weight coefficient, a weighted sum of the unclean area ratio of the unclean segmented region and the lesion area ratio of the lesion segmented region of the fourth endoscopic detection image is taken to obtain the image cleanliness of the fourth endoscopic detection image, wherein the larger the unclean area ratio, the smaller the image cleanliness; and the larger the lesion area ratio, the greater the image cleanliness.

[0072] Among them, the greater the image cleanliness, the cleaner the image.

[0073] The preset weight coefficients include a first weight coefficient w1, a second weight coefficient w2 and a third weight coefficient w3. The first weight coefficient, the second weight coefficient and the third weight coefficient are 2, 0.1 and -2 respectively.

[0074] Specifically, the calculation formula of image cleanliness S is S=w1*A+w2*(1-AC)+w3*C.

[0075] Among them, w1 is the first weight coefficient, the second weight coefficient is w2, and the third weight coefficient is w3. The first weight coefficient w1 and the second weight coefficient w2 are positive numbers, and the third weight coefficient is a negative number.

[0076] (4) The fourth endoscopic detection images having an image cleanliness higher than a preset cleanliness are determined as a plurality of fifth endoscopic detection images.

[0077] Among them, the preset cleanliness level can be set according to the specific situation.

[0078] (5) Based on the plurality of fifth endoscopic detection images and the corresponding image description annotation texts, determine the plurality of second endoscopic detection images and the corresponding image description annotation texts.

[0079] In a specific embodiment, the plurality of fifth endoscopic detection images and the corresponding image description annotation texts are determined as the plurality of second endoscopic detection images and the corresponding image description annotation texts.

[0080] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of fifth endoscopic detection images and corresponding image description annotation texts includes:

[0081] (1) Obtain the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score, and shooting site text score of the fifth endoscopic detection image.

[0082] In the embodiment of the present application, the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score and shooting site text score of the fifth endoscopic detection image are obtained, including:

[0083] Obtain the preset normal brightness value and the brightness value of the fifth endoscopic detection image, and determine the brightness score of the fifth endoscopic detection image based on the absolute value of the difference between the preset normal brightness value and the brightness value of the fifth endoscopic detection image, wherein the larger the absolute value of the difference, the smaller the brightness score of the fifth endoscopic detection image. The brightness value of the fifth endoscopic detection image is the average value of the brightness values ​​of each pixel on the fifth endoscopic detection image. When the brightness value is close to the preset normal brightness value, it is determined that the fifth endoscopic detection image is normal, and the highest brightness score is 1.

[0084] For example, brightness is divided into five categories: bright, bright, normal, dark, and dim, and the brightness scores correspond to -1, 0, 1, 0, and -1 points, respectively, recorded as Score_b.

[0085] A contrast value of the fifth endoscopic detection image is obtained, and a contrast score of the fifth endoscopic detection image is determined based on the contrast value of the fifth endoscopic detection image, wherein the larger the contrast value, the larger the contrast score of the fifth endoscopic detection image.

[0086] For example, contrast is divided into three categories: high, medium, and low, and the contrast scores correspond to 1, 0, and -1 points, respectively, recorded as Score_c.

[0087] The size of the noise region on the fifth endoscopic detection image is obtained, and the noise score of the fifth endoscopic detection image is determined based on the size of the noise region, wherein the larger the size of the noise region, the larger the noise score of the fifth endoscopic detection image. Specifically, the proportion of the size of the noise region to the noise area of ​​the entire image is calculated, and the larger the proportion of the noise area, the smaller the noise score of the fifth endoscopic detection image.

[0088] The noise is segmented into a noise area through a noise segmentation model, and then the ratio of the noise area to the noise area of ​​the entire image is calculated S2, and finally 1-S2 is determined as the noise score, recorded as Score_n.

[0089] The size of the artifact region on the fifth endoscopic detection image is obtained, and the artifact score of the fifth endoscopic detection image is determined based on the size of the artifact region, wherein the larger the size of the artifact region, the smaller the artifact of the fifth endoscopic detection image. Specifically, the proportion of the size of the artifact region to the artifact area of ​​the entire image is calculated, and the larger the proportion of the artifact area, the smaller the artifact score of the fifth endoscopic detection image.

[0090] The artifact is segmented into an artifact area through an artifact segmentation model, and then the proportion of the artifact area size to the artifact area of ​​the entire image S3 is calculated. Finally, 1-S3 is determined as the artifact score, recorded as Score_a.

[0091] The mucosal condition is divided into two categories: clear and blurred. The mucosal area clarity scores correspond to 1 and -1 points respectively. The mucosal area clarity score is recorded as Score_m.

[0092] The presence or absence of lesions in the mucosal area was divided into two categories: yes and no, and the abnormal scores of the mucosal area corresponded to 1 and 0 points, respectively. The abnormal score of the mucosal area was recorded as Score_f.

[0093] The total area ratio of the lesion region on the fifth endoscopic detection image is obtained, and the lesion score of the fifth endoscopic detection image is determined based on the total area ratio of the lesion region, wherein the greater the total area ratio of the lesion region, the greater the lesion score of the fifth endoscopic detection image. Specifically, the fifth endoscopic detection image is segmented to obtain the unclean segmented area and the lesion segmented area of ​​the fifth endoscopic detection image; the unclean area ratio of the unclean segmented area and the lesion area ratio of the lesion segmented area of ​​the fifth endoscopic detection image are determined. The lesion area ratios of each lesion segmented area on the fifth endoscopic detection image are summed to obtain the total area ratio of the lesion region.

[0094] The shooting part text of the fifth endoscopic detection image is obtained, and a shooting part text score of the fifth endoscopic detection image is determined based on the shooting part text, wherein different shooting part texts correspond to different shooting part text scores, and wherein the shooting part text score when the shooting part text is esophagus, cardia and duodenum is greater than the shooting part text score when the shooting part text is other parts.

[0095] The positions are divided into 7 major parts: esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, duodenum, and other parts. When the shooting part text is esophagus, cardia, and duodenum, the shooting part text score is 1.5, and when the shooting part text is other parts, the shooting part text score is 1, which is recorded as Score_1.

[0096] (2) Determine the overall feature score of the fifth endoscopic detection image based on the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score, and shooting site text score of the fifth endoscopic detection image, and obtain the overall feature score of each fifth endoscopic detection image.

[0097] Specifically, the calculation formula of the overall feature score P of the fifth endoscopic detection image is:

[0098] P=(Score_b+Score_c+Score_n+Score_a) / 4+(Score_m+Score_f+Score_s+Score_l) / 4.

[0099] Among them, the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score and shooting site text score of the fifth endoscopic detection image are Score_b, Score_c, Score_n, Score_a, Score_m, Score_f, Score_s and Score_l respectively.

[0100] If the mucosa is hazy, without lesions, or the location is other, the overall feature score is set to 0.

[0101] (3) The fifth endoscopic detection images and the corresponding image description annotation texts having an overall feature score higher than a preset score are determined as a plurality of third endoscopic detection images and the corresponding image description annotation texts.

[0102] Among them, the preset score can be set according to the situation and is not limited here.

[0103] (4) Based on the plurality of third endoscopic detection images and the corresponding image description annotation texts, determine the plurality of second endoscopic detection images and the corresponding image description annotation texts.

[0104] In a specific embodiment, the plurality of third endoscopic detection images and the corresponding image description annotation texts are determined as the plurality of second endoscopic detection images and the corresponding image description annotation texts.

[0105] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of third endoscopic detection images and corresponding image description annotation texts includes:

[0106] (1) Determine a plurality of detection image groups based on a plurality of third endoscopic detection images, wherein the detection image groups include two third endoscopic detection images.

[0107] In a specific embodiment, any two third endoscopic detection images of the plurality of third endoscopic detection images are determined as one detection image group or a plurality of detection image groups.

[0108] (2) Calculate the first similarity of the two third endoscopic detection images in the detection image group to obtain the first similarity of the detection image group.

[0109] In the embodiment of the present application, calculating the first similarity of two third endoscopic detection images in the detection image group to obtain the first similarity of the detection image group includes:

[0110] (3) Feature extraction is performed on the two third endoscopic detection images respectively to obtain two detection image feature maps of the two third endoscopic detection images.

[0111] In the embodiment of the present application, the two third endoscopic detection images are respectively input into the feature extraction network for feature extraction to obtain two detection image feature maps of the two third endoscopic detection images.

[0112] (4) Map the two detection image feature maps to one-dimensional vectors to obtain two one-dimensional feature vectors corresponding to the two detection image feature maps.

[0113] (3) The Manhattan distance between the two one-dimensional feature vectors corresponding to the two detection image feature maps of the two third endoscopic detection images is input into the fully connected layer to obtain a fully connected vector.

[0114] (4) Use the sigmoid function to map the fully connected vector to the range of [0-1] to obtain the first similarity of the detection image group.

[0115] The larger the Manhattan distance between the two one-dimensional feature vectors corresponding to the two detection image feature maps of the two third endoscopic detection images, the smaller the first similarity between the two third endoscopic detection images.

[0116] (5) Obtaining lesion segmentation regions on two third endoscopic detection images of the detection image group obtained by performing lesion segmentation on the two third endoscopic detection images respectively.

[0117] In the embodiment of the present application, image segmentation is performed on the third endoscopic detection image to obtain a lesion segmentation area of ​​the third endoscopic detection image.

[0118] (6) Determine a second similarity between the two third endoscopic detection images based on the lesion segmentation regions on the two third endoscopic detection images to obtain a second similarity between the detection image group.

[0119] (7) Determine a third similarity of the detection image groups based on the first similarity of the detection image groups and the second similarity of the detection image groups, and obtain third similarities of the plurality of detection image groups.

[0120] In the embodiment of the present application, an average value of the first similarity and the second similarity of the detection image group is determined as the third similarity of the detection image group.

[0121] (8) Deleting at least one third endoscopic detection image in the detection image group whose third similarity is greater than a preset similarity from the plurality of third endoscopic detection images to obtain a plurality of second endoscopic detection images, and obtaining a plurality of second endoscopic detection images and corresponding image description annotation texts.

[0122] In a specific embodiment, a random third endoscopic detection image in each detection image group whose third similarity is greater than a preset similarity is deleted from multiple third endoscopic detection images to obtain multiple seventh endoscopic detection images, and the multiple seventh endoscopic detection images are deduplicated to obtain multiple fourth endoscopic detection images.

[0123] In another specific embodiment, all images in the detection image group whose third similarity is greater than a preset similarity are put into a candidate image set, the occurrence frequency of each third endoscopic detection image in the candidate image set is calculated, and the third endoscopic detection images with a frequency higher than the preset frequency are deleted from the multiple third endoscopic detection images to obtain multiple seventh endoscopic detection images, and the multiple seventh endoscopic detection images are deduplicated to obtain multiple fourth endoscopic detection images.

[0124] In the embodiment of the present application, determining the second similarity of the two third endoscopic detection images based on the lesion segmentation areas on the two third endoscopic detection images to obtain the second similarity of the detection image group includes:

[0125] (1) Obtain the number of lesion segmentation regions on the two third endoscopic detection images of the detection image group respectively.

[0126] (2) When the number of lesion segmentation regions on two third endoscopic detection images of the detection image group is the same, the two third endoscopic detection images of the detection image group are respectively determined as the first target image and the second target image.

[0127] When the number of lesion segmentation regions on the two third endoscopic detection images of the detection image group is different, the third endoscopic detection image with the larger number of lesion segmentation regions among the two third endoscopic detection images of the detection image group is determined as the third target image, the third endoscopic detection image with the larger number of lesion segmentation regions among the two third endoscopic detection images of the detection image group is determined as the second target image, the number of lesion segmentation regions of the second target image is determined as the first preset number, the lesion segmentation regions on the third target image are clustered into the first preset number of first segmentation region clusters based on the centroid positions of the lesion segmentation regions on the third target image, each first segmentation region cluster includes at least one lesion segmentation region. Each first segmentation region cluster is merged into a new lesion segmentation region to obtain the first preset number of new lesion segmentation regions on the third target image, and the third target image containing the first preset number of new lesion segmentation regions is determined as the first target image, so that the number of lesion segmentation regions on the first target image and the second target image is greater. Specifically, each first segmentation region cluster is merged into a new lesion segmentation region to obtain a first preset number of new lesion segmentation regions on the third target image, including: obtaining the centroid of each lesion segmentation region in the first segmentation region cluster, connecting the centroids of each lesion segmentation region in the first segmentation region cluster in sequence to obtain a first polygonal region, and geometrically enlarging the first polygonal region to obtain a second polygonal region, wherein the second polygonal region and the first polygonal region have the same centroid, and the second polygonal region is a minimum polygonal region that is a geometrically enlarged version of the first polygonal region and contains each lesion segmentation region in the first segmentation region cluster.

[0128] In a specific embodiment, clustering the lesion segmentation regions on the third target image into a first preset number of first segmentation region clusters based on the centroid position of the lesion segmentation regions on the third target image includes: randomly dividing the lesion segmentation regions on the third target image into Q segmentation region sets. Clustering the Q segmentation region sets respectively, clustering each segmentation region set into a first preset number of second segmentation region clusters. Determining each second segmentation region cluster as a target segmentation region cluster, and calculating the region cluster similarity between the target segmentation region cluster and the second segmentation region clusters in the Q segmentation region sets respectively. Merging the second segmentation region clusters with the greatest similarity to the target segmentation region cluster in each segmentation region set to obtain the first segmentation region cluster corresponding to the target segmentation region cluster, obtaining the first segmentation region cluster corresponding to each second segmentation region cluster, and obtaining the first preset number of first segmentation region clusters corresponding to the first preset number of second segmentation region clusters.

[0129] (3) Determine each lesion segmentation region of the first target image as a target lesion segmentation region, and determine a matching lesion segmentation region on the second target image that matches the target lesion segmentation region.

[0130] Specifically, the lesion segmentation region distances between the target lesion segmentation region and each lesion segmentation region on the second target image are calculated respectively, wherein the lesion segmentation region distance is the distance between the centroids of the two lesion segmentation regions. The lesion segmentation region on the second target image having the largest lesion segmentation region distance with the target lesion segmentation region is determined as the matching lesion segmentation region of the target lesion segmentation region.

[0131] (4) The lesion similarity index between the target lesion segmentation region and the matching lesion segmentation region is determined as the lesion similarity index of the target lesion segmentation region, and the lesion similarity index of each lesion segmentation region on the first target image is obtained.

[0132] In a specific embodiment, the minimum bounding rectangle area of ​​the target lesion segmentation area and the minimum bounding rectangle area of ​​the matching lesion segmentation area are obtained, the union of the minimum bounding rectangle area of ​​the target lesion segmentation area and the minimum bounding rectangle area of ​​the matching lesion segmentation area is determined as the comparison area, the fourth similarity between the image in the comparison area of ​​the first target image and the image in the comparison area of ​​the second target image is calculated, and the lesion similarity index is determined based on the fourth similarity. The fourth similarity can be calculated by SSIM (structural similarity metric) and MS-SSIM (multi-scale structural similarity metric).

[0133] The fourth similarity can be determined as the lesion similarity index. Further, the lesion intersection-and-union ratio of the target lesion segmentation region and the matching lesion segmentation region is calculated, and the lesion intersection-and-union ratio is the ratio of the intersection of the target lesion segmentation region and the matching lesion segmentation region to the union of the target lesion segmentation region and the matching lesion segmentation region. The lesion similarity index is determined based on the lesion intersection-and-union ratio and the fourth similarity. Specifically, the average value of the lesion intersection-and-union ratio and the fourth similarity is determined as the lesion similarity index.

[0134] (5) Based on the lesion weight coefficients of the lesion segmented regions on the first target image, weighted summation is performed on the lesion similarity indexes of the lesion segmented regions on the first target image to obtain a second similarity of the detection image group.

[0135] In the embodiment of the present application, the lesion area ratio of each lesion segmentation area on the first target image is obtained, and the lesion area ratio of each lesion segmentation area on the first target image is determined as the lesion weight coefficient of each lesion segmentation area. Based on the lesion weight coefficient of each lesion segmentation area on the first target image, the lesion similarity index of each lesion segmentation area on the first target image is weighted and summed to obtain the second similarity of the detection image group.

[0136] 203. Split the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio.

[0137] The training set includes a plurality of second endoscopic detection images and corresponding image description annotation texts. The training set includes a plurality of second endoscopic detection images and corresponding image description annotation texts. The training set includes a plurality of second endoscopic detection images and corresponding image description annotation texts. The training set includes a plurality of second endoscopic detection images and corresponding image description annotation texts.

[0138] In an embodiment of the present application, the plurality of second endoscopic detection images and the corresponding image description annotation texts are split into a training set and a test set according to a preset ratio. For example, the preset ratio is 7:3. The ratio of the number of the plurality of second endoscopic detection images in the training set to the number of the plurality of second endoscopic detection images in the test set is 7:3.

[0139] 204. Fine-tune the preset neural network model based on the training set to obtain a fine-tuned model.

[0140] Among them, the preset neural network model can be a large language model.

[0141] In the embodiment of the present application, a preset neural network model is fine-tuned based on a training set to obtain a fine-tuned model, including:

[0142] (1) Divide the training set into K equal subsets.

[0143] Wherein, K is an integer greater than 2.

[0144] (2) The K subsets are respectively determined as validation sets, and the subsets of the K subsets other than the validation sets are combined to form a combined training set, thereby obtaining K validation sets and corresponding combined training sets.

[0145] (3) Train the preset neural network model based on the combined training set to obtain the combined training model corresponding to the combined training set.

[0146] (4) Verifying the combined training model corresponding to the combined training set based on the verification set corresponding to the combined training set, obtaining the model verification parameters of the combined training model corresponding to the combined training set, and obtaining the model verification parameters of the combined training models corresponding to the multiple combined training sets.

[0147] The model validation parameter may be the sum of the accuracy (Precision) and the recall (Recall) of the combined training model on the validation set.

[0148] (5) The combined training model with the highest model verification parameter is determined as the fine-tuning model.

[0149] 205. Test and fine-tune the model based on the test set to obtain the model test results.

[0150] In the embodiment of the present application, the fine-tuning model is tested based on the test set to obtain the model test results, including:

[0151] (1) The second endoscopic detection image in the test set is input into the fine-tuning model to obtain the image description prediction text corresponding to the second endoscopic detection image.

[0152] The image description annotation text corresponding to the second endoscopic detection image is: the label is that an ulcer is visible, with a regular shape, shallow depression, a base covered with thin white fur, and accompanied by bleeding.

[0153] The image description prediction text corresponding to the second endoscopic detection image is: It is predicted that an ulcer is visible, with a regular shape, deep depression, a base covered with dirty moss, and visible bleeding.

[0154] (2) Calculate a text similarity index between the image description annotation text corresponding to the second endoscopic detection image and the image description prediction text corresponding to the second endoscopic detection image.

[0155] Specifically, both the image description annotation text and the image description prediction text include a disease feature field and multiple lesion feature fields.

[0156] First, identify the disease characteristic fields. The disease characteristic fields are all ulcers, that is, when the disease characteristic fields are the same, the disease score is 1, and if they are different, it is 0.

[0157] Secondly, the feature accuracy is evaluated based on multiple feature fields describing the lesions: the multiple feature fields describing the lesions in the image description annotation text are {'morphology': 'shallow depression', 'border': 'regular shape', 'bleeding': 'yes', 'white coating': 'thin white coating', 'benign and malignant': 'benign'}, and the predicted multiple feature fields describing the lesions are {'morphology': 'deep depression', 'border': 'regular shape', 'bleeding': 'yes', 'white coating': 'dirty coating', 'benign and malignant': 'neoplastic'}.

[0158] Specifically, calculating the text similarity index between the image description annotation text corresponding to the second endoscopic detection image and the image description prediction text corresponding to the second endoscopic detection image includes:

[0159] (1) Determine a weight coefficient based on whether the disease feature field of the image description annotation text and the disease feature field of the image description prediction text are the same, wherein if the disease feature field of the image description annotation text and the disease feature field of the image description prediction text are the same, the weight coefficient is a first weight; if the disease feature field of the image description annotation text and the disease feature field of the image description prediction text are different, the weight coefficient is a second weight, and the first weight is greater than the second weight.

[0160] For example, the first weight is 1 and the second weight is 0.9.

[0161] (2) Determine the proportion of identical fields describing lesion features in the image description annotation text and the image description prediction text.

[0162] For example, the number of identical fields is 2, the total number is 5, and the percentage of identical fields is 0.4.

[0163] (3) The proportion of the same fields is weighted based on the weight coefficient, and the obtained weighted value is determined as the text similarity index between the image description annotation text and the image description prediction text.

[0164] Specifically, the text similarity index is: the proportion of the same fields * weight coefficient.

[0165] Furthermore, the bleu-4 and rouge-l scores are calculated. In this example, rouge-l is 0.78 and bleu-4 is 0.63. The text similarity index is: (same field proportion + bleu-4 score + rouge-l score) / 3*weight coefficient. If the disease feature fields are the same and the disease score is 1, the final score is (same field proportion + bleu-4 score + rouge-l score) / 3. If the disease feature fields are different and the disease score is 0, the final score is (same field proportion + bleu-4 score + rouge-l score) / 3*0.9, so as to implement the penalty for the disease score.

[0166] (4) The second endoscopic detection image whose text similarity index is greater than a preset similarity is determined as the second endoscopic detection image that is predicted correctly.

[0167] (5) The percentage of the second endoscopic detection images predicted correctly in the test set is determined as the model test result.

[0168] 206. When the model test result does not meet the preset condition, obtain multiple sixth endoscopic detection images and corresponding multiple image description annotation texts.

[0169] The preset condition may be that the proportion of the number of correctly predicted second endoscopic detection images in the test set is less than a preset proportion.

[0170] In a specific embodiment, the plurality of sixth endoscopic detection images and the corresponding plurality of image description annotation texts may be pre-annotated data.

[0171] In another specific embodiment, acquiring a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts further includes:

[0172] (1) Obtaining an endoscopic detection report, which includes a plurality of endoscopic detection image areas and an image description text area.

[0173] like Figure 4 As shown, the endoscopic detection report includes 6 endoscopic detection image areas and image description text areas. For example, the endoscopic detection image area is an image of a gastric angle, and the text in the image description text area is: "Gastric angle: arc-shaped, smooth".

[0174] (2) The endoscopic detection image area and the image description text area are cropped from the endoscopic detection report to obtain a plurality of first endoscopic detection images and image description text area images within the plurality of endoscopic detection image areas.

[0175] In a specific embodiment, the endoscopic detection report is segmented to obtain multiple endoscopic detection image areas and image description text areas. The endoscopic detection image areas and image description text areas are cropped to obtain multiple first endoscopic detection images and image description text area images in the multiple endoscopic detection image areas.

[0176] (3) Perform text recognition on the image description text area to obtain the overall image description text.

[0177] The overall image description text includes image description texts of multiple first endoscopic detection images. For example, the overall image description text includes image description texts of the esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, etc.

[0178] (4) Determine the image description annotation text corresponding to each first endoscopic detection image from the overall image description text, and obtain each first endoscopic detection image and the corresponding image description annotation text.

[0179] In a specific embodiment, keyword detection is performed on the overall image description text to obtain each shooting part text in the overall image description text. Each shooting part text may be esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, etc. It is determined whether the number of shooting part texts is the same as the number of first endoscopic detection images. If the number of shooting part texts is the same as the number of first endoscopic detection images, the current arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image are obtained, and each shooting part text is matched with each first endoscopic detection image based on the current arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image to obtain each shooting part text and the corresponding first endoscopic detection image, and the detection condition description text corresponding to each shooting part text is obtained. The detection condition description text is located between two shooting part texts, and the shooting part text and the corresponding detection condition description text are determined as an image description annotation text to obtain each first endoscopic detection image and the corresponding image description annotation text.

[0180] Among them, the further to the left the shooting part text is and the higher the shooting part text is, the earlier the current arrangement order of the shooting part text is; the further to the left the first endoscopic detection image is and the higher the first endoscopic detection image is, the earlier the current arrangement order of the first endoscopic detection image is.

[0181] In a specific embodiment, each shooting part text is matched with each first endoscopic detection image based on the current text arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image, including: matching each shooting part text with each first endoscopic detection image one by one in the order of the current arrangement order to obtain each first endoscopic detection image and the corresponding image description annotation text.

[0182] However, it is possible that the doctor will not place the endoscopic detection images in the corresponding order, and the shooting part text is generally generated by a template, and the user generally will not adjust it. Therefore, according to the order of the current arrangement order, each shooting part text is not matched with each first endoscopic detection image one by one. Therefore, in order to improve the matching success rate, in a specific embodiment, each shooting part text is matched with each first endoscopic detection image based on the current arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image, including: obtaining the issuing user who issued the endoscopic detection report, obtaining the target correspondence between the image arrangement order and the text arrangement order of the issuing user, and matching each shooting part text with each first endoscopic detection image based on the target correspondence between the image arrangement order and the text arrangement order of the issuing user. Specifically, when the issuing user has issued the endoscopic detection report multiple times in history, the historical correspondence between the historical image arrangement order and the historical text arrangement order is obtained, and the historical correspondence with the highest frequency is determined as the target correspondence. For example, the historical correspondence is: the historical text arrangement order of the texts of each shooting part is esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum; and the corresponding historical image arrangement order is gastric antrum, esophagus, gastric fundus, cardia, gastric body, gastric angle.

[0183] Since it is possible that some endoscopic detection images have no problems, the doctor did not put them up, resulting in the number of shooting part texts being different from the number of the first endoscopic detection images, and they cannot be directly matched at this time. If the number of shooting part texts is different from the number of the first endoscopic detection images, the detection status description text corresponding to each shooting part text is obtained, and the no-abnormal template description text corresponding to each shooting part text is obtained, wherein the no-abnormal template description text is the description text automatically generated by the system to save the doctor's time when there is no abnormality in the shooting part. The number of the first endoscopic detection images is determined as the second preset number, and the text similarity between the no-abnormal template description text corresponding to the shooting part text and the detection status description text corresponding to the shooting part text is calculated, and the shooting part texts of the second preset number that are ranked higher in text similarity from small to large are matched with the first endoscopic detection images of the second preset number to obtain the first endoscopic detection images of the second preset number and the corresponding shooting part texts.

[0184] In an embodiment of the present application, a plurality of endoscopic detection reports are obtained; a first endoscopic detection image and a corresponding image description annotation text are obtained from the detection report, thereby obtaining a plurality of first endoscopic detection images and a corresponding image description annotation text.

[0185] In another specific embodiment, the image description annotation text includes the shooting site text on the sixth endoscopic detection image, and acquiring a plurality of sixth endoscopic detection images and a corresponding plurality of image description annotation texts includes:

[0186] (1) Obtaining the site occurrence frequency of each captured site text on a plurality of first endoscopic detection images.

[0187] The site appearance frequency of the photographing site text is the number of first endoscopic detection images in which the photographing site corresponding to the photographing site text appears.

[0188] (2) The captured part texts whose part occurrence frequency is lower than a preset frequency are determined as the parts to be enhanced.

[0189] The preset frequency may be the median of the occurrence frequency of each shooting part text.

[0190] (3) Performing image enhancement on a plurality of first endoscopic detection images whose captured part texts are parts to be enhanced, to obtain a plurality of endoscopic enhanced images.

[0191] In an embodiment of the present application, image enhancement is performed on multiple first endoscopic detection images whose shooting site text is the site to be enhanced to obtain multiple endoscopically enhanced images, including: performing at least one of rotation, cropping, scaling, brightness adjustment, and color adjustment on each of the first endoscopic detection images whose shooting site text is the site to be enhanced to perform image enhancement and obtain multiple endoscopically enhanced images.

[0192] (4) Determine the plurality of endoscopically enhanced images and the corresponding image description annotation texts as a plurality of sixth endoscopically detected images and the corresponding plurality of image description annotation texts.

[0193] 207. Add multiple sixth endoscopic detection images and corresponding multiple image description annotation texts into the training set to obtain a new training set.

[0194] 208. Determine a new model test result corresponding to the new fine-tuning model based on the new training set and the test set, until the new model test result meets the preset conditions.

[0195] The steps of determining a new model test result corresponding to a new fine-tuned model based on the new training set and test set can refer to 204-206.

[0196] 209. A new fine-tuned model whose new model test results meet preset conditions is determined as a target anomaly detection model.

[0197] 210. When the endoscopic detection image to be identified is obtained, the endoscopic detection image to be identified is input into the target anomaly detection model for anomaly detection to obtain an anomaly detection description text corresponding to the endoscopic detection image to be identified.

[0198] The target anomaly detection model can provide auxiliary diagnosis services and provide personalized services for different user groups.

[0199] In order to facilitate better implementation of the anomaly detection method based on a large language model provided in the embodiment of the present application, the embodiment of the present application also provides an anomaly detection device based on a large language model based on the above anomaly detection method based on a large language model. The meaning of the nouns is the same as that in the above anomaly detection method based on a large language model. For specific implementation details, please refer to the description in the above method embodiment.

[0200] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of an anomaly detection device based on a large language model provided in an embodiment of the present application, the anomaly detection device based on a large language model may include:

[0201] A first acquisition module 701 is used to acquire a first endoscopic detection image and a corresponding image description annotation text;

[0202] A first determination module 702 is used to determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of first endoscopic detection images and corresponding image description annotation texts;

[0203] A splitting module 703 is used to split the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio, wherein the training set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts, and the test set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts;

[0204] A fine-tuning module 704, configured to fine-tune a preset neural network model based on a training set to obtain a fine-tuned model;

[0205] A testing module 705 is used to test the fine-tuning model based on the test set to obtain a model test result;

[0206] The second acquisition module 706 is used to acquire a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts when the model test result does not meet the preset condition;

[0207] An adding module 707, used to add a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts into the training set to obtain a new training set;

[0208] A second determination module 708 is used to determine a new model test result corresponding to the new fine-tuning model based on the new training set and the test set, until the new model test result meets a preset condition;

[0209] The third determination module 709 is used to determine the new fine-tuning model whose new model test results meet the preset conditions as the target anomaly detection model;

[0210] The anomaly detection module 710 is used to input the endoscopic detection image to be identified into the target anomaly detection model for anomaly detection when the endoscopic detection image to be identified is acquired, so as to obtain an anomaly detection description text corresponding to the endoscopic detection image to be identified.

[0211] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0212] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the large language model-based anomaly detection method provided in this embodiment by calling a computer program stored in the memory.

[0213] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0214] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0215] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0216] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0217] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0218] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0219] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the anomaly detection method based on the large language model provided in this application, such as:

[0220] Acquire a first endoscopic detection image and a corresponding image description annotation text; determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on the plurality of first endoscopic detection images and the corresponding image description annotation texts; split the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio, wherein the training set includes a plurality of second endoscopic detection images and the corresponding image description annotation texts, and the test set includes a plurality of second endoscopic detection images and the corresponding image description annotation texts; fine-tune a preset neural network model based on the training set to obtain a fine-tuned model; test the fine-tuned model based on the test set to obtain a model test result; and when the model test result is not satisfactory, When the preset conditions are met, multiple sixth endoscopic detection images and corresponding multiple image description annotation texts are obtained; the multiple sixth endoscopic detection images and the corresponding multiple image description annotation texts are added to the training set to obtain a new training set; based on the new training set and the test set, a new model test result corresponding to the new fine-tuning model is determined until the new model test result meets the preset conditions; the new fine-tuning model whose new model test result meets the preset conditions is determined as the target abnormality detection model; when the endoscopic detection image to be identified is obtained, the endoscopic detection image to be identified is input into the target abnormality detection model for abnormality detection, and the abnormality detection description text corresponding to the endoscopic detection image to be identified is obtained.

[0221] It should be noted that the electronic device provided in the embodiment of the present application belongs to the same concept as the anomaly detection method based on a large language model in the above embodiment. The specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0222] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on the processor of the electronic device provided in the embodiment of the present application, the processor of the electronic device executes the steps in the anomaly detection method based on a large language model provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0223] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the above-mentioned anomaly detection method based on a large language model.

[0224] The above is a detailed introduction to the anomaly detection method and device based on a large language model provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0225] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for anomaly detection based on a large language model, characterized in that: include: Acquire a first endoscopic detection image and a corresponding image description annotation text; Determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on the plurality of first endoscopic detection images and corresponding image description annotation texts; Splitting the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio, wherein the training set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts, and the test set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts; Fine-tune the preset neural network model based on the training set to obtain a fine-tuned model; Testing the fine-tuning model based on the test set to obtain a model test result; When the model test result does not meet the preset conditions, a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts are obtained, wherein the image description annotation text includes the shooting part text on the sixth endoscopic detection image, and the part occurrence frequency of each shooting part text on the plurality of first endoscopic detection images is obtained; the shooting part text with a part occurrence frequency lower than the preset frequency is determined as the part to be enhanced; image enhancement is performed on the plurality of first endoscopic detection images with the shooting part text as the part to be enhanced to obtain a plurality of endoscopic enhanced images; the plurality of endoscopic enhanced images and the corresponding image description annotation texts are determined as a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts; Adding a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts into the training set to obtain a new training set; Determine a new model test result corresponding to the new fine-tuning model based on the new training set and test set, until the new model test result meets the preset conditions; Determine the new fine-tuned model whose new model test results meet the preset conditions as the target anomaly detection model; When the endoscopic detection image to be identified is obtained, the endoscopic detection image to be identified is input into the target anomaly detection model for anomaly detection, and an anomaly detection description text corresponding to the endoscopic detection image to be identified is obtained.

2. The anomaly detection method based on a large language model according to claim 1, characterized in that: The step of testing the fine-tuning model based on the test set to obtain a model test result includes: Inputting the second endoscopic detection image in the test set into the fine-tuning model to obtain an image description prediction text corresponding to the second endoscopic detection image; Calculating a text similarity index between the image description annotation text corresponding to the second endoscopic detection image and the image description prediction text corresponding to the second endoscopic detection image; Determining the second endoscopic detection image whose text similarity index is greater than a preset similarity as the second endoscopic detection image that is predicted correctly; The proportion of the second endoscopic detection images predicted correctly in the test set is determined as the model test result.

3. The anomaly detection method based on a large language model according to claim 2, characterized in that: The image description annotation text and the image description prediction text both include a disease feature field and a plurality of lesion feature fields, and the calculating of a text similarity index between the image description annotation text corresponding to the second endoscopic detection image and the image description prediction text corresponding to the second endoscopic detection image includes: Based on whether the disease type feature field of the image description annotation text and the disease type feature field of the image description prediction text are the same, a weight coefficient is determined, wherein if the disease type feature field of the image description annotation text and the disease type feature field of the image description prediction text are the same, the weight coefficient is a first weight, and if the disease type feature field of the image description annotation text and the disease type feature field of the image description prediction text are different, the weight coefficient is a second weight, and the first weight is greater than the second weight; Determine the proportion of the same fields describing the lesion features in the image description annotation text and the image description prediction text; The proportion of the same field is weighted based on the weight coefficient, and the obtained weighted value is determined as the text similarity index of the image description annotation text and the image description prediction text.

4. The anomaly detection method based on a large language model according to claim 2, characterized in that: The step of acquiring a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts further includes: Obtaining an endoscopic detection report, wherein the endoscopic detection report includes a plurality of endoscopic detection image areas and an image description text area; Cutting out the endoscopic detection image area and the image description text area from the endoscopic detection report to obtain a plurality of first endoscopic detection images and image description text area images in the plurality of endoscopic detection image areas; Perform text recognition on the image description text area image to obtain the overall image description text; The image description annotation text corresponding to each first endoscopic detection image is determined from the overall image description text to obtain each first endoscopic detection image and the corresponding image description annotation text.

5. The anomaly detection method based on a large language model according to claim 4, characterized in that: The step of performing image enhancement on the plurality of first endoscopic detection images whose captured part texts are parts to be enhanced to obtain a plurality of endoscopic enhanced images comprises: The first endoscopic detection images whose shooting part text is the part to be enhanced are respectively subjected to at least one of rotation, cropping, scaling, brightness adjustment, and color adjustment to perform image enhancement to obtain a plurality of endoscopic enhanced images.

6. The anomaly detection method based on a large language model according to claim 5, characterized in that: The step of fine-tuning the preset neural network model based on the training set to obtain a fine-tuned model includes: Divide the training set into K subsets; Determine the K subsets as validation sets respectively, combine the subsets of the K subsets other than the validation set to determine a combined training set, and obtain K validation sets and a corresponding combined training set; Training the preset neural network model based on the combined training set to obtain a combined training model corresponding to the combined training set; Verifying the combined training model corresponding to the combined training set based on the verification set corresponding to the combined training set, obtaining model verification parameters of the combined training model corresponding to the combined training set, and obtaining model verification parameters of the combined training models corresponding to multiple combined training sets; The combined trained model with the highest model validation parameters is determined as the fine-tuning model.

7. An anomaly detection device based on a large language model, characterized in that: include: A first acquisition module is used to acquire a first endoscopic detection image and a corresponding image description annotation text; A first determination module, configured to determine a plurality of second endoscopic detection images and corresponding image description annotation texts based on a plurality of first endoscopic detection images and corresponding image description annotation texts; A splitting module, used to split the plurality of second endoscopic detection images and the corresponding image description annotation texts into a training set and a test set according to a preset ratio, wherein the training set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts, and the test set includes the plurality of second endoscopic detection images and the corresponding image description annotation texts; A fine-tuning module, used to fine-tune the preset neural network model based on the training set to obtain a fine-tuning model; A testing module, used to test the fine-tuning model based on the test set to obtain a model testing result; A second acquisition module is used for acquiring a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts when the model test result does not meet the preset conditions, wherein the image description annotation text includes the shooting part text on the sixth endoscopic detection image, and acquiring the part occurrence frequency of each shooting part text on the plurality of first endoscopic detection images; determining the shooting part text with a part occurrence frequency lower than the preset frequency as the part to be enhanced; performing image enhancement on the plurality of first endoscopic detection images with the shooting part text as the part to be enhanced to obtain a plurality of endoscopic enhanced images; determining the plurality of endoscopic enhanced images and the corresponding image description annotation texts as a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts; An adding module, used for adding a plurality of sixth endoscopic detection images and a plurality of corresponding image description annotation texts into the training set to obtain a new training set; A second determination module is used to determine a new model test result corresponding to the new fine-tuning model based on the new training set and the test set, until the new model test result meets a preset condition; A third determination module is used to determine a new fine-tuning model whose new model test results meet preset conditions as a target anomaly detection model; The anomaly detection module is used to input the endoscopic detection image to be identified into the target anomaly detection model for anomaly detection when the endoscopic detection image to be identified is acquired, so as to obtain the anomaly detection description text corresponding to the endoscopic detection image to be identified.

8. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the anomaly detection method based on a large language model according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the anomaly detection method based on a large language model according to any one of claims 1 to 6.

Images