A method and device for constructing a large model image-text pair data set

Abstract

The present application discloses a method and device for constructing a large model image-text pair data set, wherein multiple detection image groups are determined based on multiple second endoscopic detection images; the first similarity of the detection image groups is calculated; the second similarity of the detection image groups is determined based on the lesion segmentation areas on the two second endoscopic detection images; the third similarity of the detection image groups is determined based on the first similarity of the detection image groups and the second similarity of the detection image groups, and the third similarity of the multiple detection image groups is obtained; at least one second endoscopic detection image in the detection image group whose third similarity is greater than the preset similarity is deleted from the multiple second endoscopic detection images, and multiple third endoscopic detection images are obtained; a third endoscopic detection image and the corresponding image description text are determined as large model image-text pair data, and a large model image-text pair data set containing multiple large model image-text pair data is obtained. The present application can improve the construction efficiency and data quality of the large model image-text pair data set.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for constructing a large-model image-text pair data set. Background Art

[0002] At present, traditional detection technologies based on deep learning methods have many shortcomings, such as poor interpretability. It is urgent to develop an auxiliary detection method and system based on a multimodal large language model to automatically identify whether the digestive tract examination images are abnormal and give a standardized examination report. High-quality image-text data sets often determine the effect of multimodal large language models. Traditional data set construction methods consider fewer factors and rely on manual annotation, resulting in low efficiency in constructing large model image-text data sets and uneven data quality. Summary of the invention

[0003] The embodiments of the present application provide a method and device for constructing a large model image-text pair data set, which can improve the construction efficiency and data quality of the large model image-text pair data set.

[0004] In the first aspect, the method for constructing a large model image-text pair dataset provided by the present application includes:

[0005] Acquire a first endoscopic detection image and a corresponding image description text;

[0006] Determining a plurality of second endoscopic detection images and corresponding image description texts based on the plurality of first endoscopic detection images and corresponding image description texts;

[0007] determining a plurality of detection image groups based on a plurality of the second endoscopic detection images, wherein the detection image groups include two of the second endoscopic detection images;

[0008] calculating a first similarity between two of the second endoscope detection images in the detection image group to obtain a first similarity of the detection image group;

[0009] Acquire lesion segmentation regions on two second endoscopic detection images of the detection image group obtained by respectively performing lesion segmentation on the two second endoscopic detection images;

[0010] determining a second similarity between the two second endoscopic detection images based on the lesion segmentation regions on the two second endoscopic detection images, and obtaining a second similarity between the detection image group;

[0011] determining a third similarity of the detection image groups based on the first similarity of the detection image groups and the second similarity of the detection image groups, and obtaining third similarities of a plurality of the detection image groups;

[0012] Deleting at least one of the second endoscopic detection images in the detection image group whose third similarity is greater than a preset similarity from the plurality of second endoscopic detection images to obtain a plurality of third endoscopic detection images;

[0013] A third endoscopic detection image and the corresponding image description text are determined as large model image-text pair data, and a large model image-text pair data set containing multiple large model image-text pair data is obtained.

[0014] In the second aspect, the large model image-text pair data set construction device provided by the present application includes:

[0015] A first acquisition module, used to acquire a first endoscopic detection image and a corresponding image description text;

[0016] A first determination module, configured to determine a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of first endoscopic detection images and corresponding image description texts;

[0017] a second determination module, configured to determine a plurality of detection image groups based on a plurality of the second endoscopic detection images, wherein the detection image groups include two of the second endoscopic detection images;

[0018] a calculation module, used for calculating a first similarity between two of the second endoscopic detection images in the detection image group to obtain the first similarity of the detection image group;

[0019] A second acquisition module, used for acquiring lesion segmentation areas on two second endoscopic detection images of the detection image group obtained by respectively performing lesion segmentation on the two second endoscopic detection images;

[0020] A third determination module is used to determine a second similarity between two second endoscopic detection images based on the lesion segmentation areas on the two second endoscopic detection images to obtain a second similarity between the detection image group;

[0021] a fourth determining module, configured to determine a third similarity of the detection image group based on the first similarity of the detection image group and the second similarity of the detection image group, to obtain third similarities of a plurality of the detection image groups;

[0022] a deleting module, configured to delete at least one of the second endoscopic detection images in the detection image group whose third similarity is greater than a preset similarity from the plurality of second endoscopic detection images, so as to obtain a plurality of third endoscopic detection images;

[0023] The fifth determination module is used to determine a third endoscopic detection image and a corresponding image description text as large model image-text pair data, and obtain a large model image-text pair data set containing multiple large model image-text pair data.

[0024] In a third aspect, the electronic device provided in the present application includes a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the method for constructing a large model image and text data set provided in the present application.

[0025] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the method for constructing a large model image-text pair data set provided in the present application.

[0026] In the fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implements the steps in the method for constructing a large model image-text pair data set provided in the present application.

[0027] In the present application, compared with the related art, a first endoscopic detection image and a corresponding image description text are obtained; based on the plurality of first endoscopic detection images and the corresponding image description texts, a plurality of second endoscopic detection images and the corresponding image description texts are determined; based on the plurality of second endoscopic detection images, a plurality of detection image groups are determined, wherein the detection image group includes two second endoscopic detection images; a first similarity of the two second endoscopic detection images in the detection image group is calculated to obtain a first similarity of the detection image group; lesion segmentation areas on the two second endoscopic detection images obtained by respectively performing lesion segmentation on the two second endoscopic detection images of the detection image group are obtained; based on the two second endoscopic detection images, a plurality of detection image groups are determined, wherein the detection image group includes two second endoscopic detection images; a first similarity of the two second endoscopic detection images in the detection image group is calculated to obtain a first similarity of the detection image group; a lesion segmentation area on the two second endoscopic detection images obtained by respectively performing lesion segmentation on the two second endoscopic detection images of the detection image group is obtained; based on the two second endoscopic detection images, a plurality of detection image groups are determined, wherein the detection image group includes two second endoscopic detection images; a first similarity of the two second endoscopic detection images in the detection image group is calculated to obtain a first similarity of the detection image group; a first similarity of the two second endoscopic detection images in the detection image group is calculated to obtain a first similarity of the two second endoscopic detection images in ... The lesion segmentation area on the detection image determines the second similarity of the two second endoscopic detection images, and obtains the second similarity of the detection image group; based on the first similarity of the detection image group and the second similarity of the detection image group, the third similarity of the detection image group is determined to obtain the third similarity of multiple detection image groups; at least one second endoscopic detection image in the detection image group whose third similarity is greater than the preset similarity is deleted from the multiple second endoscopic detection images to obtain multiple third endoscopic detection images; a third endoscopic detection image and the corresponding image description text are determined as large model image-text pair data to obtain a large model image-text pair data set containing multiple large model image-text pair data. The present application can improve the construction efficiency and data quality of the large model image-text pair data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1It is a scene schematic diagram of a large model image-text pair data set construction system provided in an embodiment of the present application;

[0030] Figure 2 It is a flow chart of an embodiment of a method for constructing a large model image-text pair data set provided in an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of a first endoscopic detection image in an embodiment of a method for constructing a large model image-text pair data set provided in an embodiment of the present application;

[0032] Figure 4 It is a schematic diagram of calculating the first similarity in an embodiment of the method for constructing a large model image-text pair data set provided in an embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of an unclean segmented area and a lesion segmented area of ​​a fourth endoscopic detection image in one embodiment of a method for constructing a large model image-text pair data set provided by an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of a circumscribed rectangle of a lesion segmentation area on two second endoscopic detection images of a detection image group in one embodiment of a method for constructing a large model image-text pair data set provided by an embodiment of the present application;

[0035] Figure 7 It is a structural schematic diagram of a large model image-text pair data set construction device provided in an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0038] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0041] In order to improve the effect of constructing a large model image-text pair data set, the embodiment of the present application provides a large model image-text pair data set construction method, a large model image-text pair data set construction device, an electronic device, a computer-readable storage medium, and a computer program product. Among them, the large model image-text pair data set construction method can be executed by the large model image-text pair data set construction device, or by an electronic device integrated with the large model image-text pair data set construction device.

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0043] Please refer to Figure 1 ,This application also provides a large model image-text pair dataset construction system, such as Figure 1 As shown, the large model image-text pair data set construction system electronic device 100, the electronic device 100 is integrated with the large model image-text pair data set construction device provided by the present application.

[0044] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0045] In addition, if Figure 1 As shown, the large model image-text data set construction system may also include a memory 200 for storing original data, intermediate data and result data.

[0046] In the embodiment of the present application, the memory 200 may be a cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems and other functions to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and jointly provide data storage and business access functions to the outside world.

[0047] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume, and the physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0048] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is divided into stripes in advance. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0049] It should be noted that Figure 1 The scenario diagram of the large model image-text dataset construction system shown is merely an example. The large model image-text dataset construction system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the large model image-text dataset construction system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0050] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0051] Please refer to Figure 2 , Figure 2 : is a flow chart of an embodiment of a method for constructing a large model image-text pair data set provided in an embodiment of the present application, such as Figure 2 As shown, the process of the method for constructing a large model image-text pair dataset provided by this application is as follows:

[0052] 201. Obtain a first endoscopic detection image and a corresponding image description text.

[0053] In the embodiment of the present application, the plurality of first endoscopic detection images may be endoscopic images obtained by endoscopy, images obtained by gastroscopy, or images obtained by colonoscopy, which may be set according to specific circumstances. The image description text corresponding to the first endoscopic detection image is the text determined by the doctor based on the first endoscopic detection image to describe the first endoscopic detection image.

[0054] like Figure 3 As shown, Figure 3 A plurality of first endoscopic detection images are schematically shown.

[0055] In a specific embodiment, the first endoscopic detection image and the corresponding image description text may be input by a user.

[0056] In another specific embodiment, an endoscopic detection report is obtained, and the endoscopic detection report generally includes an endoscopic detection image area and an image description text area. For example, the endoscopic detection image area is an image of a stomach angle, and the text in the image description text area is: "Stomach angle: arc-shaped, smooth". Specifically, the endoscopic detection image area and the image description text area are cut out from the endoscopic detection report to obtain a first endoscopic detection image and a corresponding image description text area image, and text recognition is performed on the image description text area image to obtain the image description text corresponding to the first endoscopic detection image, thereby obtaining the first endoscopic detection image and the corresponding image description text.

[0057] In an embodiment of the present application, a plurality of endoscopic detection reports are obtained; a first endoscopic detection image and a corresponding image description text are obtained from the detection report, thereby obtaining a plurality of first endoscopic detection images and corresponding image description texts.

[0058] Specifically, the image description text includes an endoscopic part text corresponding to the first endoscopic detection image and a detection status description text of the endoscopic part text. The endoscopic detection image, the endoscopic part text and the detection status description text of the endoscopic part are cut out from the endoscopic detection report to obtain an endoscopic detection image, an endoscopic part area image and a detection status description text area image, and text recognition is performed on the endoscopic part area image and the detection status description text area image respectively to obtain the endoscopic part text corresponding to the first endoscopic detection image and the detection status description text of the endoscopic part text.

[0059] 202. Determine a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of first endoscopic detection images and corresponding image description texts.

[0060] In a specific embodiment, a plurality of first endoscopic detection images and corresponding image description texts are determined as a plurality of second endoscopic detection images and corresponding image description texts.

[0061] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of first endoscopic detection images and corresponding image description texts includes:

[0062] (1) Obtaining an image resolution of each first endoscopic detection image and a preset resolution threshold of each first endoscopic detection image.

[0063] The preset resolution threshold of each first endoscopic detection image may be the same threshold, which can be set according to specific circumstances. For example, the preset resolution threshold of each first endoscopic detection image is 5 million ppi (pixels per inch).

[0064] In a specific embodiment, the preset resolution thresholds of the first endoscopic detection images may not be the same threshold.

[0065] Specifically, the shooting part of each first endoscopic detection image is detected to obtain the shooting part of each first endoscopic detection image; the shooting part of the first endoscopic detection image is determined based on the shooting part of the first endoscopic detection image, and different shooting parts correspond to different preset resolution thresholds; the preset resolution threshold of each first endoscopic detection image is determined based on the shooting part of each first endoscopic detection image. For example, the shooting part may be the esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, and duodenum. When the shooting part is the esophagus, the corresponding preset resolution threshold is 4 million ppi; when the shooting part is the cardia, the corresponding preset resolution threshold is 6 million ppi. Different shooting parts correspond to different preset resolution thresholds to avoid screening out too many images.

[0066] (2) Determine the first endoscopic detection images whose image resolution is higher than the corresponding preset resolution threshold as a plurality of fourth endoscopic detection images.

[0067] Specifically, determine whether the image resolution of the first endoscopic detection image is higher than the preset resolution threshold of the first endoscopic detection image. If the image resolution of the first endoscopic detection image is higher than the preset resolution threshold of the first endoscopic detection image, the first endoscopic detection image is determined as the fourth endoscopic detection image. If the image resolution of the first endoscopic detection image is not higher than the preset resolution threshold of the first endoscopic detection image, the first endoscopic detection image is not determined as the fourth endoscopic detection image.

[0068] (3) Determining a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of fourth endoscopic detection images and corresponding image description texts.

[0069] In a specific embodiment, the plurality of fourth endoscopic detection images and the corresponding image description texts are determined as the plurality of second endoscopic detection images and the corresponding image description texts.

[0070] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of fourth endoscopic detection images and corresponding image description texts includes:

[0071] (1) Performing image segmentation on the fourth endoscopic detection image to obtain an unclean segmented area and a lesion segmented area of ​​the fourth endoscopic detection image.

[0072] Pre-train an image segmentation model, such as a Segment Anything Model (SAM) model, a Unet model, a mask-rcnn model, etc. The image segmentation model is used to segment the unclean area, the lesion area, and the normal mucosal area of ​​the endoscopic detection image. The unclean segmentation area includes areas containing artifacts, flushing, biopsy, surgery, bubbles, and residues. The lesion segmentation area is the area where the lesion is detected.

[0073] like Figure 5 As shown, Figure 5 The unclean segmented area and the lesion segmented area of ​​the fourth endoscope detection image are shown, wherein the area segmented on the left is the lesion segmented area, the area segmented on the right is the unclean segmented area, and the other areas are normal mucosa areas.

[0074] (2) Determine an unclean area ratio of an unclean segmented region and a lesion area ratio of a lesion segmented region of the fourth endoscopic detection image.

[0075] Among them, the unclean area ratio of the unclean segmented area is the area ratio of the unclean segmented area to the fourth endoscopic detection image. The normal mucosal area ratio of the normal mucosal area is the area ratio of the normal mucosal area to the fourth endoscopic detection image. The lesion segmentation area ratio of the lesion segmentation area is the area ratio of the lesion segmentation area to the fourth endoscopic detection image. For example, the lesion area ratio of the lesion segmentation area is A, the unclean segmentation area ratio is C, and the normal mucosal area ratio is B, because A+B+C=1, B is also equal to =1-AC.

[0076] (3) Based on a preset weight coefficient, a weighted sum of the unclean area ratio of the unclean segmented region and the lesion area ratio of the lesion segmented region of the fourth endoscopic detection image is taken to obtain the image cleanliness of the fourth endoscopic detection image, wherein the larger the unclean area ratio, the smaller the image cleanliness; and the larger the lesion area ratio, the greater the image cleanliness.

[0077] Among them, the greater the image cleanliness, the cleaner the image.

[0078] The preset weight coefficients include a first weight coefficient w1, a second weight coefficient w2 and a third weight coefficient w3. The first weight coefficient, the second weight coefficient and the third weight coefficient are 2, 0.1 and -2 respectively.

[0079] Specifically, the calculation formula of image cleanliness S is S=w1*A+w2*(1-AC)+w3*C.

[0080] Among them, w1 is the first weight coefficient, the second weight coefficient is w2, and the third weight coefficient is w3. The first weight coefficient w1 and the second weight coefficient w2 are positive numbers, and the third weight coefficient is a negative number.

[0081] (4) The fourth endoscopic detection images having an image cleanliness higher than a preset cleanliness are determined as a plurality of fifth endoscopic detection images.

[0082] Among them, the preset cleanliness level can be set according to the specific situation.

[0083] (5) Based on the plurality of fifth endoscopic detection images and the corresponding image description texts, determine a plurality of second endoscopic detection images and the corresponding image description texts.

[0084] In a specific embodiment, the plurality of fifth endoscopic detection images and the corresponding image description texts are determined as the plurality of second endoscopic detection images and the corresponding image description texts.

[0085] In another specific embodiment, determining a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of fifth endoscopic detection images and corresponding image description texts includes:

[0086] (1) Obtain the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score, and shooting site score of the fifth endoscopic detection image.

[0087] In the embodiment of the present application, the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score and shooting site score of the fifth endoscopic detection image are obtained, including:

[0088] Obtain the preset normal brightness value and the brightness value of the fifth endoscopic detection image, and determine the brightness score of the fifth endoscopic detection image based on the absolute value of the difference between the preset normal brightness value and the brightness value of the fifth endoscopic detection image, wherein the larger the absolute value of the difference, the smaller the brightness score of the fifth endoscopic detection image. The brightness value of the fifth endoscopic detection image is the average value of the brightness values ​​of each pixel on the fifth endoscopic detection image. When the brightness value is close to the preset normal brightness value, it is determined that the fifth endoscopic detection image is normal, and the highest brightness score is 1.

[0089] For example, brightness is divided into five categories: bright, bright, normal, dark, and dim, and the brightness scores correspond to -1, 0, 1, 0, and -1 points, respectively, recorded as Score_b.

[0090] A contrast value of the fifth endoscopic detection image is obtained, and a contrast score of the fifth endoscopic detection image is determined based on the contrast value of the fifth endoscopic detection image, wherein the larger the contrast value, the larger the contrast score of the fifth endoscopic detection image.

[0091] For example, contrast is divided into three categories: high, medium, and low, and the contrast scores correspond to 1, 0, and -1 points, respectively, recorded as Score_c.

[0092] The size of the noise region on the fifth endoscopic detection image is obtained, and the noise score of the fifth endoscopic detection image is determined based on the size of the noise region, wherein the larger the size of the noise region, the larger the noise score of the fifth endoscopic detection image. Specifically, the proportion of the size of the noise region to the noise area of ​​the entire image is calculated, and the larger the proportion of the noise area, the smaller the noise score of the fifth endoscopic detection image.

[0093] The noise is segmented into a noise area through a noise segmentation model, and then the ratio of the noise area to the noise area of ​​the entire image is calculated S2, and finally 1-S2 is determined as the noise score, recorded as Score_n.

[0094] The size of the artifact region on the fifth endoscopic detection image is obtained, and the artifact score of the fifth endoscopic detection image is determined based on the size of the artifact region, wherein the larger the size of the artifact region, the smaller the artifact of the fifth endoscopic detection image. Specifically, the proportion of the size of the artifact region to the artifact area of ​​the entire image is calculated, and the larger the proportion of the artifact area, the smaller the artifact score of the fifth endoscopic detection image.

[0095] The artifact is segmented into an artifact area through an artifact segmentation model, and then the proportion of the artifact area size to the artifact area of ​​the entire image S3 is calculated. Finally, 1-S3 is determined as the artifact score, recorded as Score_a.

[0096] The mucosal condition is divided into two categories: clear and blurred. The mucosal area clarity scores correspond to 1 and -1 points respectively. The mucosal area clarity score is recorded as Score_m.

[0097] The presence or absence of lesions in the mucosal area was divided into two categories: yes and no, and the abnormal scores of the mucosal area corresponded to 1 and 0 points, respectively. The abnormal score of the mucosal area was recorded as Score_f.

[0098] The total area ratio of the lesion region on the fifth endoscopic detection image is obtained, and the lesion score of the fifth endoscopic detection image is determined based on the total area ratio of the lesion region, wherein the greater the total area ratio of the lesion region, the greater the lesion score of the fifth endoscopic detection image. Specifically, the fifth endoscopic detection image is segmented to obtain the unclean segmented area and the lesion segmented area of ​​the fifth endoscopic detection image; the unclean area ratio of the unclean segmented area and the lesion area ratio of the lesion segmented area of ​​the fifth endoscopic detection image are determined. The lesion area ratios of each lesion segmented area on the fifth endoscopic detection image are summed to obtain the total area ratio of the lesion region.

[0099] The shooting part of the fifth endoscopic detection image is obtained, and a shooting part score of the fifth endoscopic detection image is determined based on the shooting part, wherein different shooting parts correspond to different shooting part scores, and wherein the shooting part score when the shooting part is the esophagus, cardia and duodenum is greater than the shooting part score when the shooting part is other parts.

[0100] The locations are divided into 7 major parts: esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, duodenum, and other parts. When the shooting location is esophagus, cardia, and duodenum, the shooting location score is 1.5, and when the shooting location is other parts, the shooting location score is 1, recorded as Score_1.

[0101] (2) Determine the overall feature score of the fifth endoscopic detection image based on the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score, and shooting location score of the fifth endoscopic detection image, and obtain the overall feature score of each fifth endoscopic detection image.

[0102] Specifically, the calculation formula of the overall feature score P of the fifth endoscopic detection image is:

[0103] P=(Score_b+Score_c+Score_n+Score_a) / 4+(Score_m+Score_f+Score_s+Score_l) / 4.

[0104] Among them, the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score and shooting site score of the fifth endoscopic detection image are Score_b, Score_c, Score_n, Score_a, Score_m, Score_f, Score_s and Score_l respectively.

[0105] If the mucosa is hazy, without lesions, or the location is other, the overall feature score is set to 0.

[0106] (3) The fifth endoscopic detection images and the corresponding image description texts having an overall feature score higher than a preset score are determined as a plurality of second endoscopic detection images and the corresponding image description texts.

[0107] Among them, the preset score can be set according to the situation and is not limited here.

[0108] 203. Determine a plurality of detection image groups based on the plurality of second endoscopic detection images, wherein the detection image group includes two second endoscopic detection images.

[0109] In a specific embodiment, any two second endoscopic detection images of the plurality of second endoscopic detection images are determined as one detection image group or a plurality of detection image groups.

[0110] 204. Calculate a first similarity between two second endoscope detection images in the detection image group to obtain a first similarity of the detection image group.

[0111] In the embodiment of the present application, calculating the first similarity of two second endoscope detection images in the detection image group to obtain the first similarity of the detection image group includes:

[0112] (1) Feature extraction is performed on the two second endoscopic detection images respectively to obtain two detection image feature maps of the two second endoscopic detection images.

[0113] In the embodiment of the present application, the two second endoscopic detection images are respectively input into the feature extraction network for feature extraction to obtain two detection image feature maps of the two second endoscopic detection images.

[0114] like Figure 4 As shown, the two second endoscopic detection images are respectively input into two feature extraction networks as x1 and x2 to obtain two detection image feature maps of the two second endoscopic detection images.

[0115] (2) Map the two detection image feature maps to one-dimensional vectors to obtain two one-dimensional feature vectors corresponding to the two detection image feature maps.

[0116] like Figure 4 As shown, the two second endoscopic detection images are respectively input into two feature extraction networks as x1 and x2 to obtain two detection image feature maps of the two second endoscopic detection images, and the two detection image feature maps of the two second endoscopic detection images are mapped to one-dimensional vectors to obtain two one-dimensional feature vectors corresponding to the two detection image feature maps.

[0117] (3) The Manhattan distance between the two one-dimensional feature vectors corresponding to the two detection image feature maps of the two second endoscopic detection images is input into the fully connected layer to obtain a fully connected vector.

[0118] like Figure 4 As shown, the Manhattan distance of two one-dimensional feature vectors corresponding to the two detection image feature maps of the two second endoscopic detection images is input into the fully connected layer to obtain a fully connected vector.

[0119] (4) Use the sigmoid function to map the fully connected vector to the range of [0-1] to obtain the first similarity of the detection image group.

[0120] like Figure 4 As shown, the sigmoid function is used to map the fully connected vector to the range of [0-1] to obtain the first similarity P (x1, x2) of the detection image group.

[0121] The larger the Manhattan distance between the two one-dimensional feature vectors corresponding to the two detection image feature maps of the two second endoscopic detection images, the smaller the first similarity between the two second endoscopic detection images.

[0122] 205. Obtain lesion segmentation regions on two second endoscopic detection images obtained by respectively performing lesion segmentation on two second endoscopic detection images of the detection image group.

[0123] In the embodiment of the present application, image segmentation is performed on the second endoscopic detection image to obtain a lesion segmentation area of ​​the second endoscopic detection image.

[0124] 206. Determine a second similarity between the two second endoscopic detection images based on the lesion segmentation regions on the two second endoscopic detection images to obtain a second similarity between the detection image group.

[0125] In the embodiment of the present application, the second similarity of the two second endoscopic detection images is determined based on the lesion segmentation areas on the two second endoscopic detection images to obtain the second similarity of the detection image group, including:

[0126] (1) Obtain the number of lesion segmentation regions on the two second endoscopic detection images of the detection image group respectively.

[0127] like Figure 6 As shown, Figure 6 The circumscribed rectangles of the lesion segmentation areas on the two second endoscopic detection images of the detection image group are schematically shown.

[0128] (2) When the number of lesion segmentation regions on the two second endoscopic detection images of the detection image group is the same, the two second endoscopic detection images of the detection image group are respectively determined as the first target image and the second target image.

[0129] When the number of lesion segmentation regions on the two second endoscopic detection images of the detection image group is different, the second endoscopic detection image with the larger number of lesion segmentation regions among the two second endoscopic detection images of the detection image group is determined as the third target image, the second endoscopic detection image with the larger number of lesion segmentation regions among the two second endoscopic detection images of the detection image group is determined as the second target image, the number of lesion segmentation regions of the second target image is determined as the first preset number, the lesion segmentation regions on the third target image are clustered into the first preset number of first segmentation region clusters based on the centroid positions of the lesion segmentation regions on the third target image, each first segmentation region cluster includes at least one lesion segmentation region. Each first segmentation region cluster is merged into a new lesion segmentation region to obtain the first preset number of new lesion segmentation regions on the third target image, and the third target image containing the first preset number of new lesion segmentation regions is determined as the first target image, so that the number of lesion segmentation regions on the first target image and the second target image is greater. Specifically, each first segmentation area cluster is merged into a new lesion segmentation area to obtain a first preset number of new lesion segmentation areas on the third target image, including: obtaining the centroid of each lesion segmentation area in the first segmentation area cluster, connecting the centroids of each lesion segmentation area in the first segmentation area cluster in sequence to obtain a first polygonal area, and geometrically enlarging the first polygonal area to obtain a second polygonal area, wherein the second polygonal area and the first polygonal area have the same centroid, the second polygonal area is a minimum polygonal area that is geometrically enlarged by the first polygonal area and contains each lesion segmentation area in the first segmentation area cluster, and the second polygonal area is determined as a new lesion segmentation area after the lesion segmentation areas in the first segmentation area cluster are merged to obtain a first preset number of new lesion segmentation areas on the third target image.

[0130] In a specific embodiment, clustering the lesion segmentation regions on the third target image into a first preset number of first segmentation region clusters based on the centroid position of the lesion segmentation regions on the third target image includes: randomly dividing the lesion segmentation regions on the third target image into Q segmentation region sets. Clustering the Q segmentation region sets respectively, clustering each segmentation region set into a first preset number of second segmentation region clusters. Determining each second segmentation region cluster as a target segmentation region cluster, and calculating the region cluster similarity between the target segmentation region cluster and the second segmentation region clusters in the Q segmentation region sets respectively. Merging the second segmentation region clusters with the greatest similarity to the target segmentation region cluster in each segmentation region set to obtain the first segmentation region cluster corresponding to the target segmentation region cluster, obtaining the first segmentation region cluster corresponding to each second segmentation region cluster, and obtaining the first preset number of first segmentation region clusters corresponding to the first preset number of second segmentation region clusters.

[0131] (3) Determine each lesion segmentation region of the first target image as a target lesion segmentation region, and determine a matching lesion segmentation region on the second target image that matches the target lesion segmentation region.

[0132] Specifically, the lesion segmentation region distances between the target lesion segmentation region and each lesion segmentation region on the second target image are calculated respectively, wherein the lesion segmentation region distance is the distance between the centroids of the two lesion segmentation regions. The lesion segmentation region on the second target image having the largest lesion segmentation region distance with the target lesion segmentation region is determined as the matching lesion segmentation region of the target lesion segmentation region.

[0133] (4) The lesion similarity index between the target lesion segmentation region and the matching lesion segmentation region is determined as the lesion similarity index of the target lesion segmentation region, and the lesion similarity index of each lesion segmentation region on the first target image is obtained.

[0134] In a specific embodiment, the minimum bounding rectangle area of ​​the target lesion segmentation area and the minimum bounding rectangle area of ​​the matching lesion segmentation area are obtained, the union of the minimum bounding rectangle area of ​​the target lesion segmentation area and the minimum bounding rectangle area of ​​the matching lesion segmentation area is determined as the comparison area, the fourth similarity between the image in the comparison area of ​​the first target image and the image in the comparison area of ​​the second target image is calculated, and the lesion similarity index is determined based on the fourth similarity. The fourth similarity can be calculated by SSIM (structural similarity metric) and MS-SSIM (multi-scale structural similarity metric).

[0135] The fourth similarity can be determined as the lesion similarity index. Further, the lesion intersection-and-union ratio of the target lesion segmentation region and the matching lesion segmentation region is calculated, and the lesion intersection-and-union ratio is the ratio of the intersection of the target lesion segmentation region and the matching lesion segmentation region to the union of the target lesion segmentation region and the matching lesion segmentation region. The lesion similarity index is determined based on the lesion intersection-and-union ratio and the fourth similarity. Specifically, the average value of the lesion intersection-and-union ratio and the fourth similarity is determined as the lesion similarity index.

[0136] (5) Based on the lesion weight coefficients of the lesion segmented regions on the first target image, weighted summation is performed on the lesion similarity indexes of the lesion segmented regions on the first target image to obtain a second similarity of the detection image group.

[0137] In the embodiment of the present application, the lesion area ratio of each lesion segmentation area on the first target image is obtained, and the lesion area ratio of each lesion segmentation area on the first target image is determined as the lesion weight coefficient of each lesion segmentation area. Based on the lesion weight coefficient of each lesion segmentation area on the first target image, the lesion similarity index of each lesion segmentation area on the first target image is weighted and summed to obtain the second similarity of the detection image group.

[0138] 207 . Determine a third similarity of the detection image groups based on the first similarity of the detection image groups and the second similarity of the detection image groups, and obtain third similarities of the plurality of detection image groups.

[0139] In the embodiment of the present application, an average value of the first similarity and the second similarity of the detection image group is determined as the third similarity of the detection image group.

[0140] 208. Delete at least one second endoscopic detection image in the detection image group having a third similarity greater than a preset similarity from the plurality of second endoscopic detection images to obtain a plurality of third endoscopic detection images.

[0141] In a specific embodiment, a random second endoscopic detection image in each detection image group whose third similarity is greater than a preset similarity is deleted from multiple second endoscopic detection images to obtain multiple seventh endoscopic detection images, and the multiple seventh endoscopic detection images are deduplicated to obtain multiple fourth endoscopic detection images.

[0142] In another specific embodiment, all images in the detection image group whose third similarity is greater than a preset similarity are put into a candidate image set, the occurrence frequency of each second endoscopic detection image in the candidate image set is calculated, and the second endoscopic detection images with a frequency higher than the preset frequency are deleted from the multiple second endoscopic detection images to obtain multiple seventh endoscopic detection images, and the multiple seventh endoscopic detection images are deduplicated to obtain multiple fourth endoscopic detection images.

[0143] 209. Determine a third endoscopic detection image and a corresponding image description text as large model image-text pair data, and obtain a large model image-text pair data set containing a plurality of large model image-text pair data.

[0144] Furthermore, we built a unified annotation template based on professional knowledge, invited multiple experts to annotate the large model image-text pair data set, and conducted consistency evaluation after the annotation by multiple experts. We also conducted expert review on the labeled image-text pairs with poor consistency to form a preliminary image-text pair annotation data set. Based on actual training and clinical application needs, we used methods such as rotation, flipping, cropping, scaling, adjusting brightness and color to enhance the unbalanced data types, and finally formed a high-quality image-text pair data set.

[0145] Furthermore, the plurality of third endoscopic detection images and the corresponding image description texts are split into a training set and a test set according to a preset ratio.

[0146] In an embodiment of the present application, the plurality of third endoscopic detection images and the corresponding image description texts are split into a training set and a test set according to a preset ratio. For example, the preset ratio is 7:3. The ratio of the number of the plurality of third endoscopic detection images in the training set to the number of the plurality of third endoscopic detection images in the test set is 7:3.

[0147] The preset neural network model is fine-tuned based on the training set to obtain a fine-tuned model.

[0148] Among them, the preset neural network model can be a large language model.

[0149] In the embodiment of the present application, a preset neural network model is fine-tuned based on a training set to obtain a fine-tuned model, including:

[0150] (1) Divide the training set into K equal subsets.

[0151] Wherein, K is an integer greater than 2.

[0152] (2) The K subsets are respectively determined as validation sets, and the subsets of the K subsets other than the validation sets are combined to form a combined training set, thereby obtaining K validation sets and corresponding combined training sets.

[0153] (3) Train the preset neural network model based on the combined training set to obtain the combined training model corresponding to the combined training set.

[0154] (4) Verifying the combined training model corresponding to the combined training set based on the verification set corresponding to the combined training set, obtaining the model verification parameters of the combined training model corresponding to the combined training set, and obtaining the model verification parameters of the combined training models corresponding to the multiple combined training sets.

[0155] The model validation parameter may be the sum of the accuracy (Precision) and the recall (Recall) of the combined training model on the validation set.

[0156] (5) The combined training model with the highest model verification parameter is determined as the fine-tuning model.

[0157] The fine-tuned model is tested based on the test set to obtain the model test results.

[0158] In the embodiment of the present application, the fine-tuning model is tested based on the test set to obtain the model test results, including:

[0159] (1) The third endoscopic detection image in the test set is input into the fine-tuning model to obtain the image description prediction text corresponding to the third endoscopic detection image.

[0160] The image description text corresponding to the third endoscopic detection image is: the label is that an ulcer is visible, with a regular shape, shallow depression, a thin white coating on the base, and bleeding.

[0161] The image description prediction text corresponding to the third endoscopic detection image is: It is predicted that an ulcer is visible, with a regular shape, deep depression, a base covered with dirty moss, and visible bleeding.

[0162] (2) Calculating a text similarity index between the image description text corresponding to the third endoscopic detection image and the image description prediction text corresponding to the third endoscopic detection image.

[0163] Specifically, both the image description text and the image description prediction text include a disease feature field and a plurality of lesion feature fields.

[0164] Specifically, calculating the text similarity index between the image description text corresponding to the third endoscopic detection image and the image description prediction text corresponding to the third endoscopic detection image includes:

[0165] (1) Determine a weight coefficient based on whether the disease feature field of the image description text and the disease feature field of the image description prediction text are the same, wherein if the disease feature field of the image description text and the disease feature field of the image description prediction text are the same, the weight coefficient is a first weight; if the disease feature field of the image description text and the disease feature field of the image description prediction text are different, the weight coefficient is a second weight, and the first weight is greater than the second weight.

[0166] For example, the first weight is 1 and the second weight is 0.9.

[0167] (2) Determine the proportion of identical fields describing lesion features in the image description text and the image description prediction text.

[0168] For example, the number of identical fields is 2, the total number is 5, and the percentage of identical fields is 0.4.

[0169] (3) The proportion of the same fields is weighted based on the weight coefficient, and the obtained weighted value is determined as the text similarity index between the image description text and the image description prediction text.

[0170] Specifically, the text similarity index is: the proportion of the same fields * weight coefficient.

[0171] (4) The third endoscopic detection image whose text similarity index is greater than a preset similarity is determined as the third endoscopic detection image that is predicted correctly.

[0172] (5) The percentage of the third endoscopy detection images predicted correctly in the test set is determined as the model test result.

[0173] When the model test result does not meet the preset condition, a plurality of sixth endoscopic detection images and a corresponding plurality of image description texts are obtained.

[0174] The preset condition may be that the proportion of the number of correctly predicted third endoscopic detection images in the test set is less than a preset proportion.

[0175] In a specific embodiment, the plurality of sixth endoscopic detection images and the corresponding plurality of image description texts may be pre-labeled data.

[0176] In another specific embodiment, acquiring a plurality of sixth endoscopic detection images and a plurality of corresponding image description texts further includes:

[0177] (1) Obtaining an endoscopic detection report, which includes a plurality of endoscopic detection image areas and an image description text area.

[0178] (2) The endoscopic detection image area and the image description text area are cropped from the endoscopic detection report to obtain a plurality of first endoscopic detection images and image description text area images within the plurality of endoscopic detection image areas.

[0179] In a specific embodiment, the endoscopic detection report is segmented to obtain multiple endoscopic detection image areas and image description text areas. The endoscopic detection image areas and image description text areas are cropped to obtain multiple first endoscopic detection images and image description text area images in the multiple endoscopic detection image areas.

[0180] (3) Perform text recognition on the image description text area to obtain the overall image description text.

[0181] The overall image description text includes image description texts of multiple first endoscopic detection images. For example, the overall image description text includes image description texts of the esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, etc.

[0182] (4) Determine the image description text corresponding to each first endoscopic detection image from the overall image description text, and obtain each first endoscopic detection image and the corresponding image description text.

[0183] In a specific embodiment, keyword detection is performed on the overall image description text to obtain the texts of each shooting part in the overall image description text. Each shooting part text may be esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum, etc. It is determined whether the number of shooting part texts is the same as the number of first endoscopic detection images. If the number of shooting part texts is the same as the number of first endoscopic detection images, the current arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image are obtained, and each shooting part text is matched with each first endoscopic detection image based on the current arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image to obtain each shooting part text and the corresponding first endoscopic detection image, and the detection condition description text corresponding to each shooting part text is obtained. The detection condition description text is located between two shooting part texts, and the shooting part text and the corresponding detection condition description text are determined as one image description text to obtain each first endoscopic detection image and the corresponding image description text.

[0184] Among them, the further to the left the shooting part text is and the higher the shooting part text is, the earlier the current arrangement order of the shooting part text is; the further to the left the first endoscopic detection image is and the higher the first endoscopic detection image is, the earlier the current arrangement order of the first endoscopic detection image is.

[0185] In a specific embodiment, each shooting part text is matched with each first endoscopic detection image based on the current text arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image, including: matching each shooting part text with each first endoscopic detection image one by one in the order of the current arrangement order to obtain each first endoscopic detection image and the corresponding image description text.

[0186] However, it is possible that the doctor will not place the endoscopic detection images in the corresponding order, and the shooting part text is generally generated by a template, and the user generally will not adjust it. Therefore, according to the order of the current arrangement order, each shooting part text is not matched with each first endoscopic detection image one by one. Therefore, in order to improve the matching success rate, in a specific embodiment, each shooting part text is matched with each first endoscopic detection image based on the current arrangement order of each shooting part text and the current arrangement order of each first endoscopic detection image, including: obtaining the issuing user who issued the endoscopic detection report, obtaining the target correspondence between the image arrangement order and the text arrangement order of the issuing user, and matching each shooting part text with each first endoscopic detection image based on the target correspondence between the image arrangement order and the text arrangement order of the issuing user. Specifically, when the issuing user has issued the endoscopic detection report multiple times in history, the historical correspondence between the historical image arrangement order and the historical text arrangement order is obtained, and the historical correspondence with the highest frequency is determined as the target correspondence. For example, the historical correspondence is: the historical text arrangement order of the texts of each shooting part is esophagus, cardia, gastric fundus, gastric body, gastric angle, gastric antrum; and the corresponding historical image arrangement order is gastric antrum, esophagus, gastric fundus, cardia, gastric body, gastric angle.

[0187] Since it is possible that some endoscopic detection images have no problems, the doctor did not put them up, resulting in the number of shooting part texts being different from the number of the first endoscopic detection images, and they cannot be directly matched at this time. If the number of shooting part texts is different from the number of the first endoscopic detection images, the detection status description text corresponding to each shooting part text is obtained, and the no-abnormal template description text corresponding to each shooting part text is obtained, wherein the no-abnormal template description text is the description text automatically generated by the system to save the doctor's time when there is no abnormality in the shooting part. The number of the first endoscopic detection images is determined as the second preset number, and the text similarity between the no-abnormal template description text corresponding to the shooting part text and the detection status description text corresponding to the shooting part text is calculated, and the shooting part texts of the second preset number that are ranked higher in text similarity from small to large are matched with the first endoscopic detection images of the second preset number to obtain the first endoscopic detection images of the second preset number and the corresponding shooting part texts.

[0188] In an embodiment of the present application, a plurality of endoscopic detection reports are obtained; a first endoscopic detection image and a corresponding image description text are obtained from the detection report, thereby obtaining a plurality of first endoscopic detection images and corresponding image description texts.

[0189] Add multiple sixth endoscopic detection images and corresponding multiple image description texts into the training set to obtain a new training set. Determine a new model test result corresponding to the new fine-tuning model based on the new training set and the test set, until the new model test result meets the preset conditions. Determine the new fine-tuning model whose new model test result meets the preset conditions as the target abnormality detection model. When the endoscopic detection image to be identified is obtained, the endoscopic detection image to be identified is input into the target abnormality detection model for abnormality detection, and obtain the abnormality detection description text corresponding to the endoscopic detection image to be identified.

[0190] In order to facilitate the implementation of the large model image-text pair data set construction method provided in the embodiment of the present application, the embodiment of the present application also provides a large model image-text pair data set construction device based on the large model image-text pair data set construction method. The meaning of the nouns is the same as that in the large model image-text pair data set construction method. For specific implementation details, please refer to the description in the above method embodiment.

[0191] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of a large model image-text pair data set construction device provided in an embodiment of the present application, the large model image-text pair data set construction device may include:

[0192] A first acquisition module 701 is used to acquire a first endoscopic detection image and a corresponding image description text;

[0193] A first determination module 702, configured to determine a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of first endoscopic detection images and corresponding image description texts;

[0194] A second determination module 703 is used to determine a plurality of detection image groups based on the plurality of second endoscopic detection images, wherein the detection image group includes two second endoscopic detection images;

[0195] A calculation module 704 is used to calculate a first similarity between two second endoscope detection images in the detection image group to obtain the first similarity of the detection image group;

[0196] The second acquisition module 705 is used to acquire lesion segmentation areas on two second endoscopic detection images obtained by respectively performing lesion segmentation on the two second endoscopic detection images of the detection image group;

[0197] A third determination module 706 is used to determine a second similarity of the two second endoscopic detection images based on the lesion segmentation regions on the two second endoscopic detection images to obtain a second similarity of the detection image group;

[0198] A fourth determination module 707, configured to determine a third similarity of the detection image groups based on the first similarity of the detection image groups and the second similarity of the detection image groups, to obtain third similarities of the plurality of detection image groups;

[0199] A deleting module 708, configured to delete at least one second endoscopic detection image in the detection image group whose third similarity is greater than a preset similarity from the plurality of second endoscopic detection images, to obtain a plurality of third endoscopic detection images;

[0200] The fifth determination module 709 is used to determine a third endoscopic detection image and the corresponding image description text as large model image-text pair data, and obtain a large model image-text pair data set containing multiple large model image-text pair data.

[0201] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0202] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the method for constructing a large model image-text pair data set provided in this embodiment by calling a computer program stored in the memory.

[0203] Please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0204] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0205] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0206] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0207] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0208] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0209] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the method for constructing a large model image-text pair data set provided by the application, such as:

[0210] Acquire a first endoscopic detection image and a corresponding image description text; determine a plurality of second endoscopic detection images and corresponding image description texts based on the plurality of first endoscopic detection images and the corresponding image description texts; determine a plurality of detection image groups based on the plurality of second endoscopic detection images, wherein the detection image group includes two second endoscopic detection images; calculate a first similarity of the two second endoscopic detection images in the detection image group to obtain a first similarity of the detection image group; obtain lesion segmentation areas on two second endoscopic detection images obtained by respectively performing lesion segmentation on the two second endoscopic detection images in the detection image group; and obtain a lesion segmentation area on the two second endoscopic detection images based on the lesion segmentation areas on the two second endoscopic detection images. The lesion segmentation area determines the second similarity of the two second endoscopic detection images to obtain the second similarity of the detection image group; based on the first similarity of the detection image group and the second similarity of the detection image group, the third similarity of the detection image group is determined to obtain the third similarity of multiple detection image groups; at least one second endoscopic detection image in the detection image group whose third similarity is greater than the preset similarity is deleted from the multiple second endoscopic detection images to obtain multiple third endoscopic detection images; a third endoscopic detection image and the corresponding image description text are determined as large model image-text pair data to obtain a large model image-text pair data set containing multiple large model image-text pair data.

[0211] It should be noted that the electronic device provided in the embodiment of the present application belongs to the same concept as the large model image-text data set construction method in the above embodiment. The specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0212] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on the processor of the electronic device provided in the embodiment of the present application, the processor of the electronic device executes the steps in the method for constructing a large model image-text pair data set provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0213] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the above-mentioned large model image-text pair data set construction method.

[0214] The above is a detailed introduction to a large-model graphic-text data set construction method and device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0215] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A method for constructing a large model image-text pair data set, characterized in that: include: Acquire a first endoscopic detection image and a corresponding image description text; Based on a plurality of first endoscopic detection images and corresponding image description texts, a plurality of second endoscopic detection images and corresponding image description texts are determined; wherein, the image resolution of each of the first endoscopic detection images and the preset resolution threshold of each of the first endoscopic detection images are obtained; the first endoscopic detection images whose image resolution is higher than the corresponding preset resolution threshold are determined as a plurality of fourth endoscopic detection images; the fourth endoscopic detection image is segmented to obtain unclean segmented areas and lesion segmented areas of the fourth endoscopic detection image; the unclean area ratio of the unclean segmented areas and the lesion area ratio of the lesion segmented areas of the fourth endoscopic detection image are determined; based on a preset weight coefficient, the unclean area ratio of the unclean segmented areas and the lesion area ratio of the lesion segmented areas of the fourth endoscopic detection image are weighted and summed to obtain the image cleanliness of the fourth endoscopic detection image, wherein the larger the unclean area ratio, the smaller the image cleanliness; the larger the lesion area ratio, the larger the image cleanliness; the fourth endoscopic detection image whose image cleanliness is higher than the preset cleanliness is determined as a plurality of fifth endoscopic detection images; based on a plurality of fifth endoscopic detection images and corresponding image description texts, a plurality of second endoscopic detection images and corresponding image description texts are determined; determining a plurality of detection image groups based on a plurality of the second endoscopic detection images, wherein the detection image groups include two of the second endoscopic detection images; calculating a first similarity between two of the second endoscope detection images in the detection image group to obtain a first similarity of the detection image group; Acquire lesion segmentation regions on two second endoscopic detection images of the detection image group obtained by respectively performing lesion segmentation on the two second endoscopic detection images; determining a second similarity between the two second endoscopic detection images based on the lesion segmentation regions on the two second endoscopic detection images, and obtaining a second similarity between the detection image group; determining a third similarity of the detection image groups based on the first similarity of the detection image groups and the second similarity of the detection image groups, and obtaining third similarities of a plurality of the detection image groups; Deleting at least one of the second endoscopic detection images in the detection image group whose third similarity is greater than a preset similarity from the plurality of second endoscopic detection images to obtain a plurality of third endoscopic detection images; A third endoscopic detection image and the corresponding image description text are determined as large model image-text pair data, and a large model image-text pair data set containing multiple large model image-text pair data is obtained.

2. The method for constructing a large model image-text pair dataset according to claim 1, characterized in that: The calculating the first similarity of two second endoscopic detection images in the detection image group to obtain the first similarity of the detection image group includes: Performing feature extraction on the two second endoscopic detection images respectively to obtain two detection image feature graphs of the two second endoscopic detection images; Mapping the two detection image feature maps to one-dimensional vectors to obtain two one-dimensional feature vectors corresponding to the two detection image feature maps; Inputting the Manhattan distance of two one-dimensional feature vectors corresponding to the two detection image feature maps of the two second endoscopic detection images into a fully connected layer to obtain a fully connected vector; The fully connected vector is mapped to the range of [0-1] using the sigmoid function to obtain the first similarity of the detection image group, wherein the larger the Manhattan distance between the two one-dimensional feature vectors corresponding to the two detection image feature maps of the two second endoscopic detection images, the smaller the first similarity of the two second endoscopic detection images.

3. The method for constructing a large model image-text pair data set according to claim 2, characterized in that: The determining the second similarity of the two second endoscopic detection images based on the lesion segmentation areas on the two second endoscopic detection images to obtain the second similarity of the detection image group includes: Respectively obtaining the number of lesion segmentation regions on two of the second endoscopic detection images of the detection image group; When the numbers of lesion segmentation regions on the two second endoscopic detection images of the detection image group are the same, determining the two second endoscopic detection images of the detection image group as the first target image and the second target image respectively; Determine each lesion segmentation region of the first target image as a target lesion segmentation region, and determine a matching lesion segmentation region on the second target image that matches the target lesion segmentation region; Determining a lesion similarity index between the target lesion segmentation region and the matching lesion segmentation region as a lesion similarity index of the target lesion segmentation region, and obtaining lesion similarity indexes of each lesion segmentation region on the first target image; The second similarity of the detection image group is obtained by weighted summing of lesion similarity indicators of each lesion segmentation area on the first target image based on the lesion weight coefficient of each lesion segmentation area on the first target image.

4. The method for constructing a large model image-text pair data set according to claim 3, characterized in that: The determining of a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of fifth endoscopic detection images and corresponding image description texts comprises: Obtaining a brightness score, a contrast score, a noise score, an artifact score, a mucosal area clarity score, a mucosal area abnormality score, a lesion area score, and a shooting site score of the fifth endoscopic detection image; Determine the overall feature score of the fifth endoscopic detection image based on the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score and shooting site score of the fifth endoscopic detection image, and obtain the overall feature score of each of the fifth endoscopic detection images; The fifth endoscopic detection image and the corresponding image description text having an overall feature score higher than a preset score are determined as a plurality of second endoscopic detection images and the corresponding image description text.

5. The method for constructing a large model image-text pair data set according to claim 4, characterized in that: The step of obtaining the brightness score, contrast score, noise score, artifact score, mucosal area clarity score, mucosal area abnormality score, lesion area score, and shooting site score of the fifth endoscopic detection image includes: Obtaining a preset normal brightness value and a brightness value of the fifth endoscopic detection image, and determining a brightness score of the fifth endoscopic detection image based on an absolute value of a difference between the preset normal brightness value and the brightness value of the fifth endoscopic detection image, wherein the larger the absolute value of the difference, the smaller the brightness score of the fifth endoscopic detection image; Acquiring a contrast value of the fifth endoscopic detection image, and determining a contrast score of the fifth endoscopic detection image based on the contrast value of the fifth endoscopic detection image, wherein the greater the contrast value, the greater the contrast score of the fifth endoscopic detection image; Acquiring the size of the noise region on the fifth endoscopic detection image, and determining the noise score of the fifth endoscopic detection image based on the size of the noise region, wherein the larger the size of the noise region, the smaller the noise score of the fifth endoscopic detection image; Acquiring the size of the artifact region on the fifth endoscopic detection image, and determining the artifact score of the fifth endoscopic detection image based on the size of the artifact region, wherein the larger the size of the artifact region, the smaller the artifact score of the fifth endoscopic detection image; Obtaining a total area ratio of the lesion region on the fifth endoscopic detection image, and determining a lesion score of the fifth endoscopic detection image based on the total area ratio of the lesion region, wherein the greater the total area ratio of the lesion region, the greater the lesion score of the fifth endoscopic detection image; The shooting part of the fifth endoscopic detection image is obtained, and a shooting part score of the fifth endoscopic detection image is determined based on the shooting part, wherein different shooting parts correspond to different shooting part scores, and wherein the shooting part score when the shooting part is the esophagus, cardia and duodenum is greater than the shooting part score when the shooting part is other parts.

6. A large-model image-text pair data set construction device, characterized in that: include: A first acquisition module, used to acquire a first endoscopic detection image and a corresponding image description text; The first determination module is used to determine a plurality of second endoscopic detection images and corresponding image description texts based on a plurality of first endoscopic detection images and corresponding image description texts; wherein the image resolution of each of the first endoscopic detection images and a preset resolution threshold of each of the first endoscopic detection images are obtained; the first endoscopic detection images having an image resolution higher than the corresponding preset resolution threshold are determined as a plurality of fourth endoscopic detection images; image segmentation is performed on the fourth endoscopic detection images to obtain unclean segmented areas and lesion segmented areas of the fourth endoscopic detection images; and unclean surfaces of the unclean segmented areas of the fourth endoscopic detection images are determined. The fourth endoscopic detection image is characterized by the following aspects: the image cleanliness of the fourth endoscopic detection image is obtained by weighting the unclean area ratio of the unclean segmented area and the lesion area ratio of the lesion segmented area based on the preset weight coefficient, wherein the larger the unclean area ratio, the smaller the image cleanliness; the larger the lesion area ratio, the greater the image cleanliness; the fourth endoscopic detection image having an image cleanliness higher than the preset cleanliness is determined as a plurality of fifth endoscopic detection images; based on the plurality of fifth endoscopic detection images and the corresponding image description texts, a plurality of second endoscopic detection images and the corresponding image description texts are determined; a second determination module, configured to determine a plurality of detection image groups based on a plurality of the second endoscopic detection images, wherein the detection image groups include two of the second endoscopic detection images; a calculation module, used for calculating a first similarity between two of the second endoscopic detection images in the detection image group to obtain the first similarity of the detection image group; A second acquisition module, used for acquiring lesion segmentation areas on two second endoscopic detection images of the detection image group obtained by respectively performing lesion segmentation on the two second endoscopic detection images; A third determination module is used to determine a second similarity between two second endoscopic detection images based on the lesion segmentation areas on the two second endoscopic detection images to obtain a second similarity between the detection image group; a fourth determining module, configured to determine a third similarity of the detection image group based on the first similarity of the detection image group and the second similarity of the detection image group, to obtain third similarities of a plurality of the detection image groups; a deleting module, configured to delete at least one of the second endoscopic detection images in the detection image group whose third similarity is greater than a preset similarity from the plurality of second endoscopic detection images, so as to obtain a plurality of third endoscopic detection images; The fifth determination module is used to determine a third endoscopic detection image and a corresponding image description text as large model image-text pair data, and obtain a large model image-text pair data set containing multiple large model image-text pair data.

7. An electronic device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the method for constructing a large model image-text pair data set as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the steps in the method for constructing a large model image-text pair data set as described in any one of claims 1 to 5.

Images