Endoscopic image searching system and method
By using a deep learning model to calculate the similarity of endoscopic images, the problem of high difficulty in image comparison during endoscopic examinations has been solved, enabling fast and effective image comparison and improving examination efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INVENTEC PUDONG TECH CORPOARTION
- Filing Date
- 2024-11-18
- Publication Date
- 2026-05-19
AI Technical Summary
Existing endoscopic equipment cannot access and compare images from previous examinations in real time during the examination, which makes image comparison difficult, labor-intensive, and difficult to locate, reducing examination efficiency and potentially causing important symptoms to be overlooked.
A deep learning model is used to extract image features. The image similarity is calculated through the feature extraction model, and the source image with the highest similarity is selected as the comparison result to achieve real-time image comparison.
It enables rapid and effective comparison of endoscopic images, improves examination efficiency, and ensures that important symptoms are not overlooked.
Smart Images
Figure CN122066976A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of endoscopic imaging and image comparison technology, and in particular to an endoscopic image search system and method. Background Technology
[0002] Endoscopy is primarily used to examine the health of the digestive tract to detect signs of ulcers, tumors, or other lesions. During the endoscopic examination, the doctor will take photos to document any abnormalities found, recording the results and providing a reference for subsequent diagnosis and treatment planning.
[0003] However, existing endoscopic equipment cannot access and compare images from previous examinations in real time during the procedure. Comparing medical images from different times and locations is crucial for clinical diagnosis and treatment decisions, effectively assisting physicians in their analysis. However, the characteristics of endoscopic images make comparison challenging: on the one hand, a single examination often generates dozens of images, far more than other medical images such as X-rays, increasing the workload for screening and comparison; on the other hand, endoscopic images cannot display the overall outline of organs, making localization difficult and hindering the direct identification of corresponding locations between two images.
[0004] Therefore, image comparison is a burden for physicians both during and after the examination, and there is currently a lack of effective and rapid solutions. This not only reduces the efficiency of endoscopic examinations, but may also lead to the neglect of important symptoms due to limited information flow. Summary of the Invention
[0005] The purpose of this application is to provide an endoscopic image search system and method that can effectively and quickly compare endoscopic images and provide physicians with real-time comparison information.
[0006] In a first aspect, this application provides a method for searching endoscopic images, comprising multiple steps executed by a computing device, the steps including: a feature extraction model generating a target feature value, multiple first feature values, and multiple second feature values based on a target image, multiple first source images, and multiple second source images, wherein the multiple first source images and the multiple second source images each belong to an endoscopic image of a different region of a human organ; calculating multiple first similarities and second similarities between each of the multiple first feature values and the multiple second feature values and the target feature value; calculating a first reference value and a second reference value based on the multiple first similarities and the multiple second similarities; when the first reference value is greater than the second reference value, selecting the first source image corresponding to the maximum value among the multiple first similarities as the search result; when the first reference value is not greater than the second reference value, selecting the second source image corresponding to the maximum value among the multiple second similarities as the search result.
[0007] Secondly, this application provides an endoscopic image search system, including a storage device and a computing device. The storage device is used to store multiple first source images and multiple second source images. Each of the multiple first source images and multiple second source images belongs to an endoscopic image of a different region of a human organ. The computing device is electrically connected to the storage device. The computing device is used to generate a target feature value, multiple first feature values, and multiple second feature values based on a target image, multiple first source images, and multiple second source images using a feature extraction model; calculate multiple first similarities and second similarities between each of the multiple first feature values and multiple second feature values and the target feature value; calculate a first reference value and a second reference value based on the multiple first similarities and multiple second similarities; when the first reference value is greater than the second reference value, select the first source image corresponding to the maximum value among the multiple first similarities; when the first reference value is not greater than the second reference value, select the second source image corresponding to the maximum value among the multiple second similarities as the search result.
[0008] In summary, the endoscopic image search system and method proposed in this application use a deep learning model to extract image features, specifically extracting feature values from source images (e.g., image records from previous examinations) and target images (e.g., real-time endoscopic images, or the latest examination record), which are then used as comparison references in subsequent processes. Furthermore, the target image is compared with each source image to calculate its similarity, and the overall similarity between the target image and a single region is calculated. Finally, the source image with the highest similarity is selected from the region with the highest similarity.
[0009] The above description of the disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principles of this application, and to provide a further explanation of the scope of this patent application. Attached Figure Description
[0010] Figure 1 This is a block architecture diagram of an endoscopic image search system according to an embodiment of this application.
[0011] Figure 2 This is a flowchart of an endoscopic image search method according to an embodiment of this application.
[0012] Figure 3 and Figure 4 These are flowcharts of two implementations of training a feature extraction model.
[0013] Component designation explanation
[0014] 1. Storage device
[0015] 3. Computing device
[0016] Steps S1-S6, T1-T3, U1-U4 Detailed Implementation
[0017] The following detailed description of the features and advantages of this application in the embodiments is sufficient to enable anyone skilled in the art to understand the technical content of this application and implement it accordingly. Furthermore, based on the content disclosed in this specification, the scope of the patent application, and the drawings, anyone skilled in the art can easily understand the relevant objectives and advantages of this application. The following embodiments are for further detailed explanation of the viewpoints of this application, but are not intended to limit the scope of this application in any way.
[0018] Figure 1 This is a block architecture diagram of an endoscopic image searching system according to an embodiment of this application. Figure 1 As shown, the system includes a storage device 1 and a computing device 3.
[0019] Storage device 1 is used to store multiple first-source images, multiple second-source images, and analog or hybrid images generated based on the original images. Each of the multiple first-source images and multiple second-source images belongs to an endoscope image of a different region of a human organ. For example, the multiple first-source images are multiple endoscopic images of the anterior segment of the intestine, and the multiple second-source images are multiple endoscopic images of the posterior segment of the intestine. Specifically, storage device 1 stores all previously captured endoscopic images, each image having a user-given image label indicating which region of a human organ it belongs to. In other words, for the same organ, the endoscopic images in storage device 1 can be divided into multiple images of region 1, multiple images of region 2, ..., and multiple images of region N. For ease of explanation, the following examples use two regions, but this application does not limit the upper limit of the number of regions (N).
[0020] Storage device 1 may be a memory in a computer, a hard disk, or an external storage device connected to a computer. In one embodiment, storage device 1 may be implemented using at least one of the following examples: flash memory, hard disk (HDD), solid-state drive (SSD), dynamic random access memory (DRAM), static random access memory (SRAM), or other non-volatile memory. However, this application is not limited to the above examples.
[0021] like Figure 1 As shown, the computing device 3 is electrically connected to the storage device 1. The computing device 3 is used to train and execute the feature extraction model, calculate the overall similarity between the target image and different regions, calculate the individual similarity between the target image and each source image, and select the region and / or source image with the highest similarity as the search result.
[0022] Specifically, the feature extraction model executed by the computing device 3 generates a target feature value, multiple first feature values, and multiple second feature values based on a target image, multiple first source images, and multiple second source images. The computing device 3 calculates multiple first similarities and second similarities between each of the multiple first feature values and the multiple second feature values and the target feature value. The computing device 3 calculates a first reference value and a second reference value based on the multiple first similarities and the multiple second similarities. When the first reference value is greater than the second reference value, the computing device 3 selects the first source image corresponding to the maximum value among the multiple first similarities as the search result; when the first reference value is not greater than the second reference value, the computing device 3 selects the second source image corresponding to the maximum value among the multiple second similarities as the search result. It should be noted that the above feature values can also be presented in the form of high-dimensional vectors, matrices, tensors, etc. In other words, the feature extraction model converts the image into a high-dimensional data form, and this application does not limit this.
[0023] In one embodiment, the computing device 3 may be at least one of the following: a personal computer, a network server, a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller (MCU), an application processor (AP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-a-chip (SOC), a deep learning accelerator, or any electronic device with similar functions. This application does not limit the hardware type of the computing device 3.
[0024] Figure 2 This is a flowchart of an endoscopic image searching method according to an embodiment of the present application, including steps S1 to S5 executed by the computing device 3.
[0025] In step S1, the computing device 3 executes a feature extraction model, which generates a target feature value, a plurality of first feature values, and a plurality of second feature values based on a target image, a plurality of first source images, and a plurality of second source images.
[0026] In one embodiment, a feature extraction model is trained using self-supervised learning, with contrastive learning as the underlying architecture. This application proposes various image transformation methods to modify the original image to generate multiple new images while preserving the meaning of the image content. In addition to common methods such as rotation and cropping, this application proposes the following two embodiments based on the features of endoscopic images. Figure 3 and Figure 4 These are flowcharts of two implementations of training a feature extraction model.
[0027] Figure 3 The embodiment includes steps T1 to T3. In step T1, the endoscope takes a picture of a human organ based on a first light source to generate a raw image. In step T2, the computing device 3 generates a simulated image based on the raw image. The simulated image is a simulation of the endoscope taking a picture of a human organ based on a second light source. In step T3, the computing device 3 trains a feature extraction model based on the simulated image and the raw image. Specifically, in one embodiment, the first light source is, for example, white light, and the second light source is, for example, a narrow band image (NBI), but this application is not limited thereto. During endoscopic examination, the physician switches between these two light sources, so the raw image may be taken based on one of these two light sources. In order for the feature extraction model to be able to extract features from images of the same location under different light sources, a [specific method is used]. Figure 3 The process involves using image conversion methods based on two light sources to simulate the imaging effect of the same image under a different light source. In one embodiment, the computational unit 3 modifies specific color values in the original image, such as reducing red or increasing green to simulate narrowband images. The specific adjustment parameters must be determined according to the imaging differences between the first and second light sources.
[0028] Figure 4The embodiment includes steps U1 to U4. In step U1, an endoscope is used to photograph a human organ to generate a first original image and a second original image. The first original image may be an endoscope image with abnormal areas, and the second original image may be an endoscope image without abnormal areas. Abnormal areas in the original images may be manually marked or automatically marked using existing image recognition algorithms and models. In step U2, a computing device 3 extracts a block from the first original image based on a location. The location is associated with an abnormal area of the photographed human organ, such as inflammation or intestinal metaplasia. In step U3, the computing device 3 replaces any location in the second original image with this block to generate a hybrid image, where the location in the second original image belongs to a normal area. Therefore, when preparing the dataset, images of all abnormal areas need to be extracted beforehand. During the training phase, a random number is generated for each original image. When the random number is greater than a predetermined threshold, an abnormal area image is randomly selected from the aforementioned abnormal area image library and overlaid on any location of this original image. In step U4, the computing device 3 trains a feature extraction model based on the first original image, the second original image, and the mixed image.
[0029] pass Figure 3 and Figure 4 The two image conversion methods shown can increase the amount of data available to the computing device 3 when training the feature extraction model. The deep learning model learns that the feature values of different enhanced images originating from the same image should be very similar, while differing significantly from the feature values of other images. Furthermore, it can be applied multiple times to different parts of the same human organ. Figure 3 or Figure 4 This is an example of how to amplify the training dataset for different parts of the dataset. Next, please return to... Figure 2 The flowchart.
[0030] In step S2, the computing device 3 calculates multiple first similarities and second similarities between each of the plurality of first feature values and the plurality of second feature values and the target feature value. In other words, the computing device 3 calculates the similarity between each source image and the target image based on the feature values. In one embodiment, the first similarity or the second similarity is Euclidean distance or cosine similarity.
[0031] In step S3, the computing device 3 calculates a first reference value and a second reference value based on the plurality of first similarities and the plurality of second similarities, respectively. In one embodiment, the first reference value and the second reference value are respectively the average or median of the plurality of first similarities and the plurality of second similarities. The first / second reference values correspond to the overall similarity between the target image and the regions to which the first / second source images belong.
[0032] In step S4, the arithmetic unit 3 determines the magnitude of the first reference value and the second reference value. If the first reference value is greater than the second reference value, then proceed to step S5. If the first reference value is not greater than the second reference value, then proceed to step S6.
[0033] In step S5, when the first reference value is greater than the second reference value, the computing device 3 selects the first source image corresponding to the maximum value among the plurality of first similarities as the search result.
[0034] In step S6, when the first reference value is not greater than the second reference value, the computing device 3 selects the second source image corresponding to the maximum value among the plurality of second similarities as the search result.
[0035] The above process encompasses two stages: comparing regional similarity (step S4) and comparing the similarity of individual images (step S5 or step S6). This design aims to prevent highly similar images from being scattered across different regions, thus ensuring regional consistency in the returned images. In one embodiment, after identifying the region most similar to the target image, the computing device 3 selects the region with the highest similarity as the target for return, and returns the image similarity ranking of multiple images within that region, as well as the similarity score (reference value) for each region.
[0036] In summary, the endoscopic image search system and method proposed in this application use a deep learning model to extract image features, specifically extracting feature values from source images (e.g., image records from previous examinations) and target images (e.g., real-time endoscopic images, or the latest examination record), which are then used as comparison references in subsequent processes. Furthermore, the target image is compared with each source image to calculate its similarity, and the overall similarity between the target image and a single region is calculated. Finally, the source image with the highest similarity is selected from the region with the highest similarity.
[0037] While the foregoing embodiments of this application have been disclosed above, they are not intended to limit this application. Any modifications and refinements made without departing from the spirit and scope of this application are within the scope of patent protection of this application. For the scope of protection defined in this application, please refer to the appended patent application claims.
Claims
1. A method for searching endoscopic images, characterized in that, Multiple steps performed by a computing device, the multiple steps including: A feature extraction model generates a target feature value, multiple first feature values, and multiple second feature values based on a target image, multiple first source images, and multiple second source images, respectively, wherein the multiple first source images and the multiple second source images each belong to endoscopic images of different regions of a human organ. Calculate multiple first similarities and second similarities between each of the plurality of first feature values and the plurality of second feature values and the target feature value; Calculate the first reference value and the second reference value based on the plurality of first similarities and the plurality of second similarities, respectively; When the first reference value is greater than the second reference value, one of the plurality of first source images corresponding to the maximum value among the plurality of first similarities is selected as a search result; and When the first reference value is not greater than the second reference value, one of the multiple second source images corresponding to the maximum value among the multiple second similarities is selected as the search result.
2. The method for searching endoscopic images according to claim 1, characterized in that, Also includes: The human organ is photographed using an endoscope based on a first light source to produce a raw image; The computing device generates a simulated image based on the original image, wherein the simulated image simulates the result of the endoscope taking a picture of the human organ based on a second light source. The computing device trains the feature extraction model based on the simulated image and the original image.
3. The method for searching endoscopic images according to claim 1, characterized in that, Also includes: The human organs were photographed using an endoscope to produce a first primary image and a second primary image: The computing device extracts a block of the first original image based on a location; The block is used to replace the position of the second original image to generate a composite image; and The computing device trains the feature extraction model based on the first original image, the second original image, and the mixed image.
4. The method for searching endoscopic images according to claim 1, characterized in that, The first reference value and the second reference value are respectively the average or median of the plurality of first similarities and the plurality of second similarities.
5. The method for searching endoscopic images according to claim 1, characterized in that, The plurality of first similarities and the plurality of second similarities are Euclidean distances or cosine similarities.
6. A system for searching endoscopic images, characterized in that, include: A storage device for storing a plurality of first source images and a plurality of second source images, wherein the plurality of first source images and the plurality of second source images each belong to endoscopic images of different regions of a human organ; and A computing device electrically connected to the storage device is configured to generate a target feature value, a plurality of first feature values, and a plurality of second feature values based on a target image, a plurality of first source images, and a plurality of second source images using a feature extraction model; calculate a plurality of first similarities and second similarities between each of the plurality of first feature values and the plurality of second feature values and the target feature value; calculate a first reference value and a second reference value based on the plurality of first similarities and the plurality of second similarities; when the first reference value is greater than the second reference value, select one of the plurality of first source images corresponding to the maximum value among the plurality of first similarities as a search result; and when the first reference value is not greater than the second reference value, select one of the plurality of second source images corresponding to the maximum value among the plurality of second similarities as the search result.
7. The endoscopic image searching system according to claim 6, characterized in that, The storage device is further configured to store an original image of the human organ captured by an endoscope based on a first light source, and the computing device is further configured to generate a simulated image based on the original image, and train the feature extraction model based on the simulated image and the original image, wherein the simulated image simulates the result of the endoscope capturing the human organ based on a second light source.
8. The endoscopic image searching system according to claim 6, characterized in that, The storage device is further configured to store a first original image and a second original image generated by the endoscope capturing the human organ, and the computing device is further configured to extract a block of the first original image based on a position, replace the position of the second original image with the block to generate a composite image, and train the feature extraction model based on the first original image, the second original image and the composite image.
9. The endoscopic image searching system according to claim 6, characterized in that, The first reference value and the second reference value are respectively the average or median of the plurality of first similarities and the plurality of second similarities.
10. The endoscopic image searching system according to claim 6, characterized in that, The plurality of first similarities or the plurality of second similarities are Euclidean distances or cosine similarities.