Method, device and equipment for automatic positioning of lesion area based on image fusion

Abstract

The present invention discloses a method, device and equipment for automatically locating a lesion area based on image fusion, the method comprising: collecting a confocal microendoscopic image and a white light endoscopic inspection image of a target object during the inspection process, if the two images have the same diagnostic results for the same lesion area, locating the lesion area of ​​the white light endoscopic inspection image, and generating a reference white light endoscopic image; acquiring a real-time biopsy image of the target object through a white light endoscope, calculating the similarity between the real-time biopsy image and the reference white light endoscopic image, if the similarity is greater than a target similarity threshold, determining the lesion area in the real-time biopsy image as a target lesion area. The present invention adopts a method of fusing confocal microendoscopy with white light endoscope to automatically locate the lesion area, thereby matching the real-time acquired white light endoscopic image with the pre-located reference white light endoscopic image before biopsy sampling, accurately locating the target lesion area to be sampled, and improving the efficiency and accuracy of biopsy sampling.

Description

Technical Field

[0001] The present invention relates to the field of endoscopic lesion positioning, and in particular to an automatic lesion area positioning method, device and equipment based on image fusion. Background Art

[0002] Confocal laser endoscopy is a real-time, non-invasive, high-definition endoscopic examination technology. When it is performed synchronously with white light endoscopy, it can well observe various lesions of the digestive tract. During the examination, after the lesion area is preliminarily determined by confocal laser endoscopy and white light endoscopy, it is necessary to extract living tissue from the lesion area for biopsy to obtain the most accurate diagnosis of the nature of the lesion. When the existing technology is used for biopsy sampling, it is impossible to accurately locate the lesion area that has been observed in advance under confocal endoscopy and white light endoscopy, which can easily lead to misdiagnosis and missed diagnosis. Summary of the invention

[0003] The present invention provides a method, device and equipment for automatically locating a lesion area based on image fusion, which solves the above-mentioned technical problems.

[0004] A first aspect of an embodiment of the present invention provides a method for automatically locating a lesion area based on image fusion, comprising the following steps:

[0005] Step 1, collecting a confocal endomicroscope image and a white-light endoscopic examination image of the target object during the examination process, if the confocal endomicroscope image and the white-light endoscopic examination image have the same diagnosis result for the same lesion area, locating the lesion area in the white-light endoscopic examination image, and generating a reference white-light endoscopic image;

[0006] Step 2: Before biopsy sampling, a real-time biopsy image of the target object is obtained through white light endoscopy, and the similarity between the real-time biopsy image and the reference white light endoscopy image is calculated. If the similarity is greater than a target similarity threshold, the lesion area in the real-time biopsy image is determined as the target lesion area.

[0007] A second aspect of an embodiment of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned method for automatically locating a lesion area based on image fusion is implemented.

[0008] A third aspect of an embodiment of the present invention provides an automatic lesion area positioning device based on image fusion, comprising a computer-readable storage medium and a processor, wherein the processor implements the steps of the above-mentioned automatic lesion area positioning method based on image fusion when executing the computer program on the computer-readable storage medium.

[0009] A fourth aspect of the embodiments of the present invention provides an automatic lesion area positioning device based on image fusion, comprising a positioning module and a sampling auxiliary module.

[0010] The positioning module is used to collect confocal endomicroscope images and white-light endoscopic images of the target object during the inspection process. If the confocal endomicroscope image and the white-light endoscopic image have the same diagnosis result for the same lesion area, the lesion area in the white-light endoscopic image is positioned, and a reference white-light endoscopic image is generated;

[0011] The sampling auxiliary module is used to obtain a real-time biopsy image of the target object through a white light endoscope before biopsy sampling, calculate the similarity between the real-time biopsy image and the reference white light endoscope image, and if the similarity is greater than a target similarity threshold, determine the lesion area in the real-time biopsy image as a target lesion area.

[0012] The beneficial effects of the present invention are as follows: the present invention provides a method, device and equipment for automatically locating a lesion area based on image fusion, which automatically locates the lesion area by fusing confocal microscopy with white light endoscopy, thereby matching the real-time acquired white light endoscopy image with the pre-positioned reference white light endoscopy image during the biopsy sampling process, accurately locating the lesion area to be sampled, and improving the efficiency and accuracy of biopsy sampling.

[0013] In order to make the above-mentioned objects, features and advantages of the invention more obvious and easy to understand, the preferred embodiments of the present invention are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 is a schematic diagram of a flow chart of a method for automatically locating a lesion area based on image fusion provided by an embodiment;

[0016] Figure 2 An embodiment provides a flow chart of step 1 in a method for automatically locating a lesion area based on image fusion;

[0017] Figure 3 An embodiment provides a flow chart of step 2 in a method for automatically locating a lesion area based on image fusion;

[0018] Figure 4is a schematic structural diagram of a device for automatically locating a lesion area based on image fusion provided by an embodiment;

[0019] Figure 5 The present invention is a schematic diagram of the structure of an automatic lesion area positioning device based on image fusion provided by an embodiment. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0021] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the scope of protection of the present invention. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order from the module division in the device or the flow chart. Furthermore, the words "first", "second", "third", etc. used in the present invention do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0022] Figure 1 FIG. 1 is a flow chart of a method for automatically locating a lesion region based on image fusion provided in Example 1. Figure 1 As shown, the following steps are included:

[0023] Step 1, collecting a confocal endomicroscope image and a white-light endoscopic examination image of the target object during the examination process, if the confocal endomicroscope image and the white-light endoscopic examination image have the same diagnosis result for the same lesion area, locating the lesion area in the white-light endoscopic examination image, and generating a reference white-light endoscopic image;

[0024] Step 2: Before biopsy sampling, a real-time biopsy image of the target object is obtained through white light endoscopy, and the similarity between the real-time biopsy image and the reference white light endoscopy image is calculated. If the similarity is greater than a target similarity threshold, the lesion area in the real-time biopsy image is determined as the target lesion area, i.e., the area to be sampled.

[0025] The above embodiment provides a method for automatically locating the lesion area based on image fusion, which automatically locates the lesion area by fusing confocal microscopy and white light endoscopy, so that during the biopsy sampling process, the real-time white light endoscopy image is matched with the pre-positioned reference white light endoscopy image, and the lesion area to be sampled is accurately located, thereby improving the efficiency and accuracy of biopsy sampling.

[0026] Each step of the above method is described in detail below using specific embodiments.

[0027] As known to those skilled in the art, confocal endoscopy is a new type of endoscope that can magnify images more than 1000 times, and can magnify and display tissues, cells and subcellular structures while performing white light endoscopy, thereby accurately identifying normal, hyperplastic, neoplastic and cancerous areas.

[0028] It is understandable that the present invention can collect the white light endoscopic image of the current position of the confocal microendoscopic probe in real time, and associate and cache the confocal microendoscopic image and the white light endoscopic image by setting the same identifier, thereby achieving synchronization of the confocal microendoscopic image and the white light endoscopic image. In a specific embodiment, Figure 2 As shown, the above step 1 specifically includes the following steps:

[0029] Step 101, after starting the confocal microendoscopic examination, collect confocal microendoscopic images during the examination at a preset frequency. In one embodiment, a digital video signal / analog video signal in a preset format can be collected first, and after reading the video signal through opencv or other methods, it is transcoded into an image format frame by frame, and then all images are key frame screened to obtain the confocal microendoscopic image.

[0030] Step 102, using the first preset diagnostic model to determine whether there is a lesion area in the confocal microendoscopic image, if so, generate a first diagnostic result, and drive the white light endoscope to automatically save the white light endoscope inspection image at the corresponding moment, and establish an association relationship between the white light endoscope inspection image and the confocal microendoscopic image. The first diagnostic result includes the location of the lesion area identified by the first preset diagnostic model and the most likely lesion type. If the confocal diagnostic result is no diagnostic information or normal (no lesion area), continue to identify and judge the next frame of the image.

[0031] Step 103: Use a second preset diagnostic model to identify the lesion area in the associated white light endoscopy image and generate a second diagnostic result, which includes whether the white light endoscopy image contains a lesion area and the most likely lesion type.

[0032] Step 104, determine whether the lesion area position and lesion type in the first diagnosis result and the second diagnosis result are the same. If so, use a preset area positioning model to locate the lesion area in the white light endoscopy image, and generate and save the positioning information of the lesion area.

[0033] It is understandable that the first preset diagnostic model, the second preset diagnostic model and the preset area positioning model in this embodiment can all adopt trained convolutional neural network models. Specifically, for the first preset diagnostic model, a large number of confocal images are first acquired and annotated by professionals, and the annotation results include images with no diagnostic information, images with lesion areas and images that are normal (no lesion areas), thereby obtaining a confocal image lesion data set. Then, an initial convolutional neural network is established, and the initial convolutional neural network is trained using the confocal image lesion data set, thereby optimizing the structure and parameters of the initial convolutional neural network, and obtaining a confocal image lesion diagnostic model that meets the preset accuracy, that is, the first preset diagnostic model.

[0034] In a preferred embodiment, the model structure of the first preset diagnostic model is optimized and designed, and has higher detection efficiency and more accurate detection results than other classic neural network models. Specifically, the first preset diagnostic model includes a region generation module, a feature extraction module, an attention generation module and a classification module.

[0035] The region generation module is used to crop a plurality of sub-regions of different scales and sizes from different positions of the input image (i.e., the confocal image) by step size control to cover all identifiable regions of the input image;

[0036] The feature extraction module is used to extract features from each sub-region and generate a feature map corresponding to each sub-region;

[0037] The attention generation module is used to extract the attention map of each feature map through the channel attention model and the spatial attention model;

[0038] The classification module is used to fuse the feature map and the attention map corresponding to each sub-region, and generate a classification result, wherein the classification result includes that the input image has no diagnostic information, the input image is normal, or there is a lesion area in the input image.

[0039] It is understandable that in other embodiments, similar neural network construction and training methods are used to obtain the second preset diagnostic model and preset area positioning model mentioned above. The model structure can adopt YOLO, Faster-RCNN, etc., which will not be described in detail here.

[0040] In a preferred embodiment, multiple lesion areas in the white light endoscopy image may be identified by the second preset diagnostic model, and different disease diagnosis results may exist in the multiple lesion areas, which require manual confirmation. Specifically, step 1 also includes a supplementary positioning step, which is:

[0041] If the first diagnosis result and the second diagnosis result have at least one undetermined lesion area of ​​different number, different location and / or different lesion type, a first prompt instruction is generated to prompt manual identification;

[0042] The manual recognition result is received, and at least one pending lesion area in the white light endoscopy image is ignored or supplemented with positioning processing according to the manual recognition result, so as to obtain and mark a more accurate diagnosis result.

[0043] Then, step 2 is performed. During the biopsy sampling process, the real-time biopsy image obtained by white-light endoscopy and the reference white-light endoscopy image where the lesion area has been located are matched for similarity. It can be understood that the similarity calculation selects the color histogram to extract features and the gray-level co-occurrence matrix to extract features. The color histogram extraction feature is more robust to image rotation, scaling, and slight changes in illumination, while the gray-level co-occurrence matrix can extract image texture features. Using the two to compare image similarity can make the algorithm more accurate and more robust.

[0044] In a specific embodiment, Figure 3 As shown, the similarity between the real-time biopsy image and the reference white light endoscopic image is calculated as follows:

[0045] Step 201, obtaining a first color histogram feature of the real-time biopsy image and a second color histogram feature of the reference white-light endoscopic image, and calculating a cosine similarity between the first color histogram feature and the second color histogram feature, which is recorded as a first similarity;

[0046] Step 202, obtaining a first gray level co-occurrence matrix texture feature of the real-time biopsy image and a second gray level co-occurrence matrix texture feature of the reference white light endoscopy image, calculating a Bhattacharyya distance between the first gray level co-occurrence matrix texture feature and the second gray level co-occurrence matrix texture feature, and generating a second similarity;

[0047] Step 203, the first similarity and the second similarity are combined, such as obtaining the weight of the first similarity and the second similarity and performing weighted calculation, so as to generate the target similarity between the real-time biopsy image and the reference white light endoscopy image. When the target similarity is greater than a preset target similarity threshold, the lesion area in the real-time biopsy image is determined as the target lesion area, and a sampling prompt instruction can be generated to remind the operator that the target area to be biopsied has been reached.

[0048] In a preferred embodiment, the biopsy sampling process also includes a path guidance step, specifically:

[0049] generating a path guidance instruction according to the positioning information of the lesion area, wherein the path guidance instruction is used to guide the white light endoscope to extend from the current position to the lesion area;

[0050] During the path guidance process, the similarity between the real-time biopsy image and the reference white-light endoscope image is obtained, and the difference between the similarity and the target similarity threshold is calculated. If the difference is less than the preset deviation value (i.e., the lower limit of the deviation value), a second prompt instruction is generated. By setting the preset deviation value and the second prompt instruction, the operator can be informed that the current position of the white-light endoscope is close to the target sampling area, thereby reminding the operator to reduce the movement amplitude to prevent the white-light endoscope from deviating from the target sampling area again. In other embodiments, when the difference is greater than the second preset deviation value (i.e., the upper limit of the deviation value), a prompt instruction is generated to inform the operator that the target sampling area has been deviated at this time, thereby further improving the sampling efficiency.

[0051] It is understandable that in order to ensure sampling accuracy and subsequent follow-up results, the preferred embodiment can also retain corresponding sampling photos during biopsy sampling, such as before biopsy, during biopsy (i.e., the moment when the biopsy clamp touches the tissue), and after biopsy. Therefore, in a preferred embodiment, step 2 also includes the following steps:

[0052] When the lesion area in the real-time biopsy image is determined to be the target lesion area, a sampling prompt instruction is generated, and a first sampling image, a second sampling image and a third sampling image corresponding to before sampling, during sampling and after sampling are respectively collected at preset times;

[0053] Using a preset regional positioning model to respectively locate the lesion areas of the first sampling image, the second sampling image, and the third sampling image, and generate corresponding positioning results;

[0054] If the positioning result of any sampling image is inconsistent with the lesion area of ​​the reference white light endoscopic image, a third prompt instruction is generated to remind the operator to check and ensure the accuracy of the sampling result.

[0055] In a preferred embodiment, the biopsy sampling method further includes a threshold adjustment step, specifically:

[0056] Generate a target similarity threshold corresponding to the lesion area position and / or the lesion type, and update the target similarity threshold according to the biopsy interval (i.e., the interval between the examination time and the biopsy time) and / or the disease progression status;

[0057] And / or if there is no real-time biopsy image with a similarity greater than the target similarity threshold during the biopsy sampling process, the maximum similarity corresponding to the real-time biopsy image is updated to the target similarity threshold, thereby further improving the efficiency of biopsy sampling.

[0058] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0059] The embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above-mentioned method for automatically locating a lesion area based on image fusion is implemented.

[0060] Figure 4 is a schematic diagram of the structure of a device for automatically locating a lesion area based on image fusion provided by an embodiment. Figure 4 As shown, it includes a positioning module 100 and a sampling auxiliary module 200,

[0061] The positioning module 100 is used to collect confocal endomicroscope images and white-light endoscopic images of the target object during the inspection process. If the confocal endomicroscope image and the white-light endoscopic image have the same diagnosis result for the same lesion area, the lesion area in the white-light endoscopic image is positioned, and a reference white-light endoscopic image is generated;

[0062] The sampling auxiliary module 200 is used to obtain a real-time biopsy image of the target object through a white light endoscope before biopsy sampling, calculate the similarity between the real-time biopsy image and the reference white light endoscope image, and if the similarity is greater than a target similarity threshold, determine the lesion area in the real-time biopsy image as a target lesion area.

[0063] The above embodiment provides an automatic lesion area positioning device based on image fusion, which automatically locates the lesion area by fusing confocal microscopy and white light endoscopy, thereby matching the real-time acquired white light endoscopy image with the pre-positioned reference white light endoscopy image during the biopsy sampling process, accurately locating the lesion area to be sampled, and improving the efficiency and accuracy of biopsy sampling.

[0064] In a preferred embodiment, the positioning module 100 specifically includes:

[0065] An acquisition unit, used for acquiring confocal endomicroscopy images during the inspection process at a preset frequency after the confocal endomicroscopy inspection is started;

[0066] a first diagnostic unit, configured to use a first preset diagnostic model to determine whether there is a lesion area in the confocal microendoscopic image, and if so, to generate a first diagnostic result, and to drive the white light endoscope to automatically save the white light endoscope examination image at a corresponding moment;

[0067] a second diagnosis unit, configured to identify a lesion area in the white light endoscopy image using a second preset diagnosis model and generate a second diagnosis result;

[0068] A positioning unit is used to determine whether the position of the lesion area and the lesion type in the first diagnosis result and the second diagnosis result are the same. If so, a preset area positioning model is used to locate the lesion area in the white light endoscopy image, and the positioning information of the lesion area is generated and saved.

[0069] In a preferred embodiment, the positioning module 100 also includes a supplementary positioning unit, which is used to generate a first prompt instruction when there is at least one pending lesion area of ​​different numbers, different positions and / or different lesion types in the first diagnostic result and the second diagnostic result; and is used to receive manual recognition results and ignore or supplement the positioning of at least one pending lesion area in the white light endoscopy image based on the manual recognition results.

[0070] In a preferred embodiment, the sampling auxiliary module 200 specifically includes:

[0071] a first calculation unit, configured to obtain a first color histogram feature of the real-time biopsy image and a second color histogram feature of the reference white-light endoscopic image, and calculate a cosine similarity between the first color histogram feature and the second color histogram feature, recorded as a first similarity;

[0072] a second calculation unit, configured to obtain a first gray-level co-occurrence matrix texture feature of the real-time biopsy image and a second gray-level co-occurrence matrix texture feature of the reference white-light endoscopic image, calculate a Bhattacharyya distance between the first gray-level co-occurrence matrix texture feature and the second gray-level co-occurrence matrix texture feature, and generate a second similarity;

[0073] A third calculation unit is configured to generate a target similarity between the real-time biopsy image and the reference white light endoscopy image by combining the first similarity and the second similarity.

[0074] In a preferred embodiment, the auxiliary sampling module 200 also includes a path guidance unit, which is used to generate a path guidance instruction according to the positioning information of the lesion area during the biopsy sampling process; and in the path guidance process, obtain the similarity between the real-time biopsy image and the reference white light endoscope image, calculate the difference between the similarity and the target similarity threshold, and generate a second prompt instruction if the difference is less than a preset deviation value; the path guidance instruction is used to guide the white light endoscope to extend from the current position to the lesion area.

[0075] In a preferred embodiment, the auxiliary sampling module 200 also includes a sampling verification unit, which is specifically used to collect the first sampling image, the second sampling image and the third sampling image corresponding to before sampling, during sampling and after sampling at preset times after generating a sampling prompt instruction; and to use a preset area positioning model to locate the lesion areas of the first sampling image, the second sampling image and the third sampling image, respectively, to generate corresponding positioning results; and when the positioning result of any sampling image is inconsistent with the lesion area of ​​the reference white light endoscopic image, a third prompt instruction is generated.

[0076] In a preferred embodiment, the biopsy sampling device further includes a threshold adjustment module, and the threshold adjustment module specifically includes:

[0077] an updating unit, for generating a target similarity threshold corresponding to the lesion area position and / or the lesion type, and updating the target similarity threshold according to the biopsy interval (i.e., the interval between the examination time and the biopsy time) and / or the disease progression status; and for updating the maximum similarity corresponding to the real-time biopsy image to the target similarity threshold when there is no similarity of the real-time biopsy image greater than the target similarity threshold during the biopsy sampling process, thereby further improving the efficiency of biopsy sampling.

[0078] An embodiment of the present invention also provides an automatic lesion area positioning device based on image fusion, including a computer-readable storage medium and a processor. When the processor executes the computer program on the computer-readable storage medium, it implements the steps of the above-mentioned automatic lesion area positioning method based on image fusion.

[0079] Figure 5 FIG. 1 is a schematic diagram of a structure of an automatic lesion area positioning device based on image fusion provided by an embodiment of the present invention. Figure 5As shown, the lesion area automatic positioning device 8 based on image fusion of this embodiment includes: a processor 80, a readable storage medium 81, and a computer program 82 stored in the readable storage medium 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned various method embodiments are implemented, for example Figure 1 Alternatively, when the processor 80 executes the computer program 82, the functions of the modules in the above-mentioned device embodiments are realized, for example Figure 4 Functionality of the modules shown.

[0080] Exemplarily, the computer program 82 may be divided into one or more modules, which are stored in the readable storage medium 81 and executed by the processor 80 to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 82 in the image fusion-based lesion area automatic positioning device 8.

[0081] The image fusion-based lesion area automatic positioning device 8 may include, but is not limited to, a processor 80 and a readable storage medium 81. Those skilled in the art will understand that Figure 5 It is only an example of the automatic lesion area positioning device 8 based on image fusion, and does not constitute a limitation of the automatic lesion area positioning device 8 based on image fusion. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the automatic lesion area positioning device based on image fusion may also include a power management module, an operation processing module, input and output devices, a network access device, a bus, etc.

[0082] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0083] The readable storage medium 81 may be an internal storage unit of the image fusion-based lesion area automatic positioning device 8, such as a hard disk or memory of the image fusion-based lesion area automatic positioning device 8. The readable storage medium 81 may also be an external storage device of the image fusion-based lesion area automatic positioning device 8, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the image fusion-based lesion area automatic positioning device 8. Further, the readable storage medium 81 may also include both the internal storage unit of the image fusion-based lesion area automatic positioning device 8 and an external storage device. The readable storage medium 81 is used to store the computer program and other programs and data required by the image fusion-based lesion area automatic positioning device. The readable storage medium 81 may also be used to temporarily store data that has been output or is to be output.

[0084] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0085] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0086] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0087] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.

Claims

1. A method for automatically locating a lesion area based on image fusion, characterized in that: The following steps are involved: Step 1: Acquire a confocal endomicroscope image and a white-light endoscopic image of the target object during the examination process. If the confocal endomicroscope image and the white-light endoscopic image have the same diagnostic results for the same lesion area, locate the lesion area in the white-light endoscopic image and generate a reference white-light endoscopic image. Step 2: Before biopsy sampling, a real-time biopsy image of the target object is obtained through white-light endoscopy, and the similarity between the real-time biopsy image and the reference white-light endoscopy image is calculated. If the similarity is greater than a target similarity threshold, the lesion area in the real-time biopsy image is determined to be the target lesion area.

2. The method for automatically locating a lesion area based on image fusion according to claim 1, characterized in that: The step 1 is specifically as follows: Step 101 , after starting a confocal endomicroscopy examination, collecting confocal endomicroscopy images during the examination process at a preset frequency; Step 102: using a first preset diagnostic model to determine whether a lesion area exists in the confocal endomicroscopy image; if so, generating a first diagnostic result and driving the white light endoscope to automatically save the white light endoscopy image at the corresponding moment; Step 103: using a second preset diagnostic model to identify the lesion area in the white light endoscopy image and generate a second diagnostic result; Step 104, determine whether the lesion area position and lesion type in the first diagnostic result and the second diagnostic result are the same. If so, use a preset area positioning model to locate the lesion area in the white light endoscopy image, and generate and save the positioning information of the lesion area.

3. The method for automatically locating a lesion area based on image fusion according to claim 2, characterized in that: The first preset diagnostic model includes a region generation module, a feature extraction module, an attention generation module and a classification module. The region generation module is used to crop multiple sub-regions of different scales and sizes from different positions of the input image through step size control to cover all recognizable areas of the input image; The feature extraction module is used to extract features from each sub-region and generate a feature map corresponding to each sub-region; The attention generation module is used to extract the attention map of each feature map through the channel attention model and the spatial attention model; The classification module is used to fuse the feature map and attention map corresponding to each sub-region and generate a classification result, which includes that the input image has no diagnostic information, the input image is normal, or there is a diseased area in the input image.

4. The method for automatically locating a lesion area based on image fusion according to claim 2, characterized in that: The step 1 also includes a supplementary positioning step, specifically: If the first diagnosis result and the second diagnosis result contain at least one undetermined lesion area with different numbers, different locations and / or different lesion types, a first prompt instruction is generated; A manual recognition result is received, and according to the manual recognition result, at least one undetermined lesion area in the white light endoscopy image is ignored or supplemented with a positioning process.

5. The method for automatically locating a lesion area based on image fusion according to any one of claims 1 to 4, characterized in that: The similarity between the real-time biopsy image and the reference white-light endoscopic image is calculated as follows: Step 201: obtaining a first color histogram feature of the real-time biopsy image and a second color histogram feature of the reference white-light endoscopic image, and calculating a cosine similarity between the first color histogram feature and the second color histogram feature, which is recorded as a first similarity; Step 202: obtaining a first gray-level co-occurrence matrix texture feature of the real-time biopsy image and a second gray-level co-occurrence matrix texture feature of the reference white-light endoscopic image, calculating a Bhattacharyya distance between the first gray-level co-occurrence matrix texture feature and the second gray-level co-occurrence matrix texture feature, and generating a second similarity; Step 203 : The first similarity and the second similarity are combined to generate a target similarity between the real-time biopsy image and the reference white-light endoscopic image.

6. The method for automatically locating a lesion area based on image fusion according to claim 5, characterized in that: The step 2 also includes a path guidance step, specifically: generating a path guidance instruction according to the positioning information of the lesion area, wherein the path guidance instruction is used to guide the white light endoscope to extend from a current position to the lesion area; During the path guidance process, the similarity between the real-time biopsy image and the reference white-light endoscopic image is obtained, and the difference between the similarity and the target similarity threshold is calculated. If the difference is less than a preset deviation value, a second prompt instruction is generated.

7. The method for automatically locating a lesion area based on image fusion according to claim 5, characterized in that: The step 2 further comprises: When the lesion area in the real-time biopsy image is determined to be the target lesion area, a sampling prompt instruction is generated, and a first sampling image, a second sampling image, and a third sampling image corresponding to before sampling, during sampling, and after sampling are respectively collected at preset times; Using a preset region positioning model to respectively locate the lesion regions of the first sampling image, the second sampling image, and the third sampling image, and generate corresponding positioning results; If the positioning result of any sample image is inconsistent with the lesion area of ​​the reference white-light endoscopic image, a third prompt instruction is generated.

8. An automatic lesion area positioning device based on image fusion, characterized in that: Including positioning module and sampling auxiliary module, The positioning module is used to collect confocal endomicroscopy images and white-light endoscopic images of the target object during the examination process, and if the confocal endomicroscopy image and the white-light endoscopic image have the same diagnosis result for the same lesion area, the lesion area in the white-light endoscopic image is located, and a reference white-light endoscopic image is generated; The sampling auxiliary module is used to obtain a real-time biopsy image of the target object through white light endoscopy before biopsy sampling, calculate the similarity between the real-time biopsy image and the reference white light endoscopy image, and if the similarity is greater than a target similarity threshold, determine the lesion area in the real-time biopsy image as the target lesion area.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for automatically locating a lesion area based on image fusion according to any one of claims 1 to 7 is implemented.

10. An automatic lesion area positioning device based on image fusion, comprising a computer-readable storage medium and a processor, characterized in that: When the processor executes the computer program on the computer-readable storage medium, the processor implements the steps of the method for automatic lesion area positioning based on image fusion according to any one of claims 1 to 7.

Images