A lesion boundary joint positioning method, device, medium and equipment

Abstract

The present application relates to the field of confocal endomicroscopy, and discloses a lesion boundary joint positioning method, device, medium and equipment. First, a white light endoscope is used for macroscopic screening to obtain the best observation angle and the best working distance of the lesion. Then, a target scanning path of the confocal endomicroscopy probe under the best observation angle and the best working distance is established. Subsequently, the confocal endomicroscopy probe is guided to perform microscopic scanning along the target scanning path to obtain micro-mutation point information. Finally, the macroscopic lesion boundary and the micro-mutation point are fused by an algorithm to realize rapid and accurate positioning of the suspicious lesion boundary, improve the accuracy of lesion boundary identification, and thus improve the surgical resection effect and the prognosis of patients.

Description

Technical Field

[0001] This invention relates to the field of microendoscopy, and more particularly to a method, apparatus, medium, and device for joint localization of lesion boundaries. Background Technology

[0002] White light endoscopy is currently the most commonly used and fundamental endoscopic technique for diagnosing digestive system diseases. It uses a light source at the tip of the endoscope to emit white light, illuminating the digestive system area and allowing for direct observation of its morphology and lesions using optical imaging principles, providing a real-time macroscopic view for clinical diagnosis. Confocal microscopy is an advanced endoscopic technique that uses a fiber optic bundle to insert a miniature probe deep into the body for cellular-level microscopic imaging of tissues. It can be used in conjunction with conventional white light endoscopy; the confocal microscopy probe is inserted into the body through the biopsy channel of the white light endoscope to observe the lesion. While confocal microscopy can provide cellular-level pathological imaging, its field of view is small and scanning is time-consuming. Relying solely on it for exploring the boundaries of large lesions is inefficient and cannot quickly and accurately locate lesion boundaries. Therefore, there is an urgent clinical need for a combined localization method based on the macroscopic view of white light endoscopy and the microscopic view of confocal microscopy, thereby achieving efficient, accurate, and automated identification of lesion boundaries. Summary of the Invention

[0003] This invention provides a method, apparatus, medium, and device for joint localization of lesion boundaries, which solves the technical problems mentioned above.

[0004] A first aspect of this invention provides a method for joint localization of lesion boundaries, comprising the following steps:

[0005] Step 1: Adjust the current working distance and / or current observation angle of the white light endoscope to the target state;

[0006] Step 2: Acquire white light endoscopic images and locate the initial lesion boundary of suspicious lesions based on the white light endoscopic images;

[0007] Step 3: Identify the geometric center point of the initial lesion boundary, obtain the longest and shortest center line directions of the initial lesion boundary based on the geometric center point, and establish the target scanning path of the confocal endoscope probe.

[0008] Step 4: Position the center of the probe of the white light endoscope to the geometric center point, and insert the confocal microendoscopic probe along the biopsy channel of the white light endoscope. Control the confocal microendoscopic probe to scan sequentially along the target scanning path to obtain at least one abrupt change point on the target scanning path.

[0009] Step 5: Based on the at least one mutation point, the initial lesion boundary is corrected to generate the target lesion boundary of the suspected lesion.

[0010] A second aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for joint localization of lesion boundaries.

[0011] A third aspect of the present invention provides a lesion boundary joint localization device, including a computer-readable storage medium and a processor, wherein the processor executes a computer program on the computer-readable storage medium to implement the steps of the lesion boundary joint localization method described above.

[0012] A fourth aspect of this invention provides a lesion boundary joint localization device, including an adjustment module, an initial segmentation module, a path establishment module, a mutation point acquisition module, and a correction module.

[0013] The adjustment module is used to adjust the current working distance and / or current observation angle of the white light endoscope to the target state;

[0014] The initial segmentation module is used to acquire white light endoscope images and locate the initial lesion boundary of suspicious lesions based on the white light endoscope images;

[0015] The path establishment module is used to identify the geometric center point of the initial lesion boundary, obtain the longest centerline direction and the shortest centerline direction of the initial lesion boundary based on the geometric center point, and establish the target scanning path of the confocal endoscope probe.

[0016] The mutation point acquisition module is used to position the probe center of the white light endoscope to the geometric center point, and extend the confocal microendoscopic probe along the biopsy channel of the white light endoscope, and control the confocal microendoscopic probe to scan sequentially along the target scanning path to acquire at least one mutation point on the target scanning path.

[0017] The correction module is used to correct the initial lesion boundary based on the at least one mutation point, and generate the target lesion boundary of the suspected lesion.

[0018] The beneficial effects of this invention are as follows: This invention provides a method, device, medium, and equipment for joint localization of lesion boundaries. First, a macroscopic screening is performed using a white light endoscope to obtain the optimal observation angle and working distance of the lesion. Then, a confocal microscopic endoscope probe is guided to perform microscopic scanning along a critical path to obtain information on microscopic mutation points. Finally, an algorithm is used to fuse the macroscopic lesion boundary with the microscopic mutation points, thereby achieving rapid and accurate localization of suspicious lesion boundaries, improving the accuracy of lesion boundary identification, thereby improving surgical resection results and enhancing patient prognosis.

[0019] To make the above-mentioned objects, features and advantages of the invention more apparent and understandable, preferred embodiments of the invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1a This is a product diagram illustrating the combined use of a cholangioscope and a confocal microendoscopy in the combined positioning method of this invention.

[0022] Figure 1b This is a magnified view of the fiber optic probe of a confocal endoscope passing through the working channel of the cholangioscope.

[0023] Figure 2 It is to utilize Figure 1a The schematic diagram of the principle of joint localization of lesion boundaries using the product;

[0024] Figure 3 It is to utilize Figure 1a A flowchart illustrating the method for joint localization of lesion boundaries using the product;

[0025] Figure 4 yes Figure 3 The provided method includes a schematic diagram of the movement of the co-focusing microendoscopic probe;

[0026] Figure 5 It is to utilize Figure 1a A schematic diagram of the structure of the product's lesion boundary joint positioning device;

[0027] Figure 6 It is to utilize Figure 1a A schematic diagram of the structure of the product's lesion boundary joint positioning device. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0029] It should be noted that, unless otherwise specified, the various features in the embodiments of this invention can be combined with each other, all of which are within the protection scope of this invention. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this invention do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0030] Figure 1a This is a product diagram showing the combined use of a choledochoscope and a confocal endoscope in the lesion boundary joint localization method of the present invention. Both are products developed by our company and are used for diagnosing lesions in the biliary and pancreatic regions. The choledochoscope 3 is a white light endoscope used for macroscopic exploration of lesions, while the confocal endoscope is a confocal endoscope used for microscopic exploration of lesion boundary tissues. Figure 1b This is a magnified view of the fiber optic probe 1 of a confocal endoscope passing through the working channel 4 of the cholangioscope, as shown in the image. Figure 1a , Figure 1b As shown, when used in combination, the distal end 8 of the fiber optic probe 1 (i.e. the other end opposite to the fiber optic probe handle 2) enters through the working channel 4 of the cholangioscope 3 until it exits through the distal end 7 of the cholangioscope 3, and the direction of the distal end detection is adjusted by the first adjustment knob 5 and the second adjustment knob 6 of the cholangioscope 3.

[0031] Figure 2 This is a schematic diagram illustrating the principle of using the product shown in Figure 1 for joint localization of lesion boundaries. The left image in the figure is a macroscopic view of a cholangioscopy examining a cancerous bile duct. Because the working distance of the cholangioscopy ranges from a few centimeters to tens of centimeters, the boundary between normal and malignant tissue (tissue diameter in the micrometer range) cannot be clearly distinguished from this image. In this case, a confocal endoscope is used to examine the tissue around the boundary. Since the working distance of the confocal microendoscopy is in the micrometer range, the image of the normal bile duct shown in the upper right corner represents normal tissue, while the image of the malignant bile duct shown in the lower right corner represents malignant tissue. The boundary between the normal and malignant bile ducts is the lesion boundary (shown by the dotted line in the figure).

[0032] To achieve efficient, accurate, and automated identification of lesion boundaries, this invention provides a flowchart illustrating a method for joint localization of lesion boundaries. (See attached diagram.) Figure 3 As shown, it includes the following steps:

[0033] Step 1: Adjust the current working distance and / or current observation angle of the white light endoscope to the target state;

[0034] Step 2: Acquire white light endoscopic images and locate the initial lesion boundary of suspicious lesions based on the white light endoscopic images;

[0035] Step 3: Identify the geometric center point of the initial lesion boundary, obtain the longest and shortest center line directions of the initial lesion boundary based on the geometric center point, and establish the target scanning path of the confocal endoscope probe.

[0036] Step 4: Position the center of the probe of the white light endoscope to the geometric center point, and insert the confocal microendoscopic probe along the biopsy channel of the white light endoscope. Control the confocal microendoscopic probe to scan sequentially along the target scanning path to obtain at least one abrupt change point on the target scanning path.

[0037] Step 5: Based on the at least one mutation point, the initial lesion boundary is corrected to generate the target lesion boundary of the suspected lesion.

[0038] The above embodiments provide a method for joint localization of lesion boundaries. First, a macroscopic screening is performed using a white light endoscope to obtain the optimal observation angle and working distance of the lesion. Then, the confocal microscopic endoscope probe is guided to perform microscopic scanning along a key path to obtain information on microscopic mutation points. Finally, the macroscopic lesion boundary and the microscopic mutation points are fused using an algorithm to achieve rapid and accurate localization of suspicious lesion boundaries, improve the accuracy of lesion boundary identification, thereby improving the surgical resection effect and improving the patient's prognosis.

[0039] The following specific embodiments will be used to describe each step of the above method in detail.

[0040] For example, in one embodiment, step 1 of adjusting the working distance of the white light endoscope specifically includes the following steps:

[0041] Step 101: After the white light endoscope probe enters the area to be detected in the human body, control the current working distance of the white light endoscope to the initial working distance corresponding to the current scene, so as to observe the overall state of the tissue at the initial working distance.

[0042] Step 102: Continuously observe the area to be detected and obtain at least one suspicious lesion marking result in the white light endoscope image at the initial working distance. Typically, the initial working distance is set relatively large, thus performing a panoramic scan of the tissue area to be detected from a wide field of view, ensuring no blind spots are missed, and quickly locating and marking suspicious lesions with abnormal color or shape, such as marking tissues that are reddish, whitish, have abnormal vascular structures, or have lost surface structures. For example, physicians can manually mark suspicious lesion locations during white light endoscope observation, such as manually marking the boundaries of suspicious lesion areas, or the white light endoscope image corresponding to the initial working distance can be automatically acquired, and a pre-trained neural network model can be used to identify and mark suspicious lesions in the white light endoscope image.

[0043] Step 103: Invoke the preset working distance adjustment scheme to perform local detailed examination of the marked suspicious lesions at multiple working distances, generating local images of the suspicious lesions corresponding to different working distances. For example, the initial working distance can be gradually reduced, enlarged, and then reduced again at the suspicious lesion site to perform local detailed examination.

[0044] Step 104: Perform local feature recognition on the local image of the suspected lesion to generate a target working distance that meets preset feature conditions, and adjust the current working distance of the white light endoscope to the target working distance. In this way, by judging whether the local features of the lesion, such as the clarity of the lesion and surrounding tissues, meet the preset conditions, the optimal target working distance is obtained, and the endoscope is stopped and locked at the target working distance to ensure that the lesion and surrounding tissues are clearly visible.

[0045] Once a suspicious lesion is located, the target working distance is kept constant, meaning the endoscope body remains stationary, neither advancing nor retracting. Then, the angle of the white light endoscope probe is adjusted at the suspicious lesion to change the observation direction, allowing the lesion to be viewed from multiple angles (up, down, left, and right). By comparing these observations, the operator can find the target observation angle with the largest area of ​​the suspicious lesion.

[0046] In other embodiments, to obtain a more accurate target observation angle, an image recognition method can be used. This involves keeping the target working distance of the endoscope constant while changing the observation angle of the endoscope probe. The observation angle corresponding to the largest suspicious lesion area in the endoscope image is then used as the target observation angle. Specifically, the curved part at the endoscope tip is rotated using a bend angle button to capture images of the same suspicious lesion from different angles, acquiring a series of images, for example, one image every 10-15°. The boundaries of the suspicious lesion area in the endoscope image—the dividing line between abnormal and normal points—are then segmented in real time. The pixel percentage of the suspicious lesion area under different observation angles is calculated, thus assisting the doctor in determining the observation angle with the largest suspicious lesion area. In a preferred embodiment, during image capture, the integrity of the suspicious lesion is identified after each angle adjustment, and incomplete images of the suspicious lesion are deleted, further improving recognition efficiency.

[0047] For example, in one specific embodiment, the observation angle corresponding to the largest area of ​​suspicious lesions in the white light endoscopic image is taken as the target observation angle, specifically:

[0048] Step 105: Obtain the white light endoscope image corresponding to each observation angle at the target working distance, and perform preprocessing, including filtering, distortion correction, etc., and retain only the effective circular field of view.

[0049] Step 106: Identify suspicious lesion regions in the preprocessed white light endoscopy image using a preset neural network model, and segment the boundaries of the suspicious lesion regions in real time to generate the number of lesion pixels corresponding to the suspicious lesion regions. For example, a binary mask can be set on the segmentation result, where white pixels (value 1) represent lesions and black pixels (value 0) represent the background, thereby calculating the number of lesion pixels and the total number of image pixels.

[0050] Step 107: Calculate the percentage of lesion pixels in each white light endoscope image under different observation angles, and take the observation angle with the largest percentage of lesion pixels as the target observation angle. The percentage of lesion pixels = number of lesion pixels / total number of image pixels.

[0051] In the above embodiments, since the target working distance is fixed, the relative physical distance between the endoscope probe and the mucosal surface remains constant. Therefore, the actual physical size represented by each pixel in the white light endoscope image is essentially consistent across images taken from different angles. Consequently, the number of pixels in the lesion area directly reflects its lesion area from the current viewing angle. This embodiment can determine which angle makes the lesion appear largest by comparing changes in this number of pixels, thereby finding the best observation angle.

[0052] For example, in a preferred embodiment, a white light working distance optimization step is also included, specifically:

[0053] Keep the target observation angle of the white light endoscope unchanged, and increase or decrease the current working distance of the white light endoscope, i.e. the target working distance, according to the preset adjustment range;

[0054] In response to user commands, the working distance corresponding to the target proportion of the field of view is used as the optimized working distance.

[0055] In the above embodiments, the target observation angle of the white light endoscope is kept unchanged, that is, the observation direction is locked, so that the white light endoscope probe observes in the locked direction. At the same time, the current working distance of the white light endoscope is adjusted again. That is, the endoscope is slowly advanced or retracted according to the preset adjustment range at the target working distance to ensure that all suspicious lesions are placed in the center of the white light endoscope's field of view and are of appropriate size, such as occupying 1 / 3 to 2 / 3 of the field of view. When this is the case, the doctor controls the corresponding button to generate user instructions and generate an optimized working distance to avoid distortion caused by being too close or blurry details caused by being too far away.

[0056] For example, in a preferred embodiment, the target scanning path of the confocal endoscope probe is established as follows:

[0057] Determine whether the initial lesion boundary is elliptical, i.e., whether it is a regular lesion. If so, use the minimum bounding rectangle method to determine the direction of the longest centerline (i.e., the major axis direction) and the direction of the shortest centerline (i.e., the minor axis direction) of the initial lesion boundary. If not, establish several scan lines at different angles starting from the geometric center point and determine the direction of the longest centerline and the direction of the shortest centerline.

[0058] Specifically, for elliptical or approximately elliptical suspicious lesion areas, a minimum bounding rectangle can be established to encompass the entire boundary region of the suspicious lesion. The direction of the long side of the minimum bounding rectangle is the major axis direction, and the direction of the short side is the minor axis direction. For irregular suspicious lesion areas, starting from the geometric center point, n scan lines can be generated by rotating n times at a preset rotation angle. Each scan line will intersect the initial lesion boundary at two points, namely P1 and P2. The distance between P1 and P2 at each angle is calculated and sorted to obtain several larger distance values ​​greater than a first preset threshold and several smaller distance values ​​less than a second preset threshold. This determines several longest and shortest centerline directions, thereby establishing a target scan path that includes the longest and shortest centerline directions.

[0059] In a preferred embodiment, target lesion points with a classification probability greater than a preset value on the initial lesion boundary can also be obtained, i.e., pixels with a high lesion probability. The target lesion points are connected to the geometric center point to generate several preferred scanning directions and add them to the target scanning path of the confocal microendoscopic probe, thereby further increasing the accuracy and effectiveness of the confocal microendoscopic probe scanning process and ensuring that as many effective mutation points as possible can be scanned.

[0060] Then, the white light endoscope can be locked at the target observation angle and target / optimized working distance. Simultaneously, the probe center of the white light endoscope is positioned at the geometric center point of the initial lesion boundary. The confocal microendoscopic probe is then inserted through the biopsy channel of the white light endoscope, and the probe is controlled to advance / retreat linearly along the target scanning path. That is, the confocal microendoscopic probe is probed along the longest centerline direction, the shortest centerline direction, and the preferred scanning direction, respectively. Figure 4 As shown, the probe continuously acquires a preset number of confocal endoscopic images each time it advances a preset distance. Then, through doctor observation or a preset neural network model, at least one mutation point is identified on the scanning path—a point where characteristics such as cell morphology, glandular structure, and blood vessel density change from normal to abnormal. This mutation point is then used to accurately locate the lesion's extent. Confocal microscopy is a point-scanning imaging technique with an extremely small field of view but extremely high resolution. Directly scanning the entire lesion would be extremely time-consuming. In this embodiment, a white light endoscope is first used to determine the macroscopic outline of the lesion, and then a confocal microscopy is used to sample mutation points along the main path, thereby obtaining accurate boundaries of suspected lesions more quickly.

[0061] In a further preferred embodiment, the initial lesion boundary is corrected based on the at least one mutation point, specifically as follows:

[0062] Step 501: Fit the initial lesion boundary to generate an initial lesion curve;

[0063] Step 502: Transform the at least one mutation point into a white light endoscope image, and obtain the matching point on the initial lesion curve that is closest to each mutation point to establish a point-to-point relationship;

[0064] Step 503: Calculate the displacement vector of each mutation point according to the point-to-point relationship, and establish a boundary correction model based on the displacement vector. The boundary correction model is used to correct the initial lesion boundary, so that the initial lesion boundary fits into these mutation points. This utilizes the global information of white light endoscopy and incorporates the local precision of confocal endoscopy, further improving the efficiency and accuracy of lesion boundary identification.

[0065] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0066] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for joint localization of lesion boundaries.

[0067] Figure 5 This is a schematic diagram of the lesion boundary joint localization device provided by the present invention, including an adjustment module 100, an initial segmentation module 200, a path establishment module 300, a mutation point acquisition module 400, and a correction module 500.

[0068] The adjustment module 100 is used to adjust the current working distance and / or current observation angle of the white light endoscope to the target state;

[0069] The initial segmentation module 200 is used to acquire white light endoscope images and locate the initial lesion boundary of suspicious lesions based on the white light endoscope images;

[0070] The path establishment module 300 is used to identify the geometric center point of the initial lesion boundary, obtain the longest center line direction and the shortest center line direction of the initial lesion boundary based on the geometric center point, and establish the target scanning path of the confocal endoscope probe.

[0071] The mutation point acquisition module 400 is used to position the probe center of the white light endoscope to the geometric center point, and extend the confocal microendoscopic probe along the biopsy channel of the white light endoscope, and control the confocal microendoscopic probe to scan sequentially along the target scanning path to acquire at least one mutation point on the target scanning path.

[0072] The correction module 500 is used to correct the initial lesion boundary based on the at least one mutation point, and generate the target lesion boundary of the suspected lesion.

[0073] The above embodiments provide a lesion boundary joint localization device. First, a macroscopic screening is performed using a white light endoscope to obtain the optimal observation angle and working distance of the lesion. Then, the confocal microscopic endoscope probe is guided to perform microscopic scanning along a key path to obtain information on microscopic mutation points. Finally, the macroscopic lesion boundary and the microscopic mutation points are fused by an algorithm to achieve rapid and accurate localization of suspicious lesion boundaries, improve the accuracy of lesion boundary identification, thereby improving the surgical resection effect and improving the patient's prognosis.

[0074] In a preferred embodiment, the adjustment module 100 includes a working distance adjustment unit, specifically comprising:

[0075] The first adjustment unit is used to control the current working distance of the white light endoscope to the initial working distance corresponding to the current scene after the white light endoscope probe enters the detection area of ​​the human body.

[0076] The first acquisition unit is used to continuously observe the area to be detected and acquire at least one suspicious lesion marker result in the white light endoscope image at the initial working distance;

[0077] The second adjustment unit calls the preset working distance adjustment scheme to conduct local detailed examination of the marked suspicious lesions at multiple working distances, and generates local images of suspicious lesions corresponding to different working distances.

[0078] The first acquisition unit is used to perform local feature recognition on the local image of the suspected lesion, generate a target working distance that meets the preset feature conditions, and adjust the current working distance of the white light endoscope to the target working distance.

[0079] In a preferred embodiment, the adjustment module 100 includes an observation angle adjustment unit, which is used to keep the target working distance of the white light endoscope constant, change the observation angle of the white light endoscope probe, and take the observation angle corresponding to the largest suspicious lesion area in the white light endoscope image as the target observation angle.

[0080] In a preferred embodiment, the observation angle adjustment unit specifically includes:

[0081] The second acquisition unit is used to acquire white light endoscope images corresponding to each observation angle at the target working distance and to perform preprocessing.

[0082] The segmentation unit is used to identify suspicious lesion areas in the preprocessed white light endoscope image through a preset neural network model, and to segment the boundaries of the suspicious lesion areas in real time to generate the number of lesion pixels corresponding to the suspicious lesion areas.

[0083] The calculation unit is used to calculate the proportion of lesion pixels in each white light endoscope image under different observation angles, and to take the observation angle with the largest proportion of lesion pixels as the target observation angle. The proportion of lesion pixels = number of lesion pixels / total number of image pixels.

[0084] In a preferred embodiment, the adjustment module 100 further includes a white light working distance optimization unit, which is specifically used to keep the target observation angle of the white light endoscope unchanged, increase and decrease the current working distance of the white light endoscope according to a preset adjustment range, and respond to user commands to use the working distance corresponding to the target proportion of the field of view as the optimized working distance.

[0085] In a preferred embodiment, the path establishment module 300 specifically includes:

[0086] The judgment unit is used to determine whether the initial lesion boundary is elliptical. If so, the minimum bounding rectangle method is used to determine the direction of the longest center line and the direction of the shortest center line of the initial lesion boundary. If not, several scan lines with different angles are established from the geometric center point to determine the direction of the longest center line and the direction of the shortest center line.

[0087] The third acquisition unit is used to acquire target lesion points on the initial lesion boundary with a classification probability greater than a preset value, connect the target lesion points with the geometric center point, and generate a preferred scanning direction.

[0088] The construction unit is used to establish the target scanning path of the confocal microendoscopic probe, wherein the target scanning path includes the longest centerline direction, the shortest centerline direction, and the preferred scanning direction.

[0089] In a preferred embodiment, the correction module 500 specifically includes:

[0090] A fitting unit is used to fit the initial lesion boundary to generate an initial lesion curve;

[0091] A matching unit is used to transform the at least one mutation point into a white light endoscopic image and obtain the matching point on the initial lesion curve that is closest to each mutation point, and establish a point-to-point relationship.

[0092] The correction unit is used to calculate the displacement vector of each mutation point according to the point pair relationship, and establish a boundary correction model based on the displacement vector, so as to correct the initial lesion boundary through the boundary correction model and generate the target lesion boundary.

[0093] It should be noted that the foregoing explanation of the lesion boundary joint localization method embodiment also applies to the lesion boundary joint localization device of the above embodiment, and will not be repeated here.

[0094] This invention also provides a lesion boundary joint localization device, including a computer-readable storage medium and a processor, wherein the processor executes a computer program on the computer-readable storage medium to implement the steps of the lesion boundary joint localization method described above.

[0095] Figure 6 This is a schematic diagram of the lesion boundary joint localization device provided by the present invention. The lesion boundary joint localization device 8 includes: a processor 80, a readable storage medium 81, and a computer program 82 stored in the readable storage medium 81 and executable on the processor 80. When the processor 80 executes the computer program 82, it implements the steps in the above-described method embodiments, for example... Figure 3The steps shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module in the above-described device embodiments, for example... Figure 5 The functions of the module shown.

[0096] For example, the computer program 82 may be divided into one or more modules, which are stored in the readable storage medium 81 and executed by the processor 80 to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 82 in the lesion boundary joint localization device 8.

[0097] The lesion boundary joint localization device 8 may include, but is not limited to, a processor 80 and a readable storage medium 81. Those skilled in the art will understand that... Figure 6 This is merely an example of the lesion boundary joint positioning device 8 and does not constitute a limitation on the lesion boundary joint positioning device 8. It may include more or fewer components than shown, or combine certain components, or different components. For example, the lesion boundary joint positioning device may also include a power management module, a computing processing module, input / output devices, network access devices, buses, etc.

[0098] The processor 80 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0099] The readable storage medium 81 can be an internal storage unit of the lesion boundary joint localization device 8, such as a hard disk or memory of the lesion boundary joint localization device 8. The readable storage medium 81 can also be an external storage device of the lesion boundary joint localization device 8, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the lesion boundary joint localization device 8. Further, the readable storage medium 81 can include both internal storage units and external storage devices of the lesion boundary joint localization device 8. The readable storage medium 81 is used to store the computer program and other programs and data required by the lesion boundary joint localization device. The readable storage medium 81 can also be used to temporarily store data that has been output or will be output.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0101] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0102] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0103] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0104] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0105] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0106] The present invention is not limited to the description in the specification and embodiments, and thus other advantages and modifications can be readily realized by those skilled in the art. Therefore, the present invention is not limited to the specific details, representative devices and illustrated examples shown and described herein without departing from the spirit and scope of the general concept as defined by the claims and their equivalents.

Claims

1. A method for joint localization of lesion boundaries, characterized in that, Includes the following steps: Step 1: Adjust the current working distance and / or current observation angle of the white light endoscope to the target state; Step 2: Acquire white light endoscopic images and locate the initial lesion boundary of suspicious lesions based on the white light endoscopic images; Step 3: Identify the geometric center point of the initial lesion boundary, obtain the longest and shortest center line directions of the initial lesion boundary based on the geometric center point, and establish the target scanning path of the confocal endoscope probe. Step 4: Position the center of the probe of the white light endoscope to the geometric center point, and insert the confocal microendoscopic probe along the biopsy channel of the white light endoscope. Control the confocal microendoscopic probe to scan sequentially along the target scanning path to obtain at least one abrupt change point on the target scanning path. Step 5: Based on the at least one mutation point, the initial lesion boundary is corrected to generate the target lesion boundary of the suspected lesion; Step 1 includes the white light working distance optimization step, specifically: Keep the target observation angle of the white light endoscope unchanged, and increase or decrease the current working distance of the white light endoscope according to the preset adjustment range; In response to user commands, the working distance corresponding to the target proportion of the field of view is used as the optimized working distance; Step 5 modifies the initial lesion boundary based on the at least one mutation point, specifically as follows: The initial lesion boundary is fitted to generate an initial lesion curve; Transform the at least one mutation point into a white light endoscopic image, and obtain the matching point on the initial lesion curve that is closest to each mutation point to establish a point-to-point relationship; The displacement vector of each mutation point is calculated based on the point-to-point relationship, and a boundary correction model is established based on the displacement vector. The initial lesion boundary is corrected by the boundary correction model to generate the target lesion boundary.

2. The lesion boundary joint localization method according to claim 1, characterized in that, Step 1: Adjust the working distance of the white light endoscope, specifically as follows: Once the white light endoscope probe enters the area to be detected on the human body, the current working distance of the white light endoscope is controlled to be the initial working distance corresponding to the current scene. The area to be detected is continuously observed to obtain at least one suspicious lesion marker result in the white light endoscope image at the initial working distance; The preset working distance adjustment scheme is invoked to conduct local detailed examination of the marked suspicious lesions at multiple working distances, generating local images of suspicious lesions corresponding to different working distances; The local image of the suspected lesion is used to identify the local features of the lesion, generate a target working distance that meets the preset feature conditions, and adjust the current working distance of the white light endoscope to the target working distance.

3. The method for joint localization of lesion boundaries according to claim 1, characterized in that, Step 1: Adjust the observation angle of the white light endoscope, specifically as follows: Keeping the target working distance of the white light endoscope constant, the observation angle of the white light endoscope probe is changed, and the observation angle corresponding to the largest suspicious lesion area in the white light endoscope image is taken as the target observation angle.

4. The lesion boundary joint localization method according to claim 3, characterized in that, The observation angle corresponding to the largest area of ​​suspicious lesions in the white light endoscopy image is specifically taken as the target observation angle: Acquire white light endoscope images corresponding to each observation angle at the target working distance, and perform preprocessing; The suspicious lesion areas in the preprocessed white light endoscope image are identified by a preset neural network model, and the boundaries of the suspicious lesion areas are segmented in real time to generate the number of lesion pixels corresponding to the suspicious lesion areas. Calculate the percentage of lesion pixels in each white light endoscope image under different observation angles, and take the observation angle with the largest percentage of lesion pixels as the target observation angle. The percentage of lesion pixels = number of lesion pixels / total number of image pixels.

5. The method for joint localization of lesion boundaries according to any one of claims 1-4, characterized in that, Establish the target scanning path for the confocal endoscope probe, specifically as follows: Determine whether the initial lesion boundary is elliptical. If so, use the minimum bounding rectangle method to determine the direction of the longest and shortest center line of the initial lesion boundary. If not, establish several scan lines at different angles starting from the geometric center point and determine the direction of the longest and shortest center line among them. Obtain target lesion points with a classification probability greater than a preset value on the initial lesion boundary, connect the target lesion points with the geometric center point, and generate a preferred scanning direction; A target scanning path for a confocal endoscope probe is established, the target scanning path including the longest centerline direction, the shortest centerline direction, and the preferred scanning direction.

6. A lesion boundary joint localization device, based on the lesion boundary joint localization method according to any one of claims 1-5, characterized in that, It includes an adjustment module, an initial segmentation module, a path establishment module, a mutation point acquisition module, and a correction module. The adjustment module is used to adjust the current working distance and / or current observation angle of the white light endoscope to the target state; The initial segmentation module is used to acquire white light endoscope images and locate the initial lesion boundary of suspicious lesions based on the white light endoscope images; The path establishment module is used to identify the geometric center point of the initial lesion boundary, obtain the longest centerline direction and the shortest centerline direction of the initial lesion boundary based on the geometric center point, and establish the target scanning path of the confocal endoscope probe. The mutation point acquisition module is used to position the probe center of the white light endoscope to the geometric center point, and extend the confocal microendoscopic probe along the biopsy channel of the white light endoscope, and control the confocal microendoscopic probe to scan sequentially along the target scanning path to acquire at least one mutation point on the target scanning path. The correction module is used to correct the initial lesion boundary based on the at least one mutation point, and generate the target lesion boundary of the suspected lesion.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the lesion boundary joint localization method according to any one of claims 1-5.

8. A lesion boundary joint localization device, comprising a computer-readable storage medium and a processor, characterized in that, When the processor executes the computer program on the computer-readable storage medium, it implements the steps of the lesion boundary joint localization method according to any one of claims 1-5.

Images