Biological tissue molecular fingerprint extraction method and system for intraoperative margin assessment

Abstract

The application discloses a kind of biological tissue molecular fingerprint extraction methods and systems for intraoperative margin evaluation, the method includes rough division and subdivision of margin, adopts projection technology, and the near-infrared autofluorescence signal contrast image of tissue is presented in the operation field of doctor, and the tumor / cancer and normal tissue margin coordinate points that are delineated by probe track tracking method are presented in the operation field of doctor, in intraoperative margin evaluation, realize the fluorescence imaging of biological tissue without staining, the imaging depth of big biological tissue is big, realizes deep biological tissue raman spectrum molecular fingerprint detection;Combined with projection technology, the scanning track of probe to tissue is projected and superimposed on tissue sample, cooperatively realize deep biological tissue raman spectrum molecular fingerprint detection and its guided imaging, to help doctor to judge margin position.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of biomedical technology and the field of optical technology, and relates to a biomolecular fingerprint extraction method and system for intraoperative margin assessment, in particular to a Raman spectrum acquisition method and system guided by a large field of view near-infrared autofluorescence functional imaging. BACKGROUND

[0002] Local excision surgery for early-stage tumors (such as breast ductal carcinoma in situ) not only can achieve a postoperative survival rate comparable to that of total excision surgery, but also can significantly shorten the postoperative recovery time of patients and reduce the surgical cost, greatly improving the postoperative life quality of patients. During local excision surgery, the surgeon resects the tumor tissue and a small amount of surrounding healthy tissue to ensure that all tumor tissue is resected and as little healthy tissue as possible is resected to minimize the functional loss caused by the resection of normal tissue: that is, the surgeon needs to perform intraoperative margin assessment.

[0003] Currently, surgeons mainly use preoperative images (such as CT, MRI, and ultrasound imaging) to plan the surgical path, or use intraoperative frozen sections and cytological prints to determine the pathological type of the resected biological tissue. However, the relative positional relationship of the biological tissue during actual surgery (such as the resection of the breast skin during breast-conserving surgery for breast cancer) inevitably differs from that observed in the preoperative images, making the surgical path planned by the preoperative images not entirely applicable to the actual surgery. The application of intraoperative frozen sections and cytological prints to the resected biological tissue in actual surgery effectively avoids the difficulties faced by preoperative images, but it is time-consuming and significantly prolongs the operation time. Therefore, it is necessary to solve the actual problems faced by the existing margin assessment methods that rely on preoperative images, intraoperative frozen sections, and cytological prints.

[0004] Raman spectroscopy is a spectroscopy technique based on inelastic Raman scattering, and its spectral signal is derived from different vibration modes of molecules. Therefore, Raman spectroscopy can realize the detection of molecular fingerprints such as proteins, lipids, DNA, and water molecules in biological tissues. The molecular fingerprints of tumor tissues and normal tissues are different, laying a pathological and physical foundation for intraoperative margin assessment based on Raman spectroscopy. For example, existing studies have used Raman spectroscopy for margin assessment in breast-conserving surgery for breast cancer, and have preliminarily demonstrated the potential of intraoperative margin assessment based on Raman spectroscopy molecular fingerprint detection. However, existing Raman spectroscopy techniques are limited by resolution and acquisition time, and their application to intraoperative margin assessment faces the following problems: (1) they mainly detect the molecular fingerprints of local points, and cannot effectively record the positions of molecular fingerprint detection; (2) they are slow and cannot meet the real clinical needs of rapid intraoperative margin assessment of a large volume of surgical cavities in breast-conserving surgery.

[0005] Near-infrared autofluorescence imaging technology has the characteristics of time saving and high spatial resolution, but the specificity and sensitivity of near-infrared autofluorescence imaging technology are low, and it is not suitable for intraoperative application alone. Therefore, it is necessary to combine the selective sampling Raman spectroscopy technology of Raman spectroscopy and near-infrared autofluorescence imaging. Under the guidance of near-infrared autofluorescence imaging, the number of Raman measurements on a larger area of breast tissue can be greatly reduced, thereby saving the time of breast-conserving surgery. In recent years, the research on selective sampling Raman spectroscopy technology has been continuously developed, aiming to identify a larger breast tissue in the operation with the highest possible accuracy and reasonable time range (for example, less than 30 minutes). However, these near-infrared autofluorescence images are excited by ultraviolet light, which is less safe for surgeons than near-infrared light (785 nm). However, since the emission light of near-infrared light (greater than 810 nm) is not visible to the naked eye, the near-infrared autofluorescence image of the tissue is not in the surgical field of view of the surgeon, thereby increasing the clinical translation time. The overlay tissue imaging method can solve this problem, but they do not consider the visualization of breast tissue Raman spectroscopy. Some studies that consider the visualization of tissue Raman spectroscopy only superimpose the probe trajectory on the imaging video according to the type of Raman spectroscopy, and do not present in the surgical field of view of the surgeon, and near-infrared fluorescence imaging needs to apply a fluorescence enhancer (such as protoporphyrin IX) on the tissue in advance, which increases the operation time.

[0006] In summary, the existing problems of the edge point evaluation technology for early tumors (such as breast ductal carcinoma in situ) in breast-conserving surgery are as follows:

[0007] 1. Near-infrared autofluorescence imaging relies on the application of a fluorescence enhancer (such as protoporphyrin IX) on the tissue, which increases the time of breast-conserving surgery and the workload of the doctor's operation.

[0008] 2. The detection depth of the fiber Raman spectroscopy probe used for margin assessment is limited, and it cannot detect the Raman spectroscopy molecular fingerprint of deep biological tissue; the detection depth of the fiber Raman spectroscopy probe used for margin assessment: the skin is ~1cm, and it is dependent on the tissue, and cannot detect the Raman spectroscopy molecular fingerprint of deep biological tissue.

[0009] 3. When performing margin assessment, the probe trajectory coordinate points corresponding to different tissue types of different colors cannot be directly presented in the surgical field of view of the doctor, but the probe trajectory is superimposed on the clinical imaging video, which increases the time and workload of the doctor's clinical translation. SUMMARY

[0010] In view of the problems in the prior art, the present application aims to provide a biological tissue molecular fingerprint extraction method and system for intraoperative margin evaluation, which realizes biological tissue fluorescence imaging with large imaging depth without staining during intraoperative margin evaluation, and combines projection technology to project the fluorescence image onto the tissue sample, guide the realization of deep biological tissue Raman spectrum molecular fingerprint detection, and realize the projection visualization of Raman spectrum track points on the biological tissue.

[0011] To achieve the above-mentioned purpose, the present application provides a biological tissue molecular fingerprint extraction method for intraoperative margin evaluation, which comprises the following steps:

[0012] S100. Rough division of the margin, specifically comprising:

[0013] S101. Laser pen positioning working surface; two laser pens are used for positioning, so that the two laser points intersect on the working surface, to determine whether the system working distance is at the set distance;

[0014] S102. Position registration of near-infrared camera and projector; a preset registration pattern is used, and a visible light camera is used as an intermediate medium for registration, to perform image registration on the near-infrared camera and the projector;

[0015] S103. Laser excitation of tissue; under the excitation of the laser, the tissue generates near-infrared autofluorescence signals to form a fluorescence image;

[0016] S104. Near-infrared camera acquires fluorescence image; after filtering, the near-infrared camera receives the near-infrared fluorescence contrast image generated by the tissue;

[0017] S105. The fluorescence contrast image is cropped, contrast is improved, binarized, denoised, matched with the field of view of the projector, and the edge of the fluorescence region is drawn;

[0018] S106. The projector projects the image obtained in step S105 onto the tissue surface to realize rough division of abnormal tissue and normal tissue;

[0019] S200. Subdivision of the margin, specifically comprising:

[0020] S201. Position registration of visible light camera and projector; a preset registration pattern is used to perform image registration on the visible light camera and the projector to obtain a registration matrix;

[0021] S202A. Probe collects tissue Raman spectrum; under the excitation of narrowband laser, the probe collects the Raman spectrum signal generated by the tissue and received by the spectrometer;

[0022] S202B. Synchronizing with step S202A, the visible light camera collects images of the probe scanning the abnormal tissue and normal tissue margin track while recording the entire imaging process;

[0023] S203A. Classifying the Raman spectrum collected in S202A;

[0024] S203B. Synchronizing with step S203A, after each frame of image of the probe track scanning is processed by the probe tip coordinate point finding algorithm, the probe scanning track coordinate points are obtained;

[0025] S204. Combining the spectrum classification result of step S203A and the probe scanning track coordinate points obtained in step S203B, different colors are given to different spectral points, thereby generating a probe track coordinate point graph of different tissue types corresponding to different color points;

[0026] S205. Using the registration matrix obtained in step S201, the different color probe track scanning coordinate point graph obtained in step S204 and the projector are matched in the field of view;

[0027] S206. Projecting the track coordinate point graph of different colors onto the tissue surface, thereby helping the doctor accurately determine the tissue type of the margin.

[0028] Further, in step S101, two 650nm laser pens are used for positioning, and when the two laser points intersect on the working surface, the working distance of the system is 30cm, that is, the distance from the near-infrared camera lens surface to the working surface is 30cm.

[0029] Further, in step S103, under the excitation of a 785nm laser, a near-infrared autofluorescence signal greater than 810nm is generated according to the characteristics of the tissue itself.

[0030] Further, in step S202A, a 785nm laser or a 1064nm laser is used for excitation.

[0031] Further, in step S202B, the probe tip coordinate point finding algorithm process is as follows:

[0032] P1. Color marker making; first, color blocks are used to mark on the probe according to the geometric shape of the probe; the color blocks are composed of a first color block above and a second color block below;

[0033] P2. Frame image size reduction; reduce the input image size to one half of the original size;

[0034] P3. HSV color block segmentation; in the HSV color space, the specific color parts corresponding to the first color block and the second color block are respectively reserved, and morphological opening and closing operations are performed, and after noise removal of the segmented image, the two largest segmented regions are respectively the HSV segmented regions corresponding to the first color block and the second color block;

[0035] P4. Judgment of whether there is shielding; the ratio of the image areas of the second color block and the first color block calculated in P3 is judged, if less than a set threshold, it is judged that there is shielding, the next step is P5, if greater than or equal to the threshold, there is no shielding, the next steps are P5-P6 in turn;

[0036] P5*. Deletion of the probe picture and the corresponding spectrum collected by shielding;

[0037] P5. Corner point coordinate searching; the minimum circumscribed rectangle algorithm in OpenCv is selected to obtain the four corner point coordinates of the first color block, so as to calculate the center coordinates of the first color block; the minimum circumscribed triangle algorithm in OpenCv is selected to obtain the three corner point coordinates corresponding to the second color block;

[0038] P6. Obtaining the probe tip coordinate point; the tip coordinate of the probe is the corner point coordinate farthest from the center coordinates of the first color block among the three corner point coordinates of the second color block.

[0039] Further, in step S103, in the autofluorescence imaging part, 365nm or 405nm laser is used to excite biological tissues to generate a fluorescence image, and the tissue fluorescence signal wave band generated under the excitation of this wave band is respectively in the range of 442nm-488nm and 480nm-562nm.

[0040] Further, in step S204, for the probe trajectory coordinate points of different colors after classification, the probe trajectory points are superimposed on the imaging video.

[0041] On the other hand, the present application provides a biomolecular fingerprint extraction system for intraoperative margin evaluation, which is used to realize the biomolecular fingerprint extraction method for intraoperative margin evaluation according to the present application.

[0042] Further, the system comprises a near-infrared autofluorescence imaging subsystem and a Raman spectrum subsystem, wherein the near-infrared autofluorescence imaging subsystem comprises a 785nm excitation light source, a near-infrared camera and a long-pass filter; the Raman spectrum subsystem comprises a visible light camera, a 785nm or 1064nm narrow-band excitation light source, a Raman probe and a Raman spectrometer.

[0043] Further, the Raman probe comprises a band-pass filter, a dichroic mirror, a color block, a multi-mode optical fiber loaded into a 24G needle tube and a small ball at the end, and a long-pass filter.

[0044] Advantages:

[0045] (1) In part of the near-infrared autofluorescence image, a more friendly 785nm near-infrared light source is used to excite the doctor instead of an ultraviolet light source;

[0046] (2) Near-infrared autofluorescence imaging is used instead of protoporphyrin IX fluorescence imaging, that is, without the need to apply a fluorescent agent to the tissue in advance, but according to the characteristics that the tissue generates an autofluorescence signal greater than 810nm under 785nm laser excitation, a contrast fluorescence image of deep layer (about 3cm) breast tumor / cancer and normal tissue is generated.

[0047] (3) By putting the multimode optical fiber with a small ball at the end into a 24G thin needle, the working distance between the probe and the tissue is controlled, and the molecular fingerprint detection of the biological tissue Raman spectrum is realized.

[0048] (4) The projection technology is used to present the near-infrared autofluorescence signal contrast image of the tissue and the tumor / cancer and normal tissue margin coordinate points drawn by the probe track tracking method in the surgical field of the doctor, so as to reduce the time and workload of the doctor in clinical translation, thereby reducing the time of confirming the tumor / cancer and normal tissue margin in the operation. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The system diagram of the near-infrared autofluorescence imaging and probe tracking identification and projection of the tumor / cancer and normal tissue margin in the biological molecular fingerprint extraction system for intraoperative margin evaluation according to the present application is shown;

[0050] Figure 2 The overall algorithm flow chart of the biological tissue molecular fingerprint extraction for intraoperative margin evaluation according to the present application is shown;

[0051] Figure 3 The near-infrared image and projector registration flow chart according to the present application is shown;

[0052] Figure 4 The near-infrared camera, visible light camera and projector position registration process acquisition diagram according to the present application is shown; wherein: Figure 4 a. Near-infrared camera registration pattern original drawing; Figure 4 b. Visible light camera registration pattern original drawing; Figure 4 c. Projection drawing of the visible light camera shooting Figure 4 a; Figure 4 d. Projection drawing of the visible light camera shooting Figure 4 b; Figure 4 e. SURF algorithm transformation to the projection drawing under the perspective of the near-infrared camera; Figure 4 f. Near-infrared camera registration result drawing;Figure 4 g. Visible light camera registration result image;

[0053] Figure 5 The figure shows the human breast cancer / normal tissue autofluorescence image projection process according to the present application; wherein: Figure 5 a. Near-infrared camera collects human breast cancer / normal tissue near-infrared autofluorescence image;

[0054] Figure 5 b. Figure 5 a. Cropped image; Figure 5 c. Near-infrared autofluorescence image after contrast enhancement; Figure 5 d. Binary NIRAF image; Figure 5 e. NIRAF image of the largest area of the reserved contour area for denoising, i.e. breast tumor tissue; Figure 5 f. NIRAF image of the registered breast tumor tissue; Figure 5 g. NIRAF image of the bold red edge contour image; Figure 5 h. Figure 5 f and Figure 5 g. Superimposed image;

[0055] Figure 6 The figure shows the probe tip coordinate point finding algorithm flow chart according to the present application;

[0056] Figure 7 The figure shows the probe tip coordinate point finding algorithm specific implementation diagram according to the present application; wherein, Figure 7 a. Probe original image with color block installed; Figure 7 b. Blue cylindrical segmentation image in HSV space; Figure 7 c. Cyan conical segmentation image in HSV space; Figure 7 d. Denoised image after morphological opening and closing operation of the blue cylindrical segmentation image; Figure 7 e. Denoised image after morphological opening and closing operation of the cyan conical segmentation image; Figure 7 f. Probe coordinate point schematic diagram.

[0057] Figure 8 The figure shows the trajectory + fluorescence projection image obtained by the probe tip coordinate point finding algorithm according to the present application; wherein, Figure 8 a. Visible light camera records probe scanning trajectory coordinate points and coordinate point finding image; Figure 8 b. Probe trajectory point coordinate image after visible light camera and projector registration, classification and different color assignment; Figure 8 c. Probe trajectory coordinate point and near-infrared autofluorescence superimposed image; Figure 8 d. Final projection result image.

[0058] Figure 9Scheme 2 flow chart showing the biomolecular fingerprint extraction method according to the present application for intraoperative margin assessment;

[0059] Figure 10 Scheme 3 flow chart showing the biomolecular fingerprint extraction method according to the present application for intraoperative margin assessment;

[0060] Figure 11 Scheme 4 flow chart showing the biomolecular fingerprint extraction method according to the present application for intraoperative margin assessment;

[0061] Figure 12 Scheme 5 flow chart showing the biomolecular fingerprint extraction method according to the present application for intraoperative margin assessment.

[0062] The relevant reference signs are as follows: 1, near-infrared autofluorescence imaging subsystem: 101, 650 nm positioning laser pen; 102, 785 nm excitation light source for near-infrared autofluorescence imaging; 103, 785 nm band-pass filter for eliminating Raman and fluorescence background signals generated by optical fibers; 104, 3-layer long-pass filter of the near-infrared autofluorescence imaging subsystem; 105, near-infrared imaging camera; 106, 830 nm LED lamp bead for image registration of the near-infrared camera and the projector; 2, Raman spectrum subsystem: 201, 785 nm narrow-band excitation light source of the Raman spectrum subsystem; 202, band-pass filter for passing only 785 nm laser; 203, dichroic mirror; 204, color block for probe identification, which is composed of a blue-marked cylindrical shape and a cyan-marked round table shape; 205, multi-mode optical fiber end is burned into a small ball, and the optical fiber is installed into a 24G needle tube for collecting Raman spectrum of deep tissue; 206, long-pass filter for passing only fluorescence signals generated by tissues to be received by the Raman spectrometer; 207, spectrometer for collecting tissue Raman spectrum. Among them, 3, projector shared by the near-infrared autofluorescence imaging subsystem and the Raman spectrum subsystem; 4, visible light camera shared by the near-infrared autofluorescence imaging subsystem and the Raman spectrum subsystem. 5, uninterruptible power supply. 601, computer host; 602, computer display screen; 603, near-infrared fluorescence image, visible light image, and Raman spectrum collection interface. DETAILED DESCRIPTION

[0063] The technical solutions of the present application will be described clearly and completely below in combination with the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0064] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0065] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0066] The specific embodiments of the present application will be described in detail below. Figures 1-12 The specific embodiments described herein are intended to illustrate and explain the present application, and are not intended to limit the present application.

[0067] The biomolecule fingerprint extraction method and system for intraoperative margin evaluation of the present application is aimed at completely removing tumor or cancer tissue during breast-conserving surgery to reduce the recurrence rate and secondary resection rate of breast cancer. A more efficient method is to combine the near-infrared autofluorescence image of the tissue and the selective Raman sampling spectrum technology of the tissue, without losing the spatial information, resolution and specificity and sensitivity of tissue classification, to realize non-destructive, rapid and accurate tissue margin identification and confirmation.

[0068] As Figure 1As shown, the biomolecular fingerprint extraction system for intraoperative resection margin evaluation of the application includes a near-infrared autofluorescence imaging subsystem 1 and a Raman spectrum subsystem 2, which are placed on a small trolley. Among them, the near-infrared autofluorescence imaging subsystem 1 includes a 650nm positioning laser pen 101, a 785nm excitation light source 102, a band-pass filter 103, a 3-layer long-pass filter 104, a near-infrared camera 105, an 830nm LED lamp bead 106, a projector 3, and a visible light camera 4; the Raman spectrum subsystem 2 includes a 785nm narrow-band excitation light source 201, a Raman probe (consisting of a band-pass filter 202, a dichroic mirror 203, a color block 204, a multi-mode optical fiber 205 packed into a 24G needle tube and a small ball burned at the end, a long-pass filter 206), a Raman spectrometer 207, a projector 3, and a visible light camera 4.

[0069] Among them, the 650nm positioning laser pen 101 of the near-infrared autofluorescence imaging subsystem 1 is used to determine whether the system working distance is at 30cm, that is, when the two laser points intersect on the working surface, the distance from the near-infrared camera lens to the working surface is 30cm. The 3-layer long-pass filter 104 is used to filter out signals outside the waveband of the fluorescence signals generated by the tissue, reducing interference. The near-infrared imaging camera 105 is used to receive the near-infrared autofluorescence image generated by the sample tissue under 785nm excitation. The 830nm LED lamp bead 106 is turned on when the near-infrared camera and the projector are registered, which facilitates the near-infrared camera 105 with a long-pass filter to capture a registered image. The dichroic mirror 203 is used to separate the 785nm narrow-band excitation light and the Raman signal (greater than 810nm) generated by the tissue, so as to realize the collection of the tissue Raman spectrum. The projector 3 is used to project the near-infrared autofluorescence contrast image of the tissue into the surgical field in the near-infrared autofluorescence imaging subsystem 1, which facilitates the doctor to make a preliminary judgment on abnormal and normal tissues (abnormal tissues include tumor / cancer tissues), that is, to highlight and project the lesion area in the near-infrared fluorescence imaging area; in the Raman spectrum subsystem 2, it is used to project the different color resection margin coordinate points of the abnormal and normal tissues drawn by the probe into the doctor's surgical field of view, which facilitates the doctor to make accurate judgment on tumor / cancer and normal tissues in the field of view of the surgical cavity. The visible light camera 4 acts as an intermediate medium for image registration of the near-infrared camera and the projector in the near-infrared autofluorescence imaging subsystem 1 and is used to record the projected image of the tissue, and in the Raman spectrum subsystem 2, it is used to record the probe scanning trajectory point image, so as to facilitate subsequent probe coordinate point searching and projection of different color probe trajectory points. The uninterruptible power supply 5 supplies power to the integrated system during work. The computer host 601 is used to control the operation of the whole system. The display screen 602 is used to display the collected images, spectra, projection operations, etc.

[0070] As Figure 2As shown, the procedure for extracting the molecular fingerprint of biological tissue for intraoperative margin assessment is as follows, including two main steps: coarse margin classification (S100) and fine margin classification (S200). Specifically:

[0071] S100. Coarse division of the cutting edge, including:

[0072] S101. Laser pointer positioning of the working surface; Two 650nm laser pointers are used for positioning to determine whether the system working distance is 30cm, that is, when the two laser points intersect on the working surface, the distance from the lens surface of the near-infrared camera to the working surface is 30cm.

[0073] S102. Position registration of the near-infrared camera and projector; using a preset registration pattern and with the aid of a visible light camera as an intermediate registration medium, image registration is performed between the near-infrared camera and the projector to prepare for subsequent fluorescence signal contrast image transformation and projection; the flowchart of this registration is as follows. Figure 3 As shown, a detailed description follows:

[0074] S1. Registration Pattern Acquisition. The specific implementation of step S1 is as follows, including:

[0075] S1.1. Near-infrared camera captures the registration pattern. Turn on the 830nm near-infrared LED, use the preset registration pattern, and have the near-infrared camera capture the registration pattern to obtain the registration pattern acquired by the near-infrared camera (e.g., ...). Figure 4 (as shown in a).

[0076] S1.2. Visible light camera captures the registration pattern. Using a preset registration pattern, the visible light camera captures the registration pattern, obtaining the registration pattern acquired by the visible light camera (e.g., ...). Figure 4 (as shown in b).

[0077] S1.3. Visible light camera shooting Figure 4 The projection image of a. The projector shows the registration pattern captured by the near-infrared camera. Figure 4 a) Projected onto the working surface and captured by a visible light camera. Figure 4 The projection pattern of a ( Figure 4 c).

[0078] S2. Automatic registration of near-infrared and visible light images. Using the SURF automatic registration algorithm, the near-infrared camera image (…) is first obtained. Figure 4 a) and visible light camera ( Figure 4 b) The registration matrix T1 of the acquired images.

[0079] S3. Automatic image registration between the near-infrared camera and the projector. The specific implementation of step S3 is as follows, including:

[0080] S3.1. Conversion of the near-infrared camera's projected image. This is achieved using the image obtained in step S1.3. Figure 4 c and the registration matrix T1 in step S2 are used to obtain the projected image under the field of view of the near-infrared camera. Figure 4 e).

[0081] S3.2. Position registration matrix of near-infrared camera and projector. Using the SURF automatic registration algorithm, the position registration matrix of the near-infrared camera (…) is obtained. Figure 4 a) and projector ( Figure 5 e) Position registration matrix T2.

[0082] The near-infrared autofluorescence images acquired by the near-infrared camera are then converted to the projector's viewpoint and projected onto the surgeon's surgical field of view, and matched with the location of the tumor / cancer in the ex vivo tissue.

[0083] S103. Laser-excited tissue; Under 785nm laser excitation, near-infrared autofluorescence signals greater than 810nm are generated according to the characteristics of the tissue itself, and the intensity of the fluorescence signal varies among different tissue types. For the portion where autofluorescence images are obtained, a safer near-infrared light source (785nm) is used for excitation instead of an ultraviolet light source.

[0084] S104. Near-infrared camera acquires fluorescence images; after passing through three long-pass filters (greater than 808nm), the fluorescence contrast signal is acquired by the near-infrared camera, forming a near-infrared autofluorescence contrast image of the tissue. In breast tissue, the fluorescence signal of tumor / cancer tissue is stronger than that of normal tissue (such as fat, stroma, etc.).

[0085] S105. Fluorescence image projection processing; cropping of fluorescence contrast images ( Figure 5 b) Improve contrast ( Figure 5 c) Binarization Figure 5 d) Noise reduction ( Figure 5 e) Projector field of view matching ( Figure 5 f), and draw the approximately 2mm edge of the fluorescent area in red (f). Figure 5 g、 Figure 4 h). Specifically, when binarizing the fluorescence image, the portion of the fluorescence signal exceeding a set threshold is identified as abnormal tissue, and this region is assigned white while other regions are black. This pixel threshold is derived from an empirical threshold, preferably 120. Pixel values ​​greater than 120 are identified as abnormal tissues such as tumors / cancer tissue, while values ​​less than or equal to 120 are normal tissues. The fluorescence contrast image is then matched to the projector's field of view. This specifically involves using the position registration matrix T2 between the near-infrared camera and the projector obtained in step S102 to convert the fluorescence contrast image to the projector's viewing angle.

[0086] S106. The projector projects the image obtained in step S105 onto the tissue surface, i.e. the surgeon's surgical field, to help the surgeon preliminarily determine the approximate location of abnormal tissue (e.g. tumor / cancer) on the tissue, and achieve the rough division of abnormal tissue and normal tissue. Meanwhile, the preliminary determination is also used to guide the probe to quickly draw the margin.

[0087] S200. Subdivision of the margin. It includes:

[0088] S201. Visible light camera and projector position registration; adopt a preset registration pattern to perform image registration on the visible light camera and the projector, and obtain a registration matrix T3 to prepare for subsequent trajectory point transformation and projection. The specific steps are as follows:

[0089] T1. Adjust the parameters (exposure time, etc.) of the visible light camera to capture the preset registration pattern, such as Figure 4 b;

[0090] T2. Project the pattern obtained in step T1 through the projector into the field of view of the visible light camera, and capture the projected image again with the visible light camera, such as Figure 4 d;

[0091] T3. Adopt a SURF automatic registration algorithm to obtain the position registration matrix T3 of the visible light camera Figure 4 b) and the projector Figure 8 d).

[0092] S202A. Excite the tissue under the 785nm narrowband laser, and the probe collects the Raman spectrum signal generated by the tissue and is received by the Raman spectrometer, so as to obtain the Raman spectrum of the tissue;

[0093] S202B. Synchronously with step S202A, the visible light camera records the entire imaging process and also collects the image of the probe scanning the abnormal tissue and normal tissue margin trajectory coordinate points;

[0094] S203A. Classify the Raman spectrum collected in S202A, and adopt a spectral classification model (such as a partial least squares discriminant analysis model) to divide the collected Raman spectrum into three categories: fat tissue, normal tissue, and abnormal tissue (tumor / cancer).

[0095] S203B. Synchronously with step S203A, after each frame of image scanned by the probe trajectory is processed by the probe tip coordinate point finding algorithm, the probe scanning trajectory coordinate points are obtained, as shown in Figure 6 a; wherein the specific process of the probe tip coordinate point finding algorithm is shown in Figure 7 , and the details are as follows:

[0096] P1. Color marker is made. First, the geometry of the probe and prior knowledge are combined, and a color block is marked on the probe, which is composed of a first color block (blue color block) above and a second color block (cyan color block) below. The color block in this embodiment is a 3D printed piece of resin material processed in combination with the size of the probe, and then a blue waterproof cloth-based tape is pasted on the cylinder above the printed piece, and a cyan waterproof cloth-based tape is pasted on the circular truncated cone below to distinguish from biological tissues, as shown in Figure 7 a. Among them, the visible light camera collects a two-dimensional image of the color marker printed piece.

[0097] P2. Frame image size reduction. Reduce the size of the input image to one half of the original to speed up the subsequent processing speed of the algorithm.

[0098] P3. HSV color block segmentation. The parts of the blue and cyan colors of interest are retained in the HSV color space, as shown in Figure 7 b, 7c. After noise removal of the segmented image, the first and second color blocks are the largest image segmentation blocks, as shown in Figure 7 d, 7e.

[0099] P4. Judgment of whether there is an occlusion. Since when the cyan color block is occluded, the probe tip coordinate point cannot be found or cannot be found, the ratio of the areas of the cyan color block and the blue color block calculated in P3 is judged, and if it is less than the ratio threshold value 1.5, it is judged that there is an occlusion, and the next step is P5, that is, deleting the picture and the corresponding spectrum, and if it is greater than or equal to the threshold value, it is judged that there is no occlusion, and then P5-P6 are performed in turn.

[0100] P5*. Delete the probe picture and the corresponding spectrum collected.

[0101] P5. Corner point coordinate search. Select the minimum enclosing rectangle algorithm in OpenCv to obtain the 4 corner point coordinates of the blue rectangular block: P1(x1, y1), P2(x2, y2), P3(x3, y3), P4(x4, y4), and select the minimum enclosing triangle algorithm in OpenCv to obtain the 3 corner point coordinates of the cyan color block: T1(x1, y1), T2(x2, y2), T3(x3, y3), Figure 7 f.

[0102] P6. Obtain the probe tip coordinate point. The center coordinates P c (x c , y c ) of the blue rectangular block are calculated, as shown in Figure 8 f, and the calculation formula is shown in formula (1); T1, T2, and T3 to Pc Given the distances d1, d2, and d3, the coordinates of the probe tip P are... T (x T y T ) is the coordinate of the corner point that is farthest from the center point of the rectangle. The distance calculation formula is shown in formula (2), where * represents points 1, 2, and 3.

[0103]

[0104]

[0105] S204. Combining the spectral classification results of step S203A and the probe scanning trajectory coordinates obtained in step S203B, different colors are assigned to different spectral points to generate probe trajectory scanning coordinate point maps corresponding to different color points in different tissue types.

[0106] S205. Using the registration matrix T3 obtained in step S201, perform field-of-view matching between the different color probe trajectory scanning point maps obtained in step S204 and the projector. The trajectory point transformation results are as follows: Figure 8 As shown in b;

[0107] S206. Project this trajectory onto the tissue surface, i.e., into the surgeon's field of vision, thereby helping the surgeon accurately determine the location of the cutting margin, such as... Figure 9 As shown in d, the near-infrared camera, visible light camera, and Raman probe spectral acquisition are controlled by a pre-written interface program.

[0108] In step S103, in the autofluorescence imaging section, a 365nm or 405nm laser can be used instead of a 785nm laser to excite biological tissue to produce a fluorescence image, such as... Figure 10 Scheme 2, as shown, generates tissue fluorescence signals in the ranges of 442nm-488nm and 480nm-562nm under excitation at this wavelength, which are within the visible light band. This means there is no need to use a projection system to project the tissue fluorescence image into the surgeon's operating cavity. However, ultraviolet light sources are not as safe as near-infrared light sources, and ultraviolet light sources require operation by professional technicians.

[0109] In step S202A, in the part where the tissue is excited to generate a Raman spectrum, the excitation light can be a 1064nm narrowband laser instead of the 785nm narrowband laser in the original scheme. For example... Figure 11 Scheme 3 is shown.

[0110] In steps S205-S206, for the probe trajectory points of different colors after classification, the probe trajectory points can be superimposed on the imaging video, instead of projecting the classified probe trajectory points onto the sample tissue in the original scheme. For example... Figure 12 Scheme 4 is shown.

[0111] ​ In the illustrated scheme five, 1064nm is used instead of 785nm narrowband laser in the original scheme to excite the tissue to generate corresponding Raman signals. At the same time, for the probe trajectory points of different colors after classification, the probe trajectory points are superimposed on the imaging video, instead of projecting the classified probe trajectory points onto the sample tissue in the original scheme.

[0112] The technical advantages of the present application are as follows:

[0113] (1) A safer and time-saving near-infrared autofluorescence imaging and projection system is proposed.

[0114] The near-infrared autofluorescence imaging and projection system mainly consists of a 785nm excitation light source, a long-pass filter, a near-infrared sensitive camera, a projector, and a visible light camera. The safety of the system refers to the use of a 785nm near-infrared excitation light source, which is more friendly to doctors, instead of a ultraviolet excitation light source when obtaining autofluorescence images of the tissue. The other aspect refers to the use of 785nm laser to excite the excised tissue of the patient without the need for additional application of a fluorescence enhancer. Instead, the 785nm laser directly generates autofluorescence signal images greater than 810nm according to the characteristics of the tissue, which greatly saves the time for preliminary judgment of the type of tissue during the fluorescence imaging of the tissue.

[0115] (2) A system combining near-infrared autofluorescence imaging technology and selective Raman spectrum sampling technology is proposed to reduce the secondary resection rate and recurrence rate of tumor / cancer and normal tissue in breast-conserving surgery.

[0116] (3) A visible light camera is used as an intermediate medium to realize the position registration algorithm of the fluorescence image and the projection image collected by the near-infrared camera packaged with a long-pass filter (greater than 808nm).

[0117] In this system, the near-infrared camera cannot obtain the projection image of the visible light band because three 808nm long-pass filters have been packaged in front of the near-infrared camera lens. By using the visible light camera, the position registration of the near-infrared camera and the projector is realized, so that the fluorescence image obtained by the near-infrared camera can be transformed to the perspective of the projector to project the fluorescence image of the excised tissue into the surgeon's surgical field, reducing the misjudgment rate of the tumor / cancer by the doctor and saving the time for confirming the tumor / cancer and normal tissue during the operation.

[0118] (4) A Raman probe trajectory coordinate point tracking projection algorithm is proposed.

[0119] Through the algorithm, the tumor / cancer and normal tissue margin coordinate points in the tissue can be identified, depicted in the surgery under the premise that the near-infrared autofluorescence image of the ex situ tissue has been obtained, and the margin coordinate points can be projected into the surgical field of the doctor, reducing the time of clinical translation of the doctor and saving the time of confirming the tumor / cancer and normal tissue margin in the surgery.

[0120] Any procedural or methodological description in the flowchart of the present application or otherwise described herein can be understood as representing a subsystem, fragment or portion of code comprising one or more executable instructions for implementing a specific logical function or procedure, which can be implemented in any computer readable medium for an instruction execution system, device or apparatus, which can be any medium containing storage, communication, propagation or transmission programs for use by an execution system, device or apparatus, including read-only memory, magnetic or optical disk, etc.

[0121] In the description of the present application, the description referring to the terms "embodiment", "example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily directed to the same embodiment or example. In addition, those skilled in the art can combine or combine the different embodiments or examples described in the present specification and the features therein without producing contradictions.

[0122] Although the above has shown and described the embodiments of the present application, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and modifications of the above embodiments within the scope of the present application.

Claims

1. A method for extracting a biological tissue molecular fingerprint for intraoperative margin assessment, characterized in that, The method comprises the following steps: S100. Coarse division of the incisal edge, specifically comprising: S101. Laser pen positioning working surface; two laser pens are used for positioning, so that the two laser points intersect on the working surface, to determine whether the system working distance is at the set distance; S102. Position registration of near-infrared camera and projector; a preset registration pattern is used, with the aid of a visible light camera as an intermediate registration medium, to perform image registration on the near-infrared camera and the projector; S103. Laser excitation of tissue; under the excitation of the laser, the tissue generates a near-infrared autofluorescence signal to form a fluorescence image; S104. Near-infrared camera acquires fluorescence image; after filtering, the near-infrared fluorescence contrast image of the tissue is generated by the near-infrared camera; S105. The fluorescence contrast image is cropped, contrast-enhanced, binarized, denoised, and matched with the field of view of the projector, and the edge of the fluorescence region is drawn; S106. The projector projects the image obtained in step S105 onto the tissue surface, to achieve coarse division of abnormal tissue and normal tissue; S200. Fine division of the incisal edge, specifically comprising: S201. Position registration of visible light camera and projector; a preset registration pattern is used to perform image registration on the visible light camera and the projector, to obtain a registration matrix; S202A. Probe acquires tissue Raman spectrum; the tissue is excited under narrowband laser, and the probe collects the Raman spectrum signal generated by the tissue and received by the spectrometer; S202B. Synchronized with step S202A, the visible light camera records the entire imaging process and also acquires images of the probe scanning abnormal tissue and normal tissue incisal edge track points; S203A. Classification of the Raman spectrum acquired in S202A; S203B. Synchronized with step S203A, after each frame of image of the probe track scanning is processed by a probe tip coordinate point finding algorithm, the probe scanning track coordinate points are obtained; S204. Combining the spectral classification results of step S203A and the probe scanning track coordinate points obtained in step S203B, different colors are given to different spectral points, to generate a probe track coordinate point graph with different color points corresponding to different tissue types; S205. Using the registration matrix obtained in step S201, the different color probe track scanning coordinate point graph obtained in step S204 is matched with the field of view of the projector; S206. Projecting the track coordinate points onto the tissue surface, to help the doctor accurately determine the incisal edge position.

2. The method for biological tissue molecular fingerprint extraction for intraoperative margin assessment according to claim 1, wherein, In step S101, two 650nm laser pens are used for positioning, and when the two laser points intersect on the working surface, the working distance of the system is 30cm.

3. The bio tissue molecular fingerprint extraction method for intraoperative margin assessment according to claim 1, wherein, In step S103, under the excitation of 785nm laser, the near-infrared autofluorescence signal greater than 810nm is generated according to the characteristics of the tissue itself.

4. The bio tissue molecular fingerprint extraction method for intraoperative margin assessment according to claim 1, wherein, In step S202A, 785nm laser or 1064nm laser is used for excitation.

5. The bio tissue molecular fingerprint extraction method for intraoperative margin assessment according to claim 1, wherein, In step S203B, the probe tip coordinate point finding algorithm process is as follows: P1. Color marker is made; first, color blocks are marked on the probe according to the geometry of the probe; The color blocks are composed of an upper first color block and a lower second color block; P2. Frame image downsize; reduce the input image size to half of the original size; P3. HSV color block segmentation; Reserve the specific color part corresponding to the first color block and the second color block respectively in the HSV color space, and perform opening and closing operations of morphology on the segmented image. After noise removal, the two largest segmented regions are the HSV segmented regions corresponding to the first color block and the second color block respectively; P4. Determine whether there is an occlusion; determine the ratio of the image area of the second color block to the first color block calculated in P3. If it is less than a set threshold, it is determined that there is an occlusion, and the next step is P5*. If it is greater than or equal to the threshold, there is no occlusion, and the next steps are P5-P6 in turn; P5*. Delete the occluded probe picture and the corresponding spectrum collected; P5. Corner point coordinate search; select the minimum circumscribed rectangle algorithm in OpenCv to obtain the 4 corner point coordinates of the first color block, and calculate the center coordinates of the first color block; select the minimum circumscribed triangle algorithm in OpenCv to obtain the 3 corner point coordinates corresponding to the second color block; P6. Obtain the probe tip coordinate point; the tip coordinate of the probe is the corner point coordinate farthest from the center coordinate of the first color block among the 3 corner point coordinates of the second color block.

6. The bio tissue molecular fingerprint extraction method for intraoperative margin assessment according to claim 1, wherein, In step S103, in the autofluorescence imaging part, 365nm or 405nm laser is used to excite biological tissue to generate a fluorescence image. The tissue fluorescence signal generated under this excitation is in the range of 442nm-488nm and 480nm-562nm respectively.

7. The bio tissue molecular fingerprint extraction method for intraoperative margin assessment according to claim 1, wherein, For the different color probe trajectory points obtained in step S204, the probe trajectory points are superimposed on the imaging video.

8. A biomolecule fingerprint extraction system for intraoperative margin assessment, comprising: The system is used to realize the biological tissue molecular fingerprint extraction method for intraoperative margin evaluation according to any one of claims 1-7.

9. The biomolecule fingerprint extraction system for intraoperative margin assessment of claim 8, wherein, The system comprises a near-infrared autofluorescence imaging subsystem and a Raman spectrum subsystem, wherein the near-infrared autofluorescence imaging subsystem comprises a 650nm positioning laser pen, a 785nm excitation light source, a band-pass filter, a long-pass filter, a near-infrared camera, a visible light camera, and a projector; the Raman spectrum subsystem comprises a 785nm narrow-band excitation light source, a Raman probe, a Raman spectrometer, a visible light camera, and a projector.

10. The biomolecule fingerprint extraction system for intraoperative margin assessment of claim 9, wherein, The Raman probe comprises a band-pass filter, a dichroic mirror, a color block, a multi-mode optical fiber inserted into a 24G needle tube and a small ball at the end, and a long-pass filter.

Images