Dual-mode imaging chip and its application in intraoperative navigation endoscope and cell microscopic high-content analyzer

Abstract

The present application relates to a kind of dual-mode imaging chip and its application in intraoperative navigation endoscope and cell microscopic high content analyzer, dual-mode imaging chip includes first imaging core particle, second imaging core particle and imaging servo core particle;Intraoperative navigation endoscope is configured dual-mode imaging chip, runs in intraoperative navigation application scene, executes tissue morphology observation and fluorescent label boundary identification, is used for tumor resection surgery navigation;The present application solves the engineering problems such as fiber-optic light source coupling, real-time navigation image generation, tumor cell edge display superposition in endoscope production;Cell microscopic high content analyzer is configured dual-mode imaging chip, runs in cell microscopic high content analysis application scene, cooperates with multi-well plate automatic sampling and environment control module, executes cell multi-parameter quantitative analysis and whole plate scanning, is used for drug research and development high-throughput screening;The present application solves the engineering problems such as optical mechanism complexity, mass data analysis, storage difficulty, phototoxicity, photobleaching in instrument research and development.

Description

Technical Field

[0001] This invention belongs to the field of medical device manufacturing and biomedical instrument R&D technology, specifically relating to a dual-mode imaging chip and its application in intraoperative navigation endoscopy and high-content cell microscopy analyzers. This invention is applicable to the manufacturing of medical devices for tumor resection surgical navigation and the development of high-throughput screening bioanalytical instruments for drug development. Background Technology

[0002] Currently, the manufacturing bottlenecks of existing intraoperative navigation endoscopes are as follows: (1) Limitations on probe miniaturization. Existing surgical microscopes or endoscope systems rely on traditional CCD / CMOS cameras and external image processing workstations. The probe diameter is usually greater than 5mm, which makes it impossible to enter narrow cavities such as those in neurosurgery and ophthalmology. Furthermore, fluorescence lifetime information needs to be fitted offline, which cannot guide the resection boundary in real time. The core reason is that the image sensor and processing unit are physically separated. Data needs to be transmitted over a long distance to the back-end FPGA / GPU for processing, with a delay of more than 10ms, which cannot meet the requirements of real-time navigation.

[0003] (2) Lack of cellular-level precision. Traditional intraoperative navigation relies on the surgeon's visual observation or preoperative MRI / CT images, with resolution limited to the tissue level (>1mm). It cannot identify the microscopic invasion boundaries of early cancerous cells, leading to incomplete or excessive resection. The fundamental bottleneck lies in the lack of in-situ intelligent processing capabilities of the front-end sensors, which cannot perform cellular-level morphological analysis while performing surgical navigation.

[0004] (3) Lagging human-computer interaction. The existing system uses a 2D display to present intraoperative images, requiring the surgeon to frequently switch between the screen and the surgical field of view, increasing the operational burden; and it cannot directly map cell classification labels onto physical tissues. The technical root cause is that the image fusion algorithm runs on the back-end industrial control computer, and the front end only outputs the raw video stream, which cannot superimpose biological information labels in real time.

[0005] The limitations of existing high-content cell microscopy analyzers are as follows: (1) Conflict between throughput and sensitivity. The frame rate and sensitivity of general-purpose scientific research-grade sCMOS cameras cannot be simultaneously achieved. Multi-well plate scanning relies on mechanical displacement stages for point-by-point imaging, and the scanning time of a 96-well plate is more than 30 minutes, which cannot meet the high-throughput requirements of drug screening. The core problem is that the optical structure of the image sensor is complex, lacks in-situ buffering and parallel processing capabilities, and each frame of data needs to be completely read into the back-end memory before analysis, making data transfer a bottleneck.

[0006] (2) Separate environmental control. In the existing system, the culture chamber and imaging module are separated. During cell transport, the temperature / CO2 fluctuations affect the quality of long-term imaging of live cells, resulting in a decline in imaging quality. Furthermore, the imaging module lacks embedded intelligence and cannot adaptively adjust exposure and gain according to environmental parameters.

[0007] (3) Lack of data management compliance. Drug screening data must comply with FDA 21 CFR Part 11 or NMPA electronic record specifications. Existing systems mostly use local storage and manual backup, which poses risks of tampering and difficulties in traceability. The root cause lies in the lack of chip-level trusted computing integration between the analysis unit and the storage unit. Summary of the Invention

[0008] The purpose of this invention is to provide a dual-mode imaging chip and its application in intraoperative navigation endoscopes and cell microscopic high-content analyzers, solving the problems of excessively large probe diameter, lack of cell-level precision, and lagging human-computer interaction in the manufacturing of existing intraoperative navigation endoscopes, as well as the problems of complex optical mechanisms, limited throughput, separate environmental control, and lack of data management compliance in the development of cell microscopic high-content analyzers.

[0009] The dual-mode imaging sensor processing chip for cell microscopy achieves a four-in-one integration of "optics-sensing-electronics-intelligence". Its system-level application manufacturing needs to solve the following problems: the manufacturing of intraoperative navigation endoscopes requires the dual-mode imaging chip to be integrated into the front end of the surgical instrument in a cylindrical miniaturized package (outer diameter ≤3mm), and engineering problems such as fiber optic light source coupling, real-time navigation image generation, and tumor cell edge display superposition need to be solved; the development of cell microscopy high content analysis instruments requires the dual-mode imaging chip to be placed in a three-dimensional rectangular package under the sample pool, and combined with modules such as multi-well plate automatic sample introduction, closed-loop environmental control, multi-parameter quantitative analysis, and pharmacodynamic evaluation, and solve engineering problems such as complex optical structure, difficulty in analyzing and storing massive amounts of data, phototoxicity, and photobleaching in the instrument development.

[0010] The core of the aforementioned system-level application relies on the independent caches and microprocessors of the first and second imaging chips of the dual-mode imaging chip to achieve parallel data computation, ensuring mode switching latency ≤300ns and real-time performance.

[0011] In this invention, a chiplet, a modular chip design method, refers to optoelectronic heterogeneous dies manufactured using different integrated circuit processes, which are then integrated into a system-on-a-chip through 2.5D / 3D packaging processes.

[0012] A dual-mode imaging sensor chip for cell microscopy was designed using optoelectronic heterogeneous bare dies, hereinafter referred to as: dual-mode imaging chip.

[0013] The technical solution of this invention is as follows: A dual-mode imaging chip includes a first imaging chip, a second imaging chip, and an imaging servo chip, wherein the first imaging chip, the second imaging chip, and the imaging servo chip are packaged on the same two-dimensional planar carrier. The first imaging chip is vertically stacked and integrated from multiple photoelectric heterogeneous chips, used for bright-field imaging signal acquisition, image processing, and bio-information label fusion. The second imaging chip is also vertically stacked and integrated from multiple photoelectric heterogeneous chips, used for fluorescence imaging digital signal acquisition and time-correlated single-photon counting. The AI ​​inference chip integrated in the internal time-domain processing chip is responsible for cell morphology classification and cancer cell identification. The imaging servo chip is composed of multiple heterogeneous chips, used for mode switching between bright-field imaging mode and fluorescence imaging mode and I / O interface control, as well as for inter-chip clock synchronization, energy distribution, and data path reconstruction.

[0014] According to a preferred embodiment of the present invention, the first imaging chip, the second imaging chip, and the imaging servo chip are heterogeneously integrated into the same package through a standardized die-to-die interface, and the multiple optoelectronic heterogeneous functional chips inside the first imaging chip, the second imaging chip, and the imaging servo chip are packaged using 2.5D or 3D chiplet packaging technology.

[0015] According to a preferred embodiment of the present invention, the clock synchronization function of the imaging servo chip is as follows: a multi-phase clock tree is generated through an on-chip PLL to provide a common clock for the first imaging chip, the second imaging chip, and the external interface, thereby eliminating clock drift between chips; the power consumption allocation function is as follows: the power supply voltage and operating frequency of each chip are adjusted in real time according to the current imaging mode through DVFS dynamic voltage and frequency adjustment technology, reducing the power consumption of the first imaging chip in bright field mode and reducing the power consumption of the second imaging chip in fluorescence mode; the data path reconstruction function is as follows: data routing between chips is realized through a high-speed chiplet interconnect channel.

[0016] An intraoperative navigation endoscope, employing the aforementioned dual-mode imaging chip, includes: The probe module, namely the dual-mode imaging chip, is packaged in a cylindrical three-dimensional stack and placed at the front end of the surgical instrument. The first imaging chip is used for large-field tissue localization, the second imaging chip is used for cell-level morphology analysis, and the imaging servo chip is used to drive the switching between bright field imaging mode and fluorescence imaging mode in real time. The light source module integrates a white laser LED and a four-color laser LED. The white laser LED is used for bright field illumination, and the wavelength of the four-color laser LED is matched with the blue, green, yellow, and red fluorescent filters of the dual-mode imaging chip. The display module includes a second image fusion unit and a cancer boundary recognition unit. The second image fusion unit is used to superimpose the navigation image data stream output by the first imaging core and the fluorescence detection data stream output by the second imaging core at the pixel level to display the cancer boundary probability heat map and fluorescence lifetime map, thereby realizing intraoperative navigation. Cancer boundary recognition unit, used to identify cancer boundaries.

[0017] According to a preferred embodiment of the present invention, the cylindrical package forms a rigid coaxial connection with the front end of the surgical instrument.

[0018] According to a preferred embodiment of the present invention, the data parallel computing architecture of the first imaging chip is as follows: the first imaging chip has an independent frame buffer and an embedded microprocessor built in, and completes the acquisition of color video signals, Bayer decoding, white balance correction and image compression at the front end of the probe, and outputs the compressed navigation image data stream. The data parallel computing architecture of the second imaging chip is as follows: The second imaging chip has an independent SPAD array photosensitive area, a four-color filter array, a single photon counting buffer and a time-to-digital conversion microprocessor. It completes the timestamp recording of fluorescent photons, fluorescence lifetime histogram construction, lifetime fitting and carcinogenesis probability inference calculation at the front end of the probe, and outputs a carcinogenesis boundary probability heat map and a fluorescence lifetime map to achieve in-situ intelligent analysis with cell-level precision. The imaging servo core mode switching control includes: the imaging servo core integrates a light source control circuit and a hardware timing controller, which controls the mode switching between bright field imaging mode and fluorescence imaging mode through line and field drive signals, with a switching delay of no more than 300ns.

[0019] According to a preferred embodiment of the present invention, the four-color laser LED includes four bands: 450nm±10nm, 520nm±10nm, 590nm±10nm, and 650nm±10nm. Each band is pulse-modulated by independent current driving, and the pulse frequency is synchronized with the gate window of the SPAD array.

[0020] According to a preferred embodiment of the present invention, intraoperative fluorescence lifetime navigation is achieved through a second imaging chip; including: using a closed-loop control method of photostimulation and fluorescence response to collect cell fluorescence lifetime information in real time during tumor resection: after the photostimulation pulse is emitted, the fluorescence emitted by the tissue cells triggers the SPAD array to enter a high temporal resolution detection window; the time-to-digital converter microprocessor built into the second imaging chip records a timestamp for each photon pulse, constructs a time-photon number histogram in the local cache of the chip, and fits the fluorescence lifetime curve; when the fluorescence lifetime value deviates from the normal tissue baseline value τ0 by more than the threshold Δτ (0.5~5ns), the inference microprocessor built into the second imaging chip determines that the corresponding area is cancerous tissue, and marks it with red highlight on the displayed image to guide the surgical resection boundary.

[0021] Further preferred, the data interface includes USB 3.1 Gen2 and 10GigE Vision.

[0022] A high-content cell microscopy analyzer, employing the aforementioned dual-mode imaging chip, includes: The microscopic imaging module, also known as the dual-mode imaging chip, is packaged in a three-dimensional rectangular package and placed below the sample pool. The first imaging chip performs low-magnification global localization scanning of each well, while the second imaging chip performs high-magnification fluorescence imaging of the target well. The imaging servo chip is used to switch the well synchronization mode. Environmental control module; used for closed-loop PID temperature control, infrared CO2 concentration monitoring and humidity control; High-content analysis module; used for cell segmentation and tracking, multi-parameter quantitative analysis, and pharmacodynamic evaluation; The data management module is used to store raw images, analysis results, experimental metadata, and blockchain evidence.

[0023] According to a preferred embodiment of the present invention, the environmental control module includes a PID temperature controller, a CO2 concentration monitor, and a humidity controller; The system is equipped with closed-loop PID temperature control and infrared CO2 concentration monitoring. The temperature is controlled at 37℃±0.5℃, the CO2 concentration is controlled at 5±0.2%, and the humidity is maintained at no less than 85% RH through water bath humidification to ensure long-term imaging of live cells.

[0024] According to a preferred embodiment of the present invention, the first imaging chip performs a low-magnification global localization scan on each well location; including: the first imaging chip uses an independent frame buffer and a microprocessor to quickly scan the entire multi-well plate, completes image preprocessing and well edge detection locally on the chip, and outputs the coordinate index and contour mask of each well location to provide localization guidance for fluorescence lifetime imaging of the second imaging chip; The second imaging chip performs high-magnification fluorescence imaging of the target pore locations; including: the second imaging chip receives the coordinate index of each pore location output by the first imaging chip, and performs multi-channel parallel fluorescence acquisition of the target area of ​​each pore; the SPAD array photosensitive area, together with the four-color filter array, realizes four-channel parallel fluorescence detection, and the photon count data of each channel is buffered in the independent buffer area of ​​the second imaging chip, and the embedded microprocessor performs fluorescence intensity integration, fluorescence lifetime fitting and cell segmentation operations in parallel, and outputs a multi-dimensional cell phenotypic feature vector composed of cell area, contour perimeter, texture feature parameters, fluorescence intensity and fluorescence lifetime; Cross-frame trajectory reconstruction of AI inference cores; including: the AI ​​inference core integrated in the temporal processing core inside the second imaging core receives the multidimensional cell phenotypic feature vector, performs morphological classification and cross-frame trajectory reconstruction: cross-frame identity preservation for individual cells, using the Hungarian algorithm to match cell positions and morphological similarities in adjacent frames, reconstructing cell movement trajectories and lineage differentiation paths; outputting cell phenotypic feature vectors and trajectory data for dynamic drug response analysis; The imaging servo core's aperture position synchronization mode switching includes: the imaging servo core switches between bright-field imaging mode and fluorescence imaging mode through hardware timing control, and the switching timing is synchronized with the mechanical displacement cycle of the multi-well plate; after the first imaging core completes the bright-field positioning scan of a certain aperture position, the imaging servo core triggers the mode switch, and the second imaging core then enters the fluorescence imaging mode to perform single-photon digital signal acquisition; after the acquisition is completed, it switches back to the bright-field mode, and the displacement stage moves to the next aperture position.

[0025] According to a preferred embodiment of the present invention, the independent buffer of the first imaging chip includes a Bayer raw frame buffer and a compressed bitstream buffer, and the microprocessor of the first imaging chip includes an image signal processing (ISP) and a JPEG encoder, which perform parallel operations of bright-field image acquisition, decoding, correction and compression, and output a compressed navigation image data stream. The independent cache of the second imaging chip includes a SPAD photon counting cache and a lifetime histogram cache. The microprocessor includes a TDC time-to-digital converter, a lifetime fitting engine and an inference accelerator, which performs parallel operations of fluorescence photon detection, timestamps, histograms, data fitting and cell classification, and outputs fluorescence lifetime maps, cancer probability maps and phenotypic feature vectors. The imaging servo chip acts as a central scheduler, coordinating the frame buffer access of the first and second imaging chips through a time-division multiplexing bus. It transmits the parallel output data of the first and second imaging chips to the back end after pixel-level registration through a unified fusion output interface.

[0026] According to a preferred embodiment of the present invention, the high-content analysis module includes a cell segmentation and tracking unit, a multi-parameter quantitative analysis unit, and a pharmacodynamic evaluation unit; Cell segmentation and tracking are achieved through a cell segmentation and tracking unit, including: multi-dimensional cell phenotypic feature vectors output by the second imaging core, cross-frame trajectory reconstruction of the AI ​​inference core, identity preservation of individual cells, and reconstruction of cell movement trajectory and lineage differentiation path; Multi-parameter quantitative analysis is achieved through a multi-parameter quantitative analysis unit, including: extracting five parameters: cell area, contour perimeter, texture feature parameters, fluorescence intensity, and fluorescence lifetime, and constructing a multidimensional cell phenotypic feature vector; Pharmacokinetic evaluation is performed using a pharmacokinetic assessment unit, including: generating a compound concentration-response curve using a four-parameter logistic model based on a time-varying parameter set, and calculating the half-maximal effective concentration (EC50). 50 With half-maximal inhibitory concentration (IC50) 50 .

[0027] The beneficial effects of this invention are as follows: 1. Intraoperative navigation with cellular-level precision. Through the single-photon sensitivity of the second imaging core and the in-situ inference of the microprocessor embedded in the core, the cancer boundary identification accuracy reaches the cellular level (<10μm), which is two orders of magnitude higher than traditional tissue-level navigation (>1mm).

[0028] 2. Miniaturized probe. The cylindrical package has an outer diameter of ≤3mm, which is 40% smaller than the traditional system (>5mm), allowing it to enter narrow cavities in neurosurgery, ophthalmology, and other fields.

[0029] 3. Real-time fluorescence lifetime navigation. The second imaging core is independently cached and the microprocessor enables in-situ fitting of fluorescence lifetime, with a mode switching delay of ≤300ns, which is 7 orders of magnitude better than traditional offline fitting (>10min).

[0030] 4. Improved throughput for high-content analysis. The parallel computing architecture of the first and second imaging cores enables full-panel scanning of a 96-well plate in less than 5 minutes, a 6-fold improvement over traditional sCMOS solutions (>30 minutes).

[0031] 5. Long-term preservation of viable cells. The environmental control module maintains 37±0.5℃ / 5±0.2%CO2, and the cell viability is >95% (observed for 72 hours).

[0032] 6. Data compliance and trusted computing. Blockchain-based evidence storage combined with SM2 electronic signature (a national cryptographic standard) meets the NMPA requirements for medical device software registration.

[0033] 7. Optimization of manufacturing and R&D costs. The same dual-mode imaging chip, through different packaging forms and firmware configurations, can be adapted to the manufacturing of intraoperative navigation endoscopes and the development of high-content cell microscopy analysis instruments, thereby reducing hardware chip costs and software development redundancy.

[0034] 8. Flexible application adaptation. Through firmware configuration, the same dual-mode imaging chip can be used in the manufacturing of other endoscopes such as laparoscopy and capsule endoscopy, as well as in the research and development of bioanalytical instruments. This simplifies the hardware design without requiring modifications, while optimizing performance, power consumption, and cost. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the dual-mode imaging chip being installed at the tip of an intraoperative navigation endoscope (Chip on Tip).

[0036] Figure 2 This is a schematic diagram of the installation of a dual-mode imaging chip in a high-content cell microscopy analyzer.

[0037] Figure 1 and Figure 2 In the middle, 61, dual-mode imaging chip, 11, first imaging chip, 12, second imaging chip, 13, imaging servo chip; Figure 3This is a diagram of the overall architecture of an intraoperative navigation endoscope based on a dual-mode imaging chip. in, 1. Probe module; 2. Control module; 21. Cancer boundary recognition unit; 22. First image fusion unit; 23. Physiological parameter detection unit; 24. AI inference core; 25. Temporal processing core; 26. Hardware timing controller. 3. Light source module, 31. White laser LED, 32. Four-color laser LED; 4. Display module; 41. Second image fusion unit; 42. Cancerous boundary recognition unit; 5. Data interface module, 51. USB 3.1 Gen2 interface, 52. 10GigE Vision video interface.

[0038] 6. Power supply module; Figure 4 This is a diagram of the overall architecture of a cell microscopic high-content analyzer based on a dual-mode imaging chip. in, 71. Automatic sample injection module; 72. Microscopic imaging module; 721. Cell culture well plate; 73. Environmental control module; 731. PID temperature controller; 732. CO2 concentration monitor; 733. Humidity controller; 74. High-content analysis module; 741. Cell segmentation and tracking unit; 742. Multi-parameter quantitative analysis unit; 743. Pharmacokinetic Evaluation Unit; 745. Bioinformatics Extraction Unit; 746. Cell Image Processing Unit; 75. Data Management Module; 751. Raw Image Storage Module; 752. Analysis Result Storage Module; 753. Experimental Setup Data Storage Module; 754. Blockchain Evidence Storage Module. 76. Peripheral device interface module. Detailed Implementation

[0039] The present invention will be further defined below with reference to the accompanying drawings and embodiments, but is not limited thereto.

[0040] Example 1 A dual-mode imaging chip includes a first imaging chip 11, a second imaging chip 12, and an imaging servo chip 13, wherein the first imaging chip 11, the second imaging chip 12, and the imaging servo chip 13 are packaged on the same two-dimensional planar carrier. The first imaging core 11 is vertically stacked and integrated from multiple cores with heterogeneous optoelectronic functions, and is used for signal acquisition, image processing, and bio-information label fusion in bright-field imaging mode. The second imaging core 12 is also vertically stacked and integrated from multiple cores with heterogeneous optoelectronic functions, and is used for digital signal acquisition and time-correlated single-photon counting in fluorescence imaging. The AI ​​inference core integrated in the internal time-domain processing core is responsible for cell morphology classification and cancer cell identification. The imaging servo core 13 is composed of multiple heterogeneous cores and is used for mode switching between bright-field imaging mode and fluorescence imaging mode and I / O interface control, as well as for clock synchronization, energy distribution, and data path reconstruction between cores.

[0041] Example 2 The difference between the dual-mode imaging chip described in Example 1 and the one in Example 1 is as follows: The first imaging chip 11, the second imaging chip 12, and the imaging servo chip 13 are heterogeneously integrated into the same package through a standardized die-to-die interface. The multiple optoelectronic heterogeneous functional chips and multiple heterogeneous chips inside the first imaging chip 11, the second imaging chip 12, and the imaging servo chip 13 are packaged using 2.5D or 3D chiplet packaging technology.

[0042] This invention uses a dual-mode imaging chip 61 as the core imaging sensor processor, and adapts to two application scenarios—the manufacturing of intraoperative navigation endoscopes and the development of high-content cell microscopy analysis instruments—through different packaging forms and firmware configurations.

[0043] The clock synchronization function of imaging servo chip 13 is as follows: A multi-phase clock tree is generated through an on-chip PLL to provide a common clock for the first imaging chip 11, the second imaging chip 12, and external interfaces, eliminating clock drift between chips. The power allocation function is as follows: Dynamic voltage and frequency adjustment technology (DVFS) is used to adjust the power supply voltage and operating frequency of each chip in real time according to the current imaging mode (bright field / fluorescence). In bright field mode, the power consumption of the first imaging chip 11 is reduced, and in fluorescence mode, the power consumption of the second imaging chip 12 is reduced. The data path reconstruction function is as follows: Data routing between chips is achieved through a high-speed chiplet interconnect channel. In intraoperative navigation scenarios, low-latency video stream transmission is prioritized; in high-content analysis scenarios, parallel throughput of batch data is prioritized.

[0044] The frame buffers of the first imaging core 11 and the second imaging core 12 exist independently, enabling time-division sampling of light and parallel data computation; the biomarker information and bright-field image information are fused together and then output to the back-end display.

[0045] The imaging servo chip 13 integrates a white laser LED7 and a four-color laser LED8. The white laser LED7 is used for bright field illumination, and the four-color laser LED8 is used for fluorescence excitation. The wavelengths of the four-color fluorescence excitation light source are matched with the blue, green, yellow, and red fluorescence filters of the dual-mode imaging chip 61. Each wavelength is driven by an independent current to achieve pulse modulation. The DVFS-μTEC collaborative temperature control module is configured to ensure the thermal stability of the LED light source and SPAD array during long-term surgery.

[0046] The first imaging core 11 comprises, from top to bottom: a first optical core, a photoelectric sensing core, and an electrical core; the first optical core is located at the top of the photoelectric sensing layer and includes an imaging microlens array core, a focusing microlens array core, and a filter microarray core; the imaging microlens array core participates in optical imaging, the focusing microlens array core is used to increase light throughput, and the filter array core includes red, green, and blue three-color filters and blue, green, yellow, and red four-color fluorescent filters to achieve imaging of different spectra; the photoelectric sensing core includes a first sensing core and a second sensing core, the first... The first sensing chip is a CMOS image sensor photodiode (PD) array chip used for light intensity imaging; the second sensing chip is a single-photon avalanche diode (SPAD) array chip used for high-sensitivity fluorescence lifetime imaging; the electrical chip is vertically stacked and integrated with the first sensing chip in three dimensions, including an analog front-end chip (AFE), an analog-to-digital converter (ADC) chip, an image processing chip (ISP), a data buffer chip, and a video generation chip. The analog front-end chip is used to amplify the output signal of the photosensitive element; the ADC chip uses a Σ-Δ ADC for signal analog-to-digital conversion; the image processing chip is used for noise reduction, sharpening, and white balance processing; the data buffer chip is used for high-speed data storage and access; and the video generation chip is used to generate cell image videos at a frame rate of 30-60 frames per second.

[0047] The second imaging chip 12 includes a second optical chip, a second sensing chip, and a time-domain processing chip. The second optical chip is located directly above the photoelectric sensing chip and contains two optical functional layers: an aspherical microlens layer chip and a multi-band fluorescence filter layer. The second sensing chip is a single-photon avalanche diode (SPAD) array chip. The time-domain processing chip and the second sensing chip are vertically stacked via wafer-level Cu-Cu hybrid bonding with a bonding spacing of <10μm. It includes a TDC (Time-to-Digital Conversion) chip, a time-correlated single-photon counting chip, a data buffer chip, a histogram data processing chip, a fluorescence image generation array chip, a point cloud processing chip, an AI inference chip, and an I / O interface chip.

[0048] The imaging servo chip 13 consists of a light source chip, a light source control chip, a temperature monitoring chip, and a micro thermoelectric cooler temperature control chip, and is manufactured using 3D chiplet packaging technology.

[0049] Example 3 An intraoperative navigation endoscope, using the dual-mode imaging chip 61 described in Example 1 or 2, such as... Figure 1 and Figure 3 As shown, a dual-mode imaging chip 61 is configured to run in intraoperative navigation application scenarios via firmware configuration, performing tissue morphology observation and fluorescently labeled boundary recognition for tumor resection surgical navigation; including: The probe module 1 (miniaturized probe with an outer diameter ≤3mm), namely the dual-mode imaging chip 61, is a cylindrical three-dimensional stacked package with an outer diameter not exceeding 3mm, and is positioned at the front end of the surgical instrument. The first imaging core 11 performs large-field tissue localization, and the second imaging core 12 performs cell-level morphological analysis, accomplished by the cancer boundary recognition unit 21, the physiological parameter detection unit 23, and the AI ​​inference core 24 (deep learning semantic segmentation network - cancer probability map output). The imaging servo core 13 drives real-time switching between bright-field imaging mode and fluorescence imaging mode.

[0050] The light source module 3 integrates a white laser LED 31 (400-700nm) and a four-color laser LED 32 (405 / 488 / 561 / 640 nm). The white laser LED 31 is used for bright field illumination, and the light output stabilization time is no longer than 10μs. The wavelength of the four-color laser LED 32 is matched with the blue, green, yellow, and red fluorescent filters of the dual-mode imaging chip 61. The display module 4 includes a second image fusion unit 41 and a cancer boundary recognition unit 42, which presents navigation images at a refresh rate of not less than 60fps. The second image fusion unit 41 is used to superimpose the navigation image data stream output by the first imaging core 11 and the fluorescence detection data stream output by the second imaging core 12 at the pixel level to display the cancer boundary probability heat map and fluorescence lifetime map, thereby realizing intraoperative navigation. The cancer boundary probability heat map is triggered by the fluorescence lifetime deviation threshold Δτ, and Δτ is dynamically set according to the preoperative normal tissue baseline value τ0, where Δτ∈[0.5ns,5ns].

[0051] The cancer boundary recognition unit 42 is used to identify the cancer boundary; it serves as a guide for doctors to remove tissue cells during surgical navigation, and the surgical resection is carried out according to the cancer boundary to avoid incomplete removal of cancer cells and recurrence, while also avoiding excessive resection and damage to normal tissue.

[0052] Control module 2 includes: a cancer boundary recognition unit 21, a first image fusion unit 22, a physiological parameter detection unit 23, an AI inference core 24 (a deep learning semantic segmentation network for outputting cancer probability maps), a temporal processing core 25 (fluorescence lifetime fitting → metabolic parameter extraction), and a hardware timing controller 26 (line and field drive mode switching). The synchronous control signal of the line and field drive controls the linkage of these units to perform data sampling, calculation, and parameter extraction. All of these operations are existing technologies.

[0053] Data interface module 5 communicates with peripheral devices, and power module 6 is responsible for supplying energy to each module.

[0054] The cylindrical package forms a rigid coaxial connection with the front end of the surgical instrument. The connection interface complies with the ISO 10993 biocompatibility standard for medical devices. The packaging material uses biocompatible epoxy resin and a medical-grade stainless steel shell, and the surface is plasma-treated to reduce protein adsorption.

[0055] Example 4 The intraoperative navigation endoscope according to Example 3 differs in that: The first imaging core 11 acquires optical signals from the tissue surface during surgery to generate a large-field-of-view color navigation image; the second imaging core 12 performs four-channel parallel fluorescence detection, and its internal AI inference core outputs cell morphology classification results and cancer boundary probability in real time; the imaging servo core 13 integrates a light source control circuit, and realizes mode switching between bright field imaging mode and fluorescence imaging mode through a hardware timing controller, with a switching delay ≤300ns; the display module 4 presents the navigation image at a refresh rate of not less than 60fps, and overlays a cancer boundary probability heatmap and a fluorescence lifetime map; the cancer boundary probability heatmap is triggered by the fluorescence lifetime deviation threshold Δτ, Δτ is dynamically set according to the preoperative normal tissue baseline value τ0, Δτ∈[0.5ns, 5ns]; The data parallel computing architecture of the first imaging chip 11 is as follows: The first imaging chip 11 has an independent frame buffer and an embedded microprocessor built in, which completes the acquisition of color video signals (4K@60fps), Bayer decoding, white balance correction and image compression at the front end of the probe, and outputs the compressed navigation image data stream; there is no need to wait for back-end processing, which greatly reduces the data transmission bandwidth requirements and processing latency.

[0056] The data parallel computing architecture of the second imaging core 12 is as follows: The second imaging core 12 has an independent SPAD array photosensitive area, a four-color filter array, a single photon counting buffer and a time-to-digital conversion microprocessor. It completes the timestamp recording of fluorescent photons, fluorescence lifetime histogram construction, lifetime fitting and carcinogenesis probability inference calculation at the front end of the probe, and outputs a carcinogenesis boundary probability heat map and a fluorescence lifetime map to achieve in-situ intelligent analysis with cell-level precision. The imaging servo chip 13 controls mode switching, including integrating a light source control circuit and a hardware timing controller. It precisely controls the switching between bright-field imaging and fluorescence imaging modes via line and field drive signals, with a switching delay of no more than 300 ns. The dual-mode imaging chip 61 has two imaging modes, but the data processing of the first imaging chip 11 and the second imaging chip 12 is independent and parallel, without blocking each other. The mode switching is achieved by the imaging servo chip 13 through hardware timing control, unaffected by firmware version.

[0057] The imaging servo chip 13 communicates with the host computer via a data interface, which is configured as a USB 3.1 Gen2 interface 51 or a 10GigE Vision video interface 52. The bandwidth of the raw data stream in the intraoperative navigation application scenario is no higher than 60% of the rated bandwidth of the data interface.

[0058] The manufacturing configuration table for intraoperative navigation endoscopes is shown in Table 1: Table 1 The four-color laser LED 32 includes four bands: 450nm±10nm, 520nm±10nm, 590nm±10nm, and 650nm±10nm. Each band is driven by an independent current to achieve pulse modulation, and the pulse frequency is precisely synchronized with the gate window of the SPAD array.

[0059] Intraoperative fluorescence lifetime navigation is achieved through the second imaging core 12. This includes: using a closed-loop control method based on photostimulation and fluorescence response to acquire cell fluorescence lifetime information in real time during tumor resection. After the photostimulation pulse is emitted, the fluorescence emitted by the tissue cells triggers the SPAD array to enter a high temporal resolution detection window (50 ps accuracy). The time-to-digital converter microprocessor built into the second imaging core 12 records a timestamp for each photon pulse, constructs a time-photon count histogram in the core's local cache, and fits the fluorescence lifetime curve. When the fluorescence lifetime value deviates from the normal tissue baseline value τ0 by more than the threshold Δτ (0.5~5 ns), the inference microprocessor built into the second imaging core 12 determines that the corresponding area is cancerous tissue and highlights it in red on the displayed image to guide the surgical resection boundary. All the above calculations for fluorescence lifetime detection, histogram construction, fluorescence lifetime fitting, and cancer determination are completed in the independent cache and microprocessor of the second imaging core 12, and the data does not need to be transferred off-film, ensuring real-time performance.

[0060] Firmware configuration for intraoperative navigation application scenarios: The chip is packaged in a cylindrical shape with an outer diameter ≤3mm; the first imaging chip 11 is enabled to continuously output 60fps color video; the second imaging chip 12 is enabled to trigger-type fluorescence lifetime detection; the light source strategy is constant white light + fluorescence pulse trigger; the trigger source for start, stop, and reset is the surgical process or surgeon's command; the load control of the data interface module 5 is set so that the original data stream bandwidth is ≤60% of the interface's rated bandwidth.

[0061] The data interface module 5 includes a USB 3.1 Gen2 interface 51 and a 10GigE Vision video interface 52.

[0062] The surgical navigation process includes: Preoperative calibration: Fluorescence lifetime measurements were performed on normal tissue surrounding the tumor at 5 sites, τ0 = 2.5 ± 0.3 ns (NADH standard), and the data were stored in the second imaging core 12 buffer as a baseline; Intraoperative navigation: The probe is inserted into the surgical cavity of the tumor. The first imaging core 11 continuously outputs color video (60fps), and the second imaging core 12 is triggered in parallel to acquire fluorescence lifetime. Cancer identification: When the built-in microprocessor of the second imaging core 12 detects τ=1.2ns (deviation from τ0 1.3ns>Δτ=1.0ns), it is determined in situ to be cancerous tissue, and a red highlight is output to the display module 4; Precise sampling: biopsy forceps channel, sampling volume 0.8mm³, sent for frozen pathology verification; Boundary confirmation: After excision, a second scan confirmed that no red-marked residue remained in the core 12.

[0063] Clinical results include: a 30% reduction in operation time (eliminating the need for repeated re-examination); a decrease in tumor residual rate from 15% to <5%; and a 20% improvement in nerve function preservation rate (due to precise resection avoiding excessive damage).

[0064] Example 5 A high-content cell microscopy analyzer, such as Figure 2 As shown. The dual-mode imaging chip 61 described in Example 1 or 2 is applied.

[0065] The firmware is configured to run in high-content cell microscopy analysis applications. The overall architecture of the high-content cell microscopy analyzer is as follows: Figure 4 As shown, the dual-mode imaging chip 61, in conjunction with the automatic sample introduction module 71 (mechanical part of a high-content cell microscopy analyzer, general technology), performs multi-parameter quantitative analysis of cells and whole-plate scanning for high-throughput screening in drug development; including:

[0066] The microscopic imaging module 72, also known as the dual-mode imaging chip 61, is a three-dimensional rectangular package placed below the sample cell. The rectangular three-dimensional structure measures 20mm × 15mm and is compatible with 96-well or 384-well plate automated sample loading systems. The first imaging chip 11 performs low-magnification global localization scanning of each well, while the second imaging chip 12 performs high-magnification fluorescence imaging of the target wells. The imaging servo chip 13 enables synchronous switching of well positions.

[0067] Environmental control module 73; used for closed-loop PID temperature control, infrared CO2 concentration monitoring and humidity control; High-content analysis module 74; used for cell segmentation and tracking, multi-parameter quantitative analysis, and pharmacodynamic evaluation; The data management module 75 is used to store raw images (TIFF format, lossless compression), analysis results (XML format, structured annotation), experimental metadata (JSON format, conforming to the OMERO standard), and blockchain evidence storage. The experimental metadata storage complies with FDA 21 CFR Part 11 electronic record specifications, employs blockchain evidence storage and the national cryptographic standard SM2 electronic signature, and meets the NMPA medical device software registration application requirements. The data management module includes: raw image storage module 751 (TIFF format lossless compression), analysis result storage module 752 (XML format structured annotation), experimental setting data storage module 753 (JSON format OMERO standard), and blockchain evidence storage module 754 (national cryptographic standard SM2 electronic signature).

[0068] The automatic sample loading module 71 is used to perform image microscopy imaging of the culture chamber. The automatic sample loading module 71 includes a 96 / 384-well plate stage (cell culture chamber) and a precision displacement stage; it takes pictures of each well, with a well-to-well switching time of less than 2 seconds per well. The automatic sample loading module 71 is a common structure in high-content cell microscopy analyzers and is existing technology. The automatic sample loading module automatically delivers the 96 / 384-well plate to the cell culture incubator, the precision displacement stage moves the plate, and takes pictures of each well, completing the image microscopy imaging of the 96 / 384-well culture chamber.

[0069] Peripheral device interface module 76 is used to connect to workstations and other supporting equipment of the cell microscopic high content analyzer.

[0070] Example 6 The difference between the cell microscopic high-content analyzer described in Example 5 and the one described in Example 5 is as follows: The environmental control module 73 includes a PID temperature controller 731 (37±0.5℃, PID closed loop), a CO2 concentration monitor 732 (5±0.2%, infrared sensor feedback), and a humidity controller 733; The dual-mode imaging chip 61 performs microscopic imaging of cells cultured in the cell culture plate 721. The cell culture environment is regulated by the environmental control module 73, which adjusts the temperature and CO2 concentration required for cell culture. During cell culture, the cell's ecological parameters are analyzed by a conventional AI module, and the generated analysis results are output to the storage unit along with the image data for later use.

[0071] The system is equipped with closed-loop PID temperature control and infrared CO2 concentration monitoring. The temperature is controlled at 37℃±0.5℃, and the temperature fluctuations recover to the set value ±0.5℃ within 30 seconds. The CO2 concentration is controlled at 5±0.2% (volume percentage), and the CO2 concentration fluctuations recover to the set value ±0.2% within 60 seconds. The humidity is maintained at no less than 85% RH through water bath humidification to ensure long-term imaging of live cells (>72h).

[0072] The first imaging core 11 performs a low-magnification global positioning scan of each well location; this includes: the first imaging core 11 rapidly scans the entire 96 / 384-well plate, performs image preprocessing and well edge detection locally on the core, and outputs the coordinate index and contour mask of each well location (the specific implementation process is a well-known technology in the industrial and medical measurement industries), providing positioning guidance for the fluorescence lifetime imaging of the second imaging core 12; this process does not require the participation of a back-end processor, and the scanning speed is only limited by the mechanical displacement stage.

[0073] The second imaging core 12 performs high-magnification fluorescence imaging of the target pores; including: the second imaging core 12 receives the coordinate index of each pore output by the first imaging core 11, and performs multi-channel parallel fluorescence acquisition of the target area of ​​each pore; the SPAD array photosensitive area (generating SPAD data signal) works with the four-color filter array to realize four-channel parallel fluorescence detection, and the photon count data of each channel is buffered in the independent buffer area of ​​the second imaging core 12, and the embedded microprocessor performs fluorescence intensity integration, fluorescence lifetime fitting and cell segmentation operations in parallel, and outputs a multi-dimensional cell phenotypic feature vector composed of cell area, contour perimeter, texture feature parameters, fluorescence intensity and fluorescence lifetime; Cross-frame trajectory reconstruction of AI inference core 24; including: receiving multi-dimensional cell phenotypic feature vectors, performing morphological classification and cross-frame trajectory reconstruction by the AI ​​inference core 24 integrated within the temporal processing core of the second imaging core 12, performing cross-frame identity preservation for individual cells, using the Hungarian algorithm to match cell positions and morphological similarities in adjacent frames, reconstructing cell movement trajectories and lineage differentiation paths; outputting cell phenotypic feature vectors and trajectory data for dynamic drug response analysis; The imaging servo chip 13 performs aperture position synchronization mode switching, including: the imaging servo chip 13 switches between bright field imaging mode and fluorescence imaging mode through hardware timing control, and the switching timing is synchronized with the mechanical displacement cycle of the multi-well plate; after the first imaging chip 11 completes the bright field positioning scan of one aperture position, the imaging servo chip 13 triggers mode switching, and the second imaging chip 12 then enters the fluorescence imaging mode to perform single-photon digital signal acquisition; after the acquisition is completed, it switches back to the bright field mode, and the displacement stage moves to the next aperture position; this pipeline is autonomously scheduled by the hardware of the imaging servo chip 13, without the need for intervention from the host computer for each aperture.

[0074] The first imaging chip 11 and the second imaging chip 12 are each configured with independent frame buffers and embedded microprocessors, forming a data-parallel computing architecture. The independent buffer of the first imaging chip 11 includes a Bayer raw frame buffer and a compressed bitstream buffer. The microprocessor of the first imaging chip 11 includes an image signal processing (ISP) and a JPEG encoder, which perform parallel operations of bright-field image acquisition, decoding, correction, and compression, and output a compressed navigation image data stream.

[0075] The independent cache of the second imaging chip 12 includes a SPAD photon counting cache and a lifetime histogram cache. The microprocessor includes a TDC time-to-digital converter, a lifetime fitting engine and an inference accelerator, which performs parallel operations of fluorescence photon detection, timestamp, histogram, data fitting and morphological classification, and outputs fluorescence lifetime map, cancer probability map and phenotypic feature vector. The imaging servo core 13 acts as a central scheduler, coordinating frame buffer access between the first imaging core 11 and the second imaging core 12 via a time-division multiplexing bus. It transmits the parallel output data of the first imaging core 11 and the second imaging core 12 to the backend after pixel-level registration via a unified fusion output interface. Data processing by the two cores is entirely parallel, with no waiting time between them, ensuring system real-time performance.

[0076] The high-content analysis module 74 includes a cell segmentation and tracking unit 741, a multi-parameter quantitative analysis unit 742, a pharmacodynamic evaluation unit 743, a bioinformatics extraction unit 745 (cell area, contour perimeter, texture feature parameters, fluorescence intensity, fluorescence lifetime → multidimensional cell phenotypic feature vector), and a cell image processing unit 746 (image 3D reconstruction, coloring, denoising, and display); system-level features and compliance design; including: full-plate scanning time <5 min for 96-well plates and <15 min for 384-well plates; long-term viable cell protection >72 h; viability >95%; data compliance: FDA 21 CFR part11 / NMPA; Cell segmentation and tracking are achieved through cell segmentation and tracking unit 741; including: based on the multidimensional cell phenotypic feature vector output by the second imaging core 12, AI inference core 24 cross-frame trajectory reconstruction, identity preservation of individual cells, and reconstruction of cell movement trajectory and lineage differentiation path; Multi-parameter quantitative analysis is achieved through the multi-parameter quantitative analysis unit 742, including: extracting five parameters: cell area, contour perimeter, texture feature parameters, fluorescence intensity, and fluorescence lifetime, and constructing a multidimensional cell phenotypic feature vector; Pharmacokinetic evaluation is performed using pharmacokinetic evaluation unit 743, including: generating a compound concentration-response curve using a four-parameter logistic model based on a time-varying parameter set, and calculating the half-maximal effective concentration (EC50). 50 With half-maximal inhibitory concentration (IC50) 50 The four-parameter Logistic model includes four fitting parameters: minimum effect concentration, maximum effect concentration, half-effect concentration, and Hill coefficient.

[0077] Hardware configuration and workflow for high-content cell microscopy analysis: The chip is packaged as a three-dimensional rectangle and placed below the sample pool; the first imaging chip 11 is used for low-magnification scanning and positioning; the second imaging chip 12 is used for full-well multi-channel parallel acquisition; the light control strategy is multi-color parallel excitation and fluorescence filter filtering; the mode switching trigger source is well position index or stage beat; the whole plate scan data is transmitted in a lossless compression format.

[0078] Extended application scenario hardware configurations: For laparoscopic applications, the chip is packaged in a cylindrical shape with an outer diameter ≤5mm, adapted to the laparoscopic instrument channel. The first imaging chip 11 outputs 4K@60fps video (analog video signal), and the second imaging chip 12 performs two-channel fluorescence detection. For capsule endoscopy applications, the chip is packaged in a capsule shape with an outer diameter ≤5mm, integrating a battery and wireless transmission module. The first imaging chip 11 performs intermittent low-power imaging, and the second imaging chip 12 performs single-channel fluorescence labeling detection. For other bioanalytical instrument R&D applications, hardware parameters are configured according to specific instrument requirements. The above extended application scenarios are achieved through different firmware configurations without modifying the dual-mode imaging chip 61 hardware design.

[0079] The imaging servo chip 13 communicates with the host computer via the peripheral device interface module 76, which is configured with USB 3.1 Gen2 and 10 GigE Vision. In intraoperative navigation applications, the bandwidth of the raw data stream does not exceed 60% of the rated bandwidth of the data interface 6, with reserved bandwidth for control commands and emergency transmission. In high-content cell microscopy analysis applications, whole-plate scan data is transmitted in a lossless compression format with a compression ratio of at least 2:1 to ensure data integrity. The system is configured with software interfaces, data formats, and compliance documents that meet the requirements for medical device registration, including: software interfaces conforming to the YY / T 0664 Medical Device Software Lifecycle Standard; data formats conforming to the DICOM 3.0 Medical Digital Imaging Standard; and compliance documents including risk management reports, clinical evaluation reports, and software verification and validation reports, meeting the NMPA requirements for Class II or Class III medical device registration.

[0080] A DVFS-μTEC collaborative temperature control module is configured to ensure the thermal stability of the LED light source and SPAD array during prolonged surgery. The DVFS-μTEC collaborative temperature control module includes: a DVFS dynamic voltage and frequency adjustment unit, which adjusts the LED driving voltage and switching frequency of the imaging servo chip 13 in real time based on the chip junction temperature, reducing heat source power consumption; a μTEC miniature thermoelectric cooler, mounted on the back of the SPAD array, which stabilizes the SPAD junction temperature within the range of 25±0.5℃ through the Peltier effect; and a temperature feedback loop, which collects the LED and SPAD temperatures in real time through a temperature monitoring chip integrated into the imaging servo chip 13, and controls the collaborative operation of DVFS and μTEC in a closed loop. In intraoperative navigation scenarios, the temperature control module ensures that the SPAD dark count rate change is <5% during 4 hours of continuous surgery; in high-content analysis scenarios, the temperature control module ensures that the fluorescence lifetime measurement drift is <0.1ns during 72 hours of continuous imaging.

[0081] Whole-plate scan data for high-content cell microscopy analysis applications are transmitted in a lossless compression format; the system is configured with software interface, data format, and compliant documents that meet the requirements for medical device registration and application.

[0082] The configuration of the high-content cell microscopy analyzer is shown in Table 2. Table 2 The drug screening process includes: Cell seeding: HeLa cells were seeded in 96-well plates (5000 cells / well) and cultured for 24 h; Drug treatment: Add paclitaxel (concentration gradient 0.1~100 nM) and allow to act for 48 hours; Fluorescent labeling: EdU incorporation (2h) + DAPI counterstaining (15min); Full-plate scanning: The first imaging core 11 is used for panoramic positioning scanning (the bright field profile of each well is cached locally in the core), and the second imaging core 12 is used for multi-channel parallel acquisition (the fluorescence data of each well is cached locally in the core, and the phenotypic feature vector is calculated in situ). Data analysis: AI inference chip 24 outputs cell cycle classification (G1 / S / G2 / M / apoptosis), and calculates EC. 50 =2.3nM; Data notarization: The hash value of the original image is uploaded to the blockchain, and the analysis result is signed with SM2, which complies with FDA 21 CFR Part 11.

[0083] The screening results are as follows: the whole plate scanning time of 96-well plate is 4.5 min, which is 7.8 times better than the traditional sCMOS solution (35 min); the cell recognition accuracy is >95% (compared with flow cytometer); the data audit and tracking time is shortened from 2 days to 10 minutes (blockchain is tamper-proof).

Claims

1. A dual-mode imaging chip, characterized in that, It includes a first imaging core, a second imaging core, and an imaging servo core, all of which are packaged on the same two-dimensional planar carrier. The first imaging chip is vertically stacked and integrated from multiple photoelectric heterogeneous chips, used for bright-field imaging signal acquisition, image processing, and bio-information label fusion. The second imaging chip is also vertically stacked and integrated from multiple photoelectric heterogeneous chips, used for fluorescence imaging digital signal acquisition and time-correlated single-photon counting. The AI ​​inference chip integrated in the internal time-domain processing chip is responsible for cell morphology classification and cancer cell identification. The imaging servo chip is composed of multiple heterogeneous chips, used for mode switching between bright-field imaging mode and fluorescence imaging mode and I / O interface control, as well as for inter-chip clock synchronization, energy distribution, and data path reconstruction.

2. The dual-mode imaging chip according to claim 1, characterized in that, The first imaging chip, the second imaging chip, and the imaging servo chip are heterogeneously integrated into the same package through a standardized die-to-die interface. The multiple optoelectronic heterogeneous chips inside the first imaging chip, the second imaging chip, and the imaging servo chip are packaged using 2.5D or 3D chiplet packaging technology. Further preferably, the clock synchronization function of the imaging servo chip is as follows: a multi-phase clock tree is generated through an on-chip PLL to provide a common clock for the first imaging chip, the second imaging chip, and the external interface, eliminating clock drift between chips; the power consumption allocation function is as follows: the power supply voltage and operating frequency of each chip are adjusted in real time according to the current imaging mode through DVFS dynamic voltage and frequency adjustment technology, reducing the power consumption of the first imaging chip in bright field mode and reducing the power consumption of the second imaging chip in fluorescence mode; the data path reconstruction function is as follows: data routing between chips is realized through the chiplet interconnect high-speed channel.

3. An intraoperative navigation endoscope, employing the dual-mode imaging chip as described in claim 1 or 2, characterized in that, include: The probe module, also known as the dual-mode imaging chip, is a cylindrical three-dimensional stacked package of the dual-mode imaging chip, which is then placed at the front end of the surgical instrument. The first imaging core is used for large-field tissue localization, the second imaging core is used for cell-level morphology analysis, and the imaging servo core is used to drive the mode switching between bright field imaging mode and fluorescence imaging mode in real time. The light source module integrates a white laser LED and a four-color laser LED. The white laser LED is used for bright field illumination, and the wavelength of the four-color laser LED is matched with the blue, green, yellow, and red fluorescent filters of the dual-mode imaging chip. The display module includes a second image fusion unit and a cancer boundary recognition unit. The second image fusion unit is used to superimpose the navigation image data stream output by the first imaging chip and the fluorescence detection data stream output by the second imaging chip at the pixel level to display the cancer boundary probability heat map and fluorescence lifetime map, thereby realizing intraoperative navigation. Cancer boundary recognition unit, used to identify cancer boundaries.

4. The intraoperative navigation endoscope according to claim 3, characterized in that, The cylindrical package forms a rigid coaxial connection with the front end of the surgical instrument.

5. The intraoperative navigation endoscope according to claim 3, characterized in that, The data parallel computing architecture of the first imaging core is as follows: The first imaging core has an independent frame buffer and an embedded microprocessor built in, which completes the acquisition of color video signals, Bayer decoding, white balance correction and image compression at the front end of the probe, and outputs the compressed navigation image data stream. The data parallel computing architecture of the second imaging chip is as follows: The second imaging chip has an independent SPAD array photosensitive area, a four-color filter array, a single photon counting buffer and a time-to-digital conversion microprocessor. It completes the timestamp recording of fluorescent photons, fluorescence lifetime histogram construction, lifetime fitting and carcinogenesis probability inference calculation at the front end of the probe, and outputs a carcinogenesis boundary probability heat map and a fluorescence lifetime map to achieve in-situ intelligent analysis with cell-level precision. The imaging servo core mode switching control includes: the imaging servo core integrates a light source control circuit and a hardware timing controller, which controls the mode switching between bright field imaging mode and fluorescence imaging mode through line and field drive signals, with a switching delay of no more than 300ns.

6. The intraoperative navigation endoscope according to claim 3, characterized in that, Intraoperative fluorescence lifetime navigation is achieved through a second imaging chip; this includes: using a closed-loop control method of photostimulation and fluorescence response to collect cell fluorescence lifetime information in real time during tumor resection: after the photostimulation pulse is emitted, the fluorescence emitted by the tissue cells triggers the SPAD array to enter a high temporal resolution detection window; the time-to-digital converter microprocessor built into the second imaging chip records a timestamp for each photon pulse, constructs a time-photon number histogram in the chip's local cache, and fits the fluorescence lifetime curve; when the fluorescence lifetime value deviates from the normal tissue baseline value τ0 by more than the threshold Δτ, the inference microprocessor built into the second imaging chip determines that the corresponding area is cancerous tissue, and marks it with red highlighting on the displayed image to guide the surgical resection boundary.

7. The intraoperative navigation endoscope according to claim 3, characterized in that, The four-color laser LED includes four bands: 450nm±10nm, 520nm±10nm, 590nm±10nm, and 650nm±10nm. Each band is driven by an independent current to achieve pulse modulation, and the pulse frequency is synchronized with the gate window of the SPAD array. Further preferred, the data interface includes USB 3.1 Gen2 and 10GigE Vision.

8. A cell microscopic high-content analyzer, using the dual-mode imaging chip as described in claim 1 or 2, characterized in that, include: The microscopic imaging module, also known as the dual-mode imaging chip, is packaged in a three-dimensional rectangular package and placed below the sample pool. The first imaging chip performs low-magnification global localization scanning of each well, while the second imaging chip performs high-magnification fluorescence imaging of the target well. The imaging servo chip is used to switch the well synchronization mode. Environmental control module; used for closed-loop PID temperature control, infrared CO2 concentration monitoring and humidity control; High-content analysis module; used for cell segmentation and tracking, multi-parameter quantitative analysis, and pharmacodynamic evaluation; The data management module is used to store raw images, analysis results, experimental metadata, and blockchain evidence. More preferably, the environmental control module includes a PID temperature controller, a CO2 concentration monitor, and a humidity controller; The system is equipped with closed-loop PID temperature control and infrared CO2 concentration monitoring. The temperature is controlled at 37℃±0.5℃, the CO2 concentration is controlled at 5±0.2%, and the humidity is maintained at no less than 85% RH through water bath humidification to ensure long-term imaging of live cells.

9. A cell microscopic high-content analyzer according to claim 8, characterized in that, The first imaging chip performs a low-magnification global localization scan of each well location; including: the first imaging chip uses an independent frame buffer and microprocessor to quickly scan the entire multi-well plate, completes image preprocessing and well edge detection locally on the chip, and outputs the coordinate index and contour mask of each well location to provide localization guidance for fluorescence lifetime imaging of the second imaging chip. The second imaging chip performs high-magnification fluorescence imaging of the target pore locations; including: the second imaging chip receives the coordinate index of each pore location output by the first imaging chip, and performs multi-channel parallel fluorescence acquisition of the target area of ​​each pore; the SPAD array photosensitive area, together with the four-color filter array, realizes four-channel parallel fluorescence detection, and the photon count data of each channel is buffered in the independent buffer area of ​​the second imaging chip, and the embedded microprocessor performs fluorescence intensity integration, fluorescence lifetime fitting and cell segmentation operations in parallel, and outputs a multi-dimensional cell phenotypic feature vector composed of cell area, contour perimeter, texture feature parameters, fluorescence intensity and fluorescence lifetime; Cross-frame trajectory reconstruction of AI inference cores; including: the AI ​​inference core integrated in the temporal processing core inside the second imaging core receives the multidimensional cell phenotypic feature vector, performs morphological classification and cross-frame trajectory reconstruction: cross-frame identity preservation for individual cells, using the Hungarian algorithm to match cell positions and morphological similarities in adjacent frames, reconstructing cell movement trajectories and lineage differentiation paths; outputting cell phenotypic feature vectors and trajectory data for dynamic drug response analysis; The imaging servo core's aperture position synchronization mode switching includes: the imaging servo core switches between bright-field imaging mode and fluorescence imaging mode through hardware timing control, and the switching timing is synchronized with the mechanical displacement cycle of the multi-well plate; after the first imaging core completes a bright-field positioning scan of a certain aperture, the imaging servo core triggers mode switching, and the second imaging core then enters fluorescence imaging mode to perform single-photon digital signal acquisition; after the acquisition is completed, it switches back to bright-field mode, and the displacement stage moves to the next aperture position.

10. A cell microscopic high-content analyzer according to claim 8 or 9, characterized in that, The independent buffer of the first imaging chip includes a Bayer raw frame buffer and a compressed bitstream buffer. The microprocessor of the first imaging chip includes an image signal processing (ISP) and a JPEG encoder, which perform parallel operations of bright-field image acquisition, decoding, correction, and compression, and output a compressed navigation image data stream. The independent cache of the second imaging chip includes a SPAD photon counting cache and a lifetime histogram cache. The microprocessor includes a TDC time-to-digital converter, a lifetime fitting engine and an inference accelerator, which performs parallel operations of fluorescence photon detection, timestamps, histograms, data fitting and cell classification, and outputs fluorescence lifetime maps, cancer probability maps and phenotypic feature vectors. The imaging servo chip acts as a central scheduler, coordinating the frame buffer access of the first and second imaging chips through a time-division multiplexing bus, and transmitting the parallel output data of the first and second imaging chips to the back end after pixel-level registration through a unified fusion output interface. Further preferred, the high-content analysis module includes a cell segmentation and tracking unit, a multi-parameter quantitative analysis unit, and a pharmacodynamic evaluation unit; Cell segmentation and tracking are achieved through a cell segmentation and tracking unit, including: multi-dimensional cell phenotypic feature vectors output by the second imaging core, cross-frame trajectory reconstruction of the AI ​​inference core, identity preservation of individual cells, and reconstruction of cell movement trajectory and lineage differentiation path; Multi-parameter quantitative analysis is achieved through a multi-parameter quantitative analysis unit, including: extracting five parameters: cell area, contour perimeter, texture feature parameters, fluorescence intensity, and fluorescence lifetime, and constructing a multidimensional cell phenotypic feature vector; Pharmacokinetic evaluation is performed using a pharmacokinetic assessment unit, including: generating a compound concentration-response curve using a four-parameter logistic model based on a time-varying parameter set, and calculating the half-maximal effective concentration (EC50). 50 With half-maximal inhibitory concentration (IC50) 50 .

Images