A real-time AI detection system for oral lesions based on multispectral imaging

Abstract

The application is particularly a kind of oral cavity lesion real-time AI detection system based on multispectral imaging, relating to the medical detection technical field, comprising: an observation matrix construction module; a high-frequency noise filtering and closed solution baseline deduction module; a non-negative solution module; an abundance reconstruction and feature pseudo-color mapping module.In the application, pure matrix algebra operation that can be hardware solidified is adopted, without any iteration logic and trainable parameters, the overall processing delay is lower than the threshold value required by medical real-time image, without the support of NPU / GPU parallel computing power, and can be realized by FPGA / DSP hardware solidification, adapting to low-power portable endoscope equipment; at the same time, the whole process operation has clear physical meaning, meeting the requirements of medical device interpretability supervision, and reducing the compliance threshold of product clinical landing.

Description

Technical Field

[0001] This invention relates to the field of medical testing technology, and in particular to a real-time AI detection system for oral lesions based on multispectral imaging. Background Technology

[0002] Oral multispectral fluorescence endoscopy is a core device for achieving non-invasive, real-time early lesion screening. Its core technology lies in the spectral demixing algorithm, which separates the characteristic signals of healthy tissue and diseased tissue from multispectral images to achieve accurate identification of lesions.

[0003] Current mainstream spectral unmixing and lesion detection technologies fall into two categories, both of which have limitations in adapting to the clinical scenarios of oral endoscopy: The first type is a data-driven AI detection solution based on deep learning. It relies on tens of thousands of labeled pathological samples to complete model training. Differences in the oral mucosa spectrum of different individuals and individual differences in the light source and sensors of different devices can lead to model generalization failure. Furthermore, deep learning inference requires the support of parallel computing of NPU or GPU, and real-time processing of more than 30fps cannot be achieved on low-power portable endoscope devices. The second type is the traditional nonnegative matrix factorization (NMF) iterative demixing scheme. For a 1080P resolution, 5-channel multispectral video stream, each frame needs to process 2,073,600 pixels, and each pixel needs to be iterated at least 20 times to converge, resulting in high single-frame processing latency. Moreover, existing iterative schemes generally ignore the effects of physical noise and light source power fluctuations in the actual hardware, and the demixing error generally exceeds 30%, making them completely unusable in clinical practice.

[0004] Therefore, a real-time AI detection system for oral lesions based on multispectral imaging is proposed to address the aforementioned problems. Summary of the Invention

[0005] The purpose of this invention is to provide a real-time AI detection system for oral lesions based on multispectral imaging in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A real-time AI detection system for oral lesions based on multispectral imaging, comprising: The observation matrix construction module is configured to pre-build and solidify the end-member matrix matching the oral endoscopy scene, collect multispectral pixel vectors and light source irradiance data in real time, complete dark current correction and irradiance compensation, and establish observation equations that match the imaging physical process. The high-frequency noise filtering and closed-off baseline subtraction module is configured to take pixel vectors and irradiance data as inputs, filter high-frequency noise, perform closed-off linear operations to strip the background baseline, and output an effective spectral vector. The non-negative solution module is configured to take the effective spectral vector as input, call the generalized pseudo-inverse matrix that is pre-calculated and fixed offline based on the endmember matrix, obtain the original abundance vector through matrix-vector multiplication, and then output the non-negative abundance vector through hardware non-negative truncation. The abundance reconstruction and feature pseudocolor mapping module is configured to take the output non-negative abundance vector as input, extract the endmember abundance components of lesion features to complete pseudocolor mapping, and output a lesion visualization image after mixing with the synchronous white light image.

[0007] Preferably, the process of obtaining the endmember matrix, multispectral pixel vector, and light source irradiance includes: Endmember matrix Construction: Endmember matrix The dimension is ,in The number of spectral channels in a multispectral sensor. The amount of the dominant substance in an oral endoscopic setting; Multispectral pixel vector Acquisition: The front-end multispectral sensor uses a global shutter CMOS sensor and employs a time-division multispectral exposure design: [Image of the same spatial location] Images from five channels are acquired sequentially within the same frame period. Therefore, the response values ​​of the five channels can be obtained for the same spatial location within one frame, forming a pixel vector with a dimension of 5×1. ,in Corresponding to the center wavelength of the 5 channels; Light source irradiance Collection: Integrating a miniature photodiode at the light source output end achieves... Real-time data collection; The irradiance of each channel was measured using a standard radiometer at the standard working distance of the equipment. Simultaneously, the output voltage of the PD was recorded, and the linear correction coefficient for each channel was obtained by fitting the data. and That is, the first Irradiance of each channel ,in The real-time output voltage of the PD has a correction coefficient stored in the device's memory.

[0008] Preferably, the established observation equation is: ; in, For the first Endmembers at wavelength The standardized spectral response at that location; For the first Class endmembers in spatial location Abundance at location; Baseline for broadband background; This is high-frequency system noise.

[0009] Preferably, the method for stripping the background baseline is as follows: First, the original pixel vector Divide each channel element by the corresponding channel The spectral vector after irradiance normalization is obtained. ; Then the other channels Subtract the reference channel Multiply by calibration factor .

[0010] Preferably, the second method is: right Irradiance normalization is obtained ; Then, normalized difference operations are performed on each band pair; Finally, the differential signals corresponding to the three endmembers are combined with the baseline subtraction results of the remaining channels to obtain a 5×1 dimension signal. vector.

[0011] Preferably, in the nonnegativity solution module, the method for calculating the pseudo-inverse matrix is ​​as follows: A 5×3 matrix Perform SVD decomposition to obtain ,in It is a 5×5 orthogonal matrix. It is a 5×3 diagonal matrix, with the elements on the diagonal being... singular values, It is a 3×3 orthogonal matrix; Will Less than the maximum singular value on the diagonal The elements are treated as zero, and the reciprocals of the remaining non-zero elements are transposed to obtain the result. ; Finally obtained pseudo-reversal The dimensions are 3×5; Will Each element is quantized, and the quantized result The matrix is ​​stored in the processor's on-chip SRAM.

[0012] Preferably, the real-time matrix multiplication is described as follows: It is a 3×5 matrix. Since the vector is 5×1, matrix multiplication involves 15 multiplications and 12 additions, specifically: ; and The computational logic is the same; for Each element is directly compared with 0 using a hardware comparator. If the element value is greater than or equal to 0, the original value is output; otherwise, 0 is output.

[0013] Preferably, the process of extracting the lesion feature endmember abundance component to complete the pseudocolor mapping includes: The vector is 3×1, where the third component The abundance of porphyrin telomeres in the corresponding lesion tissue ranges from 0 to 1, where 0 represents that the pixel has no lesion components and 1 represents that the pixel is completely composed of lesion components. Validation based on a preset number of clinical samples, set up The threshold; False color mapping will The numerical value is converted into a warning color transparency, allowing doctors to intuitively judge the location and risk level of the lesion.

[0014] Preferably, the output lesion visualization image is: For each spatial location The corresponding white light pixel value , , The Alpha value of the warning color is The RGB value of the warning color is , , The mixing formula is: ; The mixed RGB888 image is written to double-buffered video memory, writing the current frame while simultaneously outputting the previous frame to the display bus.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention employs pure matrix algebra operations that can be hard-coded in hardware, without any iterative logic or trainable parameters. The overall processing latency is lower than the threshold required for real-time medical imaging. It does not require NPU / GPU parallel computing power support and can be implemented through FPGA / DSP hardware hard-coded, making it compatible with low-power portable endoscope devices. At the same time, all operations in the entire process have clear physical meaning, which meets the regulatory requirements for interpretability of medical devices and lowers the compliance threshold for clinical application of the product.

[0016] 2. This invention models the physical process based on oral multispectral imaging and controls the unmixing error through end-member matrix metrological calibration, dark current correction, real-time light source irradiance compensation, noise filtering, and baseline closure desubtraction. The solution is unaffected by individual mucosal spectral differences, individual equipment differences, and light source power fluctuations, enabling non-invasive screening of precancerous lesions and early cancers in the oral cavity, significantly improving the early diagnosis rate of oral cancer. Attached Figure Description

[0017] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0018] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0019] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0020] Example 1

[0021] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0022] Appendix Figure 1 This invention provides a structural block diagram of a real-time AI detection system for oral lesions based on multispectral imaging, showing the connection relationship between the observation matrix construction module and the abundance reconstruction and feature pseudocolor mapping module, and annotating the main functional interaction flow of each module.

[0023] In this embodiment, it includes: The observation matrix construction module is configured to pre-build and solidify the end-member matrix matching the oral endoscopy scene, collect multispectral pixel vectors and light source irradiance data in real time, complete dark current correction and irradiance compensation, and establish observation equations that match the imaging physical process. This module is the physical foundation of the entire scheme. The validity of all subsequent algebraic operations is based on the complete match between the observation model constructed by this module and the actual hardware imaging process. The observation equation output by the module is the constraint boundary of all subsequent operations and directly determines the clinical error level of the unmixing result.

[0024] Module inputs: Three types of directly obtainable physical quantities, namely, the fixed end-member matrix obtained from the equipment calibration before leaving the factory. The raw pixel vectors output in real time by the front-end multispectral sensor Real-time irradiance data of light sources collected by photodetectors .

[0025] Module output: Observation equations that conform to the actual physical imaging process, as well as raw spectral data corrected for dark current and corresponding irradiance compensation coefficients.

[0026] Endmember matrix Construction and calibration details: Endmember matrix The dimension is ,in The number of spectral channels in a multispectral sensor. The selection of both types of parameters, which represent the dominant substances in oral endoscopic settings, is based on clear physical and clinical evidence and is not arbitrary.

[0027] Determining the value: Based on statistical results from over 1000 clinical samples in a clinical oral medicine spectral database, only three types of fluorescent / reflective substances dominate intraoral endoscopic imaging. The cumulative variance contribution of these three endmembers exceeds 98%. Adding more endmembers will not significantly improve unmixing accuracy; instead, it will increase unnecessary computation. Taking 3 as the optimal value. The three types of endmembers correspond to: Collagen in the subepithelial layer of healthy mucosa: its autofluorescence excitation peak is 405nm, and its emission peak covers 480~520nm. It is not present in diseased tissue. Broadband reflection of tooth enamel, oral mucosa surface, saliva, and food debris: its reflectance is basically constant across the entire wavelength range of 400~700nm, with no specific absorption or emission peaks. Porphyrin metabolites (mainly protoporphyrin IX) enriched in diseased tissues (including epithelial dysplasia, oral precancerous lesions, and early cancerous tissues): their excitation peak is 405 nm, and their emission peak covers 620~660 nm, which are specific markers of lesions.

[0028] Determining the value: To balance hardware implementation complexity and demixing accuracy, this scheme employs a 5-channel multispectral sensor. The center wavelengths of the bandpass filters for each channel are 450nm, 500nm, 550nm, 600nm, and 650nm, with a full width at half maximum (FWHM) of 20nm. This covers all characteristic spectral ranges of the three types of endmembers while avoiding the increased hardware cost and computational load associated with too many channels. Take 5, corresponding to The matrix has a dimension of 5×3.

[0029] Methods for measuring matrix elements: All elemental measurements were performed in a standard laboratory environment using a fiber optic spectrometer calibrated by the National Institute of Metrology, with a wavelength resolution of 1 nm and an acquisition range covering 400–700 nm. The specific measurement procedure is as follows: The first column (collagen end-members): Normal buccal mucosa samples were collected from 10 healthy volunteers without oral diseases. After thoroughly removing saliva and food debris from the surface, the average response values ​​of 5 channels were collected under 405nm standard excitation light. The average value of the 10 samples was then subtracted from the dark current of the sensor to obtain the 5 elements in the first column. The second column (broadband reflectance end element): A standard diffuse reflectance white plate (99% reflectance) that has been calibrated by measurement is placed at the standard working distance of the device (10mm from the endoscope tip to the sample, which is the commonly used working distance in clinical practice). The response values ​​of 5 channels under the same excitation light are collected and multiplied by the standard reflectance to obtain the 5 elements of the second column. The third column (porphyrin metabolite endmembers): Prepare a standard physiological saline solution of protoporphyrin IX with a concentration of 1 μg / mL, place it at the standard working distance, and collect the response values ​​of 5 channels under the same excitation light. Subtract the background response of pure physiological saline to obtain the 5 elements in the third column.

[0030] Matrix calibration per unit: Because the sensor response rate and filter transmittance of different devices vary, each device needs to be tested before leaving the factory. The matrix is ​​calibrated unit by unit. The calibration method is as follows: three standard endmember samples are placed sequentially within the imaging range of the device, the multispectral response of each endmember is acquired, and the pre-stored data is compared. The matrix is ​​linearly corrected.

[0031] Corrected The unmixing error of the matrix to the standard samples must be controlled within 5% to be considered acceptable. (After calibration) The matrix is ​​stored in the device's non-volatile memory and is automatically loaded into the on-chip SRAM when the device starts up. It will not be modified during operation.

[0032] Real-time multispectral pixel vector Details on how to obtain: The front-end multispectral sensor uses a global shutter CMOS sensor with a resolution of 1920×1080 to 3840×2160, a quantization depth of 12 bits per pixel, and employs a time-division multispectral exposure design: [the following text appears to be unrelated and possibly machine-translated gibberish: "same spatial position..."] Images from 5 channels are acquired sequentially within the same frame period, with an exposure time of 1ms for each channel and a total frame period of 33ms. Therefore, the response values ​​of 5 channels can be obtained for the same spatial location within one frame, forming a pixel vector with a dimension of 5×1. ,in The center wavelengths corresponding to the 5 channels.

[0033] To eliminate the influence of dark current noise, the device first acquires 10 frames of dark images under no-light conditions upon startup. The dark current value for each channel is taken as the average of the 10 frames. Subsequent acquisitions... Each element of the vector must first be subtracted from the dark current value of the corresponding channel, and the error of dark current correction is controlled within one quantization unit.

[0034] Light source irradiance Details of data collection and compensation: The excitation source for the oral endoscope uses a 405nm semiconductor laser. The laser's output power fluctuates with changes in operating temperature and time, with a maximum fluctuation range of ±20%. Simultaneously, the fiber optic transmission efficiency of the endoscope decreases with usage time, with an annual attenuation rate reaching 5%. Without compensation, the same abundance... The vector strength will shift significantly, resulting in a demixing error of more than 20%.

[0035] This solution achieves this by integrating a miniature photodiode (PD) at the light source output end. Real-time data acquisition is achieved, with the PD's response wavelength covering 400~700nm. A linear correlation between the PD's output voltage and the irradiance of each spectral channel is pre-established before shipment. The irradiance of each channel was measured using a standard radiometer at the standard working distance of the equipment. Simultaneously, the output voltage of the PD was recorded, and the linear correction coefficient for each channel was obtained by fitting the data. and That is, the first Irradiance of each channel ,in The real-time output voltage of the PD has a correction coefficient stored in the device's memory.

[0036] During operation, at the start of each frame acquisition, the output voltage of the PD is read first, and the voltage of the five channels of the current frame is calculated. Value. Since the power change of the light source is less than 0.1% within a 33ms frame period, all pixels within the same frame share a set of values. This value ensures both the accuracy of compensation and avoids the redundant overhead of pixel-by-pixel calculation.

[0037] The established observation equation is: ; in, For the first Endmembers at wavelength Normalized spectral response at: The spectral response values ​​of the dominant substances under unit irradiance, all Together they form a pre-stored end-member matrix (Dimensions are 5×3), the three types of endmembers correspond to: =1, the autofluorescence spectral response of healthy mucosal collagen; =2, broadband reflectance spectrum response of enamel / saliva / lens stray light; =3; Specific fluorescence spectral response of porphyrin metabolites in diseased tissue; For the first Class endmembers in spatial location Abundance at: pixel First The proportion of each substance is the core unknown to be solved in this scheme; among which The abundance of endmembers at the lesion site directly reflects the risk of lesion at that location. In the physical world, the abundance of matter is non-negative, so the legal value range is [0,1]. 0 means that the pixel does not contain the corresponding matter, and 1 means that the pixel is completely composed of the corresponding matter. The broadband background baseline mainly consists of specular reflection from saliva on the oral mucosa, enamel reflection, and stray light from the endoscope lens. Its spectral characteristics are basically constant or linearly variable across the entire wavelength range and are unrelated to the specific spectra of the substances. High-frequency system noise mainly comes from sensor shot noise and circuit thermal noise, and manifests as random high-frequency fluctuations.

[0038] The high-frequency noise filtering and closed delineation baseline subtraction module is configured to take the corrected pixel vector and irradiance data as input, filter high-frequency noise through fixed-weight spatial domain convolution, and then strip the background baseline through closed delineation operation to output an effective spectral vector. The core objective of this module is to eliminate noise through closed-form solution operations using pure algebra without employing iterative algorithms. Compared with baseline To eliminate interference and output a pure, effective spectral vector. All operations are performed at fixed cycles with no variable delay.

[0039] Module input: Pixel vectors corrected for dark current output from the observation matrix construction module. With the corresponding frame value; Module output: Dimensions and Consistent pure effective spectral vector ; The overall preprocessing error is controlled within 3%.

[0040] High frequency noise Elimination: The core reason for choosing to perform noise cancellation in the spatial domain rather than the spectral domain is that spectral domain filtering alters the specific spectral characteristics of the material, leading to increased unmixing errors in subsequent processes. In contrast, spatial domain filtering targets the spatially uncorrelated characteristics of shot noise. Shot noise is random and uncorrelated between adjacent pixels, while real tissue signals are continuous between adjacent pixels. Therefore, spatial domain filtering can effectively suppress noise without altering the spectral characteristics.

[0041] Convolution operator selection and parameter design: The 3×3 mean convolution operator is preferred. All weights of this operator are equal, with each weight being 1 / 9. During the operation, only the pixel values ​​of the same spectral channel in the 3×3 neighborhood need to be averaged. No multiplication operation is required, only addition and shift operations are needed. The convolution operation of a single pixel can be completed in one clock cycle on the FPGA.

[0042] The basis for choosing the 3×3 size is as follows: at 1080P resolution, the actual physical size of a single pixel at the standard working distance is about 5μm, and the smallest lesion size of the oral mucosa is about 50μm, corresponding to 10 pixels. Therefore, 3×3 convolution will not blur the lesion boundary, and at the same time, it can reduce the shot noise intensity to 1 / 3 of the original.

[0043] For dedicated DSP chips with high computing power, a 3×3 Gaussian convolution operator can also be used, with the standard deviation of the Gaussian kernel set to 1, and the weight distribution as follows: center weight 4, four-way neighborhood weight 2, and four corner weight 1, with the entire distribution normalized by dividing by 16. Gaussian filtering offers better edge preservation, requires only a small number of multiplication operations, and still operates on a fixed period.

[0044] Boundary handling methods: For pixels at the image edges (i.e. =0、 =1919、 =0、 (For pixels with a resolution of 1079), an edge copying filling method is used, where the pixel value outside the edge is equal to the corresponding pixel value at the outermost edge, thus avoiding dark edges or distortion in the filtering results of edge pixels.

[0045] Broadband Baseline Elimination: Based on statistics of a predetermined number of clinical samples (e.g., 1000 cases), baselines of saliva, enamel reflection, and lens stray light were obtained. The intensity variation coefficient across the five spectral channels is less than 5%, meaning it is essentially constant or varies linearly with wavelength. Therefore, it can be completely eliminated using linear operators without iterative calculations. This module provides two selectable baseline subtraction methods, adapted to different clinical scenarios and hardware computing power.

[0046] Method 1: Reference band subtraction method: Suitable for low-power FPGA scenarios with extremely high computational speed requirements.

[0047] Reference band selection: 450nm channel (i.e., the first spectral channel). The selection criteria are: the specific porphyrin fluorescence of the lesion shows no emission or absorption at 450nm, and the fluorescence intensity of collagen at 450nm is less than 5%. Therefore, the signal in this channel almost entirely comes from the broadband baseline. .

[0048] Detailed operating procedures: First, the original pixel vector Divide each channel element by the corresponding channel The spectral vector after irradiance normalization is obtained. ,Right now At this time, the baseline normalized value The stability is basically constant across all channels; Then the other channels ( =2 to 5) minus the reference channel Multiply by calibration factor ;

[0049] These are fixed values ​​calibrated before shipment. The calibration method involves collecting 100 samples containing only baseline data and no valid signal (such as a standard white board covered with saliva), and calculating the values ​​for each channel. and The average of the ratios is obtained. =0.98、 =0.98、 =0.96、 =0.95, and the baseline residue after deduction is less than 1%.

[0050] Method 2: Normalized Difference Method

[0051] Suitable for scenarios requiring higher baseline subtraction accuracy, it can further eliminate the effects of light source fluctuations and baseline gradual changes.

[0052] Band pair selection: matching based on the characteristic spectra of endmembers; Porphyrin lesion endmembers: Select the 650nm channel (channel 5) where the emission peak is located and the adjacent non-emission peak 600nm channel (channel 4). The baseline intensity difference between the two channels is less than 2%, while the response of porphyrin at 650nm is more than 10 times that at 600nm. Collagen terminus: Select the 500nm channel (channel 2) where its emission peak is located and the adjacent 550nm channel (channel 3) where the non-emission peak is located; Broadband reflective end-members: Select the 550nm channel (channel 3) and the 600nm channel (channel 4).

[0053] Detailed operating procedures: Firstly, the same applies to Irradiance normalization is obtained ; Then, a normalized difference operation is performed on each band pair. For example, the difference signal corresponding to porphyrin is... The purpose of normalization is to eliminate the influence of overall signal intensity variations, so that the difference result is only related to the relative ratio of the two channels, and further eliminates the interference of baseline and light source fluctuations; Finally, the differential signals corresponding to the three endmembers are combined with the baseline subtraction results of the remaining channels to obtain a 5×1 dimension signal. vector.

[0054] After processing by this module, the observation equation in... and It has been basically stripped away, and what was obtained It contains only valid signal terms and negligible residual errors, providing a clean input for subsequent abundance unmixing.

[0055] The non-negative solution module is configured to take the effective spectral vector as input, call the generalized pseudo-inverse matrix that is pre-calculated and fixed offline based on the endmember matrix, obtain the original abundance vector through matrix-vector multiplication, and then output the non-negative abundance vector through hardware non-negative truncation. Module input: High-frequency noise filtering and closed-loop baseline subtraction module output Vector and pre-stored Moore-Penrose generalized pseudoinverse matrix ; Module output: Final abundance vector The dimension is 3×1, corresponding to the abundance values ​​of the three types of endmembers.

[0056] Offline pre-computation of pseudo-inverse matrix Implementation details: The core of this solution lies in utilizing the endmember matrix. The fixed nature of the process shifts pseudo-inverse calculation from real-time computation to the offline stage before shipment, completely avoiding the huge overhead of real-time pseudo-inverse calculation.

[0057] Methods for calculating the pseudo-inverse matrix: Singular Value Decomposition (SVD) method is used to calculate The Moore-Penrose generalized pseudoinverse, rather than directly adopting The core reason for this method is: when When the column vectors have slight collinearity, It may be an ill-conditioned matrix, with a huge error in inversion, while the SVD method can improve numerical stability by truncating small singular values.

[0058] The specific calculation process is as follows: A 5×3 matrix Perform SVD decomposition to obtain ,in It is a 5×5 orthogonal matrix. It is a 5×3 diagonal matrix, with the elements on the diagonal being... singular values, It is a 3×3 orthogonal matrix; Build pseudo-reversal :Will Less than the maximum singular value on the diagonal The elements are treated as zero, and the reciprocals of the remaining non-zero elements are transposed to obtain the result. ; Finally obtained pseudo-reversal The dimensions are 3×5; Quantization and solidification of pseudo-inverse matrices: To adapt to fixed-point arithmetic in the hardware, Each element is quantized into a 16-bit fixed-point number, with 2 bits for the integer part and 14 bits for the fractional part. The quantized result... The matrix is ​​stored in the processor's on-chip SRAM. The access latency of the on-chip SRAM is one clock cycle, eliminating the need to access external memory. This results in fast read speeds, making it suitable for real-time computing.

[0059] Verification of the pseudo-inverse matrix: Before leaving the factory, standard samples with known abundance are used to test... Perform validation, for example, input abundance as For simulated samples, the abundance obtained after unmixing must have an error of less than 3% compared to the standard value to be considered acceptable.

[0060] After pretreatment The effective signal term in the vector and observation equations satisfies This is an overdetermined equation ( Its least squares optimal solution is This operation requires only matrix-vector multiplication and has no iterative logic; the original abundance vector is... .

[0061] Calculation process: It is a 3×5 matrix. Since the vector is 5×1, matrix multiplication involves 15 multiplications and 12 additions, specifically: ; and The computational logic is the same; Hardware acceleration implementation: Parallel computing architectures can be used on FPGAs, three The calculation of elements can be performed simultaneously, requiring only 5 multiplications and 4 additions, and the operation of a single pixel can be completed in a maximum of 10 clock cycles. Based on a clock frequency of 100MHz, the operation time for a single pixel is only 0.1μs. Using 128 parallel processing units, processing 2,073,600 pixels of 1080P resolution requires only about 1.6ms, far lower than the 33ms frame cycle requirement.

[0062] Precision control: Each element is a 12-bit fixed-point number. The elements are 16-bit fixed-point numbers, the multiplication result is 28 bits, the addition result is 32 bits, and finally the result is right-shifted by 14 bits (corresponding to...). (from 14 decimal places), to get 18 bits The vector contains 4 integer bits and 14 decimal bits, with precision meeting clinical needs.

[0063] Hardware implementation details of nonnegative truncation: Since the abundance of matter in the physical world cannot be negative, but pure mathematical pseudo-inverse calculations may produce negative values ​​due to residual noise and baseline errors, non-negativity constraints need to be added. This scheme abandons traditional non-negativity constraint methods such as the iterative Lagrange multiplier method and the projected gradient method, and uses a comparator implemented with pure hardware logic gates for truncation, with negligible computational overhead.

[0064] Operation logic: for Each element is directly compared with 0 using a hardware comparator. If the element value is greater than or equal to 0, the original value is output; otherwise, 0 is output, resulting in a non-negative abundance vector. : This operation requires only one clock cycle and has no computational delay.

[0065] The abundance reconstruction and feature pseudocolor mapping module is configured to take the output non-negative abundance vector as input, extract the endmember abundance components of lesion features to complete pseudocolor mapping, and output a lesion visualization image after mixing with the synchronous white light image. This module converts the unmixed abundance vectors into visual images that doctors can interpret intuitively, while also supporting the quantitative storage and output of abundance data.

[0066] Module input: the output of each pixel from the non-negative solution module. Vector-based, synchronously acquired high-resolution white light RGB images; Module output: The final display image with lesion warning colors superimposed, with a resolution of 1920×1080 and a standard RGB888 format; The display error is less than 2 gray levels.

[0067] This module has a built-in lightweight multi-feature fusion AI verification unit. While extracting the abundance component of the lesion end-member, it simultaneously calls the white light image texture features and multispectral spatial continuity features of the 3×3 neighborhood of the pixel. Through inference using a lightweight decision tree model with less than 50KB of parameters, it filters out false positive abundance signals caused by food residue and saliva bubbles. This unit is entirely a fixed lookup table operation and does not require real-time training.

[0068] Extraction and threshold setting of lesion abundance components: The vector is 3×1, where the third component The abundance of porphyrin telomeres in the corresponding lesion tissue ranges from 0 to 1, where 0 represents that the pixel has no lesion components and 1 represents that the pixel is completely composed of lesion components. Validation is based on a preset number of clinical samples (e.g., 1200 cases). The threshold, for example: when A threshold value greater than 0.15 indicates a risk of epithelial dysplasia or cancer in the corresponding tissue; therefore, the default threshold is set to 0.15. The threshold can be adjusted via the device's user interface, and the adjusted value is stored in a hardware register without requiring modification to the hardware logic.

[0069] Feature pseudocolor mapping: The core of pseudo-color mapping is to The numerical value is converted into the transparency (Alpha value) of the warning color, allowing doctors to intuitively judge the location and risk level of lesions. The mapping logic is designed entirely based on the usage habits of clinicians. when When the value is less than 0.15, the Alpha value is 0, which means it is completely transparent and does not have a warning color superimposed, thus avoiding false positive signals with low abundance from interfering with doctors' judgment. when When the alpha value is greater than or equal to 0.15 and less than or equal to 0.8, the alpha value is ( ). -0.15) / (0.8-0.15)×255, that is, linearly mapped to an 8-bit range of 0 to 255. The semi-transparent warning color can simultaneously display the location of the lesion and the morphology of the underlying mucosa. when When the value is greater than 0.8, the Alpha value is 255, which is a completely opaque warning color, indicating a high-risk lesion.

[0070] The default warning color is pure red, which is commonly used in the medical field, with an RGB value of (255,0,0). Users can also adjust the RGB value of the warning color according to their needs, and the adjusted value is stored in a register.

[0071] Alpha blending and synchronous output: To ensure the alignment accuracy between the lesion warning color and the white light mucosal image, the multispectral image and the white light image are acquired in a time-division manner using the same sensor, or by synchronously triggering acquisition using two sensors, with a trigger signal time difference of less than 1ms. Spatial registration calibration is performed before leaving the factory, with a registration error of less than 1 pixel, eliminating the need for real-time registration.

[0072] Alpha hybrid logic: For each spatial location The corresponding white light pixel value , , The Alpha value of the warning color is The RGB value of the warning color is , , The mixing formula is: ; and The computational logic is similar.

[0073] All operations are fixed-point operations. Dividing by 255 can be achieved by right shifting by 8 bits and adding a correction term, or by directly calling the hardware divider. The operation time is 1 clock cycle.

[0074] The mixed RGB888 image is written to dual-buffered video memory, simultaneously writing the current frame and outputting the previous frame to the display bus to avoid screen tearing. The display bus uses a standard HDMI or MIPI interface, with an output frame rate of 30fps and a resolution of 1920×1080, compatible with mainstream medical displays.

[0075] At the same time, the module supports converting each pixel's... The value is output as a separate 16-bit channel and stored in the device's built-in memory for subsequent clinical analysis and case recording.

[0076] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0077] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0078] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0083] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0084] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A real-time AI detection system for oral lesions based on multispectral imaging, characterized in that, include: The observation matrix construction module is configured to pre-build and solidify the end-member matrix matching the oral endoscopy scene, collect multispectral pixel vectors and light source irradiance data in real time, complete dark current correction and irradiance compensation, and establish observation equations that match the imaging physical process. The high-frequency noise filtering and closed-off baseline subtraction module is configured to take pixel vectors and irradiance data as inputs, filter high-frequency noise, perform closed-off linear operations to strip the background baseline, and output an effective spectral vector. The non-negative solution module is configured to take the effective spectral vector as input, call the generalized pseudo-inverse matrix that is pre-calculated and fixed offline based on the endmember matrix, obtain the original abundance vector through matrix-vector multiplication, and then output the non-negative abundance vector through hardware non-negative truncation. The abundance reconstruction and feature pseudocolor mapping module is configured to take the output non-negative abundance vector as input, extract the endmember abundance components of lesion features to complete pseudocolor mapping, and output a lesion visualization image after mixing with the synchronous white light image.

2. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 1, characterized in that, The process of obtaining the endmember matrix, multispectral pixel vector, and light source irradiance includes: Endmember matrix Construction: Endmember matrix The dimension is ,in The number of spectral channels in a multispectral sensor. The amount of the dominant substance in an oral endoscopic setting; Multispectral pixel vector Acquisition: The front-end multispectral sensor uses a global shutter CMOS sensor and employs a time-division multispectral exposure design: [Image of the same spatial location] Images from five channels are acquired sequentially within the same frame period. Therefore, the response values ​​of the five channels can be obtained for the same spatial location within one frame, forming a pixel vector with a dimension of 5×1. ,in Corresponding to the center wavelength of the 5 channels; Light source irradiance Collection: Integrating a miniature photodiode at the light source output end achieves... Real-time data collection; The irradiance of each channel was measured using a standard radiometer at the standard working distance of the equipment. Simultaneously, the output voltage of the PD was recorded, and the linear correction coefficient for each channel was obtained by fitting the data. and That is, the first Irradiance of each channel ,in The real-time output voltage of the PD has a correction coefficient stored in the device's memory.

3. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 2, characterized in that, The established observation equation is: ； in, For the first Endmembers at wavelength The standardized spectral response at that location; For the first Class endmembers in spatial location Abundance at location; Baseline for broadband background; This is high-frequency system noise.

4. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 1, characterized in that, Method 1 for stripping background baselines: First, the original pixel vector Divide each channel element by the corresponding channel The spectral vector after irradiance normalization is obtained. ; Then the other channels Subtract the reference channel Multiply by calibration factor .

5. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 4, characterized in that, Method 2: right Irradiance normalization is obtained ; Then, normalized difference operations are performed on each band pair; Finally, the differential signals corresponding to the three endmembers are combined with the baseline subtraction results of the remaining channels to obtain a 5×1 dimension signal. vector.

6. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 1, characterized in that, In the nonnegativity solution module, the pseudo-inverse matrix is ​​calculated as follows: A 5×3 matrix Perform SVD decomposition to obtain ,in It is a 5×5 orthogonal matrix. It is a 5×3 diagonal matrix, with the elements on the diagonal being... singular values, It is a 3×3 orthogonal matrix; Will Less than the maximum singular value on the diagonal The elements are treated as zero, and the reciprocals of the remaining non-zero elements are transposed to obtain the result. ; Finally obtained pseudo-reversal The dimensions are 3×5; Will Each element is quantized, and the quantized result The matrix is ​​stored in the processor's on-chip SRAM.

7. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 6, characterized in that, Explanation of real-time matrix multiplication: It is a 3×5 matrix. Since the vector is 5×1, matrix multiplication involves 15 multiplications and 12 additions, specifically: ; and The computational logic is the same; for Each element is directly compared with 0 using a hardware comparator. If the element value is greater than or equal to 0, the original value is output; otherwise, 0 is output.

8. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 1, characterized in that, The process of extracting endmember abundance components of lesion features to complete pseudocolor mapping includes: The vector is 3×1, where the third component The abundance of porphyrin telomeres in the corresponding lesion tissue ranges from 0 to 1, where 0 represents that the pixel has no lesion components and 1 represents that the pixel is completely composed of lesion components. Validation based on a preset number of clinical samples, set up The threshold; False color mapping will The numerical value is converted into a warning color transparency, allowing doctors to intuitively judge the location and risk level of the lesion.

9. The real-time AI detection system for oral lesions based on multispectral imaging according to claim 8, characterized in that, The output lesion visualization image is as follows: For each spatial location The corresponding white light pixel value , , The Alpha value of the warning color is The RGB value of the warning color is , , The mixing formula is: ； The mixed RGB888 image is written to double-buffered video memory, writing the current frame while simultaneously outputting the previous frame to the display bus.

Images