An AI-powered test strip detection system for abnormal states based on multimodal data fusion

Abstract

This invention discloses an AI-powered test strip detection system for abnormal states based on multimodal data fusion, comprising a display module, an image acquisition module, and a data processing module. The data processing module integrates a virtual biochemical analysis device. Its fluid physics feature extraction module acquires a sequence of biochemical test strip images to output a fluid physics feature vector. A hardware timing and light source control module controls a multispectral light source array to project alternating pulsed light and simultaneously acquires reflected images. An optical feature reconstruction module generates a pseudo-hyperspectral feature vector based on the reflected images. A multimodal feature fusion module projects the fluid physics feature vector and the pseudo-hyperspectral feature vector into a multimodal latent space to perform cross-attention calculations, outputting a multimodal fused feature vector. A concentration regression calculation module performs forward propagation calculations on the multimodal fused feature vector to obtain a concentration calibration value. A result mapping and output module performs numerical mapping on the concentration calibration value, outputs the results, and stores the detection results.

Description

Technical Field

[0001] This invention relates to the field of biochemical detection and data processing technology, specifically to an AI-based test strip detection system for abnormal states based on multimodal data fusion. Background Technology

[0002] Biochemical test strips, as a convenient in vitro diagnostic tool, are widely used for the rapid detection of various biochemical biomarkers. Traditional test strip detection equipment typically uses an image acquisition module to read and analyze the colorimetric results of the reaction area on the test strip. In practical applications, existing test strip detection technologies have several limitations.

[0003] Most existing detection systems rely on a single optical imaging mode to acquire reaction colorimetric information. This single-modal data acquisition method is highly dependent on sample condition and external environment, and is easily affected by differences in sample matrix or stray light in the environment during actual detection. Single optical data cannot fully reflect the true state of biochemical reactions, leading to deviations in the quantitative calculation of biochemical marker concentrations, and the accuracy needs to be improved.

[0004] Meanwhile, the substrate of biochemical test strips exhibits differences in physical microstructure between different production batches, with uneven distribution of internal pores. Existing detection methods typically ignore the variations in chromatographic flow rate caused by these microscopic physical differences, failing to quantitatively compensate for the pore characteristics of the substrate. These physical differences in the substrate directly interfere with the hydrodynamic process of the test strip, leading to fluctuations in the rheological characteristics of the same type of sample on different test strips, thus reducing the consistency of test results.

[0005] Furthermore, existing test strip detection devices generally use a preset fixed waiting period as the trigger condition when controlling the acquisition of optical signals. Because different liquid samples have different actual rheological properties, the time it takes for the target sample to complete chromatography and reach stable color development on the test strip varies. Using a fixed waiting period for control cannot dynamically adjust the acquisition timing according to the actual physical rheological characteristics of the liquid, easily leading to problems such as premature imaging or color degradation, further affecting the final detection accuracy of the system. Summary of the Invention

[0006] The first aspect of the present invention provides an AI test strip detection system for abnormal states based on multimodal data fusion, which is deployed in a terminal device and includes a display module, an image acquisition module and a data processing module.

[0007] The data processing module integrates a virtual biochemical analysis device. Specifically, the fluid physics feature extraction module calls the image acquisition module to acquire a continuous image sequence of the reaction area of ​​the biochemical test strip. Based on this continuous image sequence, it performs dynamic identification of the fluid boundary and calculates and outputs a fluid physics feature vector. The hardware timing and light source control module, upon receiving an interrupt signal indicating the fluid diffusion time has elapsed, generates and sends a timing pulse sequence. This sequence controls the multispectral light source array in the terminal device to project alternating pulses of light according to a preset wavelength polling order, and simultaneously controls the image acquisition module to acquire reflection images at each preset wavelength. These reflection images are then stitched together to generate a multispectral image data cube. The optical feature reconstruction module reads the multispectral image data cube, performs spatial averaging on the pixels within the reaction area of ​​the biochemical test strip acquired by the image acquisition module to extract a multi-channel optical reflection intensity sequence, and maps this sequence to a high-dimensional space using a preset optical reconstruction matrix, generating a pseudo-hyperspectral feature vector. The multimodal feature fusion module receives fluid physics feature vectors and pseudo-hyperspectral feature vectors. It projects these vectors into a multimodal latent space of the same dimension using a fully connected layer, and performs cross-attention calculation within this space to output a multimodal fused feature vector. The concentration regression calculation module loads the multimodal fused feature vector into a pre-deployed concentration regression network for forward propagation. Through weight matrix multiplication, bias addition, and activation function mapping between the fully connected hidden layer and the linear output layer, it outputs a one-dimensional biochemical biomarker concentration calibration value. The result mapping and output module inputs the biochemical biomarker concentration calibration value into a preset standard regression model for numerical mapping, outputting the final concentration result. It extracts the system clock to generate a test timestamp, and concatenates the final concentration result, the test timestamp, and the batch identification code of the current liquid sample using feature fields to generate a structured detection result data packet. The result mapping and output module visualizes the final concentration result through a display module and writes the structured detection result data packet to the terminal device for local storage.

[0008] Furthermore, the fluid physics feature extraction module is configured as follows: controlling the display module to project a preset spatial alternating coded grating onto the biochemical test strip; extracting the optical phase distortion of the fluid wetting front region in a continuous image sequence; locating the spatial position of the fluid wetting front based on the optical phase distortion, and calculating the spatial distribution variance of the local distortion data to extract the pore correction coefficient; calculating the dynamic viscosity time series of the liquid sample on the biochemical test strip based on the spatial position of the fluid wetting front and the pore correction coefficient; performing feature extraction operations on the dynamic viscosity time series and the pore correction coefficient sequence, and concatenating the extracted features to generate a fluid physics feature vector.

[0009] Specifically, when extracting optical phase distortion variables in the fluid wetting front region of a continuous image sequence, the fluid physics feature extraction module is configured as follows: performing a two-dimensional Fourier transform on the reflected image to convert the image signal to the frequency domain; using a preset frequency domain filtering window to extract the fundamental frequency component containing the phase information of the spatially alternating coded grating, and filtering out the background zero-frequency component and high-frequency noise; performing a two-dimensional inverse Fourier transform on the extracted fundamental frequency component to calculate the encapsulated phase distribution; applying a phase unpacking algorithm to extract local phase distortion variables; obtaining the spatial gradient distribution sequence by taking the first spatial derivative of the local phase distortion variables along the fluid tomography propagation direction, and extracting the coordinates of the extreme points of the spatial gradient distribution to determine the spatial physical location of the fluid wetting front.

[0010] Furthermore, the biochemical analysis virtual device is also equipped with a timing prediction module. The timing prediction module is configured as follows: it imports the fluid physical feature vector as an input parameter into a pre-trained time prediction model to calculate the targeted color development time node; it extracts the initial timestamp when the liquid sample is added to the biochemical test strip, and sums the targeted color development time node with the initial timestamp to calculate the system absolute trigger time; it writes the system absolute trigger time into the hardware timer of the main control circuit, and when the current system time reaches the system absolute trigger time, the hardware timer sends a fluid diffusion time arrival interrupt signal to the hardware timing and light source control module.

[0011] The hardware timing and light source control module controls the power supply status of the multispectral light source array according to the level flipping logic of the timing pulse sequence; according to the preset wavelength polling order, it sequentially lights up the light-emitting diodes with different center wavelengths in the multispectral light source array with a preset pulse driving current, projecting multispectral alternating pulse light onto the surface of the biochemical test strip; the exposure control terminal of the control image acquisition module is electrically synchronized with the timing pulse sequence, and within a single pulse cycle of the multispectral alternating pulse light, the control image acquisition module performs an exposure action with a duration equal to the pulse width.

[0012] The optical feature reconstruction module calls a pre-calibrated spectral reconstruction matrix, which is an optical feature conversion base obtained by performing a broadband scan of the same type of biochemical test strip samples using a standard hyperspectral instrument and performing multiple linear regression extraction. The optical feature reconstruction module constructs a discrete multispectral intensity column vector from the multi-channel optical reflectance intensity sequence, and performs matrix multiplication operation between the discrete multispectral intensity column vector and the spectral reconstruction matrix to calculate the pseudo hyperspectral feature vector.

[0013] The multimodal feature fusion module multiplies the projected pseudo-hyperspectral feature vector with a preset query weight matrix to map it into a query matrix. Simultaneously, it multiplies the projected fluid physics feature vector with preset bond weight matrices and preset value weight matrices, mapping them into bond matrices and value matrices, respectively. Subsequently, the module multiplies the query matrix with the transpose of the bond matrix to obtain the product. It then scales the product using the square root of the dimension scaling constant of the bond matrix. A normalized exponential activation function is applied to the scaled result. The result of the normalized exponential activation function is multiplied with the value matrix to obtain the multimodal latent space feature matrix. The multimodal latent space feature matrix is ​​numerically stabilized through layer normalization. Finally, the numerically stabilized multimodal latent space feature matrix is ​​input into a feedforward neural network layer for nonlinear activation, outputting a multimodal fused feature vector.

[0014] The concentration regression calculation module controls the multimodal fusion feature vector to pass through the weight matrix multiplication, bias addition and activation function mapping of each fully connected hidden layer in sequence to extract low-dimensional feature parameters for concentration regression; it controls the linear output layer of the concentration regression network to receive the hidden state vector output by the fully connected hidden layer; it multiplies the hidden state vector with the weight parameters of the linear output layer and adds it with the bias parameters to calculate and output a one-dimensional biochemical biomarker concentration calibration value.

[0015] The fluid physics feature extraction module obtains the effective equivalent pore radius of the nitrocellulose membrane of the pre-configured biochemical test strip, the surface tension parameters of the liquid sample, and the basic constant of the solid-liquid contact angle. A pore correction coefficient is introduced to nonlinearly modify the classical capillary flow model, constructing a capillary dynamics equation with pore correction. The spatial position of the fluid wetting front is converted into a dynamic sequence of the instantaneous advance distance of the front over time. Based on the proportional relationship between the square of the instantaneous advance distance of the front and time, the capillary dynamics equation with pore correction is solved using the basic constant and the pore correction coefficient, and the dynamic viscosity time series of the liquid sample during the chromatography process is calculated.

[0016] This invention provides an AI-based test strip detection system for abnormal states based on multimodal data fusion. It has the following beneficial effects:

[0017] 1. This invention utilizes a multimodal feature fusion module to project fluid physical feature vectors and pseudo-hyperspectral feature vectors into the same multimodal latent space for cross-attention calculation. This technical solution combines the chromatographic rheological properties of liquid samples on biochemical test strips with the multi-channel optical characteristics of biochemical reaction endpoints. Through forward propagation calculation using a concentration regression network, the system integrates the data dimensions of physical and optical modes, reducing the probability of single optical imaging detection modes being easily affected by sample matrix differences or environmental stray light interference, and improving the accuracy of quantitative calculation of biochemical biomarker concentrations.

[0018] 2. This invention utilizes a fluid physics feature extraction module to extract optical phase distortion in the wetting front region of a continuous image sequence, and then calculates the spatial distribution variance of the local distortion data to extract the pore correction coefficient. By incorporating the pore correction coefficient into the capillary dynamics equation and calculating the dynamic viscosity time series, the system can dynamically quantify the microstructural changes of the porous medium in the early stages of detection. This nonlinear spatial feature correction mechanism can quantitatively compensate for the chromatographic flow rate variations caused by the non-uniformity of pores in different batches of biochemical test strips, eliminating the interference of substrate physical differences on the fluid dynamics process of the test strip.

[0019] 3. This invention uses a timing prediction module to import the fluid physics feature vector into a pre-trained time prediction model to calculate the targeted color development time node, and combines this with the initial timestamp to generate the system's absolute trigger time. When the hardware timer reaches this time node, the hardware timing and light source control module is actively triggered to control the multispectral light source array to project alternating pulse light in wavelength polling order and simultaneously acquire images. This closed-loop control timing allows the detection system to adaptively adjust the optical image acquisition node according to the true rheological properties of the target liquid sample, avoiding the problems of premature image acquisition or color degradation caused by using a fixed waiting period. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 This invention provides an AI-based test strip detection system for abnormal states based on multimodal data fusion. The system is deployed in a terminal device and may include a display module, an image acquisition module, and a data processing module.

[0023] The terminal device is an electronic device with independent computing capabilities and a human-computer interaction interface. In specific implementations, the terminal device is a smartphone, tablet computer, or portable biochemical analyzer. The terminal device integrates a power module, a storage module, and a communication module to maintain the basic operation of each hardware unit. The system operating environment is set to natural ambient light or conventional indoor lighting, and the system hardware connection structure does not include external enclosed light-proof dark box devices.

[0024] The display module is mounted on the surface of the terminal device. It employs an organic light-emitting diode (OLED) display panel or a liquid crystal display (LCD) panel. The array of pixels in the display module emits light under the control of electrical signals output from the data processing module. The display module projects a grating signal of a preset spatial frequency onto a biochemical test strip placed in its proximal detection area, and projects multispectral pulsed light with different center wavelengths at specific time intervals. The display module serves as the system's active structured light source.

[0025] The image acquisition module is located on the same side of the terminal device as the display module. The image acquisition module includes a photosensitive sensor array and an optical lens group. The detection area of ​​the terminal device is located where the effective light-emitting area of ​​the display module coincides with the field of view of the image acquisition module. The photosensitive surface of the lens of the image acquisition module faces the placement area of ​​the biochemical test strip. The image acquisition module is used to acquire the reflected and scattered images of the surface of the biochemical test strip after being illuminated by the light projected from the display module, and converts the optical signals into continuous digital video stream data output.

[0026] The data processing module is located on the internal motherboard of the terminal device and is electrically connected to the display module and image acquisition module via an internal hardware bus. Specifically, the data processing module includes a central processing unit (CPU) and a graphics processing unit (GPU). The data processing module sends backlight control commands to the display module to adjust the duty cycle, color channels, and spatial modulation patterns of the screen display. The data processing module also receives and parses the video stream data output by the image acquisition module.

[0027] The hardware structure of the terminal device does not include physical temperature sensors or microfluidic velocimetry sensors for monitoring the ambient temperature or liquid physical state of the biochemical test strips. The system relies solely on a display module and an image acquisition module to achieve a closed loop for the transmission and reception of photoelectric signals.

[0028] This invention provides an AI-powered test strip detection method for abnormal states based on multimodal data fusion. This method is executed by the aforementioned AI-powered test strip detection system for abnormal states based on multimodal data fusion and may include:

[0029] In step S100, the data processing module controls the display module to project a preset spatial alternating coding grating onto the nitrocellulose membrane area of ​​the biochemical test strip. The image acquisition module activates video acquisition to record the reflected light signal of the biochemical test strip onto the spatial alternating coding grating in a dry state, generating initial baseline data.

[0030] In step S200, after the liquid sample contacts the biochemical test strip and chromatography occurs, the image acquisition module continuously acquires a continuous video stream containing the biochemical test strip. The data processing module extracts the phase gradient change or grayscale gradient change in the fluid wetting front region of the continuous video stream. The data processing module calculates the optical distortion data of the spatially alternating coded grating based on the extracted gradient changes. The data processing module locates the spatial position of the fluid wetting front based on the optical distortion data and calculates the spatial distribution variance of the local distortion data to characterize the microscopic inhomogeneity of the membrane material pores, and then extracts the pore correction coefficient.

[0031] In step S300, the data processing module records the spatial position of the fluid wetting front as a function of time. The data processing module substitutes the pore correction coefficient into a preset capillary dynamics equation. Based on the ratio of the square of the front advancement distance to time, the data processing module calculates the dynamic viscosity parameters of the liquid sample on the biochemical test strip. The data processing module combines the dynamic viscosity parameters with the pore correction coefficient to generate a physical modal feature vector.

[0032] In step S400, the data processing module calculates the trigger time for the target analyte to reach the detection line area of ​​the biochemical test strip based on capillary dynamics equations. When this trigger time is reached, the data processing module controls the display module to stop projecting the spatially alternating coded grating. The display module then switches to emitting a combination of narrowband pulses with different center wavelengths according to a preset timing sequence. The image acquisition module simultaneously acquires multispectral reflectance images corresponding to each wavelength. The data processing module reconstructs the optical modal feature vector based on the multispectral reflectance images.

[0033] In step S500, the data processing module inputs the physical modality feature vector and the optical modality feature vector into a preset multimodal fusion neural network. The multimodal fusion neural network adjusts the channel weights of the optical modality feature vector using the physical modality feature vector. The multimodal fusion neural network calculates and outputs the quantitative concentration of the target analyte or the abnormal classification status of the biochemical test strip as the final detection result. The data processing module transmits the detection result to the display module for interface display.

[0034] This invention provides an implementation method for active illumination initialization and spatial coding grating generation, which may include:

[0035] The data processing module initiates the detection program, controlling the display module to enter the active structured light source mode. A two-dimensional spatial coordinate system is established based on the effective emitting plane of the display module, and the spatial coordinates are set as follows: The time variable is The data processing module controls the display module in spatial coordinates. With time Output a specific light intensity distribution, and define the light intensity distribution function as follows: .

[0036] In time During the system initialization phase, the data processing module generates data with a preset spatial frequency. The spatially alternating coded grating signal is converted into grayscale control signals for each array pixel of the display module, thereby controlling the display module to emit light. The display module projects the generated spatially alternating coded grating onto the nitrocellulose membrane surface of the biochemical test strip placed in the detection area.

[0037] The intensity distribution function of the initially projected grating satisfies the following relationship:

[0038]

[0039] in, Indicates the initialization time display module in spatial coordinates The intensity of light projected at a location. This represents the background light intensity parameter of the spatially alternating encoded grating. This indicates the modulation contrast of the grating signal. This represents the spatial frequency of the spatially alternating encoded grating in the projection direction. This represents the coordinate value along the direction of the spatial frequency variation of the grating. This represents the initial spatial phase parameter of the spatially alternating coding grating. The data processing module pre-configures these optical parameters based on the nominal size and working distance of the biochemical test strip.

[0040] Within the same time period during which the display module projects the spatially alternating coded grating, the data processing module sends a synchronization trigger signal to the image acquisition module. The image acquisition module receives the synchronization trigger signal and performs an image acquisition operation to obtain the reflection and scattering images of the spatially alternating coded grating on the surface of the biochemical test strip in a dry state.

[0041] The image acquisition module converts the captured reflected and scattered images into a digital pixel matrix, denoted as the initial baseline image. The data processing module receives and stores the initial baseline image. This image records the inherent optical reflectivity and physical porosity scattering characteristics of the nitrocellulose membrane surface when the biochemical test strip is not wetted by liquid. The data processing module uses this as background reference data for subsequent calculations of optical distortion.

[0042] This invention provides an implementation method for dynamic tracking and phase shift extraction of the fluid wetting front, which may include:

[0043] After the liquid sample is added to the sample application area of ​​the biochemical test strip, the liquid is chromatographically propelled through the nitrocellulose membrane due to capillary action. The porous medium in the dry area of ​​the biochemical test strip is filled with air, while the pores in the wetted area are filled with the liquid sample. The difference in optical refractive index between the air and the liquid sample causes a change in the scattering path of the spatially alternating encoded grating projected onto the surface of the nitrocellulose membrane, resulting in optical phase distortion in the fluid wetting front region at the dry-wet interface.

[0044] The image acquisition module continuously acquires a sequence of reflective images from the surface of the biochemical test strip at a preset frame rate, and transmits the image sequence containing time dimension information to the data processing module. The data processing module extracts the time. The reflected image is defined as The intensity distribution of the reflected image after being affected by fluid wetting satisfies the following equation:

[0045]

[0046] in, Indicates time In coordinates Background light intensity distribution at the location. This indicates the modulation amplitude of the reflection grating. This represents the spatial frequency of the spatially alternating encoded grating in the projection direction. This represents the initial spatial phase parameter. This indicates the local phase distortion caused by liquid wetting and abrupt changes in the refractive index of the porous medium.

[0047] The data processing module processes the reflected image. A two-dimensional Fourier transform is performed to convert the image signal from the spatial domain to the frequency domain. Subsequently, the data processing module uses a preset frequency domain filtering window to extract the fundamental frequency component containing the phase information of the spatially alternating coded grating, and filters out background zero-frequency components and high-frequency noise. The data processing module performs a two-dimensional inverse Fourier transform on the extracted fundamental frequency component to calculate the encapsulated phase distribution, and applies a phase unpacking algorithm to extract continuous local phase distortions. .

[0048] The data processing module processes local phase distortion along the direction of fluid tomography. The first spatial derivative is calculated to obtain the spatial gradient distribution sequence. Since the refractive index change in the fluid wetting front region exhibits a step-like characteristic, this physical boundary corresponds to the extreme point of the spatial gradient distribution. The data processing module extracts the coordinates of this extreme point to determine the time. The spatial physical location of the fluid wetting front.

[0049] The data processing module uses the spatial reference coordinates and time of the starting point of the biochemical test strip. The spatial physical location of the fluid wetting front was used to calculate the instantaneous advance distance of the fluid front. The data processing module synchronously traverses the image sequence output by the image acquisition module, repeating the above transformation and extraction process frame by frame, and outputs the instantaneous propulsion distance of the fluid front. Dynamic sequences changing over time and extracted local phase distortions .

[0050] This invention provides an implementation method for deriving the porosity correction coefficient based on scattering variance, which may include:

[0051] The data processing module uses the previously extracted local phase distortion variables. A local calculation window of a preset size is captured in the region behind the fluid wetting front. The local calculation window corresponds to the area of ​​the biochemical test strip where the liquid sample has been completely soaked, and the preset size is determined by the set number of pixels or physical length mapping.

[0052] The data processing module calculates the spatial distribution variance of local phase distortion within the local calculation window. The spatial distribution variance characterizes the local optical scattering intensity variation caused by microstructural inhomogeneities in porous media. The calculation of the spatial distribution variance satisfies the following formula:

[0053]

[0054] in, Indicates time Spatial distribution variance of local phase distortion. This represents a local calculation window. This represents the total number of pixels within the local calculation window. This represents the local phase distortion. Indicates time The average value of the local phase distortion within the local calculation window.

[0055] There is a physical mapping relationship between the structural porosity and local optical scattering properties of nitrocellulose membranes. The data processing module calculates the porosity correction coefficient using the spatial distribution variance. The porosity correction coefficient is calculated according to the following formula:

[0056]

[0057] in, Indicates time The porosity correction coefficient. , as well as This represents the pre-calibrated empirical constants. The data processing module obtains these empirical constants by retrieving the calibration file for the corresponding batch of biochemical test strips from the terminal device's internal storage module. , as well as empirical constants , as well as These are values ​​obtained by fitting data from multiple sets of standard concentration samples.

[0058] The data processing module calculates the porosity correction coefficient at multiple consecutive time points. The sequence undergoes time-dimensional smoothing filtering (such as a moving average filtering algorithm). The data processing module outputs a smoothed pore correction coefficient. This smoothed pore correction coefficient is used to correct the physical parameters of the capillary flow in subsequent fluid dynamics equations, thereby eliminating the interference of differences in pore structure between different batches of biochemical test strips on the detection results.

[0059] This invention provides an implementation method for constructing capillary dynamics equations incorporating pore correction, which may include:

[0060] The data processing module obtains the instantaneous propulsion distance of the fluid front calculated above. Dynamic sequence over time and smoothed porosity correction coefficient The chromatographic process of liquid samples in the nitrocellulose membrane of biochemical test strips follows the laws of capillary dynamics.

[0061] The classic capillary flow model assumes that the internal pores of the porous medium are uniformly distributed in a cylindrical shape. The data processing module introduces a smoothed pore correction coefficient. The classical capillary flow model is modified with nonlinear spatial characteristics to construct a capillary dynamics equation that incorporates pore correction.

[0062] The capillary dynamics equation with pore correction satisfies the following formula:

[0063]

[0064] in, Indicates time The instantaneous advance distance of the fluid front. This indicates the effective equivalent pore radius of the nitrocellulose membrane in the biochemical test strip. This represents the surface tension parameter of the liquid sample. This indicates the solid-liquid contact angle between the liquid sample and the nitrocellulose membrane material. Indicates time Dynamic viscosity parameters of liquid samples during chromatography. Indicates time Pore ​​correction coefficient after smoothing. This represents the time variable during which the liquid sample comes into contact with the biochemical test strip and undergoes chromatography.

[0065] Effective equivalent pore radius Surface tension parameters and solid-liquid contact angle The data processing module obtains the effective equivalent pore radius by reading the pre-calibrated and configured table of fundamental constants for specific types of liquid samples and batches of biochemical test strips from the internal storage module of the terminal device. Surface tension parameters and solid-liquid contact angle .

[0066] The data processing module establishes the instantaneous advance distance of the fluid front based on the capillary dynamics equation with pore correction. The square of the time variable The dynamic proportional relationship between them is used to characterize the changes in the rheological properties of liquid samples and to establish a mathematical mapping between the fluid propulsion image features in the spatial domain and the rheological parameters in the physical domain. This serves as the underlying computational model for subsequently deriving the physical property characteristics of liquid samples and determining the triggering sequence of optical detection.

[0067] This invention provides an implicit solution implementation method for fluid physical feature vectors, which may include:

[0068] The data processing module, based on the aforementioned capillary dynamics equation with pore correction, processes the data at time... Dynamic viscosity parameters Perform algebraic transformations and solve the problem. Dynamic viscosity parameters. The calculation satisfies the following formula:

[0069]

[0070] The definitions of the symbols in the formula are consistent with those in the aforementioned capillary dynamics equation with pore correction.

[0071] The data processing module will process the time variables corresponding to the image sequences continuously output by the image acquisition module. Instantaneous propulsion distance of the fluid front and the smoothed porosity correction coefficient Substitute the dynamic viscosity parameters The calculation formula is used to calculate the dynamic viscosity time series of the liquid sample during the chromatography process.

[0072] The data processing module processes the dynamic viscosity time series and the smoothed pore correction coefficient. The sequence performs feature extraction operations, specifically calculating the mean, variance, and time-first derivative extrema of the dynamic viscosity time series. Simultaneously, the data processing module monitors the instantaneous advance distance of the fluid front. The rate of change, when the fluid front instantaneously advances a distance When the time derivative is lower than the preset rate threshold, the chromatography process is considered to have entered the final stage, and the smoothed pore correction coefficient is extracted simultaneously. The steady-state convergence parameter of the sequence.

[0073] Subsequently, the data processing module processes the mean parameter, variance parameter, time first derivative extreme value parameter, and smoothed pore correction coefficient of the dynamic viscosity time series. The steady-state convergence parameters of the sequence are concatenated according to a preset dimensional order to generate a fluid physics feature vector. .

[0074] Fluid physical eigenvectors The data is stored in numerical form in the cache space inside the data processing module and serves as quantitative input data characterizing the interaction between the rheological properties of the liquid sample and the microporous medium of the biochemical test strip. This data is then used to input the multimodal fusion neural network in subsequent stages.

[0075] This invention provides an implementation method for real-time calculation and prediction of targeted color development time points, which may include:

[0076] The data processing module retrieves the pre-trained time prediction model from the memory and processes the fluid physics feature vectors generated by the aforementioned calculation. The pre-trained time prediction model is imported as an input parameter into the pre-trained time prediction model. The pre-trained time prediction model is an artificial neural network model (e.g., a fully connected neural network) trained based on the colorimetric reaction process data of multiple sets of liquid samples with known rheological properties on the same batch of biochemical test strips.

[0077] In the internal computational flow of the pre-trained time prediction model, the fluid physics feature vector The signal enters through the input layer, undergoes nonlinear mapping calculations using the weight matrices and activation functions of the hidden layer nodes, and finally yields the targeted color development time point from the output layer. Targeted color development time point The physical moment when the biochemical reaction in the detection area of ​​the biochemical test strip reaches the preset optical contrast requirement.

[0078] The data processing module extracts the initial timestamp recorded by the system when the liquid sample is added to the biochemical test strip, and targets the color development time point. The system's absolute trigger time is obtained by summing the timestamp with the initial timestamp.

[0079] Subsequently, the data processing module writes the system absolute trigger time into the hardware timer of the main control circuit and starts the time monitoring process. The hardware timer compares the current system time with the configured system absolute trigger time in real time according to the system clock frequency.

[0080] When the current system time reaches the system absolute trigger time, the hardware timer sends a time-reach interrupt signal to the data processing module. The data processing module receives the time-reach interrupt signal and immediately generates a detection trigger command, sending an action execution requirement to the image acquisition module to initiate optical image acquisition of the reaction endpoint state, thereby adaptively adjusting the detection waiting period for liquid samples with different physical properties.

[0081] This invention provides an implementation method for hardware interrupt control and timing pulse multispectral light source switching, which may include:

[0082] Upon receiving the aforementioned time-arrival interrupt signal, the data processing module immediately enables the internal synchronous trigger bus and sends a timing pulse sequence to the light source driving module and the image acquisition module via the synchronous trigger bus.

[0083] The light source driving module receives a timing pulse sequence and controls the power supply state of the multispectral light source array according to the level-flipping logic of the timing pulse sequence. The multispectral light source array contains multiple independent channels of light-emitting diodes (LEDs) with different center wavelengths (e.g., LEDs with center wavelengths corresponding to red, green, and blue light). The light source driving module sequentially illuminates LEDs with different center wavelengths according to a preset wavelength polling order and a preset pulse driving current to project multispectral alternating pulse light onto the surface of the biochemical test strip.

[0084] The exposure control terminal of the image acquisition module is electrically synchronized with the timing pulse sequence. Within a single pulse cycle of multispectral alternating pulsed light, the image acquisition module performs an exposure operation of the same duration as the pulse width to acquire the reflectance image of the detection area of ​​the biochemical test strip under illumination at a specific wavelength.

[0085] The image acquisition module sequentially acquires reflectance images and stitches them together according to their corresponding wavelength labels to generate a multispectral image data cube. The multispectral image data cube is then transmitted to the memory space of the data processing module as the optical basis data for subsequent quantitative analysis of biochemical biomarker concentrations.

[0086] This invention provides an implementation method for optical reconstruction of pseudo-hyperspectral eigenvectors, which may include:

[0087] The data processing module reads the aforementioned generated multispectral image data cube and extracts a multi-channel optical reflectance intensity sequence within the detection area of ​​the biochemical test strip by spatially averaging the pixels of the multispectral image data cube within the detection area. The multi-channel optical reflectance intensity sequence corresponds to the optical measurement response values ​​of the aforementioned multispectral light source array at each center wavelength.

[0088] The data processing module maps the multi-channel optical reflection intensity sequence to a high-dimensional optical feature space using a preset optical reconstruction model to generate pseudo-hyperspectral feature vectors. The calculation of the pseudo-hyperspectral feature vectors satisfies the following formula:

[0089]

[0090] in, This represents a pseudo-hyperspectral eigenvector. This represents a discrete multispectral intensity column vector composed of a multichannel optical reflection intensity sequence. This represents the pre-calibrated spectral reconstruction matrix.

[0091] The data processing module obtains the spectral reconstruction matrix by retrieving the system configuration file from internal memory. Spectral reconstruction matrix The optical feature conversion substrate was obtained by performing broadband scanning of similar biochemical test strip samples using a standard hyperspectral analyzer and extracting the scanning data through multiple linear regression.

[0092] The data processing module will calculate the pseudo-hyperspectral feature vector. The pseudo-hyperspectral feature vectors are stored in the internal cache space as numerical arrays. The quantitative optical input data, used to characterize the colorimetric state at the endpoint of a biochemical reaction, is compared with the aforementioned fluid physical feature vector generated by the solution. The features are concatenated and used together as the input feature source for the subsequent multimodal fusion neural network.

[0093] This invention provides an implementation of multimodal latent space mapping based on a cross-attention mechanism, which may include:

[0094] The data processing module reads the previously stored fluid physical feature vectors. and pseudo-hyperspectral eigenvectors And through a fully connected layer, the fluid physical feature vectors With pseudo-hyperspectral eigenvectors They are projected onto the same-dimensional multimodal latent space to achieve dimensional alignment of different data modalities.

[0095] The data processing module performs cross-attention calculation in the multimodal latent space. Specifically, it multiplies the projected pseudo-hyperspectral feature vector with the preset query weight matrix to map it into a query matrix, and simultaneously multiplies the projected fluid physics feature vector with the preset bond weight matrix and value weight matrix to map it into a bond matrix and a value matrix, respectively.

[0096] The cross-attention computation of multimodal latent space mappings satisfies the following formula:

[0097]

[0098] in, This represents the feature matrix of the fused multimodal latent space. This represents the query matrix generated by projecting pseudo-hyperspectral eigenvectors. This represents the bond matrix generated by projecting fluid physics eigenvectors. This represents the value matrix generated by projecting the fluid physics eigenvectors. This represents the dimension scaling constant of the key matrix.

[0099] The data processing module obtains the multimodal latent space feature matrix based on the cross-attention calculation formula. And through layer normalization operations, the feature matrix of the multimodal latent space is processed. Numerical stabilization processing was performed.

[0100] After processing, the data processing module will process the multimodal latent space feature matrix. The input is fed into a feedforward neural network layer for nonlinear activation to output a multimodal fusion feature vector. The multimodal fusion feature vector represents the cross-correlation features between the physical rheological properties and the optical properties of the biochemical reaction of the liquid sample, and is stored in the cache space of the data processing module as an input parameter for subsequent quantitative calculation of biochemical biomarker concentrations.

[0101] This invention provides an implementation method for physical feature-guided adaptive nonlinear calibration and network computation, which may include:

[0102] The data processing module extracts the previously stored multimodal fusion feature vector and loads it into a pre-deployed concentration regression network. The concentration regression network is a multi-layer feedforward neural network containing fully connected hidden layers and linear output layers.

[0103] The multimodal fusion feature vector is forward propagated in the concentration regression network, passing through the weight matrix multiplication, bias addition and activation function mapping of each fully connected hidden layer in turn, in order to extract low-dimensional feature parameters for concentration regression.

[0104] Subsequently, the linear output layer of the concentration regression network receives the hidden state vector output by the aforementioned fully connected hidden layer and outputs a one-dimensional biochemical biomarker concentration calibration value.

[0105] The network calculation of biochemical biomarker concentration calibration values ​​satisfies the following formula:

[0106]

[0107] in, This indicates the calibrated concentration value of biochemical markers. This represents the hidden state vector output by the last fully connected hidden layer in the concentration regression network. This represents the weight parameters of the linear output layer. This represents the bias parameters of the linear output layer.

[0108] The data processing module will calculate the calibration values ​​of biochemical marker concentrations. It is converted into a preset data format and synchronously written into the internal detection result database.

[0109] Simultaneously, the data processing module calls the display driver of the hardware motherboard to calibrate the concentration values ​​of biochemical markers. The data is sent to the underlying control circuit of the human-computer interaction panel to drive the external display screen to output the final quantitative detection result.

[0110] This invention provides an implementation method for regression and output of detection results, which may include:

[0111] The data processing module extracts the concentration calibration values ​​of the biochemical markers obtained from the aforementioned calculations. The data processing module retrieves the preset standard regression model from the internal memory and then processes the biochemical biomarker concentration calibration values. Input the data into a pre-defined standard regression model for numerical mapping to obtain the final concentration results converted to standard clinical units of measurement (e.g., mg / dL or mmol / L).

[0112] The pre-defined standard regression model is a parametric equation generated by performing benchmark tests on a series of calibration solutions with known clinical standard concentrations and then using the least squares method to perform linear or polynomial fitting on the test data.

[0113] After obtaining the final concentration result, the data processing module extracts the current system clock to generate a test timestamp, and concatenates the final concentration result, the test timestamp, and the batch identification code of the current liquid sample with feature fields to generate a structured test result data packet.

[0114] The data processing module writes the structured detection result data packets into the non-volatile memory on the system motherboard to persistently save the local detection logs. Simultaneously, the data processing module transmits the final concentration results to an external human-machine interface screen via the display bus interface, driving the screen to render and display the detection values ​​and corresponding unit information.

[0115] In addition, the data processing module sends the test result data packets to the system's network communication module. The network communication module then uploads the test result data packets to an external medical and health management system or host computer terminal according to a preset data transmission protocol, for subsequent medical analysis and data archiving by external devices.

[0116] The data processing module integrates a virtual biochemical analysis device, which can be divided into the following modules according to functional units:

[0117] The fluid physics feature extraction module is configured to call the image acquisition module to acquire a continuous image sequence of the reaction area of ​​the biochemical test strip and calculate and output a fluid physics feature vector. Specifically, this module controls the display module to project a preset spatial alternating coded grating onto the biochemical test strip; performs a two-dimensional Fourier transform on the acquired reflection image, extracts the fundamental frequency component using a preset frequency domain filtering window and performs a two-dimensional inverse Fourier transform, and applies a phase unpacking algorithm to extract the optical phase distortion of the fluid wetting front region; calculates the first spatial derivative of the local phase distortion along the fluid tomography advancement direction to obtain a spatial gradient distribution sequence, extracts the coordinates of extreme points to locate the spatial position of the fluid wetting front; calculates the spatial distribution variance of the local distortion data to extract the pore correction coefficient; obtains the fundamental constant and introduces the pore correction coefficient to construct a capillary dynamics equation combined with pore correction, and calculates the dynamic viscosity time series of the liquid sample on the biochemical test strip based on the proportional relationship between the square of the instantaneous advancement distance of the front and time; finally, performs feature extraction operations on the dynamic viscosity time series and the pore correction coefficient sequence, and concatenates the extracted features to generate a fluid physics feature vector.

[0118] The timing prediction module is configured to import the fluid physical feature vector as an input parameter into the pre-trained timing prediction model to calculate the target color development time node; extract the initial timestamp when the liquid sample is dropped onto the biochemical test strip, and sum the target color development time node and the initial timestamp to calculate the system absolute trigger time; write the system absolute trigger time into the hardware timer of the main control circuit, and when the current system time reaches the system absolute trigger time, the hardware timer sends a fluid diffusion time arrival interrupt signal to the hardware timing and light source control module.

[0119] The hardware timing and light source control module is configured to generate and send a timing pulse sequence after receiving the interrupt signal when the fluid diffusion time arrives; according to the level flipping logic of the timing pulse sequence, the multispectral light source array in the terminal device is controlled to project alternating pulse light in a preset wavelength polling order, and the exposure control terminal of the image acquisition module is synchronously controlled to acquire the reflection images at each preset wavelength, and the reflection images are stitched together to generate a multispectral image data cube.

[0120] The optical feature reconstruction module is configured to read the multispectral image data cube, perform spatial averaging on the pixels in the reaction area of ​​the biochemical test strip acquired by the image acquisition module, and extract the multi-channel optical reflectance intensity sequence; call the pre-calibrated spectral reconstruction matrix to construct the multi-channel optical reflectance intensity sequence into a discrete multispectral intensity column vector, and map it to a high-dimensional space through matrix multiplication to generate a pseudo-hyperspectral feature vector.

[0121] The multimodal feature fusion module is configured to receive fluid physics feature vectors and pseudo-hyperspectral feature vectors, and project them into a multimodal latent space of the same dimension through a fully connected layer. Cross-attention calculation is performed in the multimodal latent space to map the projected features into a query matrix, a key matrix, and a value matrix. After matrix multiplication, scaling, normalized exponential activation function calculation, and numerical stabilization processing, the features are input to a feedforward neural network layer for nonlinear activation and output as a multimodal fused feature vector.

[0122] The concentration regression calculation module is configured to load the multimodal fusion feature vector into a pre-deployed concentration regression network for forward propagation calculation. It controls the multimodal fusion feature vector to pass through the weight matrix multiplication, bias addition and activation function mapping of each fully connected hidden layer in sequence. The linear output layer receives the hidden state vector and calculates and outputs a one-dimensional biochemical biomarker concentration calibration value.

[0123] The result mapping and output module is configured to input the biochemical marker concentration calibration value into a preset standard regression model for numerical mapping and output the final concentration result; extract the system clock to generate a test timestamp; concatenate the final concentration result, test timestamp and batch identification code of the current liquid sample to generate a structured detection result data packet; visualize the final concentration result through the display module and write the structured detection result data packet to the local storage of the terminal device.

[0124] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An AI-based test strip detection system for abnormal states based on multimodal data fusion, deployed in a terminal device, characterized in that: include: The system comprises a display module, an image acquisition module, and a data processing module. The data processing module integrates a virtual biochemical analysis device, which includes: The fluid physics feature extraction module is configured to call the image acquisition module to obtain a continuous image sequence of the reaction area of ​​the biochemical test strip, perform dynamic identification of the fluid boundary based on the continuous image sequence, and calculate and output the fluid physics feature vector. The hardware timing and light source control module is configured to generate and send a timing pulse sequence after receiving the interrupt signal when the fluid diffusion time arrives; according to the timing pulse sequence, the multispectral light source array in the terminal device is controlled to project alternating pulse light in a preset wavelength polling order, and the image acquisition module is synchronously controlled to acquire the reflection images under each preset wavelength, and the reflection images are stitched together to generate a multispectral image data cube. The optical feature reconstruction module is configured to read the multispectral image data cube, perform spatial averaging on the pixels in the reaction area of ​​the biochemical test strip acquired by the image acquisition module, and extract the multi-channel optical reflectance intensity sequence; the optical feature reconstruction module maps the multi-channel optical reflectance intensity sequence to a high-dimensional space through a preset optical reconstruction matrix to generate a pseudo-hyperspectral feature vector. The multimodal feature fusion module is configured to receive fluid physical feature vectors and pseudo hyperspectral feature vectors, project the fluid physical feature vectors and pseudo hyperspectral feature vectors into the same-dimensional multimodal latent space through a fully connected layer, and perform cross-attention calculation in the multimodal latent space to output the multimodal fused feature vector. The concentration regression calculation module is configured to load the multimodal fusion feature vector into a pre-deployed concentration regression network for forward propagation calculation. Through the weight matrix multiplication, bias addition and activation function mapping of the fully connected hidden layer and the linear output layer, it outputs a one-dimensional biochemical biomarker concentration calibration value. The result mapping and output module is configured to input the biochemical marker concentration calibration value into a preset standard regression model for numerical mapping and output the final concentration result; extract the system clock to generate a test timestamp, and concatenate the final concentration result, test timestamp and batch identification code of the current liquid sample to generate a structured detection result data packet; the result mapping and output module visualizes the final concentration result through the display module and writes the structured detection result data packet to the local storage of the terminal device.

2. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 1, characterized in that: The fluid physical feature extraction module is specifically configured as follows when calculating and outputting the fluid physical feature vector: The display module is controlled to project a preset spatial alternating coded grating onto the biochemical test strip; Extract the optical phase distortion of the fluid wetting front region in the continuous image sequence; The spatial location of the fluid wetting front is determined based on the optical phase distortion, and the spatial distribution variance of the local distortion data is calculated to extract the porosity correction coefficient. Based on the spatial position of the fluid wetting front and the pore correction coefficient, the dynamic viscosity time series of the liquid sample on the biochemical test strip is calculated. Feature extraction is performed on the dynamic viscosity time series and the pore correction coefficient series, and the extracted features are concatenated to generate the fluid physics feature vector.

3. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 2, characterized in that: The fluid physics feature extraction module is specifically configured as follows when extracting the optical phase distortion of the fluid wetting front region in the continuous image sequence: Perform a two-dimensional Fourier transform on the reflected image to convert the image signal to the frequency domain; The fundamental frequency component containing the phase information of the spatially alternating coded grating is extracted using a preset frequency domain filtering window, while background zero-frequency components and high-frequency noise are filtered out. The extracted fundamental frequency component is subjected to a two-dimensional inverse Fourier transform to calculate the encapsulated phase distribution; Local phase distortion variables are extracted using a phase unpacking algorithm; The spatial gradient distribution sequence is obtained by taking the first spatial derivative of the local phase distortion along the direction of fluid tomography. The coordinates of the extreme points of the spatial gradient distribution are extracted to determine the spatial physical location of the fluid wetting front.

4. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 1, characterized in that: The biochemical analysis virtual device is also equipped with a timing prediction module, which is configured as follows: The fluid physics feature vector is imported as an input parameter into the pre-trained time prediction model to calculate the target color development time node; Extract the initial timestamp when the liquid sample is dropped onto the biochemical test strip, and sum the targeted color development time node with the initial timestamp to obtain the absolute trigger time of the system; The system absolute trigger time is written into the hardware timer of the main control circuit. When the current system time reaches the system absolute trigger time, the hardware timer sends the fluid diffusion time arrival interrupt signal to the hardware timing and light source control module.

5. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 1, characterized in that: When the hardware timing and light source control module controls the multispectral light source array to project the alternating pulse light according to a preset wavelength polling sequence, the specific configuration is as follows: The power supply state of the multispectral light source array is controlled according to the level-flipping logic of the timing pulse sequence; According to the preset wavelength polling order, the light-emitting diodes with different center wavelengths in the multispectral light source array are lit sequentially with a preset pulse driving current, and multispectral alternating pulse light is projected onto the surface of the biochemical test strip; The exposure control terminal of the image acquisition module is electrically synchronized with the timing pulse sequence. Within a single pulse cycle of the multispectral alternating pulse light, the image acquisition module is controlled to perform an exposure action with a duration equal to the pulse width.

6. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 1, characterized in that: The optical feature reconstruction module is specifically configured as follows when generating the pseudo-hyperspectral feature vector: The pre-calibrated spectral reconstruction matrix is ​​invoked. The spectral reconstruction matrix is ​​an optical feature conversion substrate obtained by performing broadband scanning of the same type of biochemical test strip samples using a standard hyperspectral analyzer and performing multiple linear regression extraction. The multi-channel optical reflection intensity sequence is constructed as a discrete multispectral intensity column vector; The pseudo-hyperspectral eigenvector is calculated by performing matrix multiplication on the discrete multispectral intensity column vector and the spectral reconstruction matrix.

7. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 1, characterized in that: When the multimodal feature fusion module performs cross-attention calculation in the multimodal latent space, the specific configuration is as follows: The projected pseudo-hyperspectral feature vector is multiplied by a preset query weight matrix to map it into a query matrix; The projected fluid physical feature vector is simultaneously multiplied by a preset bond weight matrix and a preset value weight matrix, respectively, and mapped to a bond matrix and a value matrix.

8. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 7, characterized in that: After mapping out the query matrix, key matrix, and value matrix, the multimodal feature fusion module is specifically configured as follows: Multiply the query matrix by the transpose of the key matrix to obtain the product result; The product result is scaled using the square root of the dimension scaling constant of the key matrix; The normalized exponential activation function is calculated on the scaled result; The result of the normalized exponential activation function is multiplied by the value matrix to obtain the multimodal latent space feature matrix; Numerical stabilization of the multimodal latent space feature matrix is ​​achieved through layer normalization. The numerically stabilized multimodal latent space feature matrix is ​​input into a feedforward neural network layer for nonlinear activation, outputting the multimodal fused feature vector.

9. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 1, characterized in that: The concentration regression calculation module is specifically configured as follows when outputting one-dimensional concentration calibration values ​​of the biochemical biomarker: The multimodal fusion feature vector is controlled to pass through the weight matrix multiplication, bias addition and activation function mapping of each of the fully connected hidden layers in sequence to extract low-dimensional feature parameters for concentration regression; The linear output layer of the concentration regression network is controlled to receive the hidden state vector output by the fully connected hidden layer; The hidden state vector is multiplied by the weight parameters of the linear output layer and added to the bias parameters to calculate a one-dimensional output of the biochemical biomarker concentration calibration value.

10. The abnormal state AI test strip detection system based on multimodal data fusion according to claim 2, characterized in that: The fluid physics feature extraction module is specifically configured as follows when calculating the dynamic viscosity time series of the liquid sample on the biochemical test strip: Obtain the effective equivalent pore radius of the nitrocellulose membrane of the pre-configured biochemical test strip, the surface tension parameters of the liquid sample, and the basic constant of the solid-liquid contact angle; A pore correction coefficient is introduced to correct the nonlinear spatial characteristics of the classical capillary flow model, and a capillary dynamics equation with pore correction is constructed. The spatial position of the fluid wetting front is converted into a dynamic sequence of the instantaneous advance distance of the front as a function of time; Based on the proportional relationship between the square of the instantaneous advance distance and time, and by combining the basic constant and the pore correction coefficient, the capillary dynamics equation with pore correction is solved, and the dynamic viscosity time series of the liquid sample during the chromatography process is calculated.

Images