Disease test paper detection method and system fused with deep feature extraction

Abstract

The invention discloses a disease test paper detection method and system fused with deep feature extraction. The method comprises the following steps: acquiring a test paper image through a camera, acquiring vibration and temperature signals in combination with an MEMS accelerometer and an infrared microsensor, and constructing a biochemical violation description set after determining a detection task text. A text thinking chain method is adopted to simplify the text into a short sentence set in a color-intensity-position triple form, word segmentation and coding are carried out, and a multi-head self-attention mechanism is utilized to generate a text feature vector. A test paper image is segmented into blocks which are matched and screened with text features, and image feature vectors are obtained through depth coding. Image, vibration and temperature features are fused, similarity is calculated by using a regression model, probability is predicted in combination with a classification model, and a disease occurrence probability is obtained to judge a negative and positive result. And finally, outputting a result to a user interface, and storing data for subsequent analysis and medical record tracing. By implementing the method provided by the invention, accurate feature positioning and multi-modal fusion can be realized, and the weak positive detection rate is remarkably improved.

Description

Technical Field

[0001] This invention relates to a test strip detection method, and more specifically to a disease test strip detection method and system that integrates deep feature extraction. Background Technology

[0002] Disease test strips, as a common in vitro diagnostic method, have wide applications in the medical and health field. Traditional testing methods rely on manual comparison of test strip colors with color charts, which suffers from problems such as high subjectivity, low detection accuracy, and susceptibility to ambient light interference, making it difficult to meet the high requirements of clinical clinicians for the accuracy and reliability of test results. Although image recognition technology assisted by mobile phone photography has been applied in recent years, existing technologies are mostly limited to the stage of simple classification of the entire test strip image.

[0003] Current technologies have many limitations in the detection process. They cannot accurately locate local lesion features in images, cannot utilize continuous spatiotemporal data for dynamic evidence fusion, and lack multimodal feature extraction and cross-validation methods. This results in poor performance in terms of weak positive detection rate and false negative / false positive control, making it difficult to effectively cope with complex and ever-changing real-world testing scenarios and clinical needs, thus limiting the further development and widespread application of disease test strip detection technology.

[0004] Therefore, it is necessary to design a new method to address the problems of traditional disease test strip detection, such as high subjectivity, low accuracy, susceptibility to interference, and the inability of existing technologies to accurately locate local features, lack of spatiotemporal fusion and multimodal verification, resulting in low weak positive detection rates and high false negative / false positive rates. This new method aims to achieve accurate feature localization and multimodal fusion, significantly improve the weak positive detection rate, greatly reduce false negative and false positive rates, and enhance the accuracy and reliability of the test. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a disease test strip detection method and system that integrates deep feature extraction.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a disease test strip detection method integrating deep feature extraction, comprising:

[0007] The system uses a camera to capture images of the test strip and determines the text of the detection task. Vibration and temperature signals are collected by a MEMS accelerometer and an infrared microsensor, respectively.

[0008] Based on the detection task text, a set of biochemical violation descriptions is constructed, and the text thinking chain method is used to simplify it into a set of short sentences in the form of color-intensity-location triplets;

[0009] After segmenting and encoding the short sentence set, a text feature vector is generated using a multi-head self-attention mechanism;

[0010] The test strip image is segmented into several blocks, matched and filtered with the text feature vector, and the image feature vector is obtained through block filtering and multi-head self-attention mechanism deep encoding.

[0011] By fusing image feature vectors, vibration and temperature signals, similarity is calculated using a regression model, and probability is predicted using a classification model. The weighted sum is then used to obtain the final probability of disease occurrence and to determine whether the result is positive or negative, thus obtaining the detection result.

[0012] The test results are output to the user interface, and the test data is stored for subsequent analysis and medical record tracing.

[0013] Its further technical solution is as follows: The method of acquiring test strip images using a camera and determining the detection task text, and collecting vibration and temperature signals using a MEMS accelerometer and an infrared micro-sensor respectively, includes:

[0014] The test strip image is acquired using a camera, and the detection task text is determined. A MEMS accelerometer is installed at a specific position below the microchannel of the test strip, and an infrared microsensor is set at the corresponding position to collect vibration and temperature signals respectively.

[0015] The further technical solution is as follows: Based on the detection task text, a set of biochemical violation descriptions is constructed, and the text thinking chain method is used to simplify it into a set of short sentences in the form of color-intensity-position triplets, including:

[0016] Based on the detection task text, a set of biochemical violation descriptions is constructed, wherein the set of biochemical violation descriptions includes breaking down complex medical sentences about the disease to be tested in clinical guidelines into short sentences in the form of color-intensity-location triplets to form abnormal expressions;

[0017] The textual thinking chain method is used to process the set of biochemical violation descriptions, transforming the logically complex and verbose medical descriptions into long sentences composed of simple words and relationships, resulting in a set of short sentences.

[0018] The further technical solution is as follows: after segmenting and encoding the short sentence set, a text feature vector is generated using a multi-head self-attention mechanism, including:

[0019] Each simple description in the set of short sentences is segmented into words, and medical terminology is decomposed into basic semantic units.

[0020] The semantic units obtained from word segmentation are converted into fixed-dimensional word vectors using a text encoder;

[0021] Text data with the word vectors is input into a multi-head self-attention mechanism to capture key information about the correlation and dependency between different words in the text, and generate a text feature vector that comprehensively represents the text features of the detection task.

[0022] The further technical solution is as follows: the test strip image is segmented into several blocks, matched and filtered with the text feature vector, and the image feature vector is obtained through block filtering and multi-head self-attention mechanism deep encoding, including:

[0023] The test strip image is segmented into blocks to obtain image micro-blocks;

[0024] The text feature vector is converted into a two-dimensional format. The image micro-patch is convolved with the converted text feature vector. The similarity score is calculated using the Sigmoid function to obtain the matching result.

[0025] Based on the matching results, image micro-blocks that meet the requirements are selected, flattened into one-dimensional vectors, and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors. Then, a multi-head self-attention mechanism is used to perform deep encoding on the image word vectors to mine the association and dependency relationships of feature elements and obtain image feature vectors.

[0026] The further technical solution is as follows: the fused image feature vector, vibration and temperature signals are used to calculate the similarity through a regression model, combined with the probability prediction by a classification model, and weighted summation is performed to obtain the final disease occurrence probability and determine whether it is positive or negative, so as to obtain the detection result, including:

[0027] The vibration and temperature signals are processed to obtain vibration feature vectors and temperature feature vectors;

[0028] Image feature vectors, vibration feature vectors, and temperature feature vectors are fused and input into a regression model. After deep feature extraction and flattening into a one-dimensional vector, cosine similarity is calculated.

[0029] The image feature vector, vibration feature vector, and temperature feature vector are input into the corresponding classification model to predict the probability value of disease occurrence based on the corresponding features.

[0030] Based on the cosine similarity and the probability values ​​of disease occurrence based on the corresponding features, a specific weighted fusion strategy is used to fuse the probability values ​​corresponding to the three modalities, calculate the final probability of disease occurrence, and compare it with a set threshold to determine whether the detection result is positive or negative.

[0031] The further technical solution is as follows: the processing of the vibration and temperature information to obtain vibration feature vectors and temperature feature vectors includes:

[0032] The collected vibration signals are transposed to obtain the vibration feature vector.

[0033] The acquired temperature signal is normalized to adjust its numerical range to within the set calculation interval, and the temperature signal is encoded to obtain a temperature feature vector.

[0034] This invention also provides a disease test strip detection system that integrates deep feature extraction, comprising:

[0035] The acquisition unit is used to acquire test strip images using a camera and determine the detection task text, and to collect vibration and temperature signals using a MEMS accelerometer and an infrared micro-sensor, respectively.

[0036] The set construction unit is used to construct a set of biochemical violation descriptions based on the detection task text, and to simplify it into a set of short sentences in the form of color-intensity-position triplets using the text thinking chain method;

[0037] The text feature vector generation unit is used to generate text feature vectors by means of a multi-head self-attention mechanism after the short sentence set is segmented and encoded.

[0038] The image feature vector generation unit is used to segment the test strip image into several blocks, match and filter them with the text feature vector, and obtain the image feature vector through block filtering and multi-head self-attention mechanism deep encoding.

[0039] The detection unit is used to fuse image feature vectors, vibration and temperature signals, calculate similarity through a regression model, predict probability by combining with a classification model, and obtain the final disease occurrence probability by weighted summation and determination of positive or negative to obtain the detection result;

[0040] The output unit is used to output the test results to the user interface and store the test data for subsequent analysis and medical record tracing.

[0041] The further technical solution is as follows: The acquisition unit is used to acquire test strip images using a camera and determine the detection task text. A MEMS accelerometer is installed at a specific position below the microchannel of the test strip, and an infrared microsensor is set at the corresponding position to collect vibration and temperature signals respectively.

[0042] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0043] The advantages of this invention compared to existing technologies are as follows: This invention combines test strip images acquired by a camera, vibration signals collected by a MEMS accelerometer, and temperature signals collected by an infrared microsensor. It simplifies biochemical violations into color-intensity-location triplets using a text-based thought chain method and generates text feature vectors using a multi-head self-attention mechanism. Furthermore, the test strip image is segmented into multiple blocks and matched with the text feature vectors for filtering. Block filtering and deep coding techniques are used to obtain image feature vectors. Subsequently, the image, vibration, and temperature feature vectors are fused, a regression model is used to calculate similarity, and a classification model is combined to predict probabilities. A specific weighting strategy is then used to derive the final disease occurrence probability to determine the positive or negative result. This method solves the problems of strong subjectivity, low accuracy, and susceptibility to interference in traditional disease test strip detection. It also overcomes the shortcomings of existing technologies in accurately locating local features, achieving spatiotemporal fusion and multimodal verification. This significantly improves the detection rate of weak positives and greatly reduces the error rates of false negatives and false positives, thereby enhancing the overall accuracy and reliability of the detection. Finally, the detection results can not only be output to the user interface in real time but also stored for subsequent analysis and medical record tracing.

[0044] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A schematic flowchart of the disease test strip detection method based on fusion deep feature extraction provided in an embodiment of the present invention;

[0047] Figure 2 A schematic block diagram of a disease test strip detection system with fused deep feature extraction provided in an embodiment of the present invention;

[0048] Figure 3 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0052] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0053] Please see Figure 1 , Figure 1 This is a flowchart illustrating the disease test strip detection method based on fusion deep feature extraction provided in this embodiment of the invention. This method is applied in a server. By combining test strip images acquired by a camera, vibration signals collected by a MEMS accelerometer, and temperature signals acquired by an infrared microsensor, the complex biochemical violation descriptions are simplified into color-intensity-location triplets using a text-based thought chain method, and a text feature vector is generated through a multi-head self-attention mechanism. Simultaneously, the test strip image is segmented and block-filtered, and the image feature vector is obtained by deep encoding combined with the text feature vector. Furthermore, this method fuses data from three modalities: image, vibration, and temperature. The similarity is calculated using a regression model, and the predicted probability is weighted and summed to obtain the final disease occurrence probability. This achieves accurate feature localization and spatiotemporal fusion of multimodal data, effectively solving the problems of strong subjectivity, low accuracy, and susceptibility to interference in traditional disease test strip detection. It significantly improves the weak positive detection rate, greatly reduces the incidence of false negatives and false positives, and significantly enhances the accuracy and reliability of disease detection.

[0054] Figure 1 This is a schematic flowchart of the disease test strip detection method based on fusion deep feature extraction provided in an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S110 to S160.

[0055] S110: Use a camera to acquire images of the test strip and determine the detection task text. Use a MEMS accelerometer and an infrared microsensor to collect vibration and temperature signals respectively.

[0056] In this embodiment, the test strip image refers to an image obtained by taking a picture of the disease detection test strip using a high-resolution camera. This image must be clear and complete, accurately reflecting the color changes of various reaction areas on the test strip, to facilitate subsequent analysis and processing. Simultaneously, to ensure data consistency and accuracy, a timestamp is recorded during image acquisition.

[0057] The detection task text refers to descriptive text constructed based on key information such as the specific disease name, key indicators of concern, and expected thresholds targeted in this detection. It provides clear guidelines for identifying specific biochemical parameters and evaluation criteria that need attention, thereby providing a precise basis for text feature extraction and matching in subsequent steps.

[0058] Vibration signals refer to minute physical change signals synchronously acquired by installing a MEMS accelerometer at a specific location (e.g., 0.5 mm) below the microchannels of the test strip at a sampling frequency of 2 kHz. These signals reflect the subtle vibrations that occur in the test strip during sample flow and reaction, helping to capture the dynamic characteristics of the test strip reaction process.

[0059] Temperature signals refer to the temperature changes of the test strip during the detection process, monitored and collected in real time using infrared microsensors. Since certain biochemical reactions are accompanied by heat changes, such as the aggregation of gold nanoparticles, temperature signals are crucial for understanding these processes. Specifically, a camera is used to acquire images of the test strip and define the detection task text. A MEMS accelerometer is installed at a specific location below the microchannels of the test strip, and infrared microsensors are placed at corresponding locations to collect vibration and temperature signals, respectively.

[0060] This series of operations ensures comprehensive capture of relevant information from the test strip reaction from multiple perspectives and levels, laying a solid foundation for subsequent data processing and analysis. This method not only improves the accuracy of the test but also effectively identifies weak positive results, reducing the occurrence of false negatives and false positives.

[0061] S120. Construct a set of biochemical violation descriptions based on the detection task text, and simplify it into a set of short sentences in the form of color-intensity-position triplets using the text thinking chain method.

[0062] In this embodiment, the biochemical violation description set refers to a systematic set of abnormal result representations formed by structurally parsing and decomposing complex medical descriptions from clinical guidelines, medical literature, and other materials related to the disease to be tested, based on a specific testing task text. This set specifically targets various abnormal situations that may occur during the test strip testing process, and concisely expresses these abnormal characteristics using a "color-intensity-location" triplet format. Specifically:

[0063] In one embodiment, step S120 described above may include steps S121 to S122.

[0064] S121. Based on the detection task text, construct a biochemical violation description set, wherein the biochemical violation description set includes splitting complex medical sentences about the disease to be tested in clinical guidelines into short sentences in the form of color-intensity-location triplets to form abnormal expressions.

[0065] First, based on the testing task text, we determine the disease name, key indicators, and expected thresholds relevant to this test. Then, we extract descriptions of abnormal manifestations related to the disease from clinical guidelines, expert consensus, and other authoritative medical resources. Next, we break down these complex medical sentences into more refined descriptive units, namely, "color-intensity-location" triplet phrases. For example, "a dark red spot appears in area A of the test strip" can be parsed as (color: dark red; intensity: high; location: area A). This approach ensures that each abnormal feature has a clear and easily understood description.

[0066] S122. The textual thinking chain method is used to process the set of biochemical violation descriptions, converting the logically complex and verbose medical descriptions into long sentences composed of simple words and relationships, resulting in a set of short sentences.

[0067] Building upon this, the text-based thought chain method is employed to further process the constructed set of biochemical violation descriptions. Text-based thought chains are a technique for simplifying complex logical relationships. They reconstruct the original sentence by identifying and extracting core words and their simple relationships, thus achieving simplification. For medical descriptions, this means transforming logically complex and verbose medical terms and concepts into more direct and understandable forms while maintaining their scientific accuracy and completeness. For example, "If a pale yellow change is observed in region B, it indicates the presence of a low concentration of the target substance in the sample," after simplification using text-based thought chains, might become (Color: Pale Yellow; Intensity: Low; Location: Region B).

[0068] In summary, in step S120, a set of biochemical violation descriptions is constructed based on the detection task text, and then simplified into a set of short sentences in the form of color-intensity-location triplets using the text thinking chain method. This process not only improves the understanding and recognition efficiency of anomaly detection results but also provides a clear and accurate foundation for subsequent text feature extraction, image analysis, and data matching. This method greatly enhances the automation and accuracy of data analysis, contributing to improved overall diagnostic efficiency.

[0069] S130. After segmenting and encoding the short sentence set, a text feature vector is generated using a multi-head self-attention mechanism.

[0070] In this embodiment, text feature vectors refer to the numerical representations extracted from a simplified set of descriptions through a series of natural language processing techniques (including but not limited to word segmentation, encoding, and the application of multi-head self-attention mechanisms). These vectors can capture the complex relationships and dependencies between different words within the text, thereby providing an efficient and comprehensive way to characterize the text features of the detection task, laying the foundation for subsequent matching and fusion with image features and other data.

[0071] In one embodiment, step S130 described above may include steps S131 to S133.

[0072] S131. For each simple description in the set of short sentences, perform word segmentation to decompose medical terminology into basic semantic units.

[0073] In this embodiment, a semantic unit refers to the smallest comprehensible unit of information obtained by segmenting words in the original text. In the medical field, this might involve breaking down complex medical terms or expressions into their constituent parts. For example, "deep red" can be further parsed into "deep" and "red," while "light yellow change in zone B" might be broken down into "zone B," "light yellow," and "change." This not only helps to understand the meaning of each word in greater detail but also provides more refined input for subsequent steps.

[0074] S132. Use a text encoder to convert the semantic units obtained from word segmentation into fixed-dimensional word vectors.

[0075] In this embodiment, word vectors refer to vectors that map each semantic unit obtained from word segmentation into a high-dimensional space using specific algorithms (such as Word2Vec, GloVe, or BERT). This vectorization process transforms originally incomputable textual information into a data form that can be manipulated mathematically. A key characteristic of word vectors is that they can reflect the semantic similarity and correlation between words to some extent; that is, words that are close in position in the vector space usually have similar meanings or functions. Therefore, text data processed in this step can be more easily analyzed and processed subsequently.

[0076] S133. Input the text data with the word vectors into the multi-head self-attention mechanism to capture the key information of the correlation and dependency between different words in the text, and generate a text feature vector that comprehensively represents the text features of the detection task.

[0077] In this step, a multi-head self-attention mechanism is used to delve deeper into the internal structure of the text. Multi-head self-attention allows the model to simultaneously focus on different parts of the input sequence and effectively learn the relationships and dependencies between words. In this way, higher-level abstract features can be extracted from word vectors, forming so-called text feature vectors. These feature vectors not only reflect the information of individual words but also reveal semantic information at the sentence and even paragraph levels, greatly enhancing the depth and accuracy of text content understanding. The final generated text feature vectors will become the key basis for matching and fusing with other types of data (such as image features), supporting more accurate anomaly detection result identification and interpretation.

[0078] For an input sequence of word vectors, the self-attention mechanism first generates a query vector, a key vector, and a value vector through a linear transformation. Then, by calculating the dot product between the query vector and the key vector, it obtains an attention score for each word, representing the strength of the relevance between words. Next, the attention scores are normalized (usually using the softmax function) to obtain attention weights. Finally, the attention weights are weighted and summed with the value vectors to obtain the self-attention output for each word. This output contains semantic information about the semantic association between the word and its context words, providing a more comprehensive representation of the word's meaning in the text.

[0079] Multi-head self-attention (MSA) is an extension and improvement upon traditional self-attention mechanisms. It maps input word vectors to multiple distinct attention heads, each independently calculating self-attention. This allows it to capture relationships between words from multiple perspectives and semantic levels. For example, one attention head might focus on capturing local grammatical relationships between words, while another might focus on capturing global semantic thematic relevance. In this way, MSA can more comprehensively and meticulously extract rich semantic information from text.

[0080] Multi-head self-attention mechanisms integrate feature information from multiple attention heads by concatenating or weighted summing their outputs, thereby enhancing the model's ability to express text features. This multi-angle, multi-level feature fusion approach enables the model to better understand and process complex text data, improving the modeling effect on text semantics. In medical text processing, due to the complexity and specialization of medical texts, multi-head self-attention mechanisms can more effectively capture the complex relationships between medical concepts, providing stronger support for subsequent text feature extraction and analysis.

[0081] After processing by the multi-head self-attention mechanism, the outputs of each attention head are concatenated or fused to obtain a vector representation that integrates multi-head feature information. This vector representation combines the correlation and semantic information between words captured from different perspectives, and can more comprehensively reflect the features of the text. For example, for a medical text sentence, after processing by the multi-head self-attention mechanism, the generated text feature vector can simultaneously contain the correlation information between different medical concepts such as disease name, symptom description, and examination results.

[0082] The resulting text feature vector will serve as a crucial basis for subsequent matching and fusion with other modal data, such as image features. It provides rich textual semantic information for cross-modal data association analysis, enabling the model to more accurately perform tasks such as disease detection and diagnosis by comprehensively considering multiple information sources, including text and images. For example, in disease test strip detection, the text feature vector can be fused with the feature vector of the test strip image to help the model more accurately determine whether the test strip image contains content consistent with the disease characteristics described in the text.

[0083] S140. The test strip image is segmented into several blocks, matched and filtered with the text feature vector, and the image feature vector is obtained through block filtering and multi-head self-attention mechanism deep encoding.

[0084] In this embodiment, the image feature vector refers to the numerical representation extracted from the test strip image data through a series of processing steps (including but not limited to block segmentation, feature matching and filtering, block filtering, and deep encoding). These vectors not only contain basic information about the micro-blocks of the image, but also further explore the complex relationships and dependencies between different elements within the image through a multi-head self-attention mechanism. This provides a highly abstract and comprehensive representation of features such as the key reaction areas of the test strip, providing a high-quality data foundation for subsequent analysis and evaluation.

[0085] In one embodiment, step S140 described above may include steps S141 to S143.

[0086] S141. The test strip image is segmented into blocks to obtain image micro-blocks.

[0087] In this embodiment, an image micro-patch refers to a small region obtained by uniformly dividing the original test strip image into p×p sizes. Each image micro-patch represents a local view of the original image, allowing for more detailed analysis and understanding of the image's details. This segmentation method is similar to breaking down a large image into multiple smaller but complete parts, facilitating more precise feature analysis and processing for each micro-patch. For example, when detecting a specific chemical reaction, this method can help pinpoint the exact location of the reaction on the test strip.

[0088] S142. Convert the text feature vector into a two-dimensional format, perform convolution operation between the image micro-patch and the converted text feature vector, calculate the similarity score using the Sigmoid function, and obtain the matching result.

[0089] In this embodiment, the matching result refers to the degree of correlation between the image micro-patch obtained after convolution operation and similarity scoring and the text feature vector. The specific process is as follows: First, the text feature vector is adjusted to a two-dimensional format suitable for comparison with image features. Next, convolution operation is used to slide the text features across the entire image micro-patch to find the best matching position, and the similarity score for each comparison is calculated. Finally, the sigmoid function is used to convert these scores into probability values ​​to determine which image micro-patches are most likely to contain biochemical features or anomalies that match the text description. A high score indicates that the micro-patch may contain important information or features.

[0090] S143. Based on the matching results, select the image micro-blocks that meet the requirements, flatten the image micro-blocks that meet the requirements into one-dimensional vectors, and map them to a high-dimensional space through a fully connected layer to obtain image word vectors. Then, use a multi-head self-attention mechanism to perform deep encoding on the image word vectors, mine the association and dependency relationship of feature elements, and obtain the image feature vector.

[0091] The correlation between image patches and text feature vectors is determined after convolution operations and similarity scoring. The specific process is as follows: First, the text feature vectors are adjusted to a two-dimensional format suitable for comparison with image features. Next, convolution operations are used to slide the text features across the entire image patch, finding the optimal matching position and calculating the similarity score for each comparison. Finally, the sigmoid function is used to convert these scores into probability values ​​to determine which image patches are most likely to contain biochemical features or anomalies consistent with the text description. High scores indicate that the patch may contain important information or features.

[0092] In this embodiment, the initial test strip image data is segmented into p×p micro-blocks, aiming to decompose the complex image into smaller units with relatively simple features. This facilitates subsequent refined feature analysis of each micro-block, enabling more precise location of key reaction areas on the test strip, much like accurately locating problem areas in urban management. For example, when testing for COVID-19 antigen, segmentation can divide the different reaction levels in the T and C regions into different micro-blocks for separate analysis.

[0093] The segmentation process utilizes image processing algorithms to sequentially slide windows across the image according to a preset p×p size, dividing the image into multiple small, non-overlapping or partially overlapping blocks. Overlap settings prevent target features from being segmented into different blocks, thus avoiding incomplete information. The overlap ratio can be determined based on the typical size and shape characteristics of the test strip's reaction area.

[0094] Segmentation reduces the complexity of image data, allowing computational resources to focus more on local feature extraction and improving processing efficiency. Furthermore, for test strip detection, different feature extraction strategies can be applied to different segments, enhancing the ability to capture key reaction features and providing higher-quality image data for accurate subsequent disease assessment.

[0095] Converting text feature vectors to a two-dimensional format is crucial. Image micro-patch features are typically represented as two-dimensional matrices. Converting text feature vectors to the same dimensional structure enhances compatibility in mathematical operations and logical comparisons, bridging subsequent convolution operations and similarity calculations. For example, if a text feature vector is one-dimensional, it can be transformed into two dimensions through matrix transpose and padding, ensuring that its semantic information is not lost during the conversion.

[0096] The segmented image patches are convolved with the transformed text feature vectors. The convolution kernel slides across the image patches, and the text features are matched by calculating the weighted sum of the image pixel values ​​and text feature vector values ​​within the kernel. The sigmoid function is used to calculate the similarity score because its output value is between 0 and 1, which effectively represents the degree of similarity. Patches with high similarity scores indicate that the image features in that region highly match the biochemical features described in the text, and may contain the target reactive substance. Adjacent patches with high similarity scores form continuous regions, corresponding to complete reaction bands in test strip detection, such as the T-line or C-line color development area, providing crucial clues for subsequent accurate localization and evaluation.

[0097] Image patches that meet the matching requirements—that is, patches with similarity scores reaching a set threshold—are selected based on the matching results. The aim is to remove irrelevant regions with low relevance to the target features and retain the parts containing key reaction information. For example, when testing blood glucose test strips, patches containing the colorimetric region of the glucose oxidase reaction can be selected.

[0098] Flattening the filtered image patches into one-dimensional vectors facilitates input into fully connected layers for further processing. Fully connected layers can learn the complex relationships between pixels or feature points within a patch. Mapping these relationships to a high-dimensional space yields image word vectors that more abstractly and comprehensively represent the deep features of the image patches.

[0099] A multi-head self-attention mechanism is used for deep encoding of image word vectors. This mechanism utilizes multiple attention heads working simultaneously to capture the relationships and dependencies between different feature elements within the image word vectors from various perspectives. For example, when analyzing immunofluorescence test strip images, the connections between multiple features such as fluorescence intensity, distribution location, and shape can be considered simultaneously.

[0100] After processing by the multi-head self-attention mechanism, the resulting image feature vector not only contains the appearance features of the original image micro-patch, but also integrates the inherent logical relationship between the feature elements, making the image feature vector more representative and discriminative. This provides high-quality image feature basis for subsequent comprehensive evaluation of disease detection results, and can more accurately determine key conclusions such as whether the test strip test shows a positive reaction.

[0101] S150, by fusing image feature vectors, vibration and temperature signals, similarity is calculated using a regression model, probability is predicted using a classification model, and weighted summation is performed to obtain the final probability of disease occurrence and determine whether it is positive or negative, thus obtaining the detection result.

[0102] In this embodiment, the detection result refers to the probability value of disease occurrence obtained by comprehensively analyzing data from three different modalities—image, vibration, and temperature—through a series of complex processing steps (including but not limited to processing of vibration and temperature signals, fusion of multimodal features, deep feature extraction, cosine similarity calculation, classification model prediction, and weighted probability calculation). Based on this probability value, the test strip detection result is determined to be positive or negative.

[0103] In one embodiment, step S150 described above may include steps S151 to S154.

[0104] S151. Process the vibration and temperature signals to obtain vibration feature vectors and temperature feature vectors.

[0105] In one embodiment, step S151 described above may include steps S151 to S152.

[0106] S151. Perform a matrix transpose operation on the collected vibration signal to obtain the vibration feature vector.

[0107] In this embodiment, the vibration feature vector refers to a numerical representation obtained by performing a matrix transpose operation on the original vibration signal data. This processing method changes the data organization structure, making it easier to compare and integrate with other types of feature data, while also helping to highlight key information points in the vibration signal.

[0108] S152. Normalize the acquired temperature signal, adjust its numerical range to the set calculation interval, and encode the temperature signal to obtain the temperature feature vector.

[0109] In this embodiment, the temperature feature vector refers to the normalized temperature signal data, whose values ​​are adjusted to a range suitable for subsequent calculations, and its feature representation capability is enhanced through a specific encoding method. This step allows temperature features to better participate in the multimodal feature fusion process, thereby improving the overall analysis effect.

[0110] S152. The image feature vector, vibration feature vector, and temperature feature vector are fused and input into the regression model. After deep feature extraction and flattening into a one-dimensional vector, the cosine similarity is calculated.

[0111] In this embodiment, the processed image feature vector, vibration feature vector, and temperature feature vector are first input into a regression model. This model performs deep feature extraction on the target features (such as image features), then converts these deep features, along with the vibration and temperature features, into one-dimensional vectors and calculates their cosine similarity. This process achieves effective fusion and evaluation of different modal data at the feature level.

[0112] S153. Input the image feature vector, vibration feature vector, and temperature feature vector into the corresponding classification model to predict the probability values ​​of disease occurrence based on the corresponding features.

[0113] In this embodiment, this step involves using different classification models to make independent probability predictions for each type of feature vector (i.e., image, vibration, and temperature), resulting in a first probability value Pimg based on image features, a second probability value Pvib based on vibration features, and a third probability value Ptemp based on temperature features.

[0114] S154. Based on the cosine similarity and the probability values ​​of disease occurrence based on the corresponding features, a specific weighted fusion strategy is used to fuse the probability values ​​corresponding to the three modalities, calculate the final probability of disease occurrence, and compare it with a set threshold to determine whether the detection result is positive or negative.

[0115] In this embodiment, finally, based on the cross-modal similarity result calculated in the previous step, namely the cosine similarity S, and the disease occurrence probability value corresponding to each feature, a specific weighted fusion strategy (e.g., Pfinal = 0.5 × S × (Pimg + 0.3Pvib + 0.2Ptemp)) is used to calculate the final disease occurrence probability Pfinal. Then, this final probability is compared with a preset occurrence threshold: if Pfinal is greater than the set threshold, the test strip is determined to be positive, indicating the presence of the corresponding disease feature; otherwise, it is determined to be negative. This completes the entire detection process and yields the final detection result.

[0116] In this embodiment, matrix transposition is a fundamental and effective data transformation technique during vibration signal processing. Original vibration signal data is typically organized along a time series dimension; for example, rows may represent different time sampling points, and columns may represent signal values ​​collected by multiple vibration sensors. By transposing the matrix, the dimensions of rows and columns are interchanged, changing the data organization from time-driven to sensor channel-driven, or vice versa. This change in dimensional structure provides a more suitable format for subsequent data fusion and comparison operations.

[0117] For example, when processing MEMS accelerometer signals from microchannel vibrations in test paper, the raw data might have each row representing the vibration intensity at a given time point, with three columns corresponding to the vibration components in the x, y, and z directions. The transposed matrix then becomes one where each column represents the vibration intensity at a given time point, and the rows correspond to different directions. This makes it easier to align and compare features along the direction or time dimension when fusing them with other feature data (such as image and temperature features).

[0118] The vibration features obtained after matrix transposition can highlight key characteristics in vibration signals. In test strip testing scenarios, certain disease characteristics may be closely related to vibration changes in a specific direction. Transposition allows for more intuitive observation and analysis of the time-series characteristics of vibration signals in various directions, such as the periodicity and amplitude variations of vibration. For example, when testing blood glucose strips, the vibration signal caused by blood flow in the microchannels may exhibit significant periodic changes in a specific direction. The transposed matrix helps extract this key feature, thus providing more valuable vibration characteristic information for subsequent disease detection.

[0119] Normalization of temperature signals is performed to adjust their numerical range to a suitable interval for subsequent fusion calculations. In multimodal data fusion, the numerical ranges of different feature data can vary significantly. For example, the value of an image feature vector might be between 0 and 255 (if based on pixel grayscale values), while the original temperature signal might be between 20 and 40 degrees Celsius. To ensure that temperature features have equal importance to other features during the fusion process, normalization is necessary.

[0120] Commonly used normalization methods include min-max normalization and Z-fraction normalization. In test strip detection scenarios, min-max normalization can be used to linearly map the temperature signal's numerical range to the interval [0,1] or [-1,1]. For example, assuming the collected temperature signal range is 25-35 degrees Celsius, min-max normalization can convert it into a value between 0 and 1, making the temperature feature comparable to other normalized feature data.

[0121] The purpose of encoding temperature signals is to enhance their feature representation capabilities. A method similar to the historical data encoding in technical solution 3 can be adopted, such as one-heat encoding or encoding methods based on a certain function mapping. For the temperature signal, its normalized value can be divided into several intervals, each interval corresponding to an encoding vector. For example, the temperature interval can be divided into three levels: low, medium, and high, encoded using [1,0,0], [0,1,0], and [0,0,1] respectively. This encoding method not only preserves the original information of the temperature signal but also highlights the interval characteristics of the temperature, providing richer information for subsequent multimodal feature fusion. When detecting certain temperature-sensitive disease test strips, such as those for enzymes whose activity is closely related to temperature, this encoded temperature feature can better participate in the fusion process, reflecting the probability of disease occurrence.

[0122] The processed and encoded image feature vectors, vibration feature vectors, and temperature feature vectors are jointly input into the regression model. The regression model can employ a multi-layer neural network structure, with the number of neurons in the input layer determined by the sum of the dimensions of each feature vector. Internally, the model first performs deep feature extraction on the target features (such as image features); for vibration and temperature features, the model also performs feature transformations through corresponding hidden layers to extract their deep features, enabling these features to better reflect potential disease-related information.

[0123] Deep image features, vibration features, and temperature features are flattened into one-dimensional vectors. This is because when calculating similarity, features need to be represented in vector form for vector operations. The flattening operation converts multi-dimensional feature maps or feature matrices into one-dimensional vectors while preserving the order information of the features. Then, the cosine similarity between these one-dimensional vectors is calculated. Cosine similarity is calculated by dividing the dot product of two vectors by the product of their magnitudes, yielding a value between -1 and 1, representing the degree of similarity between the vectors. In the context of test strip detection, cosine similarity can measure the correlation between feature vectors of different modalities. For example, the similarity between image feature vectors and vibration feature vectors can reflect the intrinsic relationship between image features and vibration features in the test strip's reaction area. In this way, multi-modal data can be fused and evaluated at the feature level, providing a comprehensive feature similarity basis for the final disease detection judgment. For example, when testing pregnancy strips, a high similarity between the color depth displayed by image features and vibration features (which may be related to minor structural changes caused by chemical reactions) may indicate a high level of pregnancy hormones, thus increasing the likelihood of a positive test.

[0124] Image feature vectors are input into an image classification model. This model can be a convolutional neural network (CNN) or similar model specifically designed for image feature classification. The model learns from a large amount of image data labeled with disease category information, extracting the mapping relationship between image features and disease categories. During prediction, the model calculates the probability distribution of the image belonging to different disease categories based on the input image feature vectors, outputting a probability value Pimg based on the image features. For example, when detecting images of COVID-19 antigen test strips, the classification model analyzes image information such as the color characteristics of the T and C lines to predict the presence of COVID-19 antigens, obtaining the Pimg value.

[0125] Similarly, the vibration feature vector is input into the sensor classification model. This model can be based on a recurrent neural network (RNN) or other models suitable for processing time-series data, as vibration signals are typically time-series data. The model learns the relationship between the vibration signal and disease characteristics to predict the probability of disease occurrence, Pvib, based on the vibration characteristics. For temperature characteristics, a corresponding probability value, Ptemp, can also be obtained through a suitable classification model (such as a multilayer perceptron). For example, in testing strips for certain metabolic diseases, temperature changes may be related to metabolic reaction rates; a temperature classification model can predict the probability of disease occurrence based on temperature characteristics.

[0126] A specific weighted fusion strategy is employed, such as Pfinal = 0.5 × S × (Pimg + 0.3Pvib + 0.2Ptemp), where S is the cross-modal similarity result calculated in the previous step. The choice of the weighted fusion strategy is based on the importance and reliability of different modal data in disease detection. Image features can usually intuitively reflect the reaction of the test strip, so they are given a larger weight (e.g., 0.5). Vibration features and temperature features are also important, but may be affected by more external factors, so they are given relatively smaller weights (e.g., 0.3 and 0.2), respectively. Meanwhile, the cross-modal similarity S is used as a moderating factor because similarity reflects the consistency between different modal features; higher similarity means that different modal data are more likely to jointly support a certain disease diagnosis, thus amplifying the final probability.

[0127] The final disease incidence probability, Pfinal, is calculated through weighted fusion. Pfinal is compared to a set incidence threshold. If Pfinal is greater than the threshold, the test strip result is considered positive, indicating the presence of the corresponding disease characteristic; otherwise, it is considered negative. For example, when testing for a certain cancer biomarker, if the set incidence threshold is 0.7 and the calculated Pfinal is 0.75, it is considered positive, suggesting the possible presence of the cancer biomarker and requiring further medical diagnosis. This comprehensive evaluation method fully utilizes information from multimodal data, improving the accuracy and reliability of disease detection.

[0128] S160. Output the detection results to the user interface and store the detection data for subsequent analysis and medical record tracing.

[0129] In this embodiment, the final detection results, including the identification results (positive or negative) of each detection area and the corresponding probability values, are organized and formatted, and clearly and intuitively displayed to the user through a user interface (such as a mobile APP), so that the user can quickly understand the detection situation and results.

[0130] The various data collected in this test (raw data such as images, vibrations, and temperatures, as well as processed feature data and test results) will be fed back and stored. On the one hand, this data can be used for subsequent data analysis, model optimization, and system upgrades. On the other hand, it also provides complete data support for the preservation and traceability of medical records, which helps to achieve long-term monitoring and management of diseases.

[0131] For example, use a high-resolution camera to take a picture of the blood glucose test strip, ensuring that the image is clear, complete, and accurately reflects the color changes of the reaction area on the test strip, such as the color depth of the glucose oxidase reaction area on the test strip, and record the timestamp of the image acquisition.

[0132] Based on the purpose of this blood glucose test, a descriptive text is constructed, clearly indicating the name of the disease to be concerned (hyperglycemia), the key indicator of concern (blood glucose concentration), and the expected threshold (e.g., the upper limit of the normal blood glucose concentration range).

[0133] A MEMS accelerometer was installed 0.5 mm below the microchannels of the blood glucose test strip to synchronously acquire signals of minute physical changes at a sampling frequency of 2 kHz. These signals reflect the subtle vibrations generated by the test strip during the flow of the blood sample in the microchannels and its reaction with reagents.

[0134] Infrared microsensors are used to monitor the temperature changes of the test strip in real time during the detection process, because the chemical reaction on the blood glucose test strip may be accompanied by heat changes, and the temperature signal helps to better understand the reaction process.

[0135] Based on the test task text, descriptions of abnormal manifestations in hyperglycemia testing are extracted from clinical guidelines and authoritative medical literature, breaking down complex medical sentences into short phrases consisting of "color-intensity-location". For example, "The appearance of dark brown spots in the reaction area of ​​the blood glucose test strip indicates that the blood glucose concentration may be too high" is broken down into (color: dark brown; intensity: high; location: reaction area).

[0136] The textual thinking chain method is used to further process the constructed set of biochemical violation descriptions, transforming logically complex and verbose medical descriptions into long sentences composed of simple words and relationships, resulting in a set of short sentences. For example, "If a light yellow change is observed in a certain area of ​​the test strip, it may indicate that the blood glucose concentration is in a critical state" is simplified to (color: light yellow; intensity: low; location: specific reaction area).

[0137] Each simple description in the simplified description set is segmented into basic semantic units. For example, "dark brown" can be further parsed into "dark" and "brown", while "reaction zone light yellow change" may be broken down into "reaction zone", "light yellow", and "change".

[0138] By using a text encoder to convert the semantic units obtained from word segmentation into fixed-dimensional word vectors, the originally incalculable text information is transformed into a data form that can be manipulated mathematically.

[0139] Text data with the word vectors is input into a multi-head self-attention mechanism to capture key information such as the correlation and dependency between different words in the text, and generate a text feature vector that comprehensively represents the text features of the detection task.

[0140] The blood glucose test strip image is segmented into micro-blocks. For example, it can be uniformly divided into 10×10 pixel blocks, with each micro-block representing a local view of the original image. This facilitates more precise feature analysis and processing for each micro-block in the future.

[0141] The text feature vector is converted to a two-dimensional format. Image patches are then convolved with the converted text feature vectors. A similarity score is calculated using the sigmoid function to obtain the matching result. Patches with high similarity scores may contain biochemical features that match the text description, such as color changes in the blood glucose response area.

[0142] Based on the matching results, image micro-patches that meet the requirements are selected, flattened into one-dimensional vectors, and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors. Then, a multi-head self-attention mechanism is used to perform deep encoding on the image word vectors to obtain image feature vectors.

[0143] The vibration feature vector is obtained by performing a matrix transpose operation on the vibration signal, and the temperature feature vector is obtained by normalizing and encoding the temperature signal.

[0144] Image feature vectors, vibration feature vectors, and temperature feature vectors are input into a regression model. After deep feature extraction and flattening into one-dimensional vectors, cosine similarity is calculated to achieve the fusion and evaluation of multimodal data at the feature level.

[0145] The image feature vector, vibration feature vector, and temperature feature vector are input into the corresponding classification model to predict the probability values ​​of disease occurrence based on the corresponding features, namely the first probability value Pimg based on image features, the second probability value Pvib based on vibration features, and the third probability value Ptemp based on temperature features.

[0146] A specific weighted fusion strategy (e.g., Pfinal = 0.5 × S × (Pimg + 0.3Pvib + 0.2Ptemp)) is used to fuse the probability values ​​corresponding to the three modalities, calculate the final disease occurrence probability Pfinal, and compare it with the set occurrence threshold to determine whether the test result is positive or negative.

[0147] The final test results, including the identification results (positive or negative) of each test area and the corresponding probability values, will be clearly and intuitively displayed to the user through a user interface (such as a mobile APP).

[0148] The various data collected in this test (raw data such as images, vibration, and temperature, as well as processed feature data and test results) will be fed back and stored for subsequent data analysis, model optimization, and medical record tracing.

[0149] The aforementioned disease test strip detection method, which integrates deep feature extraction, combines test strip images acquired by a camera, vibration signals collected by a MEMS accelerometer, and temperature signals collected by an infrared microsensor. It simplifies biochemical violations into color-intensity-location triplets using a text-based thought chain method and generates text feature vectors through a multi-head self-attention mechanism. Further, the test strip image is segmented into multiple blocks and matched against the text feature vectors, using block filtering and deep encoding techniques to obtain image feature vectors. Subsequently, the image, vibration, and temperature feature vectors are fused, and a regression model is used to calculate similarity, combined with a classification model to predict probabilities. A specific weighting strategy is then applied to derive the final disease probability to determine the positive or negative result. This method addresses the issues of strong subjectivity, low accuracy, and susceptibility to interference inherent in traditional disease test strip detection. It also overcomes the limitations of existing technologies in accurately locating local features, achieving spatiotemporal fusion and multimodal verification. This significantly improves the detection rate of weak positives and drastically reduces the error rates of false negatives and false positives, thereby enhancing the overall accuracy and reliability of the detection. Finally, the detection results can be output to the user interface in real time and stored for subsequent analysis and medical record tracing.

[0150] Figure 2 This is a schematic block diagram of a disease test strip detection system 300 that integrates deep feature extraction, provided in an embodiment of the present invention. Figure 2 As shown, corresponding to the above-described disease test strip detection method based on fusion deep feature extraction, the present invention also provides a disease test strip detection system 300 based on fusion deep feature extraction. This disease test strip detection system 300 includes a unit for performing the above-described disease test strip detection method based on fusion deep feature extraction, and the system can be configured in a server. Specifically, please refer to... Figure 2 The disease test strip detection system 300, which integrates deep feature extraction, includes an acquisition unit 301, an set construction unit 302, a text feature vector generation unit 303, an image feature vector generation unit 304, a detection unit 305, and an output unit 306.

[0151] The acquisition unit 301 is used to acquire test strip images using a camera and determine the detection task text, and collect vibration and temperature signals using a MEMS accelerometer and an infrared micro-sensor, respectively; the set construction unit 302 is used to construct a set of biochemical violation descriptions based on the detection task text, and simplify it into a set of short sentences in the form of color-intensity-position triplets using a text thinking chain method; the text feature vector generation unit 303 is used to generate text feature vectors by segmenting and encoding the short sentence set and using a multi-head self-attention mechanism; the image feature vector generation unit 304 is used to segment the test strip image into several blocks, match and filter them with the text feature vectors, and obtain image feature vectors through block filtering and deep encoding using a multi-head self-attention mechanism; the detection unit 305 is used to fuse the image feature vectors, vibration and temperature signals, calculate similarity using a regression model, predict probabilities using a classification model, and obtain the final disease occurrence probability by weighted summation and determine positive or negative to obtain the detection result; the output unit 306 is used to output the detection result to the user interface and store the detection data for subsequent analysis and medical record tracing.

[0152] In one embodiment, the acquisition unit 301 is used to acquire test strip images using a camera and determine the detection task text, install a MEMS accelerometer at a specific position below the microchannel of the test strip and set an infrared microsensor at a corresponding position to collect vibration and temperature signals respectively.

[0153] In one embodiment, the set construction unit 302 is used to construct a biochemical violation description set based on the detection task text. The biochemical violation description set includes splitting complex medical sentences about the disease to be tested in clinical guidelines into short sentences in the form of color-intensity-location triplets to form abnormal expressions. The biochemical violation description set is then processed using a text thinking chain method to convert the logically complex and verbose medical descriptions into long sentences composed of simple words and relationships, resulting in a short sentence set.

[0154] In one embodiment, the text feature vector generation unit 303 is used to perform word segmentation processing on each simple description of the short sentence set, decomposing medical term words into basic semantic units; using a text encoder to convert the semantic units obtained from word segmentation into fixed-dimensional word vectors; inputting the text data with the word vectors into a multi-head self-attention mechanism to capture key information on the correlation and dependency relationships between different words in the text, and generating a text feature vector that comprehensively represents the text features of the detection task.

[0155] In one embodiment, the image feature vector generation unit 304 is used to segment the test strip image into blocks to obtain image micro-blocks; the text feature vector is in two-dimensional format, and the image micro-blocks are convolved with the converted text feature vectors. A similarity score is calculated using the Sigmoid function to obtain a matching result; image micro-blocks that meet the requirements are selected based on the matching result, and the image micro-blocks that meet the requirements are flattened into one-dimensional vectors and mapped to a high-dimensional space through a fully connected layer to obtain image word vectors. Then, a multi-head self-attention mechanism is used to perform deep encoding on the image word vectors to mine the association and dependency relationships of feature elements and obtain image feature vectors.

[0156] In one embodiment, the detection unit 305 is used to process the vibration and temperature signals to obtain vibration feature vectors and temperature feature vectors; fuse the image feature vector, vibration feature vector, and temperature feature vector, and input them into a regression model. After deep feature extraction and flattening into a one-dimensional vector, the cosine similarity is calculated; the image feature vector, vibration feature vector, and temperature feature vector are respectively input into the corresponding classification model to predict the probability value of disease occurrence based on the corresponding feature; based on the cosine similarity and the probability value of disease occurrence based on the corresponding feature, a specific weighted fusion strategy is used to fuse the probability values ​​corresponding to the three modalities to calculate the final probability of disease occurrence, and compare it with a set threshold to determine whether the detection result is positive or negative.

[0157] In one embodiment, the detection unit 305 is used to perform a matrix transpose operation on the acquired vibration signal to obtain a vibration feature vector; to normalize the acquired temperature signal, adjust its numerical range to within a set calculation interval, and encode the temperature signal to obtain a temperature feature vector.

[0158] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned disease test strip detection system 300 with fusion deep feature extraction and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0159] The aforementioned disease test strip detection system 300, which integrates deep feature extraction, can be implemented as a computer program, which can perform tasks such as... Figure 3 It runs on the computer device shown.

[0160] Please see Figure 3 , Figure 3 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0161] See Figure 3 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0162] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a disease test strip detection method incorporating deep feature extraction.

[0163] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0164] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a disease test strip detection method that incorporates deep feature extraction.

[0165] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0166] The processor 502 is used to run a computer program 5032 stored in a memory to implement all steps of the disease test strip detection method based on fusion deep feature extraction.

[0167] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0168] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0169] Therefore, the present invention also provides a storage medium. This storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all steps of the disease test strip detection method based on fused deep feature extraction.

[0170] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0171] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0172] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0173] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the system of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0174] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0175] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A disease test paper detection method fusing deep feature extraction, characterized in that, The method comprises the following steps: acquiring a test paper image by using a camera and determining a detection task text, collecting vibration and temperature signals by using a MEMS accelerometer and an infrared micro sensor, respectively; constructing a biochemical rule violation description set according to the detection task text, and simplifying the set into a short sentence set in the form of a color-intensity-position triplet by using a text thinking chain method; generating a text feature vector by using a multi-head self-attention mechanism after segmenting and encoding the short sentence set; segmenting the test paper image into a plurality of blocks, matching and screening the blocks with the text feature vector, and obtaining an image feature vector through block filtering and multi-head self-attention mechanism deep encoding; fusing the image feature vector, the vibration and temperature signals, calculating the similarity by using a regression model, combining the prediction probability of a classification model, and obtaining the final disease occurrence probability and the positive and negative judgment to obtain the detection result; outputting the detection result to a user interface and storing the detection data for subsequent analysis and medical record tracing. 2.The disease test paper detection method with fusion depth feature extraction according to claim 1, characterized in that, The method of acquiring a test paper image by using a camera and determining a detection task text, collecting vibration and temperature signals by using a MEMS accelerometer and an infrared micro sensor, respectively, comprises the following steps: acquiring a test paper image by using a camera and determining a detection task text, installing a MEMS accelerometer at a specific position below the microchannel of the test strip and setting an infrared micro sensor at a corresponding position to collect vibration and temperature signals, respectively. 3.The disease test paper detection method of fusing depth feature extraction according to claim 1, characterized in that, The method of constructing a biochemical rule violation description set according to the detection task text, and simplifying the set into a short sentence set in the form of a color-intensity-position triplet by using a text thinking chain method, comprises the following steps: constructing a biochemical rule violation description set according to the detection task text, wherein the biochemical rule violation description set comprises splitting complex medical long sentences about the disease to be detected in the clinical guideline data into short sentences in the form of a color-intensity-position triplet to form abnormal expressions; processing the biochemical rule violation description set by using a text thinking chain method to convert the logically complex and lengthy medical descriptions into long sentences composed of simple words and relationships, thereby obtaining a short sentence set. 4.The disease test paper detection method with fusion depth feature extraction according to claim 1, characterized in that, The method of generating a text feature vector by using a multi-head self-attention mechanism after segmenting and encoding the short sentence set, comprises the following steps: segmenting each simple description in the short sentence set to process medical terminology and decompose it into basic semantic units; converting the semantic units obtained by segmentation into word vectors of a fixed dimension by using a text encoder; inputting the text data with the word vectors into a multi-head self-attention mechanism to capture the correlation and dependency relationship key information between different words in the text, thereby generating a text feature vector that comprehensively represents the features of the detection task text. 5.The disease test paper detection method of fusing depth feature extraction according to claim 1, characterized in that, The method of segmenting the test paper image into a plurality of blocks, matching and screening the blocks with the text feature vector, and obtaining an image feature vector through block filtering and multi-head self-attention mechanism deep encoding, comprises the following steps: segmenting the test paper image into image microblocks; performing convolution operation on the image microblocks and the converted text feature vector in a two-dimensional format, calculating the similarity score by using a Sigmoid function, and obtaining a matching result; According to the matching result, image microblocks meeting the requirements are screened out, the image microblocks meeting the requirements are flattened into one-dimensional vectors and mapped to a high-dimensional space through a fully connected layer to obtain picture word vectors, and a multi-head self-attention mechanism is used to deeply encode the picture word vectors to mine the correlation and dependency of feature elements, so as to obtain an image feature vector. 6.The disease test paper detection method of fusing depth feature extraction according to claim 1, characterized in that, The image feature vector, the vibration and temperature signals are fused, the similarity is calculated through a regression model, the prediction probability is combined through a classification model, the final disease occurrence probability is obtained by weighted summation, and the positive and negative are judged to obtain a detection result, including: The vibration and temperature signals are processed to obtain a vibration feature vector and a temperature feature vector; The image feature vector, the vibration feature vector and the temperature feature vector are fused and input into a regression model, and the cosine similarity is calculated after deep feature extraction and flattening into a one-dimensional vector; The image feature vector, the vibration feature vector and the temperature feature vector are input into the corresponding classification model respectively, and the disease occurrence probability value based on the corresponding feature is predicted; Based on the cosine similarity and the disease occurrence probability value based on the corresponding feature, a specific weighted fusion strategy is used to fuse the probability values of the three modal data, and the final disease occurrence probability is calculated and compared with the set threshold to determine the detection result as positive or negative. 7.The disease test paper detection method of fusing depth feature extraction according to claim 6, characterized in that, The vibration and temperature information are processed to obtain a vibration feature vector and a temperature feature vector, including: The collected vibration signal is subjected to a matrix transposition operation to obtain a vibration feature vector; The collected temperature signal is subjected to normalization processing, the numerical range is adjusted to a set calculation interval, and the temperature signal is encoded to obtain a temperature feature vector.

8. A disease test strip detection system fusing depth feature extraction, characterized in that, Including: An acquisition unit is configured to acquire a test paper image by using a camera, determine a detection task text, and collect vibration and temperature signals by using a MEMS accelerometer and an infrared micro sensor respectively; A set construction unit is configured to construct a biochemical violation description set according to the detection task text, and simplify the set into a short sentence set in the form of a color-intensity-position triple according to a text thinking chain method; A text feature vector generation unit is configured to perform word segmentation and encoding on the short sentence set, and generate a text feature vector by using a multi-head self-attention mechanism; An image feature vector generation unit is configured to divide the test paper image into a plurality of blocks, match and screen the blocks with the text feature vector, and obtain an image feature vector through block filtering and multi-head self-attention mechanism deep coding; A detection unit is configured to fuse the image feature vector, the vibration and temperature signals, calculate the similarity through a regression model, combine the prediction probability through a classification model, and obtain the final disease occurrence probability by weighted summation to judge the positive and negative, so as to obtain a detection result. An output unit is configured to output the detection result to a user interface and store detection data for subsequent analysis and medical record tracing. 9.The disease test paper detection system with fused depth feature extraction of claim 8, wherein, The acquisition unit is configured to acquire a test paper image by using a camera, determine a detection task text, install a MEMS accelerometer under a microchannel of a test strip at a specific position and set an infrared micro sensor at a corresponding position, and collect vibration and temperature signals respectively.

10. A computer device, comprising: The computer device comprises a memory and a processor, the memory has stored thereon a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

Images