Blood stratification method, device and medium based on multi-modal data fusion
By combining multimodal data fusion methods of blood images and centrifuge physical parameters, the problem of inaccurate blood component layering results in existing technologies is solved, achieving more accurate blood component layer location identification and improved model robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QINGDAO HAIER BIOMEDICAL TECH CO LTD
- Filing Date
- 2026-02-24
- Publication Date
- 2026-06-16
Smart Images
Figure CN122223397A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image processing technology, specifically to a blood stratification method, device, and medium based on multimodal data fusion. Background Technology
[0002] After a blood sample is obtained through a blood collection tube, it needs to be centrifuged to create different layers of blood components within the tube. These layers allow for the separation of different blood components, enabling the detection and treatment of various diseases. Therefore, accurately determining the specific locations of these different blood component layers within the centrifuged blood sample is crucial.
[0003] In existing methods, after centrifuging a blood sample, a blood image is usually obtained. The blood image is then processed using a neural network. Specifically, the color, brightness, and position information of each pixel in the image are clustered or classified to achieve image layering and determine the specific location of each blood component layer. However, when the physical parameters such as the centrifuge speed, running time, or ambient temperature change during the centrifugation of the blood sample, the layering effect of the blood sample will be significantly affected, and blurred boundaries or abnormal layering may easily occur between blood component layers.
[0004] Therefore, existing methods cannot detect the various physical parameters of the centrifuge during the centrifugation of blood samples, resulting in inaccurate stratification results of blood components and reduced robustness of neural networks.
[0005] Accordingly, there is a need in the field for a new blood stratification method based on multimodal data fusion to address the above problems. Summary of the Invention
[0006] In order to overcome the above-mentioned defects, this application is made to solve or at least partially solve the technical problem that existing methods cannot detect the various physical parameters of the centrifuge during the centrifugation of blood samples, resulting in inaccurate stratification results of blood components.
[0007] In a first aspect, a blood stratification method based on multimodal data fusion is provided, the method comprising:
[0008] After centrifuging the blood sample in the blood collection tube using a centrifuge, a blood image of the centrifuged blood sample and the physical parameters of the centrifuge during centrifugation are obtained.
[0009] The blood image and the physical parameters of the centrifuge are input into a preset image layering model for processing to obtain the blood component category corresponding to each pixel in the blood image. The blood component category includes plasma, white blood cells, and red blood cells.
[0010] Based on the blood component category corresponding to each pixel, the image layering result of the blood image is obtained. The image layering result is used to indicate the specific location of different blood component layers in the blood collection tube. The blood component layers include the plasma layer, the white membrane layer, and the red blood cell layer.
[0011] The image layering model is trained using samples of the blood images and the physical parameters of the centrifuge.
[0012] In one technical solution of the blood stratification method based on multimodal data fusion described above, the image stratification model includes an image feature extraction module, a physical parameter feature extraction module, an attention weighting module, a feature fusion module, and a classification module;
[0013] The image feature extraction module is configured to: extract image features from the blood image to obtain an image feature vector;
[0014] The physical parameter feature extraction module is configured to: extract features from the physical parameters of the centrifuge to obtain a physical parameter feature vector;
[0015] The attention weighting module is configured to: perform attention weighting on the image feature vector based on the physical parameter feature vector to obtain the attention-weighted image feature vector;
[0016] The feature fusion module is configured to: perform feature fusion on the attention-weighted image feature vector and the physical parameter feature vector to obtain a fused feature vector, wherein the feature fusion includes: obtaining the fusion weight corresponding to the attention-weighted image feature vector, obtaining the product of the fusion weight and the attention-weighted image feature vector, concatenating the product with the physical parameter feature vector to obtain a concatenated feature vector, and performing feature fusion on the concatenated feature vector to obtain a fused feature vector;
[0017] The classification module is configured to: perform classification processing on the fused feature vector to obtain the blood component category corresponding to each pixel in the blood image;
[0018] Both the image feature extraction module and the physical parameter feature extraction module employ neural networks, and the number of parameters in the physical parameter feature extraction module is less than the number of parameters in the image feature extraction module.
[0019] In one technical solution of the blood layering method based on multimodal data fusion described above, the image data of the blood image includes a multidimensional vector for each pixel in the blood image, and each multidimensional vector includes the RGB color value and position coordinates of the pixel; the image feature extraction module includes multiple parallel one-dimensional convolutional layers; the feature extraction of the blood layered image to obtain the image feature vector includes:
[0020] Based on the image data, obtain the multidimensional vector of each pixel in the blood image;
[0021] Based on the multiple parallel one-dimensional convolutional layers, feature extraction is performed on the multi-dimensional vector of each pixel to obtain the image feature vector.
[0022] In one technical solution of the blood stratification method based on multimodal data fusion described above, the physical parameters of the centrifuge include centrifuge speed setting, rotor model, rotation speed, running time, and ambient temperature; the physical parameter feature extraction module includes an embedding layer, a fully connected layer, and a fusion layer; the feature extraction of the physical parameters of the centrifuge to obtain the physical parameter feature vector includes:
[0023] Based on the embedded layer, feature mapping is performed on the centrifuge gear position and rotor model to obtain a first feature vector;
[0024] Based on the fully connected layer, feature mapping is performed on the rotation speed, running time, and ambient temperature to obtain a second feature vector;
[0025] Based on the fusion layer, the first feature vector and the second feature vector are fused to obtain the physical parameter feature vector.
[0026] In one technical solution of the blood layering method based on multimodal data fusion described above, the image layering model is trained through the following steps:
[0027] Obtain the blood images and samples of the physical parameters of the centrifuge;
[0028] Freeze the module parameters of the image feature extraction module, input the sample into the image hierarchical model to be trained, and train the image hierarchical model to be trained based on the weighted cross-entropy loss function and the online hard example mining method;
[0029] When the preset number of training iterations is reached, the module parameters of the image feature extraction module are unfrozen, and the image layering model to be trained is trained again. After training is completed, the trained image layering model is obtained.
[0030] The weighted cross-entropy loss function is a loss function obtained by weighting and calculating the first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function. The first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function are respectively used to classify plasma, white blood cells, and red blood cells in the image layering model.
[0031] In one technical solution of the blood stratification method based on multimodal data fusion described above, before freezing the module parameters of the image feature extraction module and inputting the sample into the image stratification model to be trained, the method further includes:
[0032] The blood image sample is subjected to a first preprocessing, which includes image correction of the blood image sample and obtaining the sharpness of the blood image sample. The fusion weight corresponding to the attention-weighted image feature vector is determined based on the sharpness, and the sharpness is positively correlated with the fusion weight.
[0033] The physical parameter samples are subjected to a second preprocessing, which includes standardizing the physical parameter samples and adding Gaussian noise to the physical parameter samples.
[0034] In one technical solution of the blood layering method based on multimodal data fusion described above, the image correction includes the following steps:
[0035] The specific location of the outer wall of the blood collection tube in the blood image sample was determined based on a gradient measurement method.
[0036] The tilt angle of the blood collection tube is determined based on the specific location;
[0037] The blood image sample is rotated and corrected based on the tilt angle of the blood collection tube to ensure that the blood collection tube is in a vertical position.
[0038] In one technical solution of the blood stratification method based on multimodal data fusion described above, the image feature vector includes sub-vectors of multiple image feature channels, and the attention weighting of the image feature vector based on the physical parameter feature vector includes:
[0039] Based on the physical parameter feature vector, the attention weights of the sub-vectors of each of the image feature channels are determined respectively;
[0040] The sub-vectors of each image feature channel are obtained as a product of their respective attention weights;
[0041] The attention-weighted image feature vector is obtained by multiplying the sub-vectors corresponding to each of the image feature channels.
[0042] In a second aspect, an electronic device is provided, comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described in any of the above-described technical solutions of the blood stratification method based on multimodal data fusion.
[0043] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the method described in any of the above-described technical solutions of the blood stratification method based on multimodal data fusion.
[0044] The above-described technical solutions of this application have at least one or more of the following beneficial effects:
[0045] In implementing the blood layering image segmentation method based on multi-feature fusion provided in this application, after centrifuging the blood sample in the blood collection tube using a centrifuge, a blood image of the centrifuged blood sample and the physical parameters of the centrifuge during centrifugation are obtained. The blood image and the physical parameters of the centrifuge are input into a preset image layering model for processing to obtain the blood component category corresponding to each pixel in the blood image. The blood component category includes plasma, white blood cells, and red blood cells. Based on the blood component category corresponding to each pixel, the image layering result of the blood image is obtained. The image layering result is used to indicate the specific location of different blood component layers in the blood collection tube. The blood component layers include the plasma layer, the white blood cell layer, and the red blood cell layer. Based on this application, by combining various physical parameters of the centrifuge, the specific location of each blood component layer in the image can be accurately determined, making the image layering result more accurate. Attached Figure Description
[0046] The disclosure of this application will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this application. Wherein:
[0047] Figure 1 This is a schematic flowchart of the main steps of a blood stratification method based on multimodal data fusion according to an embodiment of this application;
[0048] Figure 2 This is a schematic diagram of the main structure of an image layering model according to an embodiment of this application;
[0049] Figure 3 This is a schematic diagram of the main structure of an electronic device according to an embodiment of this application.
[0050] Figure label:
[0051] 31: Memory; 32: Processor. Detailed Implementation
[0052] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0053] In the description of this application, "processor" can include hardware, software, or a combination of both. A processor can be a central processing unit, microprocessor, graphics processor, digital signal processor, or any other suitable processor. A processor has data and / or signal processing capabilities. A processor can be implemented in software, in hardware, or a combination of both. Computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0054] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a blood stratification method based on multimodal data fusion according to an embodiment of this application. Figure 1 As shown, the blood stratification method based on multimodal data fusion in this application embodiment mainly includes the following steps S101 to S103.
[0055] Step S101: After centrifuging the blood sample in the blood collection tube using a centrifuge, obtain the blood image of the centrifuged blood sample and the physical parameters of the centrifuge during centrifugation;
[0056] Step S102: Input the blood image and the physical parameters of the centrifuge into the preset image layering model for processing to obtain the blood component category corresponding to each pixel in the blood image. The blood component category includes plasma, white blood cells, and red blood cells.
[0057] Step S103: Based on the blood component category corresponding to each pixel, obtain the image layering result of the blood image. The image layering result is used to indicate the specific location of different blood component layers in the blood collection tube. The blood component layers include the plasma layer, the white membrane layer, and the red blood cell layer. The plasma layer is composed of plasma, the white membrane layer is composed of white blood cells, and the red blood cell layer is composed of red blood cells.
[0058] Based on the method described in steps S101 to S103 above, the blood image is combined with the physical parameters of the centrifuge and input into the image layering model for processing. This obtains the blood component category corresponding to each pixel in the blood image and the image layering result of the blood image. This can accurately determine the specific location of different blood component layers in the blood collection tube, thereby making the image layering result more accurate. It avoids the problem in existing methods where changes in the physical parameters of the centrifuge affect the degree of separation of the blood sample after centrifugation, which ultimately affects the accuracy of the image layering result and the robustness of the image layering model.
[0059] Specifically, this application uses the physical parameters of the centrifuge as the input of the image layering model. When the physical parameters of the centrifuge change, the image layering model can sense the change in physical parameters and thus combine the changed physical parameters to determine the blood component category corresponding to each pixel in the blood image. Compared with the image layering model based on a single image input, the image layering model in this application has higher robustness.
[0060] Furthermore, since the image layering model in this application simultaneously accepts blood image information and centrifuge physical parameter information, the image layering model can more comprehensively grasp the layering of blood samples in the blood collection tube after centrifugation. Especially for ultra-thin layers such as the white film layer, even if the image quality is poor, the physical parameters can be combined to accurately determine the blood component category corresponding to each pixel in the blood image, thereby obtaining more accurate image layering results of the blood image.
[0061] The following description continues with an embodiment of the blood stratification method based on multimodal data fusion provided in this application, specifically describing step S101.
[0062] In one embodiment of step S101 above, the blood image of the centrifuged blood sample includes the blood sample and the blood collection tube containing the blood sample. The image data of the blood image includes a multidimensional vector of each pixel in the blood image. Each multidimensional vector includes the RGB color value of the pixel (e.g., the RGB value of H×W×3) and the position coordinates (e.g., normalized Y coordinates, where Y is the Y-axis in the image coordinate system, and the normalized Y coordinates are the coordinate values obtained after normalizing the Y-axis coordinates of each pixel). The physical parameters of the centrifuge include the centrifuge speed setting, rotor model, rotation speed, running time, ambient temperature, etc.
[0063] Furthermore, before inputting the blood images and the physical parameters of the centrifuge into a preset image layering model for processing, the method also includes:
[0064] The blood image undergoes a first preprocessing step, which includes image correction and obtaining the clarity of the blood image. Based on the clarity, the fusion weight corresponding to the attention-weighted image feature vector is determined. Clarity is positively correlated with the fusion weight. If the blood image is too blurry and has low clarity, the fusion weight corresponding to the attention-weighted image feature vector will also be lower in subsequent feature fusion.
[0065] The physical parameter samples undergo a second preprocessing step, which includes standardization or normalization of the physical parameter samples.
[0066] Furthermore, image correction includes the following steps:
[0067] The specific location of the outer wall of the blood collection tube in the blood image was determined based on the gradient measurement method;
[0068] The tilt angle of the blood collection tube is determined based on its specific location;
[0069] The blood image is rotated and corrected based on the tilt angle of the blood collection tube to ensure that the blood collection tube is in a vertical position.
[0070] The following description continues with an embodiment of the blood stratification method based on multimodal data fusion provided in this application, specifically describing step S102.
[0071] In one embodiment of step S102 above, see Appendix Figure 2 , Figure 2 This is a schematic diagram of the main structure of an image layering model according to an embodiment of this application; as shown Figure 2 As shown, the image layering model includes an image feature extraction module, a physical parameter feature extraction module, an attention weighting module, a feature fusion module, and a classification module;
[0072] The image feature extraction module is configured to: receive a blood image, extract image features from the blood image, obtain an image feature vector, and input the image feature vector into the attention weighting module;
[0073] The physical parameter feature extraction module is configured to: receive the physical parameters of the centrifuge, extract features from the physical parameters of the centrifuge, obtain the physical parameter feature vector, and input the physical parameter feature vector into the attention weighting module;
[0074] The attention weighting module is configured to: receive image feature vectors and physical parameter feature vectors, perform attention weighting on the image feature vectors based on the physical parameter feature vectors, obtain attention-weighted image feature vectors, and input the attention-weighted image feature vectors and physical parameter feature vectors into the feature fusion module;
[0075] The feature fusion module is configured to: receive the attention-weighted image feature vector and the physical parameter feature vector, perform feature fusion on the attention-weighted image feature vector and the physical parameter feature vector to obtain a fused feature vector, and input the fused feature vector into the classification module;
[0076] The classification module is configured to: receive the fused feature vector, perform classification processing on the fused feature vector, and obtain the blood component category corresponding to each pixel in the blood image;
[0077] Both the image feature extraction module and the physical parameter feature extraction module use neural networks, and the number of parameters in the physical parameter feature extraction module is less than that in the image feature extraction module.
[0078] Furthermore, the image feature extraction module includes multiple parallel one-dimensional convolutional layers to extract features from the blood layered image, obtaining image feature vectors including:
[0079] Based on the image data of the blood image, obtain the multidimensional vector of each pixel in the blood image;
[0080] Based on multiple parallel one-dimensional convolutional layers, feature extraction is performed on the multi-dimensional vector of each pixel to obtain the image feature vector;
[0081] Each one-dimensional convolutional layer uses a convolutional kernel of a different size, and the image feature vector includes sub-vectors of multiple image feature channels. The dimension of the image feature vector is H×F. img F img The number of image feature channels; for example, the kernel size = 3, 5, 7..., one-dimensional convolutional layers with small kernels are used to capture sharp boundaries (such as the surface of red blood cells), and one-dimensional convolutional layers with large kernels are used to capture blurred transition areas (such as the turbidity of blood plasma).
[0082] Furthermore, the physical parameter feature extraction module includes an embedding layer, a fully connected layer, and a fusion layer; it extracts features from the centrifuge's physical parameters to obtain physical parameter feature vectors, including:
[0083] Based on the embedding layer, feature mapping is performed on discrete variables such as centrifuge gear position and rotor model to obtain the first feature vector;
[0084] Based on the fully connected layer, feature mapping is performed on continuous variables such as rotation speed, running time and ambient temperature to obtain the second feature vector;
[0085] Based on the fusion layer, the first feature vector and the second feature vector are fused to obtain the physical parameter feature vector.
[0086] The dimension of the feature vector obtained by fusing the first and second feature vectors is 1×F. phyTo spatially align the fused feature vector with the image feature vector, after fusing the first and second feature vectors in the fusion layer, the fused feature vector needs to be copied and expanded in the spatial dimension to obtain the physical parameter feature vector. At this point, the dimension of the physical parameter feature vector becomes... , This represents the number of physical parameter characteristic channels.
[0087] Based on the image feature extraction module and the physical parameter feature extraction module, image feature vectors and physical parameter feature vectors are obtained. This allows for the combination of image information and physical parameter information, which can more accurately determine the specific location of different blood component layers in the blood collection tube, thereby making the image layering results more accurate. This avoids the problem in existing methods where changes in the physical parameters of the centrifuge affect the degree of separation of blood samples after centrifugation, ultimately impacting the accuracy of the image layering results and the robustness of the image layering model.
[0088] Furthermore, attention weighting is applied to the image feature vector based on the physical parameter feature vector, including:
[0089] Based on the physical parameter feature vector, the attention weights of the sub-vectors of each image feature channel are determined respectively;
[0090] Obtain the product of the sub-vector of each image feature channel and its attention weight;
[0091] Based on the product of the sub-vectors corresponding to each image feature channel, the attention-weighted image feature vector is obtained. For example, if the image feature channels include texture channels and color channels, when the physical parameter feature vector shows "high speed", the image layering model automatically generates attention weights. The attention weight of the sub-vector of the texture channel is generated as 0.1 (because the texture is not obvious at high speed), and the attention weight of the sub-vector of the color channel is generated as 0.9, so as to suppress the texture part in the image and enhance the color boundary part.
[0092] Based on the attention weighting module, the weights of image feature vectors can be adjusted according to the physical parameter feature vectors, thereby suppressing invalid features in the image and enhancing the valid features in the image. This avoids the impact of invalid features on the accuracy of the image layering model output results when inputting into subsequent modules for processing.
[0093] Furthermore, the feature fusion module includes a concatenation layer and a convolutional layer, which fuse the attention-weighted image feature vector and the physical parameter feature vector to obtain a fused feature vector, including:
[0094] Obtain the fusion weights corresponding to the attention-weighted image feature vectors;
[0095] Obtain the product of the fusion weights and the attention-weighted image feature vectors;
[0096] Based on the splicing layer, the product is spliced with the physical parameter feature vector to obtain the spliced feature vector;
[0097] Based on convolutional layers, feature fusion is performed on the concatenated feature vectors to obtain a fused feature vector. For example, the convolutional layers are two 1×1 layers with a fusion weight of 0.2. The attention-weighted image feature vector is U1, the physical parameter feature vector is U2, and the product of the fusion weight and the attention-weighted image feature vector is 0.2 × U1. The concatenated feature vector is... .
[0098] Furthermore, the classification module includes a normalization layer; it performs classification processing on the fused feature vector to obtain the blood component categories corresponding to each pixel in the blood image, including:
[0099] Based on the normalization layer, the probability of blood component category of each pixel in the blood image is obtained. Based on the category probability, the blood component category corresponding to each pixel is determined. For example, if the classification probabilities of plasma, white blood cells and red blood cells in a pixel are 0.1, 0.2 and 0.7 respectively, then the category of the pixel is red blood cells.
[0100] Furthermore, the image layering model is trained using samples of blood images and the physical parameters of a centrifuge; the image layering model is trained through the following steps:
[0101] Acquire blood images and samples of the physical parameters of the centrifuge;
[0102] The module parameters of the image feature extraction module are frozen, and the samples are input into the image layering model to be trained. The image layering model to be trained is trained based on the weighted cross-entropy loss function and the online hard example mining method. For example, in the first 5-10 training iterations, the module parameters of the image feature extraction module are frozen, and only the module parameters of other modules are trained. This enables the image layering model to learn to give the approximate blood component category corresponding to each pixel based on the physical parameter samples, preventing the image layering model from getting trapped in the local optimum of image noise at the beginning of training.
[0103] When the preset number of training iterations is reached, the module parameters of the image feature extraction module are unfrozen, and the image layering model to be trained continues to be trained. After training is completed, the trained image layering model is obtained.
[0104] Among them, the weighted cross-entropy loss function is a loss function obtained by weighting and calculating the first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function. The first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function are used to classify plasma, white blood cells, and red blood cells in the image layering model, respectively.
[0105] The weighted cross-entropy loss function is L represents the weighted cross-entropy loss function. , , Let these represent the first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function, respectively. , , These represent the weights of the first, second, and third cross-entropy loss functions, respectively. For example, since white blood cells constitute a relatively small proportion of a blood sample, they can be given a higher weight in the weighted cross-entropy loss function. In this case, the weights are set as follows: , , This allows the image layering model to focus on white blood cells, overcoming the problem that the white blood cell membrane layer is too thin to be recognized.
[0106] Online hard example mining methods refer to automatically selecting pixels with the largest prediction errors (usually at blurred boundaries) during the training process and calculating gradients only for these difficult samples; this allows the image layering model to focus on overcoming the most difficult-to-distinguish boundaries.
[0107] Furthermore, before freezing the module parameters of the image feature extraction module and inputting samples into the image hierarchical model to be trained, the method also includes:
[0108] The blood image sample is subjected to a first preprocessing, which includes image correction of the blood image sample and obtaining the sharpness of the blood image sample. The fusion weight corresponding to the attention-weighted image feature vector is determined based on the sharpness, and the sharpness is positively correlated with the fusion weight.
[0109] The physical parameter samples undergo a second preprocessing process, which includes standardizing or normalizing the physical parameter samples and adding Gaussian noise to the physical parameter samples.
[0110] Furthermore, image correction includes the following steps:
[0111] The specific location of the outer wall of the blood collection tube in the blood image sample was determined based on the gradient measurement method.
[0112] The tilt angle of the blood collection tube is determined based on its specific location;
[0113] The blood image sample is rotated and corrected based on the tilt angle of the blood collection tube to ensure that the blood collection tube is in a vertical position.
[0114] Based on the above model training and preprocessing steps, the accuracy of the image layering model's layering results can be improved, and the model's robustness can be enhanced. For ultra-thin layers such as the white film layer, it can also be identified more accurately. At the same time, it can more accurately identify blurred boundaries and abnormal layering between layers.
[0115] The following description continues with an embodiment of the blood stratification method based on multimodal data fusion provided in this application, specifically describing step S103.
[0116] In one embodiment of step S103 above, the blood component category of the plasma layer is plasma, the blood component category of the white membrane layer is white blood cells and platelets, and the blood component category of the red blood cell layer is red blood cells. Therefore, the blood component category also includes platelets. The image layering model can also identify platelets in the blood image. When any pixel is identified as having a blood component category of plasma, the location of that pixel is the plasma layer. When any pixel is identified as having a blood component category of white blood cells and platelets, the location of that pixel is the white membrane layer. When any pixel is identified as having a blood component category of red blood cells, the location of that pixel is the red blood cell layer.
[0117] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted solutions are equivalent to the technical solutions described in this application and therefore will also fall within the protection scope of this application.
[0118] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0119] Another aspect of this application provides a computer-readable storage medium.
[0120] In one embodiment of a computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that performs the blood stratification method based on multimodal data fusion described in the above-described method embodiments. This program can be loaded and run by a processor to implement the blood stratification method based on multimodal data fusion described above. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0121] Another aspect of this application provides an electronic device.
[0122] In one embodiment of an electronic device according to this application, the electronic device may include at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the methods described in any of the above embodiments. See Appendix Figure 3 , Figure 3 The image exemplarily illustrates a communication connection between memory 31 and processor 32 via a bus.
[0123] In some embodiments of this application, the electronic device may further include at least one sensor for sensing information. The sensor is communicatively connected to any type of processor mentioned in this application. The processor communicates with the sensor to perform the methods described in any of the above embodiments. (The electronic device described in this application may be, but is not limited to, mobile phones, tablets, desktop computers, laptops, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and this application does not limit this.)
[0124] The technical solution of this application has been described above with reference to one embodiment shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A blood stratification method based on multimodal data fusion, characterized in that, include: After centrifuging the blood sample in the blood collection tube using a centrifuge, a blood image of the centrifuged blood sample and the physical parameters of the centrifuge during centrifugation are obtained. The blood image and the physical parameters of the centrifuge are input into a preset image layering model for processing to obtain the blood component category corresponding to each pixel in the blood image. The blood component category includes plasma, white blood cells, and red blood cells. Based on the blood component category corresponding to each pixel, the image layering result of the blood image is obtained. The image layering result is used to indicate the specific location of different blood component layers in the blood collection tube. The blood component layers include the plasma layer, the white membrane layer, and the red blood cell layer. The image layering model is trained using samples of the blood images and the physical parameters of the centrifuge.
2. The method according to claim 1, characterized in that, The image layering model includes an image feature extraction module, a physical parameter feature extraction module, an attention weighting module, a feature fusion module, and a classification module; The image feature extraction module is configured to: extract image features from the blood image to obtain an image feature vector; The physical parameter feature extraction module is configured to: extract features from the physical parameters of the centrifuge to obtain a physical parameter feature vector; The attention weighting module is configured to: perform attention weighting on the image feature vector based on the physical parameter feature vector to obtain the attention-weighted image feature vector; The feature fusion module is configured to: perform feature fusion on the attention-weighted image feature vector and the physical parameter feature vector to obtain a fused feature vector, wherein the feature fusion includes: obtaining the fusion weight corresponding to the attention-weighted image feature vector, obtaining the product of the fusion weight and the attention-weighted image feature vector, concatenating the product with the physical parameter feature vector to obtain a concatenated feature vector, and performing feature fusion on the concatenated feature vector to obtain a fused feature vector; The classification module is configured to: perform classification processing on the fused feature vector to obtain the blood component category corresponding to each pixel in the blood image; Both the image feature extraction module and the physical parameter feature extraction module employ neural networks, and the number of parameters in the physical parameter feature extraction module is less than the number of parameters in the image feature extraction module.
3. The method according to claim 2, characterized in that, The image data of the blood image includes a multi-dimensional vector for each pixel in the blood image, and each multi-dimensional vector includes the RGB color value and position coordinates of the pixel; the image feature extraction module includes multiple parallel one-dimensional convolutional layers; the feature extraction of the layered blood image to obtain the image feature vector includes: Based on the image data, obtain the multidimensional vector of each pixel in the blood image; Based on the multiple parallel one-dimensional convolutional layers, feature extraction is performed on the multi-dimensional vector of each pixel to obtain the image feature vector.
4. The method according to claim 2, characterized in that, The physical parameters of the centrifuge include centrifuge speed setting, rotor model, rotation speed, running time, and ambient temperature; the physical parameter feature extraction module includes an embedding layer, a fully connected layer, and a fusion layer; the process of extracting features from the physical parameters of the centrifuge to obtain a physical parameter feature vector includes: Based on the embedded layer, feature mapping is performed on the centrifuge gear position and rotor model to obtain a first feature vector; Based on the fully connected layer, feature mapping is performed on the rotation speed, running time, and ambient temperature to obtain a second feature vector; Based on the fusion layer, the first feature vector and the second feature vector are fused to obtain the physical parameter feature vector.
5. The method according to claim 2, characterized in that, The image layering model is trained through the following steps: Obtain the blood images and samples of the physical parameters of the centrifuge; Freeze the module parameters of the image feature extraction module, input the sample into the image hierarchical model to be trained, and train the image hierarchical model to be trained based on the weighted cross-entropy loss function and the online hard example mining method; When the preset number of training iterations is reached, the module parameters of the image feature extraction module are unfrozen, and the image layering model to be trained is trained again. After training is completed, the trained image layering model is obtained. The weighted cross-entropy loss function is a loss function obtained by weighting and calculating the first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function. The first cross-entropy loss function, the second cross-entropy loss function, and the third cross-entropy loss function are respectively used to classify plasma, white blood cells, and red blood cells in the image layering model.
6. The method according to claim 5, characterized in that, Before freezing the module parameters of the image feature extraction module and inputting the samples into the image hierarchical model to be trained, the method further includes: The blood image sample is subjected to a first preprocessing, which includes image correction of the blood image sample and obtaining the sharpness of the blood image sample. The fusion weight corresponding to the attention-weighted image feature vector is determined based on the sharpness, and the sharpness is positively correlated with the fusion weight. The physical parameter samples are subjected to a second preprocessing, which includes standardizing the physical parameter samples and adding Gaussian noise to the physical parameter samples.
7. The method according to claim 6, characterized in that, The image correction includes the following steps: The specific location of the outer wall of the blood collection tube in the blood image sample was determined based on a gradient measurement method. The tilt angle of the blood collection tube is determined based on the specific location; The blood image sample is rotated and corrected based on the tilt angle of the blood collection tube to ensure that the blood collection tube is in a vertical position.
8. The method according to claim 2, characterized in that, The image feature vector includes sub-vectors from multiple image feature channels. The attention weighting of the image feature vector based on the physical parameter feature vector includes: Based on the physical parameter feature vector, the attention weights of the sub-vectors of each of the image feature channels are determined respectively; The sub-vectors of each image feature channel are obtained as a product of their respective attention weights; The attention-weighted image feature vector is obtained by multiplying the sub-vectors corresponding to each of the image feature channels.
9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores a computer program that, when executed by the at least one processor, implements the blood stratification method based on multimodal data fusion as described in any one of claims 1 to 8.
10. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the blood stratification method based on multimodal data fusion as described in any one of claims 1 to 8.