Intelligent diagnosis method, device and equipment for snake venom poisoning and medium

Abstract

The method is mainly applied to the technical field of deep learning. The invention discloses an intelligent diagnosis method, device and equipment for snake venom poisoning and a medium, and the method comprises the steps: obtaining multi-modal data which comprises a wound image, a symptom text and a blood parameter; performing feature extraction processing on the wound image, the symptom text and the blood parameters to obtain image features corresponding to the wound image, semantic features corresponding to the symptom text and high-dimensional features corresponding to the blood parameters; performing alignment and fusion processing on the image features, the semantic features and the high-dimensional features through a cross-modal attention mechanism to generate fusion features; after the fusion features are input into a preset multi-task classification model, classification is carried out in a snake species label space and a toxicity category space according to the fusion features, a poisoning grade score is determined, and a diagnosis report is generated based on the poisoning grade score. According to the invention, high-precision, rapid and intelligent diagnosis of snake venom poisoning can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a snake venom poisoning intelligent diagnosis method, device, equipment and medium. BACKGROUND

[0002] Snake venom poisoning has rapid onset and rapid progression, and the success or failure of clinical treatment often depends on whether the venom species can be accurately determined and targeted treatment is started within the golden period. Traditional diagnosis methods relying on expert experience are difficult to popularize in grassroots or field scenarios, while deep learning technology emerging in recent years has shown outstanding performance in image recognition, text analysis and other tasks, but has encountered double constraints in the field of snake venom. First, real snake bite cases are rare, and the distribution of venomous snakes in different regions is extremely unbalanced, plus professional annotation requires the involvement of a large number of snake venom experts, resulting in limited sample size for training; second, the poisoning process is reflected in the local lesions of the wound image, the symptom keywords in the patient's complaint text, and the abnormal values of the blood biochemical indicators, and any single modality cannot fully depict the evolution of the disease, so traditional single-modality models often misdiagnose or miss diagnosis. More seriously, under the condition of sample scarcity and high noise, it is difficult for conventional deep networks to complete cross-modal alignment and reliable fusion, and the model is prone to overfitting or modal bias, ultimately resulting in insufficient diagnostic accuracy, weak generalization ability, and high reasoning delay in real clinical environments. SUMMARY

[0003] The present application provides a snake venom poisoning intelligent diagnosis method, device, equipment and medium, which can realize high-precision, rapid and intelligent diagnosis of snake venom poisoning.

[0004] The present application provides a snake venom poisoning intelligent diagnosis method, which comprises: Obtaining multi-modal data, the multi-modal data comprising a wound image, a symptom text, and blood parameters; Respectively performing feature extraction processing on the wound image, the symptom text, and the blood parameters to obtain image features corresponding to the wound image, semantic features corresponding to the symptom text, and high-dimensional features corresponding to the blood parameters; Aligning and fusing the image features, the semantic features, and the high-dimensional features through a cross-modal attention mechanism to generate fusion features; After inputting the fusion features into a pre-set multi-task classification model, classifying the fusion features in a snake species label space and a toxicity category space respectively, and determining a poisoning grade score; Based on the venomous snake species, the toxicity category, and the poisoning grade score obtained by classification, a diagnosis report is generated.

[0005] Optionally, the feature extraction processing of the wound image comprises: dividing the wound image into a plurality of image blocks, and generating a spatial position identifier corresponding to each of the image blocks; mapping all the image blocks carrying the spatial position identifier into an image block embedding sequence through a linear transformation layer; inputting the image block embedding sequence into a multi-layer encoder, and capturing the pathological features of each image block layer by layer using a self-attention mechanism; normalizing and aggregating all the pathological features to generate an image feature corresponding to the wound image.

[0006] Optionally, the aligning and fusing processing of the image feature, the semantic feature, and the high-dimensional feature through the cross-modal attention mechanism to generate a fusion feature comprises: projecting the image feature, the semantic feature, and the high-dimensional feature into a vector space of the same dimension; determining the complementary semantic weight between the image feature, the semantic feature, and the high-dimensional feature in the vector space through a multi-head attention mechanism; inputting the image feature, the semantic feature, and the high-dimensional feature into a preset cross-modal learning model, and dynamically weighting and fusing according to the complementary semantic weight to generate the fusion feature.

[0007] Optionally, the intelligent snake venom poisoning diagnosis method further comprises training the multi-task classification model, comprising: constructing a weighted cross-entropy loss function, assigning loss weights to target snake species to improve the recognition sensitivity of the multi-task classification model to the target snake species; based on a preset augmented data set, making the multi-task classification model perform iterative learning of cross-modal diagnosis features under the guidance of the weighted cross-entropy loss function, and updating the network parameters of the multi-task classification model using adaptive weight optimization model; when the diagnosis accuracy of the multi-task classification model stops improving, terminating the model training, and taking the current multi-task classification model after network parameter updating as the optimal multi-task classification model.

[0008] Optionally, before generating the fusion feature, the intelligent snake venom poisoning diagnosis method further comprises: mapping the image feature, the semantic feature, and the high-dimensional feature into graph nodes, respectively; determining the semantic similarity between any two graph nodes based on an attention mechanism; based on the semantic similarity, aggregating the complementary information between adjacent graph nodes to enable each graph node to fuse multi-modal semantic features; Based on the semantic features of each of the graph nodes, a semantic mapping relationship between the image features, the semantic features and the high-dimensional features is constructed.

[0009] Optionally, the intelligent snake venom poisoning diagnosis method further comprises: The diffusion model is used to synthesize supplementary samples to perform data expansion processing on the target modality data in the multi-modal data. Based on the expanded multi-modal data, a multi-modal enhanced training set and a test set are generated. According to the multi-modal samples extracted from the training set, a training task set is constructed, and the multi-task classification model is trained using the training task set. According to the samples extracted from the test set, a test task set is constructed, and the trained multi-task classification model is tested using the test task set.

[0010] Optionally, the intelligent snake venom poisoning diagnosis method further comprises: Controllable Gaussian noise is added to the wound image, and enhanced wound images simulating different wound stages are generated by inverse denoising; Based on the symptom keyword template, diversified clinical complaint enhanced symptom texts are synthesized; The blood parameters are subjected to numerical perturbation in accordance with the toxicological feature distribution to generate enhanced blood parameters consistent with the pathology; The enhanced wound images, the enhanced symptom texts and the enhanced blood parameters are subjected to quality verification, and after passing the quality verification, they are used as training sample data of the multi-task classification model.

[0011] The application also provides an intelligent snake venom poisoning diagnosis device, which comprises: The acquisition module is used to acquire multi-modal data, including wound images, symptom texts and blood parameters; The extraction module is used to perform feature extraction processing on the wound images, the symptom texts and the blood parameters respectively to obtain image features corresponding to the wound images, semantic features corresponding to the symptom texts and high-dimensional features corresponding to the blood parameters; The fusion module is used to align and fuse the image features, the semantic features and the high-dimensional features through a cross-modal attention mechanism to generate fusion features; The classification module is used to input the fusion features into a preset multi-task classification model, and classify the fusion features in a snake species label space and a toxicity category space respectively, and determine a poisoning grade score; The generation module is used to generate a diagnosis report based on the venomous snake species, the toxicity category and the poisoning grade score obtained by classification.

[0012] The present invention also provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the intelligent diagnosis method for snake venom poisoning as described in any of the preceding claims.

[0013] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the intelligent diagnosis method for snake venom poisoning as described in any of the preceding claims.

[0014] The present invention has at least the following beneficial effects: First, multimodal data, including wound images, symptom text, and blood parameters, are acquired, covering both intuitive and physiological information about poisoning. Second, feature extraction is performed on each modality of data, yielding image features, semantic features, and high-dimensional features to accurately extract key information. Next, a cross-modal attention mechanism is used to align and fuse the multimodal features, generating fused features that fully leverage the advantages of multimodal data and compensate for the shortcomings of single-modal information. Finally, the fused features are input into a multi-task classification model, simultaneously classifying the snake species in a label space and a toxicity category space, determining the poisoning level score, and generating a diagnostic report based on the multi-dimensional classification and scoring results. The entire process fully utilizes the complementarity of multimodal data and leverages the efficient processing capabilities of deep learning models to achieve rapid and accurate diagnosis of snake poisoning, providing strong support for clinical treatment. Attached Figure Description

[0015] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0016] Figure 1 This is a flowchart illustrating the steps of an intelligent diagnostic method for snake venom poisoning.

[0017] Figure 2 This is a schematic diagram illustrating the principle of an intelligent diagnostic method for snake venom poisoning that uses image features to identify snake species. Figure 3 This is a flowchart illustrating the steps involved in feature extraction from wound images in an intelligent diagnostic method for snake venom poisoning. Figure 4 This is a flowchart of step S103 in an intelligent diagnostic method for snake venom poisoning. Figure 5 This is a flowchart illustrating the steps involved in training a multi-task classification model in an intelligent diagnostic method for snake venom poisoning. Figure 6 This is a flowchart illustrating the steps involved in generating semantic mapping relationships in an intelligent diagnostic method for snake venom poisoning. Figure 7 It is a step flow chart of multi-modal data processing in a snake venom poisoning intelligent diagnosis method. Figure 8 It is a step flow chart of wound image preprocessing in a snake venom poisoning intelligent diagnosis method. Figure 9 It is a structural schematic diagram of a snake venom poisoning intelligent diagnosis device. Figure 10 It is a structural schematic diagram of an electronic device. DETAILED DESCRIPTION

[0018] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0019] Currently, the diagnosis of snake venom poisoning patients mainly relies on artificial experience. Doctors need to judge the venomous snake species and toxicity type by comprehensively analyzing the bite site image, whole body symptoms, physiological indicators and previous case knowledge. However, the snake species distribution varies greatly in different regions, the personal experience of doctors is limited, and the venomous snake database is not complete, which leads to the problems of strong subjectivity, low accuracy and slow response in diagnosis.

[0020] The technical solution provides a snake venom poisoning intelligent diagnosis method and system based on a multi-modal large model, which can utilize wound images, symptom texts and blood tests and other multi-source heterogeneous data under the condition of scarce snake bite cases and complex clinical environment, and effectively improve the diagnosis accuracy and generalization ability of the model in real medical scenarios through diffusion model generative enhancement, cross-modal feature alignment and semantic fusion mechanism, solve the problems of data scarcity, strong model dependency and slow reasoning response in the prior art, and at least provide an intelligent diagnosis path with better robustness and clinical practical value. The following is each embodiment of the technical solution.

[0021] Please refer to Figure 1 , Figure 1 It is a step flow chart of a snake venom poisoning intelligent diagnosis method.

[0022] The embodiment provides a snake venom poisoning intelligent diagnosis method, which comprises the following steps: S101, acquiring multi-modal data, wherein the multi-modal data comprises a wound image, a symptom text and blood parameters.

[0023] S102, respectively performing feature extraction processing on the wound image, the symptom text and the blood parameters to obtain image features corresponding to the wound image, semantic features corresponding to the symptom text and high-dimensional features corresponding to the blood parameters.

[0024] S103, align and fuse the image features, semantic features, and high-dimensional features through a cross-modal attention mechanism to generate fusion features.

[0025] S104, after inputting the fusion features into a preset multi-task classification model, classify the fusion features in a snake species label space and a toxicity category space respectively, and determine a poisoning grade score.

[0026] S105, based on the venomous snake species, toxicity category, and poisoning grade score obtained by classification, generate a diagnosis report.

[0027] In this embodiment, new multi-modal case data is input into the multi-task classification model, and the multi-modal data includes three types of modal data, namely, wound images of patients, symptom text of clinical complaints, and blood parameters detected according to detection indicators, wherein the wound images are processed by ViT and denoted as The symptom text is extracted by a pre-trained language model BERT to obtain semantic features, denoted as The blood parameters such as red blood cell count, clotting time, and serum sodium concentration are input into a multi-layer perception network (MLP) to extract high-dimensional representations .

[0028] Then, a cross-modal attention mechanism (Cross-Modal Attention) is introduced to align and fuse the three types of modal encoding results, and the output of the fusion module is defined as The calculation formula is as follows:

[0029] Wherein, represents a feature concatenation operation, and CrossAttn represents a cross-modal multi-head attention operation (Multi-head Attention). This mechanism enables the model to automatically learn the key corresponding relationship between text and image, value, such as the association between “visual blurring” and “pupil abnormal image region”, and the cross-prompting between “thrombocytopenia” and “subcutaneous hemorrhage visible image”.

[0030] The above fusion representation is further input into a classifier to complete tasks such as venomous snake species identification, toxicity category judgment, or poisoning grade scoring, denoted as:

[0031] Finally, the above encoder and fusion module can quickly complete the diagnosis and prediction, and output information including venomous snake species, poisoning type, recommended serum type, etc., to provide real-time auxiliary decision-making basis for doctors or on-site first aid personnel.

[0032] It can be understood that, first, the multi-modal data such as wound image, symptom text, blood parameters, etc. are acquired, covering intuitive and physiological information of poisoning. Second, feature extraction is performed on each modal data to obtain image features, semantic features and high-dimensional features respectively, and key information is accurately extracted. Then, the multi-modal features are aligned and fused by using the cross-modal attention mechanism to generate fused features, so as to give full play to the advantages of multi-modal data and make up for the shortcomings of single modal information. Finally, the fused features are input into a multi-task classification model, and classification is performed in the snake species label space and the toxicity category space, and the poisoning grade score is determined, and a diagnosis report is generated based on the multi-dimensional classification and scoring results. The whole process makes full use of the complementarity of multi-modal data and the efficient processing capability of deep learning model, realizes the rapid and accurate diagnosis of snake venom poisoning, and provides strong support for clinical treatment.

[0033] In some embodiments, in order to acquire multi-modal data, a multi-modal acquisition platform integrating a visible light camera, a portable blood test equipment and a clinical symptom input terminal is built to realize synchronous acquisition of wound images, blood test indexes and symptom texts, complete data format standardization and structured storage.

[0034] Specifically, the acquisition equipment includes a visible light camera, an infrared thermal imager, a biochemical analyzer and a vital sign monitor, which are used to acquire multi-source data such as wound images, thermal imaging information, blood biochemical indexes and vital signs, respectively. The above acquisition equipment can be connected through a unified data interface to ensure that data of different modalities can be synchronously acquired within the same time window. The acquired data is first input to the multi-modal data layer in the original form, which contains image data, blood indexes, symptom texts and time series, etc. Then, these multi-modal data are transmitted to the data synchronization and standardization module to complete the unified format conversion, time alignment, missing value completion and other preprocessing operations. Finally, the standardized data are stored in the multi-modal database to provide structured support for subsequent data processing and model training.

[0035] In some embodiments, the way of performing feature extraction processing on the wound image includes: The wound image is divided into a plurality of image blocks, and a spatial position identifier corresponding to each image block is generated; all image blocks carrying the spatial position identifier are mapped into an image block embedding sequence through a linear transformation layer; the image block embedding sequence is input into a multi-layer encoder, and the self-attention mechanism is used to capture the pathological features of each image block layer by layer; and the all pathological features are normalized and aggregated to generate image features corresponding to the wound image.

[0036] Please refer to Figure 2 , 3The embodiment adopts the network structure of ViT, which includes an image block module, an encoding module, a Transformer encoding module, and a classification module. First, the input dent image is segmented and position encoded, then projected to the embedding space through linear transformation, and then input into the multi-layer Transformer encoder for feature extraction. Finally, the extracted features are classified and predicted by a multi-layer perceptron, and the target category is output. This structure can effectively extract global image features and is suitable for pathological feature extraction and discriminant analysis of snake bite images.

[0037] The structure is composed of an embedding layer, a normalization processing layer, a multi-channel attention mechanism layer, and a nonlinear fully connected layer, which can effectively represent deep features of image blocks, semantic text vectors and other inputs, thereby enhancing the discriminant ability of downstream diagnostic tasks. M encoding units can be stacked to improve feature expression capability, while residual connection ensures the stability of feature transmission.

[0038] It can be understood that the newly added wound image feature extraction means further improves the diagnostic accuracy based on the original technical solution. By dividing the wound image into image blocks and assigning spatial position identifiers, then mapping them into embedding sequences through a linear transformation layer, pathological features are extracted layer by layer using a multi-layer encoder combined with a self-attention mechanism, and finally normalized and aggregated to generate image features. This process not only preserves the local details and spatial information of the wound, but also accurately captures key pathological features. Combined with multi-modal data fusion and subsequent classification models, it can more accurately identify subtle differences in snake venom poisoning, further improving the high precision and reliability of diagnosis, and providing stronger technical support for fast and intelligent diagnosis.

[0039] Please refer to Figure 4 , Figure 4 A step flowchart of step S103 in the snake venom poisoning intelligent diagnosis method.

[0040] In some embodiments, step S103 includes: S201, projecting the image features, semantic features, and high-dimensional features into a vector space of the same dimension.

[0041] S202, in the vector space, determining the complementary semantic weights between the image features, semantic features, and high-dimensional features through a multi-head attention mechanism.

[0042] S203, inputting the image features, semantic features, and high-dimensional features into a pre-set cross-modal learning model, and dynamically weighting and fusing according to the complementary semantic weights to generate fused features.

[0043] In some embodiments, the calculation of the complementary semantic weight between the image features, semantic features and high-dimensional features is realized through a cross-modality multi-head attention mechanism module (CM-MHA). The specific use of the multi-head attention mechanism is as follows: using the multi-head attention mechanism in the Transformer, the feature interaction expression is constructed between different modalities (such as the image modality and the text modality, the text modality and the numerical modality). The formula is as follows:

[0044] wherein Q, K, V represent query, key, value matrices respectively, denotes the dimension of the key vector.

[0045] After projecting the image encoder output, the text encoder output and the numerical feature vector to the same dimension, the multi-modal features Q, K, V are constructed respectively, and the cross-attention is used to capture the complementary semantic correlation between modalities. The greater the obtained weight is, the higher the contribution of the modality feature to the target prediction is.

[0046] This module supports any two combinations of the image, text and numerical modalities, which are used to construct the feature fusion result in the unified semantic space.

[0047] Specifically, the dynamic weighted fusion needs to integrate the above-mentioned cross-modality attention mechanism into the multi-modal diagnostic model backbone. The backbone network structure includes ViT, medical BERT, structured numerical MLP channel compressor, unified fusion layer and decision output layer. On this basis, a sparse activation mechanism is designed to automatically suppress the parameter paths irrelevant to the target task, thereby realizing lightweight diagnostic modeling.

[0048] It can be understood that the newly added technical means further optimizes the multi-modal feature fusion process and improves the diagnostic accuracy and reliability. First, the image features, semantic features and high-dimensional features are projected to the same dimensional vector space, which facilitates subsequent fusion operations. Then, the multi-head attention mechanism is used to dynamically determine the complementary semantic weight between the features, so as to accurately capture important information of different modal data. Finally, the dynamic weighted fusion is performed according to the complementary semantic weight to generate more representative fusion features. This improvement makes the feature fusion more targeted and flexible, which can more fully play the complementary advantages of multi-modal data, so as to more accurately identify the snake venom poisoning features and further improve the high precision and rapidity of diagnosis, thereby providing stronger support for clinical treatment.

[0049] In some embodiments, the intelligent snake venom poisoning diagnosis method further comprises: building an initial model containing a cross-modality attention mechanism.

[0050] The initial model is pre-trained using a general medical dataset.

[0051] The network parameters of the initial model are updated through alternating optimization of the support set and the query set in the snake bite field task set to fine-tune the pre-trained initial model, and the fine-tuned initial model is used as the cross-modal learning model.

[0052] In this embodiment, a two-stage fine-tuning training mechanism is constructed, including a pre-training stage and a domain adaptation fine-tuning stage: In the pre-training stage, a general multi-modal representation learning is performed on the model using a large-scale public medical dataset and a self-built snake venom multi-modal corpus to learn initial parameters In the fine-tuning stage, a snake bite field specific task set (such as venom identification and toxicity grade prediction) is introduced, and the task set is defined For each task (where is the support set, is the query set), the following optimization is performed:

[0053]

[0054] where , represent the loss of task on the support set and the query set, respectively, , are the inner and outer learning rates, respectively. Through this training process, the model gradually improves the diagnostic generalization ability and cross-modal alignment ability under small sample conditions.

[0055] Please refer to Figure 5 , Figure 5 is a step flowchart for training a multi-task classification model in a snake venom poisoning intelligent diagnosis method.

[0056] In some embodiments, a snake venom poisoning intelligent diagnosis method further comprises the step of training a multi-task classification model.

[0057] The way of training a multi-task classification model includes: S301, a weighted cross-entropy loss function is constructed, and loss weights are assigned to target venomous snake categories to improve the recognition sensitivity of the multi-task classification model to target venomous snake categories.

[0058] S302, based on the pre-set augmented dataset, the multi-task classification model is guided by the weighted cross-entropy loss function to iteratively learn the model training of cross-modal diagnostic features, and the adaptive weight is used to optimize the network parameters of the model to update the multi-task classification model.

[0059] S303、When the diagnostic accuracy of the multi-task classification model stops improving, terminate the model training, and take the current multi-task classification model after updating the network parameters as the optimal multi-task classification model.

[0060] In this embodiment, the multi-task classification model training stage adopts a weighted cross entropy loss function for optimization to deal with the situation of uneven distribution of different venomous snake sample quantities. The loss function expression is:

[0061] wherein, is the total number of venomous snake species, is the real label, is the model prediction probability, is the class weighting coefficient. During the training process, the enhanced data set is used to load the support set and query set samples round by round, multiple forward propagation and backward propagation are performed, the network parameters are iteratively updated by the AdamW optimizer, and the Early Stopping strategy is used to control the training convergence.

[0062] It can be understood that by constructing the weighted cross entropy loss function, the loss weight is assigned to the target venomous snake class, which enhances the recognition sensitivity of the model to the target venomous snake class. Based on the enhanced data set, the model iteratively learns under the guidance of the weighted cross entropy loss function, optimizes the network parameters with adaptive weights, and further improves the learning ability of the model to the cross-modal diagnostic features. When the diagnostic accuracy stops improving, the training is terminated to ensure that the optimal model is obtained. This improvement makes the multi-task classification model more accurate in identifying the key features of snake venom poisoning, further improves the high precision and reliability of the diagnosis, and provides stronger technical support for fast and intelligent diagnosis.

[0063] Please refer to Figure 6 , Figure 6 is a step flowchart for generating semantic mapping relationship in the intelligent snake venom poisoning diagnosis method.

[0064] In some embodiments, an intelligent snake venom poisoning diagnosis method further comprises, before generating the fusion features: S401, mapping the image features, semantic features and high-dimensional features into graph nodes respectively.

[0065] S402, determining the semantic similarity between any two graph nodes based on the attention mechanism.

[0066] S403, based on the semantic similarity, aggregating the complementary information between adjacent graph nodes to make each graph node fuse the multi-modal semantic features.

[0067] S404, based on the semantic features of each graph node, constructing a semantic mapping relationship between the image features, semantic features and high-dimensional features.

[0068] In this embodiment, in order to further enhance the collaborative expression ability among the three modalities of images, texts and numerical values, the application introduces a modality interaction graph modeling mechanism (Modality Interaction Graph) on the basis of the encoder output, to explicitly construct the semantic dependency relationship among the modalities. Specifically, a graph structure is defined as , wherein the nodes respectively represent the embedding representations of images, texts and blood tests, and the edge set represents the similarity or interaction strength between modalities, which is automatically generated by attention scores. To learn the high-order association between modalities, a graph attention network (GAT) is introduced in the graph structure for updating, so that each modality representation can fuse the complementary information from other modalities, thereby forming a joint semantic representation of the modality after enhancement .

[0069] For example, in a typical snake venom poisoning case, the system first captures key visual features in the wound image through the ViT encoder, such as "large area of skin surface necrosis, local blackening, blurred edge", etc., which form high-confidence image modality embeddings after encoding. At the same time, the text modality BERT encoder extracts symptom keywords such as "purpura", "oozing", "blackening" from the clinical complaint description, which are converted into high-dimensional semantic representations. Combined with the blood test indicators processed by the MLP, such as "significant decrease in platelet count", "decrease in hematocrit", "prolonged clotting time", etc., physiological abnormalities form measurable toxicity reaction quantitative indicators in the numerical modality.

[0070] Through the modality interaction graph mechanism, the system constructs the semantic mapping path between the three types of information of images, texts and numerical values, and models their association strength with the help of GAT. For example, when "necrotic image area" and "purpura" keywords and "platelet reduction indicators" and other statements frequently co-occur in multiple training samples, the model will assign higher edge weights to the three features in the graph structure, so that they have stronger diagnostic directionality in the joint representation.

[0071] Finally, the collaborative relationship among the three modalities is fused into a unified diagnosis and input to the classifier for venomous snake species identification, toxicity category judgment and poisoning level scoring. This kind of joint modeling method based on multi-modality consistency enhancement not only improves the sensitivity of the model to complex poisoning symptom combinations, but also enhances the generalization ability and explainable reasoning ability of the system to atypical cases.

[0072] Please refer to Figure 7 ,Figure 7 is a step flow chart of multi-modal data processing in a snake venom poisoning intelligent diagnosis method.

[0073] In some embodiments, a snake venom poisoning intelligent diagnosis method further comprises: S501, synthesizing supplementary samples by using a diffusion model to perform data augmentation processing on target modal data in multi-modal data.

[0074] S502, generating a multi-modal enhanced training set and a test set based on the augmented multi-modal data.

[0075] S503, constructing a training task set according to multiple groups of multi-modal samples extracted from the training set, and training a multi-task classification model by using the training task set.

[0076] S504, constructing a test task set according to samples extracted from the test set, and testing the trained multi-task classification model by using the test task set.

[0077] It can be understood that the target modal data is augmented by synthesizing supplementary samples by using a diffusion model, and a multi-modal enhanced training set and a test set are generated, which provides a richer and more diverse data basis for model training. Based on the enhanced data set, a training task set and a test task set are constructed, which are respectively used for training and testing of the multi-task classification model, to ensure the generalization ability and stability of the model on multi-modal data. This improvement effectively improves the recognition accuracy and adaptability of the model to different poisoning conditions, further enhancing the high precision and rapidity of snake venom poisoning intelligent diagnosis.

[0078] In one specific embodiment, multiple groups of multi-modal samples are randomly extracted from the enhanced training set to construct a training task set, and the task is represented as

[0079] wherein, is a wound image, is a symptom text, is a blood test index, is a corresponding label, including snake species, poisoning grade, etc., and each task is divided into a support set (Support Set) and a query set (QuerySet) for fusion training of the model backbone and the modal encoder.

[0080] Sample data is randomly extracted from the enhanced test set and divided into a test support set and a test query set, and modal alignment and reasoning evaluation in a unified semantic space are performed to verify the diagnostic accuracy and generalization ability of the model in unseen data scenarios.

[0081] It can be understood that the enhanced training set and test in the embodiment are generated by using the diffusion model to perform generative enhancement on the key modal, expand the sample size of poisoning diagnosis under the condition of few samples, and thus obtain the multi-modal enhanced training set and test set.

[0082] In some embodiments, a snake venom poisoning intelligent diagnosis method further includes the step of enhancing the multi-modal data.

[0083] Specifically, the way of enhancing the multi-modal data includes: Controllable Gaussian noise is added to the wound image, and enhanced wound images simulating different wound stages are generated by reverse denoising; diversified clinical complaints are synthesized based on symptom keyword templates; blood parameters are disturbed according to toxicological characteristic distribution to generate enhanced blood parameters consistent with pathology; enhanced wound images, enhanced symptom text and enhanced blood parameters are quality checked, and after passing the quality check, they are used as training sample data of the multi-task classification model.

[0084] Please refer to Figure 8 , Figure 8 is a flow chart of the wound image preprocessing step in a snake venom poisoning intelligent diagnosis method.

[0085] To alleviate the problems of insufficient snake venom poisoning samples and uneven distribution of venom species, a multi-modal pseudo sample generation strategy based on a diffusion model is proposed, and the specific implementation is as follows: Controllable Gaussian noise is added to the wound image, and enhanced wound images simulating different wound stages are generated by reverse denoising; diversified clinical complaints are synthesized based on symptom keyword templates; blood parameters are disturbed according to toxicological characteristic distribution to generate enhanced blood parameters consistent with pathology; enhanced wound images, enhanced symptom text and enhanced blood parameters are quality checked, and after passing the quality check, they are used as training sample data of the multi-task classification model.

[0086] It can be understood that in addition to enhancing the multi-modal data, the multi-modal data can also be preprocessed in the following way: Data cleaning and normalization: For original image data, automatic cropping, brightness correction, size unification and other methods are used to remove irrelevant background areas, ensuring that the wound area information is complete and clear; text data uses word segmentation and stop word filtering algorithm to extract symptom keywords and standardize the encoding; blood test data uses Z-score standardization and unit unification as processing means to eliminate measurement bias between devices.

[0087] Missing data completion: For samples that are partially missing due to the influence of poisoning first aid environment, K-nearest neighbor average or trend interpolation based methods are used to complete the missing index filling, maximizing the retention of sample structural integrity.

[0088] Labeling and Structured Storage: Align the three types of modal data according to patient number and timestamp, unify the field names and fill in the database according to the structured template. At the same time, according to the doctor's diagnosis record and symptom characteristics, some data are labeled with auxiliary labels such as "suspected silver ring snake" and "suspected neurotoxic effect", so as to call in the subsequent data enhancement stage.

[0089] In one specific embodiment, a data processing program is set to control the operation of the snake venom poisoning intelligent diagnosis system, and wound images, clinical symptom text descriptions and blood test indicators of poisoning cases of multiple snake species (such as cobra, silver ring snake and five-step snake) are collected, respectively, to obtain n groups of image data, m groups of text data and p groups of numerical data.

[0090] The image, text and numerical data collected above are uniformly preprocessed, including format standardization, missing value filling, redundant field removal, text segmentation, image normalization, index standardization and other operations, and a diffusion model is used for generative enhancement of key modalities to expand the poisoning diagnosis sample size under the condition of few samples, to obtain a multi-modal enhanced training set and test set.

[0091] The technical solution also provides a specific embodiment of a snake venom poisoning intelligent diagnosis method implemented in an actual scene.

[0092] Firstly, a snake venom poisoning multi-modal data collection platform is built, including a wound image collection module, a blood test analysis module and a clinical symptom input module, and preliminary data format standardization processing is completed.

[0093] Secondly, a data processing program is set to preprocess the collected images, texts and structured data, including denoising, standardization, missing value completion and other operations, and based on a diffusion model for generative enhancement, the diversity of training samples in key diagnosis scenarios is expanded.

[0094] Then, a unified multi-modal feature extraction module that fuses modal-specific encoders (such as visual encoder ViT, text encoder BERT and numerical MLP) is constructed, and cross-modal attention mechanism is used to complete feature alignment; then, through a multi-level fine-tuning strategy of sparse activation mechanism, a multi-modal diagnosis large language model in a unified semantic space is trained to realize snake species recognition and poisoning grade prediction and other tasks.

[0095] Finally, a snake venom poisoning intelligent diagnosis system based on multi-modal semantic fusion and instruction-driven reasoning mechanism is built, which supports comprehensive input of images, texts and numerical values, and automatically outputs diagnosis suggestions and emergency treatment plans.

[0096] Specifically, to achieve efficient processing and intelligent inference output of multi-modal input data, the system platform builds an intelligent diagnosis system integrating diagnosis engine, data access module and user interface. The platform not only has local real-time diagnosis capability, but also takes into account the actual needs of remote deployment and edge inference. The overall structure is lightweight, fast in response, and highly adaptable, which can meet the needs of snake venom poisoning auxiliary diagnosis in multiple scenarios.

[0097] In terms of data access, the system platform supports multiple input forms, including static images, video frame screenshots, doctor-entered symptom text descriptions, and structured numerical data obtained through blood analyzers or other vital sign monitoring devices. All modal data is uniformly parsed, formatted and automatically checked after input, ensuring consistent data structure and clear fields, which is conducive to subsequent coding and fusion processing. The platform internally establishes a standardized data access protocol, describing various modal fields through JSON data structure, ensuring accurate and efficient information transmission during model invocation.

[0098] The core diagnosis part deploys the multi-modal large language model VenomDiag-LLM proposed in this application, which integrates image encoder, text semantic analysis module and numerical channel compressor, and combines cross-modal attention mechanism to complete semantic alignment and feature fusion between modalities. After receiving user input, the system will sequentially extract features of images, text and numerical values, and generate diagnosis representation through a unified fusion module, and further pass it to the multi-task decision layer to output the diagnosis result. The response time of the inference process is controlled within seconds, and the model version switching and inference logic fine-tuning are supported, which adapts to the diagnosis needs in different application environments.

[0099] The user interface part uses Qt-based graphical interaction design, with a simple and clear overall layout. The interface includes data upload and configuration module, model running and result display module, and report export module. Users can complete patient information input, image and data loading, model selection, diagnosis execution and report generation through the interface. The inference result will be presented in a combination of text and images, including the identified snake species, the predicted poisoning level and the corresponding serum recommendation scheme. The diagnosis result can be saved as a standard format PDF file for printing and archiving in primary hospitals or emergency sites.

[0100] Please refer to Figure 9 , Figure 9 is a structural schematic diagram of a snake venom poisoning intelligent diagnosis device.

[0101] The embodiment also provides a snake venom poisoning intelligent diagnosis device, which comprises: The acquisition module 601 is configured to acquire multi-modal data, and the multi-modal data comprises a wound image, a symptom text and blood parameters.

[0102] The extraction module 602 is configured to perform feature extraction on the wound image, the symptom text, and the blood parameter respectively to obtain image features corresponding to the wound image, semantic features corresponding to the symptom text, and high-dimensional features corresponding to the blood parameter.

[0103] The fusion module 603 is configured to perform alignment and fusion on the image features, the semantic features, and the high-dimensional features through a cross-modal attention mechanism to generate fusion features.

[0104] The classification module 604 is configured to input the fusion features into a preset multi-task classification model, and perform classification in a snake species label space and a toxicity category space according to the fusion features, and determine a poisoning grade score.

[0105] The generation module 605 is configured to generate a diagnosis report based on the venomous snake species, the toxicity category, and the poisoning grade score obtained through the classification.

[0106] Those skilled in the art can understand that all or some of the steps in the method disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. As known to those skilled in the art, communication media typically includes computer readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transport mechanisms, and can include any information delivery medium.

[0107] It can be understood that the contents in the above method embodiments are applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0108] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the intelligent snake venom poisoning diagnosis method according to any one of the above embodiments when executing the computer program.

[0109] Reference Figure 10 , Figure 10 The hardware structure of the electronic device of another embodiment is illustrated, which comprises: The processor 701 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to implement the technical solutions provided by the embodiments of the present application; The memory 702 can be implemented in the form of a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 702 can store operation devices and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 702 and are called and executed by the processor 701 to implement the intelligent snake venom poisoning diagnosis method of the embodiments of the present application; The input / output interface 703 is used to realize information input and output; The communication interface 704 is used to realize the communication interaction between the device and other devices, and can realize communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.); The bus 705 transmits information between various components (for example, the processor 701, the memory 702, the input / output interface 703, and the communication interface 704) of the device; The processor 701, the memory 702, the input / output interface 703, and the communication interface 704 are connected to each other through the bus 705 to realize communication connection between them in the device.

[0110] It can be understood that the contents in the above method embodiments are all applicable to the electronic device embodiments, the electronic device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0111] The embodiment of the present application further provides a computer readable storage medium, wherein a program executable by a processor is stored, and the program executable by the processor is used for implementing the intelligent diagnosis method for snake venom poisoning when executed by the processor.

[0112] The embodiment of the present application further discloses a computer program product, comprising a computer program or computer instructions, the computer program or computer instructions being stored in a computer readable storage medium, a processor of a computer device reading the computer program or computer instructions from the computer readable storage medium, and the processor executing the computer program or computer instructions, so that the computer device executes the intelligent diagnosis method for snake venom poisoning as described in any of the foregoing embodiments.

[0113] It can be understood that the contents in the method embodiments are applicable to the storage medium embodiments, the storage medium embodiments specifically implement the functions of the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0114] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so termed is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of accomplishing functionalities illustrated or claimed in any other embodiments elements. Furthermore, the terms "comprise", "have" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, article, or apparatus that comprises a list of steps or units can not necessarily be limited to those steps or units, but can include additional steps or units not expressly listed or inherent to such process, method, article, or apparatus. It should be understood that "at least one" in the present application means one or more, and "multiple" means two or more.

[0115] In several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another device, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed elements can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0117] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0118] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0119] Although the description of the present application has been quite detailed and particularly described with respect to several embodiments, it is not intended to be limited to any of these details or embodiments or any special embodiment, but should be considered to effectively cover the intended scope of the present application by referring to the appended claims, taking into account the broad interpretation of the prior art provided by the claims. In addition, the present application is described above in embodiments that the inventors can foresee, the purpose of which is to provide a useful description, and non-essential modifications to the present application that have not yet been foreseen can still represent equivalent modifications of the present application.

Claims

1. A smart diagnostic method for snake venom poisoning, characterized in that, The method includes: Acquire multimodal data, including wound images, symptom text, and blood parameters; Feature extraction processing is performed on the wound image, the symptom text, and the blood parameters respectively to obtain the image features corresponding to the wound image, the semantic features corresponding to the symptom text, and the high-dimensional features corresponding to the blood parameters; The image features, semantic features, and high-dimensional features are aligned and fused using a cross-modal attention mechanism to generate fused features. After inputting the fused features into a preset multi-task classification model, the snake species is classified in the snake species label space and the toxicity category space according to the fused features, and the poisoning level score is determined. A diagnostic report is generated based on the snake species, toxicity category, and poisoning severity score obtained from the classification.

2. The intelligent diagnostic method for snake venom poisoning according to claim 1, characterized in that, The methods for feature extraction processing of the wound image include: The wound image is divided into multiple image blocks, and a spatial location identifier is generated for each image block. A linear transformation layer maps all image blocks carrying spatial location identifiers into an image block embedding sequence. The image patch embedding sequence is input into a multi-layer encoder, and the pathological features of each image patch are captured layer by layer using a self-attention mechanism; All the pathological features are normalized and aggregated to generate image features corresponding to the wound image.

3. The intelligent diagnostic method for snake venom poisoning according to claim 1, characterized in that, The step of aligning and fusing the image features, semantic features, and high-dimensional features through a cross-modal attention mechanism to generate fused features includes: Project the image features, the semantic features, and the high-dimensional features into a vector space of the same dimension; Within the vector space, complementary semantic weights among the image features, the semantic features, and the high-dimensional features are determined through a multi-head attention mechanism. The image features, semantic features, and high-dimensional features are input into a preset cross-modal learning model, and dynamically weighted and fused according to the complementary semantic weights to generate the fused features.

4. The intelligent diagnostic method for snake venom poisoning according to claim 1, characterized in that, The method further includes training the multi-task classification model, including: A weighted cross-entropy loss function is constructed, and loss weights are assigned to the target venomous snake category to improve the recognition sensitivity of the multi-task classification model for the target venomous snake category; Based on the preset augmented dataset, the multi-task classification model is trained iteratively to learn cross-modal diagnostic features under the guidance of the weighted cross-entropy loss function, and the network parameters of the multi-task classification model are updated by an adaptive weight optimization model. When the diagnostic accuracy of the multi-task classification model stops improving, the model training is terminated, and the current multi-task classification model with updated network parameters is taken as the optimal multi-task classification model.

5. The intelligent diagnostic method for snake venom poisoning according to claim 1, characterized in that, Before generating the fusion feature, the method further includes: The image features, semantic features, and high-dimensional features are respectively mapped to graph nodes; Based on the attention mechanism, the semantic similarity between any two graph nodes is determined; Based on the semantic similarity, complementary information between adjacent graph nodes is aggregated so that each graph node integrates multimodal semantic features; Based on the semantic features of each of the graph nodes, a semantic mapping relationship is constructed between the image features, the semantic features, and the high-dimensional features.

6. The intelligent diagnostic method for snake venom poisoning according to claim 1, characterized in that, The method further includes: Supplementary samples are synthesized using a diffusion model to augment the target modality data in the multimodal data. Based on the augmented multimodal data, multimodal enhanced training and testing sets are generated; A training task set is constructed based on multiple sets of multimodal samples extracted from the training set, and the multi-task classification model is trained using the training task set. A test task set is constructed based on samples extracted from the test set, and the trained multi-task classification model is tested using the test task set.

7. The intelligent diagnostic method for snake venom poisoning according to claim 1, characterized in that, The method further includes: Controllable Gaussian noise is added to the wound image, and enhanced wound images simulating different stages of injury are generated through inverse denoising. Enhanced symptom text based on symptom keyword templates to synthesize diverse clinical complaints; The blood parameters are subjected to numerical perturbations that conform to the toxicological characteristic distribution, and pathologically consistent enhanced blood parameters are generated. The enhanced wound image, the enhanced symptom text, and the enhanced blood parameters are quality-verified, and the data that passes the quality verification are used as training sample data for the multi-task classification model.

8. A smart diagnostic device for snake venom poisoning, characterized in that, The device includes: The acquisition module is used to acquire multimodal data, including wound images, symptom text, and blood parameters. The extraction module is used to perform feature extraction processing on the wound image, the symptom text, and the blood parameters respectively to obtain the image features corresponding to the wound image, the semantic features corresponding to the symptom text, and the high-dimensional features corresponding to the blood parameters; The fusion module is used to align and fuse the image features, the semantic features, and the high-dimensional features through a cross-modal attention mechanism to generate fused features; The classification module is used to input the fused features into a preset multi-task classification model, and then classify the snakes according to the fused features in the snake species label space and the toxicity category space, and determine the poisoning level score. The generation module is used to generate a diagnostic report based on the venomous snake species, toxicity category, and poisoning level score obtained from the classification.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the intelligent diagnostic method for snake venom poisoning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the intelligent diagnosis method for snake venom poisoning as described in any one of claims 1 to 7.

Images