CIPN information fusion method and system based on multi-modal information

Abstract

The invention provides a CIPN information fusion method and system based on multi-modal information. The method comprises the following steps: evaluating a subjective questionnaire of a to-be-tested person; evaluating the walking speed and gait according to the walking video of the to-be-detected person; evaluating a finger fine movement skill of the to-be-tested person; evaluating the nerve conduction velocity of the to-be-tested person; the brain function of the to-be-tested person is evaluated; evaluating the brain structure of the to-be-tested person; and according to the subjective questionnaire, the walking speed and gait, the fine movement skills of the fingers, the nerve conduction speed, the brain function and the brain structure, information fusion is constructed. Compared with a traditional NCI-CTCAE grading system and an EORTC QLQ-CIPN20 questionnaire, the method has the advantages that the subjectivity and repeatability problems are solved, and the diagnosis accuracy and the treatment effect are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical assessment tools, and in particular to a CIPN information fusion method and system based on multimodal information. Background Art

[0002] Chemotherapeutic drug-induced peripheral neurotoxicity (CIPN) is one of the common dose-limiting non-hematological toxicities, manifested as symmetrical sensory and motor dysfunction in the distal bilateral limbs. Common symptoms include glove and stocking numbness, pain, tingling, burning sensation, and dysesthesia. Sensory nerve dysfunction is the most common, which may be accompanied by decreased motor ability and autonomic dysfunction. Traditional chemotherapy drugs such as platinum, taxanes, vinca alkaloids and protease inhibitors, as well as new ADC anti-tumor drugs can cause significant peripheral neurotoxicity. Severe CIPN not only affects the quality of life, but may also lead to treatment interruption, thereby affecting disease control.

[0003] CIPN is dose-accumulative, and timely dose reduction or drug discontinuation can relieve symptoms, but severe toxicity may persist for a long time. Studies have shown that about 20% of patients still have neurotoxicity after 18 months of using taxanes. Long-term toxicity not only reduces the quality of life, but also limits subsequent drug selection and affects the treatment effect. Therefore, timely evaluation and treatment of CIPN is crucial for the treatment of malignant tumors, especially for the goal of prolonging survival and improving the quality of life.

[0004] At present, CIPN assessment mainly relies on the NCI-CTCAE grading system, which assesses the severity of toxicity based on the patient's symptom description and the clinician's judgment, but has the problems of strong subjectivity and lack of objectivity. Although the EORTC QLQ-CIPN20 questionnaire solves the quantification problem, it still relies on the patient's self-report and cannot overcome the problems of subjectivity and repeatability. Electromyography and nerve conduction velocity are objective and quantifiable, but they can only provide local information, lack sensitivity, and are difficult to be used independently as grading tools. Therefore, the present invention designs a set of information collection systems that can objectively and quantitatively evaluate the patient's peripheral nerve and muscle function status, fine movements, gait and stride, and develops an artificial intelligence algorithm to integrate this information to form a comprehensive scoring system. This system will help clinicians to objectively and accurately evaluate the occurrence and development of CIPN and provide an important reference for intervention measures.

[0005] Patent document CN117373671A discloses a patient-reported assessment tool specifically for evaluating oxaliplatin-induced peripheral neuropathy, including a main scale, a subscale and a scoring formula. The main scale assessment content includes body organ symptoms, and the body organ symptoms are divided into levels according to the patient's subjective feelings about the frequency of their own neuropathy. The levels correspond to relative scores. The subscale includes the site of occurrence of the most serious symptoms, the duration of the most serious symptoms, and the impact of the most serious symptoms on life. This invention evaluates peripheral neuropathy in a targeted manner and integrates the features of multiple verified assessment scales. It effectively improves the accuracy and sensitivity of the peripheral neuropathy assessment results and can provide a reference for the efficacy evaluation of clinical intervention strategies. However, this invention does not take into account the correlation between multiple physiological indicators, has a strong subjectivity, and cannot meet the needs of objective evaluation of neurotoxicity. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a CIPN information fusion method and system based on multimodal information.

[0007] According to the present invention, a CIPN information fusion method based on multimodal information includes:

[0008] Step S1: Evaluate the subjective questionnaire of the person to be tested;

[0009] Step S2: evaluating walking speed and gait based on the walking video of the person to be tested;

[0010] Step S3: assessing the fine motor skills of the finger of the person to be tested;

[0011] Step S4: evaluating the nerve conduction velocity of the person to be tested;

[0012] Step S5: assessing the brain function of the person to be tested;

[0013] Step S6: evaluating the brain structure of the person to be tested;

[0014] Step S7: Information fusion is performed based on the subjective questionnaire, walking speed and gait, finger fine motor skills, nerve conduction velocity, brain function and brain structure.

[0015] Preferably, in step S1:

[0016] Step S1.1: Design a self-report scale that includes surveys on sensory symptoms, motor symptoms, autonomic symptoms, and impact on quality of life;

[0017] Step S1.2: Data collection and preprocessing:

[0018] Collect self-report questionnaire data and normalize all data;

[0019]

[0020] where x i is the i-th data in the self-report questionnaire data of the tested person, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data;

[0021] Step S1.3: Indicator weight setting:

[0022] Based on the opinions of clinical experts, the importance weight of each self-report questionnaire question was determined:

[0023]

[0024] Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the i-th indicator;

[0025] Step S1.3: Calculation of comprehensive score:

[0026] The weighted average method was used to calculate the comprehensive CIPN score, and the formula was as follows:

[0027]

[0028] Among them, w i is the importance weight of the i-th indicator, x i is the ith data in the CIPN20 questionnaire data, s1 is the score of the person being tested on this scale; M is the total number of indicators.

[0029] Preferably, in step S2:

[0030] Step S2.1: Video preprocessing

[0031] Extract K frames of images {T1,...,T K};

[0032] Step S2.2: Gait feature extraction

[0033] Use the correlation between consecutive frames to perform background subtraction, remove the static background, and retain the moving human body parts, specifically:

[0034] Step S2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image Diff between the two frames t :

[0035] Diff t =|T t -T t-1 |

[0036] Step S2.2.2: Use hard thresholding to transform the difference image Diff t Binarization, pixels greater than the threshold I are foreground:

[0037]

[0038] Among them, F t is the foreground image of frame t after removing the background;

[0039] Use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the foreground image video {(A1,B1),...,(A K ,B K )}; where (A K ,B K ) are the position coordinates of the left toe and the right toe in the Kth frame respectively;

[0040] Step S2.3: Gait cycle identification:

[0041] Calculate the distance between toe positions in two consecutive frames:

[0042]

[0043] Among them, D m is the distance between the toes of the mth frame and the m+1th frame;

[0044] Calculate the walking speed of the subject in the video:

[0045]

[0046] Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed.

[0047] Preferably, in step S3:

[0048] The user's fine motor skills can be assessed by randomly displaying click points on the phone screen and gradually shortening the time interval between the points, recording the user's reaction time. The specific steps are as follows:

[0049] Step S3.1: define the screen as an area with a width of C and a height of D;

[0050] Step S3.2: At each time interval t n At the end, a click point appears at a random screen location;

[0051] Step S3.3: Reaction time calculation:

[0052] (Δt) n =t click -t start

[0053] Among them, t start is the time when the click point appears, t click is the click time of the click point, (Δt) n is the nth reaction rate;

[0054] Step S3.4: After each click, shorten the time interval between dot appearances:

[0055] t n+1 =α×t n

[0056] Among them, α is the shortening coefficient, which is greater than 0 and less than 1; t n is the nth time interval;

[0057] Step S3.5: Calculate the average reaction time of the user and record the sequence of multiple reaction times

[0058]

[0059] Among them, s3 is the response time of the person to be tested, and n1 is the total number of time intervals.

[0060] Preferably, in step S4:

[0061] Nerve conduction velocity is calculated as:

[0062]

[0063] Among them, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end.

[0064] Preferably, in step S5:

[0065] Step S5.1: Data acquisition:

[0066] The N5-channel cerebral blood oxygen signal NIRS device was used to collect NIRS data on the frontal lobe function of N test subjects; each channel recorded the changes in the concentrations of oxyhemoglobin HbO and deoxyhemoglobin HbR. The number of data recorded in each channel of the test subject was N6, and the signal in the jth channel of the i-th test subject was:

[0067]

[0068] Among them, HbO(i) j is the oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, is the N6th oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR(i) j is the deoxyhemoglobin signal in the jth channel of the i-th person to be tested, is the N6th deoxyhemoglobin signal in the jth channel of the i-th person to be tested;

[0069] Step S5.2: Data denoising:

[0070] Use a bandpass filter to remove high-frequency noise and low-frequency drift in the signal to obtain the denoised signal, that is,

[0071] HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high )

[0072] HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high )

[0073] Among them, HbO_filtered(i) j is the filtered oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR_filtered(i) i is the filtered deoxyhemoglobin signal in the jth channel of the i-th person to be tested, BF is the bandpass filter, f low ,f high is the cut-off frequency;

[0074] Step S5.3: Brain function feature extraction

[0075] Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested:

[0076]

[0077] Among them, N5 is the number of data recorded by the channel;

[0078] Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested:

[0079]

[0080] Extract the total blood volume change s5(i) of the i-th person to be tested:

[0081] s5=ΔHbO(i)+ΔHbR(i)

[0082] Among them, mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of a signal.

[0083] Preferably, in step S6:

[0084] Step S6.1: Data collection:

[0085] Perform magnetic resonance imaging (MRI) on N persons to be tested and set scanning parameters;

[0086] Step S6.2: Structural feature extraction:

[0087] Use the automated tool FreeSurfer to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i ;

[0088] Step S6.3: Calculation of brain structural indicators:

[0089] s6=GMV i +CT i +CSA i .

[0090] Preferably, in step S7:

[0091] Step S7.1: Collect the evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested:

[0092] P={p1,...,p l …,p N},

[0093] Y={y1,...,y l …,y N},

[0094] Among them, pl ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6} T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, y l is the CIPN severity level of the lth person under test;

[0095] Step S7.2: Data standardization, the specific construction method is as follows:

[0096] Step S7.2.1: Divide the data of the N persons to be tested into a training set, a validation set and a test set according to a preset ratio;

[0097] Step S7.2.2: Normalize all the above data using the following formula:

[0098]

[0099] Among them, s ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested; s minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature;

[0100] Step S7.3: Use the Mamba model pair to build a multi-classification model:

[0101] Step S7.3.1: For each person i in the data set P, the feature vector p i ={sn i1 ,sn i2 ,sn i3 ,sn i4 ,sn i5 ,sn i6} T Embed it and convert it into a high-dimensional feature representation h i ;

[0102] -Use linear transformation to map the original normalized features to a high-dimensional embedding space:

[0103] h i =W e ·p i +b e

[0104] Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, with size d×1; d is the size of the embedding dimension;

[0105] Apply multi-head attention mechanism to capture each normalized feature s ij The relationship between them is enhanced, and the feature representation is enhanced;

[0106] For the embedded feature vector h i Compute the query Q, key K, and value V:

[0107] Q=W q ·h i ,K=W k ·h i ,V=W v ·h i

[0108] Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d;

[0109] Calculate the self-attention score Attention(Q,K,V)

[0110]

[0111] Where: d1 is the dimension of the key vector, used to scale the dot product result;

[0112] For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel:

[0113] MultiHead(Q,K,V)=Concat(head1,…,head h )·W o

[0114] Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix, size d×d; Step S7.3.2: Neural Atom Bag: NBA Feature Extraction

[0115] After the multi-head attention mechanism, NBA is introduced to extract local and global features. The feature vector after multi-head attention processing Through a set of trainable atomic mapping layers a mPerform nonlinear feature mapping:

[0116]

[0117] Among them, a m It is an atomic-level feature representation with a size of d2×1; W a and b a are the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension size of the atomic-level feature;

[0118] All atomic-level features are integrated into a global feature representation through pooling operations:

[0119]

[0120] in, is the global feature representation extracted by the neural atomic bag, with a size of d2×1; k is the number of atomic mapping layers; MaxPooling is the maximum pooling layer;

[0121] Step S7.3.3: Extract the feature vector after multi-head attention mechanism and neural atomic bag: The CIPN severity level is predicted through the classification head;

[0122] The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities:

[0123]

[0124] Among them, W c is the weight matrix of the classification head, with size C×d2; b c is the bias vector of the classification head, with size C×1; is the predicted result of CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels;

[0125] Step S7.3.4: Model training and updating:

[0126] The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is:

[0127]

[0128] Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c; C is the severity level of CIPN;

[0129] Step S7.3.5: Repeat training until convergence

[0130] Repeat steps S7.3.1 to S7.3.4 to gradually optimize the model parameters until the performance improvement efficiency on the validation set is lower than the preset standard.

[0131] According to the present invention, a CIPN information fusion system based on multimodal information includes:

[0132] Module M1: Subjective questionnaire to evaluate the person to be tested;

[0133] Module M2: Evaluate walking speed and gait based on the walking video of the person to be tested;

[0134] Module M3: Assess the fine motor skills of the test subjects;

[0135] Module M4: Assess the nerve conduction velocity of the person being tested;

[0136] Module M5: Assess the brain function of the person being tested;

[0137] Module M6: Assess the brain structure of the person being tested;

[0138] Module M7: Information fusion based on subjective questionnaires, walking speed and gait, finger fine motor skills, nerve conduction velocity, brain function and brain structure.

[0139] Preferably, in the module M1:

[0140] Module M1.1: Design a self-report scale that includes surveys on sensory symptoms, motor symptoms, autonomic symptoms, and the impact on quality of life;

[0141] Module M1.2: Data collection and preprocessing:

[0142] Collect self-report questionnaire data and normalize all data;

[0143]

[0144] where x i is the i-th data in the self-report questionnaire data of the tested person, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data;

[0145] Module M1.3: Indicator weight setting:

[0146] Based on the opinions of clinical experts, the importance weight of each self-report questionnaire question was determined:

[0147]

[0148] Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the i-th indicator;

[0149] Module M1.3: Comprehensive score calculation:

[0150] The weighted average method was used to calculate the comprehensive CIPN score, and the formula was as follows:

[0151]

[0152] Among them, w i is the importance weight of the i-th indicator, x i is the ith data in the CIPN20 questionnaire data, s1 is the score of the person to be tested on this scale; M is the total number of indicators;

[0153] In the module M2:

[0154] Module M2.1: Video Preprocessing

[0155] Extract K frames of images {T1,...,T K};

[0156] Module M2.2: Gait feature extraction

[0157] Use the correlation between consecutive frames to perform background subtraction, remove the static background, and retain the moving human body parts, specifically:

[0158] Module M2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image Diff between the two frames t :

[0159] Diff t =|T t -T t-1 |

[0160] Module M2.2.2: Use hard thresholding to convert the difference image Diff t Binarization, pixels greater than the threshold I are foreground:

[0161]

[0162] Among them, F t is the foreground image of frame t after removing the background;

[0163] Use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the foreground image video {(A1,B1),...,(A K ,B K )}; where (A K ,B K ) are the position coordinates of the left toe and the right toe in the Kth frame respectively;

[0164] Module M2.3: Gait cycle recognition:

[0165] Calculate the distance between toe positions in two consecutive frames:

[0166]

[0167] Among them, D m is the distance between the toes of the mth frame and the m+1th frame;

[0168] Calculate the walking speed of the subject in the video:

[0169]

[0170] Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed;

[0171] In the module M3:

[0172] The user's fine motor skills can be assessed by randomly displaying click points on the phone screen and gradually shortening the time interval between the points, recording the user's reaction time. The specific steps are as follows:

[0173] Module M3.1: Define the screen as an area with a width of C and a height of D;

[0174] Module M3.2: At each time interval t n At the end, a click point appears at a random screen location;

[0175] Module M3.3: Reaction time calculation:

[0176] (Δt) n =t click -t start

[0177] Among them, t start is the time when the click point appears, t click is the click time of the click point, (Δt) n is the nth reaction rate;

[0178] Module M3.4: After each click, shorten the time interval between dots:

[0179] t n+1 =α×t n

[0180] Among them, α is the shortening coefficient, which is greater than 0 and less than 1; t n is the nth time interval;

[0181] Module M3.5: Calculate the user's average reaction time and record the sequence of multiple reaction times

[0182]

[0183] Among them, s3 is the response time of the person to be tested, and n1 is the total number of time intervals;

[0184] In the module M4:

[0185] Nerve conduction velocity is calculated as:

[0186]

[0187] Wherein, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end;

[0188] In the module M5:

[0189] Module M5.1: Data Collection:

[0190] The N5-channel cerebral blood oxygen signal NIRS device was used to collect NIRS data on the frontal lobe function of N test subjects; each channel recorded the changes in the concentrations of oxyhemoglobin HbO and deoxyhemoglobin HbR. The number of data recorded in each channel of the test subject was N6, and the signal in the jth channel of the i-th test subject was:

[0191]

[0192] Among them, HbO(i) j is the oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, is the N6th oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR(i) j is the deoxyhemoglobin signal in the jth channel of the i-th person to be tested, is the N6th deoxyhemoglobin signal in the jth channel of the i-th person to be tested;

[0193] Module M5.2: Data Denoising:

[0194] Use a bandpass filter to remove high-frequency noise and low-frequency drift in the signal to obtain the denoised signal, that is,

[0195] HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high )

[0196] HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high )

[0197] Among them, HbO_filtered(i) j is the filtered oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR_filtered(i) j is the filtered deoxyhemoglobin signal in the jth channel of the i-th person to be tested, BF is the bandpass filter, f low ,f high is the cut-off frequency;

[0198] Module M5.3: Brain Function Feature Extraction

[0199] Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested:

[0200]

[0201] Among them, N5 is the number of data recorded by the channel;

[0202] Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested:

[0203]

[0204] Extract the total blood volume change s5(i) of the i-th person to be tested:

[0205] s5=ΔHbO(i)+ΔHbR(i)

[0206] Among them, mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of the signal;

[0207] In the module M6:

[0208] Module M6.1: Data Collection:

[0209] Perform magnetic resonance imaging (MRI) on N persons to be tested and set scanning parameters;

[0210] Module M6.2: Structural feature extraction:

[0211] Use the automated tool FreeSurfer to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i ;

[0212] Module M6.3: Calculation of brain structural indicators:

[0213] s6=CMV i +CT i +CSA i

[0214] In the module M7:

[0215] Module M7.1: Collect the evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested:

[0216] P={p1,...,p l …,p N},

[0217] Y={y1,...,y l …,y N},

[0218] Among them, p l ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6} T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, y l is the CIPN severity level of the lth person under test;

[0219] Module M7.2: Data standardization, the specific construction method is as follows:

[0220] Module M7.2.1: Divide the data of the N test subjects into training set, validation set and test set according to the preset ratio;

[0221] Module M7.2.2: Normalize all the above data using the following formula:

[0222]

[0223] Among them, s ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested; s minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature;

[0224] Module M7.3: Building multi-classification models using Mamba model pairs:

[0225] Module M7.3.1: For each person i in the data set P, the feature vector p i ={sn i1 ,sn i2 ,sn i3 ,sn i4 ,sn i5 ,sn i6} T Embed it and convert it into a high-dimensional feature representation h i ;

[0226] -Use linear transformation to map the original normalized features to a high-dimensional embedding space:

[0227] h i =W e ·p i +b e

[0228] Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, with size d×1; d is the size of the embedding dimension;

[0229] Apply multi-head attention mechanism to capture each normalized feature s ij The relationship between them is enhanced, and the feature representation is enhanced;

[0230] For the embedded feature vector h i Compute the query Q, key K, and value V:

[0231] Q=Wq ·h i ,K=W k ·h i ,V=W v ·h i

[0232] Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d;

[0233] Calculate the self-attention score Attention(Q,K,V)

[0234]

[0235] Where: d1 is the dimension of the key vector, used to scale the dot product result;

[0236] For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel:

[0237] MultiHead(Q,K,V)=Concat(head1,…,head h )·W o

[0238] Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix, with size d×d;

[0239] Module M7.3.2: Neural Bag of Atoms: NBA Feature Extraction

[0240] After the multi-head attention mechanism, NBA is introduced to extract local and global features. The feature vector after multi-head attention processing Through a set of trainable atomic mapping layers a m Perform nonlinear feature mapping:

[0241]

[0242] Among them, a m It is an atomic-level feature representation with a size of d2×1; W a and b a are the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension size of the atomic-level feature;

[0243] All atomic-level features are integrated into a global feature representation through pooling operations:

[0244]

[0245] in, is the global feature representation extracted by the neural atomic bag, with a size of d2×1; k is the number of atomic mapping layers; MaxPooling is the maximum pooling layer;

[0246] Module M7.3.3: Extract the feature vector after multi-head attention mechanism and neural atomic bag: The CIPN severity level is predicted through the classification head;

[0247] The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities:

[0248]

[0249] Among them, W c is the weight matrix of the classification head, with size C×d2; b c is the bias vector of the classification head, with size C×1; is the predicted result of CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels;

[0250] Module M7.3.4: Model training and updating:

[0251] The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is:

[0252]

[0253] Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c; C is the severity level of CIPN;

[0254] Module M7.3.5: Repeat training until convergence

[0255] Repeat Modules M7.3.1 to M7.3.4 to gradually optimize the model parameters until the performance improvement efficiency on the validation set is lower than the preset standard.

[0256] Compared with the prior art, the present invention has the following beneficial effects:

[0257] 1. The present invention uses artificial intelligence algorithms and precise grading systems to objectively and quantitatively evaluate the patient's peripheral nerve and muscle function, fine movements, gait and stride, and provides a comprehensive, quantitative and objective CIPN evaluation method. Compared with the traditional NCI-CTCAE grading system and EORTC QLQ-CIPN20 questionnaire, the present invention overcomes the problems of subjectivity and repeatability, and improves diagnostic accuracy and treatment effect.

[0258] 2. The present invention has high sensitivity and specificity, and can identify and evaluate the occurrence of CIPN at an early stage, helping clinicians to take timely intervention measures, reduce patient suffering and the risk of treatment interruption, and thus improve the overall treatment effect. Through continuous monitoring and evaluation, the system can be personalized according to the individual patient's situation, improving the quality of life and survival time in the treatment of advanced tumors.

[0259] 3. The present invention provides a scientific basis for drug development and clinical practice. The system can be used to evaluate drug safety and effectiveness, optimize drug dosage and treatment regimen, reduce the risk of neurotoxicity, and promote the development and application of new anti-tumor drugs. The system has a high level of automation, which reduces manual operation and subjective judgment errors and improves clinical work efficiency.

[0260] 4. The present invention is not only applicable to the evaluation of CIPN, but can also be extended to the evaluation of other nervous system diseases and injuries. By optimizing and expanding algorithms and evaluation indicators, the system provides accurate diagnosis and treatment support for more patients and promotes the development of personalized medicine. The system can acquire and analyze patient health data in real time, provide high-quality data support for clinical research, and promote the progress of medical research.

[0261] 5. This invention provides a powerful clinical tool to help doctors better manage and evaluate patients' peripheral neurotoxicity during treatment, optimize treatment plans, reduce patients' pain, and improve their quality of life. By building a comprehensive, objective, and repeatable evaluation system, the overall treatment effect is improved and the progress of tumor treatment is promoted. BRIEF DESCRIPTION OF THE DRAWINGS

[0262] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0263] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0264] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0265] Embodiment 1:

[0266] According to a CIPN information fusion method based on multimodal information provided by the present invention, Figure 1 As shown, including:

[0267] Step S1: Evaluate the subjective questionnaire of the person to be tested;

[0268] Specifically, in step S1:

[0269] Step S1.1: Design a self-report scale that includes surveys on sensory symptoms, motor symptoms, autonomic symptoms, and impact on quality of life;

[0270] Step S1.2: Data collection and preprocessing:

[0271] Collect self-report questionnaire data and normalize all data;

[0272]

[0273] where x i is the i-th data in the self-report questionnaire data of the tested person, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data;

[0274] Step S1.3: Indicator weight setting:

[0275] Based on the opinions of clinical experts, the importance weight of each self-report questionnaire question was determined:

[0276]

[0277] Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the i-th indicator;

[0278] Step S1.3: Calculation of comprehensive score:

[0279] The weighted average method was used to calculate the comprehensive CIPN score, and the formula was as follows:

[0280]

[0281] Among them, w i is the importance weight of the i-th indicator, x i is the ith data in the CIPN20 questionnaire data, s1 is the score of the person being tested on this scale; M is the total number of indicators.

[0282] Step S2: evaluating walking speed and gait based on the walking video of the person to be tested;

[0283] Specifically, in step S2:

[0284] Step S2.1: Video preprocessing

[0285] Extract K frames of images {T1,...,T K};

[0286] Step S2.2: Gait feature extraction

[0287] Use the correlation between consecutive frames to perform background subtraction, remove the static background, and retain the moving human body parts, specifically:

[0288] Step S2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image Diff between the two frames t :

[0289] Diff t =|T t -T t-1 |

[0290] Step S2.2.2: Use hard thresholding to transform the difference image Diff t Binarization, pixels greater than the threshold I are foreground:

[0291]

[0292] Among them, F t is the foreground image of frame t after removing the background;

[0293] Use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the foreground image video {(A1,B1),...,(A K ,B K )}; where (A K ,B K ) are the position coordinates of the left toe and the right toe in the Kth frame respectively;

[0294] Step S2.3: Gait cycle identification:

[0295] Calculate the distance between toe positions in two consecutive frames:

[0296]

[0297] Among them, D m is the distance between the toes of the mth frame and the m+1th frame;

[0298] Calculate the walking speed of the subject in the video:

[0299]

[0300] Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed.

[0301] Step S3: assessing the fine motor skills of the finger of the person to be tested;

[0302] Specifically, in step S3:

[0303] The user's fine motor skills can be assessed by randomly displaying click points on the phone screen and gradually shortening the time interval between the points, recording the user's reaction time. The specific steps are as follows:

[0304] Step S3.1: define the screen as an area with a width of C and a height of D;

[0305] Step S3.2: At each time interval t n At the end, a click point appears at a random screen location;

[0306] Step S3.3: Reaction time calculation:

[0307] (Δt) n =t click -t start

[0308] Among them, t start is the time when the click point appears, t click is the click time of the click point, (Δt) n is the nth reaction rate;

[0309] Step S3.4: After each click, shorten the time interval between dot appearances:

[0310] t n+1 =α×t n

[0311] Among them, α is the shortening coefficient, which is greater than 0 and less than 1; t n is the nth time interval;

[0312] Step S3.5: Calculate the average reaction time of the user and record a sequence of multiple reaction times {(Δt)1,...,(Δt) n1};

[0313]

[0314] Among them, s3 is the response time of the person to be tested, and n1 is the total number of time intervals.

[0315] Step S4: evaluating the nerve conduction velocity of the person to be tested;

[0316] Specifically, in step S4:

[0317] Nerve conduction velocity is calculated as:

[0318]

[0319] Among them, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end.

[0320] Step S5: assessing the brain function of the person to be tested;

[0321] Specifically, in step S5:

[0322] Step S5.1: Data acquisition:

[0323] The N5-channel cerebral blood oxygen signal NIRS device was used to collect NIRS data on the frontal lobe function of N test subjects; each channel recorded the changes in the concentrations of oxyhemoglobin HbO and deoxyhemoglobin HbR. The number of data recorded in each channel of the test subject was N6, and the signal in the jth channel of the i-th test subject was:

[0324]

[0325] Among them, HbO(i) j is the oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, is the N6th oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR(i) j is the deoxyhemoglobin signal in the jth channel of the i-th person to be tested, is the N6th deoxyhemoglobin signal in the jth channel of the i-th person to be tested;

[0326] Step S5.2: Data denoising:

[0327] Use a bandpass filter to remove high-frequency noise and low-frequency drift in the signal to obtain the denoised signal, that is,

[0328] HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high )

[0329] HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high )

[0330] Among them, HbO_filtered(i) j is the filtered oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR_filtered(i) j is the filtered deoxyhemoglobin signal in the jth channel of the i-th person to be tested, BF is the bandpass filter, f low ,f high is the cut-off frequency;

[0331] Step S5.3: Brain function feature extraction

[0332] Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested:

[0333]

[0334] Among them, N5 is the number of data recorded by the channel;

[0335] Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested:

[0336]

[0337] Extract the total blood volume change s5(i) of the i-th person to be tested:

[0338] s5=ΔHbO(i)+ΔHbR(i)

[0339] Among them, mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of a signal.

[0340] Step S6: evaluating the brain structure of the person to be tested;

[0341] Specifically, in step S6:

[0342] Step S6.1: Data collection:

[0343] Perform magnetic resonance imaging (MRI) on N persons to be tested and set scanning parameters;

[0344] Step S6.2: Structural feature extraction:

[0345] Use the automated tool FreeSurfer to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i ;

[0346] Step S6.3: Calculation of brain structural indicators:

[0347] s6=GMV i +CT i +CSA i .

[0348] Step S7: construct an accurate grading system based on subjective questionnaires, walking speed and gait, finger fine motor skills, nerve conduction velocity, brain function and brain structure;

[0349] Specifically, in step S7:

[0350] Step S7.1: Collect the evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested:

[0351] P={p1,...,p l …,p N},

[0352] Y={y1,...,y l …,y N},

[0353] Among them, p l ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6} T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, yl is the CIPN severity level of the lth person under test;

[0354] Step S7.2: Data standardization, the specific construction method is as follows:

[0355] Step S7.2.1: Divide the data of the N persons to be tested into a training set, a validation set and a test set according to a preset ratio;

[0356] Step S7.2.2: Normalize all the above data using the following formula:

[0357]

[0358] Among them, s ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested; s minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature;

[0359] Step S7.3: Use the Mamba model pair to build a multi-classification model:

[0360] Step S7.3.1: For each person i in the data set P, the feature vector p i ={sn i1 ,sn i2 ,sn i3 ,sn i4 ,sn i5 ,sn i6} T Embed it and convert it into a high-dimensional feature representation h i ;

[0361] -Use linear transformation to map the original normalized features to a high-dimensional embedding space:

[0362] h i =W e ·p i +b e

[0363] Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, with size d×1; d is the size of the embedding dimension;

[0364] Apply multi-head attention mechanism to capture each normalized feature s ijThe relationship between them is enhanced, and the feature representation is enhanced;

[0365] For the embedded feature vector h i Compute the query Q, key K, and value V:

[0366] Q=W q ·h i ,K=W k ·h i ,V=W v ·h i

[0367] Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d;

[0368] Calculate the self-attention score Attention(Q,K,V)

[0369]

[0370] Where: d1 is the dimension of the key vector, used to scale the dot product result;

[0371] For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel:

[0372] MultiHead(Q,K,V)=Concat(head1,…,head h )·W o

[0373] Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix, with size d×d;

[0374] Step S7.3.2: Neural Bag of Atoms: NBA feature extraction

[0375] After the multi-head attention mechanism, NBA is introduced to extract local and global features. The feature vector after multi-head attention processing Through a set of trainable atomic mapping layers a m Perform nonlinear feature mapping:

[0376]

[0377] Among them, a m It is an atomic-level feature representation with a size of d2×1; W a and b aare the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension size of the atomic-level feature;

[0378] All atomic-level features are integrated into a global feature representation through pooling operations:

[0379]

[0380] in, is the global feature representation extracted by the neural atomic bag, with a size of d2×1; k is the number of atomic mapping layers; MaxPooling is the maximum pooling layer;

[0381] Step S7.3.3: Extract the feature vector after multi-head attention mechanism and neural atomic bag: The CIPN severity level is predicted through the classification head;

[0382] The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities:

[0383]

[0384] Among them, W c is the weight matrix of the classification head, with a size of C×d2; B c is the bias vector of the classification head, with size C×1; is the predicted result of CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels;

[0385] Step S7.3.4: Model training and updating:

[0386] The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is:

[0387]

[0388] Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c; C is the severity level of CIPN;

[0389] Step S7.3.5: Repeat training until convergence

[0390] Repeat steps S7.3.1 to S7.3.4 to gradually optimize the model parameters until the performance improvement efficiency on the validation set is lower than the preset standard. The trained MANBA model is used to predict the CIPN severity level of new test subjects.

[0391] Embodiment 2:

[0392] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0393] The present invention also provides a CIPN information fusion system based on multimodal information. The CIPN information fusion system based on multimodal information can be implemented by executing the process steps of the CIPN information fusion method based on multimodal information, that is, those skilled in the art can understand the CIPN information fusion method based on multimodal information as a preferred implementation of the CIPN information fusion system based on multimodal information.

[0394] According to the present invention, a CIPN information fusion system based on multimodal information includes:

[0395] Module M1: Subjective questionnaire to evaluate the person to be tested;

[0396] Specifically, in the module M1:

[0397] Module M1.1: Design a self-report scale that includes surveys on sensory symptoms, motor symptoms, autonomic symptoms, and the impact on quality of life;

[0398] Module M1.2: Data collection and preprocessing:

[0399] Collect self-report questionnaire data and normalize all data;

[0400]

[0401] where x i is the i-th data in the self-report questionnaire data of the tested person, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data;

[0402] Module M1.3: Indicator weight setting:

[0403] Based on the opinions of clinical experts, the importance weight of each self-report questionnaire question was determined:

[0404]

[0405] Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the i-th indicator;

[0406] Module M1.3: Comprehensive score calculation:

[0407] The weighted average method was used to calculate the comprehensive CIPN score, and the formula was as follows:

[0408]

[0409] Among them, w i is the importance weight of the i-th indicator, x i is the ith data in the CIPN20 questionnaire data, s1 is the score of the person to be tested on this scale; M is the total number of indicators;

[0410] Module M2: Evaluate walking speed and gait based on the walking video of the person to be tested;

[0411] In the module M2:

[0412] Module M2.1: Video Preprocessing

[0413] Extract K frames of images {T1,...,T K};

[0414] Module M2.2: Gait feature extraction

[0415] Use the correlation between consecutive frames to perform background subtraction, remove the static background, and retain the moving human body parts, specifically:

[0416] Module M2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image Diff between the two frames t :

[0417] Diff t =|T t -T t-1 |

[0418] Module M2.2.2: Use hard thresholding to convert the difference image Diff t Binarization, pixels greater than the threshold I are foreground:

[0419]

[0420] Among them, F t is the foreground image of frame t after removing the background;

[0421] Use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the foreground image video {(A1,B1),...,(A K ,B K )}; where (A K ,B K) are the position coordinates of the left toe and the right toe in the Kth frame respectively;

[0422] Module M2.3: Gait cycle recognition:

[0423] Calculate the distance between toe positions in two consecutive frames:

[0424]

[0425] Among them, D m is the distance between the toes of the mth frame and the m+1th frame;

[0426] Calculate the walking speed of the subject in the video:

[0427]

[0428] Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed;

[0429] Module M3: Assess the fine motor skills of the test subjects;

[0430] In the module M3:

[0431] The user's fine motor skills can be assessed by randomly displaying click points on the phone screen and gradually shortening the time interval between the points, recording the user's reaction time. The specific steps are as follows:

[0432] Module M3.1: Define the screen as an area with a width of C and a height of D;

[0433] Module M3.2: At each time interval t n At the end, a click point appears at a random screen location;

[0434] Module M3.3: Reaction time calculation:

[0435] (Δt) n =t click -t start

[0436] Among them, t start is the time when the click point appears, t click is the click time of the click point, (Δt) n is the nth reaction speed; Module M3.4: After each click, shorten the time interval between dot appearances:

[0437] t n+1 =α×t n

[0438] Among them, α is the shortening coefficient, which is greater than 0 and less than 1; t nis the nth time interval;

[0439] Module M3.5: Calculate the user's average reaction time and record the sequence of multiple reaction times

[0440]

[0441] Among them, s3 is the response time of the person to be tested, and n1 is the total number of time intervals;

[0442] Module M4: Assess the nerve conduction velocity of the person being tested;

[0443] In the module M4:

[0444] Nerve conduction velocity is calculated as:

[0445]

[0446] Wherein, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end;

[0447] Module M5: Assess the brain function of the person being tested;

[0448] In the module M5:

[0449] Module M5.1: Data Collection:

[0450] The N5-channel cerebral blood oxygen signal NIRS device was used to collect NIRS data on the frontal lobe function of N test subjects; each channel recorded the changes in the concentrations of oxyhemoglobin HbO and deoxyhemoglobin HbR. The number of data recorded in each channel of the test subject was N6, and the signal in the jth channel of the i-th test subject was:

[0451]

[0452] Among them, HbO(i) j is the oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, is the N6th oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR(i) j is the deoxyhemoglobin signal in the jth channel of the i-th person to be tested, is the N6th deoxyhemoglobin signal in the jth channel of the i-th person to be tested;

[0453] Module M5.2: Data Denoising:

[0454] Use a bandpass filter to remove high-frequency noise and low-frequency drift in the signal to obtain the denoised signal, that is,

[0455] HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high )

[0456] HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high )

[0457] Among them, HbO_filtered(i) j is the filtered oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR_filtered(i) j is the filtered deoxyhemoglobin signal in the jth channel of the i-th person to be tested, BF is the bandpass filter, f low ,f high is the cut-off frequency;

[0458] Module M5.3: Brain Function Feature Extraction

[0459] Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested:

[0460]

[0461] Among them, N5 is the number of data recorded by the channel;

[0462] Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested:

[0463]

[0464] Extract the total blood volume change s5(i) of the i-th person to be tested:

[0465] s5=ΔHbO(i)+ΔHbR(i)

[0466] Among them, mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of the signal;

[0467] Module M6: Assess the brain structure of the person being tested;

[0468] In the module M6:

[0469] Module M6.1: Data Collection:

[0470] Perform magnetic resonance imaging (MRI) on N persons to be tested and set scanning parameters;

[0471] Module M6.2: Structural feature extraction:

[0472] Use the automated tool FreeSurfer to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i ;

[0473] Module M6.3: Calculation of brain structural indicators:

[0474] s6=GMV i +CT i +CSA i

[0475] Module M7: Information fusion based on subjective questionnaires, walking speed and gait, finger fine motor skills, nerve conduction velocity, brain function and brain structure.

[0476] In the module M7:

[0477] Module M7.1: Collect the evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested:

[0478] P={p1,...,p l …,p N},

[0479] Y={y1,...,y l …,y N},

[0480] Among them, p l ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6} T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, y lis the CIPN severity level of the lth person under test;

[0481] Module M7.2: Data standardization, the specific construction method is as follows:

[0482] Module M7.2.1: Divide the data of the N test subjects into training set, validation set and test set according to the preset ratio;

[0483] Module M7.2.2: Normalize all the above data using the following formula:

[0484]

[0485] Among them, s ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested; s minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature;

[0486] Module M7.3: Building multi-classification models using Mamba model pairs:

[0487] Module M7.3.1: For each person i in the data set P, the feature vector p i = {Q i1 ,sn t2 ,sn i3 ,sn i4 ,sn i5 ,sn i6} T Embed it and convert it into a high-dimensional feature representation h i ;

[0488] -Use linear transformation to map the original normalized features to a high-dimensional embedding space:

[0489] h i =W e ·p i +b e

[0490] Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, with size d×1; d is the size of the embedding dimension;

[0491] Apply multi-head attention mechanism to capture each normalized feature s ij The relationship between them is enhanced, and the feature representation is enhanced;

[0492] For the embedded feature vector h i Compute the query Q, key K, and value V:

[0493] Q=W q ·h i ,K=W k ·h i ,V=W v ·h i

[0494] Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d;

[0495] Calculate the self-attention score Attention(Q,K,V)

[0496]

[0497] Where: d1 is the dimension of the key vector, used to scale the dot product result;

[0498] For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel:

[0499] MultiHead(Q,K,V)=Concat(head1,…,head h )·W o

[0500] Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix, with size d×d;

[0501] Module M7.3.2: Neural Bag of Atoms: NBA Feature Extraction

[0502] After the multi-head attention mechanism, NBA is introduced to extract local and global features. The feature vector after multi-head attention processing Through a set of trainable atomic mapping layers a m Perform nonlinear feature mapping:

[0503]

[0504] Among them, a m It is an atomic-level feature representation with a size of d2×1; W a and b aare the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension size of the atomic-level feature;

[0505] All atomic-level features are integrated into a global feature representation through pooling operations:

[0506]

[0507] in, is the global feature representation extracted by the neural atomic bag, with a size of D2×1; k is the number of atomic mapping layers; MaxPooling is the maximum pooling layer;

[0508] Module M7.3.3: Extract the feature vector after multi-head attention mechanism and neural atomic bag: The CIPN severity level is predicted through the classification head;

[0509] The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities:

[0510]

[0511] Among them, W c is the weight matrix of the classification head, with a size of C×d2; B c is the bias vector of the classification head, with size C×1; is the predicted result of CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels;

[0512] Module M7.3.4: Model training and updating:

[0513] The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is:

[0514]

[0515] Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c; C is the severity level of CIPN;

[0516] Module M7.3.5: Repeat training until convergence

[0517] Modules M7.3.1 to M7.3.4 are triggered repeatedly to gradually optimize the model parameters until the performance improvement efficiency on the validation set is lower than the preset standard. The trained MANBA model is used to predict the CIPN severity level of new test subjects.

[0518] Embodiment 3:

[0519] Embodiment 3 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0520] In view of the defects in the prior art, the object of the present invention is to provide a CIPN information fusion system based on multimodal information.

[0521] According to the present invention, a precise grading system based on medical multimodal information is provided, comprising:

[0522] Step S1: Evaluate the subjective questionnaire of the person to be tested.

[0523] Step S2: Evaluate the walking speed and gait based on the walking video of the person to be tested.

[0524] Step S3: Evaluate the fine motor skills of the fingers of the person to be tested.

[0525] Step S4: Evaluate the nerve conduction velocity of the person to be tested.

[0526] Step S5: Evaluate the brain function of the person to be tested.

[0527] Step S6: Evaluate the brain structure of the person being tested.

[0528] Step S7: Using the evaluation values ​​of S1-S6 above, an accurate grading system is constructed.

[0529] Step S8: Using the precise grading system constructed in S7, the severity of the patient's CIPN is precisely graded.

[0530] Preferably, in step S1:

[0531] S1.1 Design of patient self-report scale

[0532] This scale includes a survey of multiple aspects of the patient, and the specific design is as follows:

[0533] Part 1: Sensory symptoms

[0534] Please select the option that best describes your feelings based on the following questions:

[0535] 0 points: None

[0536] 1 point: Mild

[0537] ·2 points: Moderate

[0538] 3 points: severe

[0539] 4 points: Very serious

[0540] Do you feel tingling in your fingers or toes?

[0541] Do you experience numbness in your fingers or toes?

[0542] Do you feel a burning sensation in your palms or soles?

[0543] Do you feel pain without any stimulation?

[0544] Part 2: Motor symptoms

[0545] Please select the option that best describes your feelings based on the following questions:

[0546] 0 points: None

[0547] 1 point: Mild

[0548] ·2 points: Moderate

[0549] 3 points: severe

[0550] 4 points: Very serious

[0551] Do you feel weakness in your hands or feet?

[0552] Do you find yourself having trouble lifting or holding objects?

[0553] Do you feel unsteady when walking?

[0554] Part 3: Autonomic Symptoms

[0555] Please select the option that best describes your feelings based on the following questions:

[0556] 0 points: None

[0557] 1 point: Mild

[0558] ·2 points: Moderate

[0559] 3 points: severe

[0560] 4 points: Very serious

[0561] Do you feel dizzy or feel faint when standing?

[0562] Do you have digestive problems such as constipation or diarrhea?

[0563] Part 4: Impact on quality of life

[0564] Please select the option that best describes your feelings based on the following questions:

[0565] 0 points: Not at all

[0566] 1 point: A little bit

[0567] 2 points: Some

[0568] 3 points: Quite a lot

[0569] 4 points: Very many

[0570] Do your neuropathy symptoms interfere with your daily activities (e.g., dressing, bathing)?

[0571] Do your neuropathy symptoms affect your social activities (such as seeing friends and family)?

[0572] Do your neuropathy symptoms affect your work or school?

[0573] Are your neuropathy symptoms affecting your sleep?

[0574] S1.2 Data collection and preprocessing

[0575] Collect patient self-report questionnaire data completed by all patients. Regularize all data;

[0576]

[0577] where x i is the ith data in the patient self-report scale questionnaire data, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data.

[0578] S1.3 Indicator weight setting

[0579] The importance weight of each patient self-report scale questionnaire question was determined based on the clinical expert's opinion.

[0580]

[0581] Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the ith indicator.

[0582] S1.3 Calculation of comprehensive score

[0583] The weighted average method was used to calculate the comprehensive CIPN score for each patient. The formula is as follows:

[0584]

[0585] Among them, w i is the importance weight of the i-th indicator, x iis the ith data in the CIPN20 questionnaire data, and s1 is the score of the person being tested on this scale.

[0586] Preferably, in step S2:

[0587] S2.1 Video Preprocessing

[0588] Extract K frames of images from the video taken by the person to be tested

[0589] {T1,...,T K}

[0590] S2.2 Gait feature extraction

[0591] First, background subtraction is performed using the correlation between consecutive frames to remove the static background and retain the moving human body parts, specifically:

[0592] Step S2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image of the two frames

[0593] Diff t =|T t -T t-1 |

[0594] Step S2.2.2: Use hard thresholding to transform the difference image Diff t Binarization, pixels greater than the threshold I are foreground:

[0595]

[0596] Among them, F t is the foreground image of frame t after removing the background.

[0597] Then, use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the above foreground image:

[0598] {(A1,B1),...,(A K ,B K )}

[0599] Among them, (A K ,B K ) are the position coordinates of the left toe and the right toe in the Kth frame respectively.

[0600] S2.3 Gait cycle recognition

[0601] Calculate the distance between the toe positions in two consecutive frames,

[0602]

[0603] Among them, D m The distance between the toes of the mth frame and the m+1th frame

[0604] Calculate the walking speed of the subject in the video:

[0605]

[0606] Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed.

[0607] Preferably, in step S3:

[0608] The user's fine motor skills are assessed by recording the user's reaction time by randomly appearing click points on the phone screen and gradually shortening the time interval between the points. The specific steps are as follows.

[0609] S3.1 defines the screen as an area with a width of C and a height of D.

[0610] S3.2 At each time interval t n At the end, a click point appears at a random screen location.

[0611] S3.3 Reaction time calculation

[0612] (Δt) n =t click -t start

[0613] t start is the time when the click point appears, t click is the click time of the click point, Δt is the nth reaction speed

[0614] S3.4 After each click, shorten the time interval between dot appearances.

[0615] t n+1 =α×t n

[0616] α is the shortening coefficient, which is greater than 0 and less than 1

[0617] S3.5 calculates the average reaction time of the user. Record a sequence of multiple reaction times {(Δt)1,...,(Δt) n},

[0618]

[0619] Where s3 is the response time of the person to be tested.

[0620] Preferably, in step S4:

[0621] Nerve conduction velocity is calculated as:

[0622]

[0623] Among them, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end.

[0624] Preferably, in step S5:

[0625] S5.1 Data Collection

[0626] Using the N5-channel cerebral blood oxygen signal (NIRS) device, NIRS data collection is performed on the frontal lobe function of N test subjects. Each channel records the changes in the concentration of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR). The number of data recorded in each channel of the test subject is N6. The signal in the jth channel of the i-th test subject is:

[0627]

[0628] S5.2 Data denoising

[0629] Use a bandpass filter to remove high-frequency noise and low-frequency drift in the above signal to obtain the denoised signal, that is,

[0630] HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high )

[0631] HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high )

[0632] Where BF is a bandpass filter, f low ,f high is the cut-off frequency.

[0633] S5.3 Brain function feature extraction

[0634] Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested:

[0635]

[0636] Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested:

[0637]

[0638] Extract the total blood volume change s5(i) of the i-th person to be tested:

[0639] s5=ΔHbO(i)+ΔHbR(i)

[0640] where mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of a signal.

[0641] Preferably, in step S6:

[0642] S6.1 Data collection

[0643] Magnetic resonance imaging (MRI) was performed on N persons to be tested, and the scanning parameters were set as follows: scanning mode: T1-weighted image; scanning plane: axial plane; slice thickness: 1 mm; matrix size: 256 x 256; field of view (FOV): 240 mm.

[0644] S6.2 Structural feature extraction

[0645] Use an automated tool (FreeSurfer) to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i .

[0646] S6.3 Calculation of brain structural indicators

[0647] s6=GMV i +CT i +CSA i

[0648] Preferably, in step S7:

[0649] S7.1 Collect the above S1-S6 evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested:

[0650] P={p1,...,p l …,p N},

[0651] Y={y1,...,y l …,y N},

[0652] where p l ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6} T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, y l It is the CIPN severity level of the lth person being tested.

[0653] S7.2 data standardization, the specific construction method is as follows:

[0654] Step S7.2.1: Divide the data of the N persons to be tested into a training set, a validation set and a test set according to a certain ratio;

[0655] Step S7.2.2: Normalize all the above data using the following formula:

[0656]

[0657] Among them, s Ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested. minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature.

[0658] S7.3 Use Mamba model pairs to build multi-classification models

[0659] S7.3.1 For each person i in the data set P, the feature vector p i ={sn i1 ,sn i2 ,sn i3 ,sn i4 ,sn i5 ,sn i6} T Embed it and convert it into a high-dimensional feature representation h i .

[0660] -Use linear transformation to map the original normalized features to a high-dimensional embedding space:

[0661] h i =W e ·p i +b e

[0662] Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, of size d×1; d is the size of the embedding dimension.

[0663] Apply a multi-head attention mechanism to capture each normalized feature s ij The relationship between them is enhanced, and the feature representation is enhanced.

[0664] For the embedded feature vector h i Compute the query Q, key K, and value V:

[0665] Q=W q ·h i ,K=W k ·h i ,V=W v ·h i

[0666] Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d.

[0667] Calculate the self-attention score Attention(Q,K,V)

[0668]

[0669] Where: d1 is the dimension of the key vector, used to scale the dot product result.

[0670] For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel:

[0671] MultiHead(Q,K,V)=Concat(head1,…,head h )·W o

[0672] Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix with size d×d.

[0673] S7.3.2 Neural Bag of Atoms (NBA) feature extraction

[0674] After the multi-head attention mechanism, NBA is introduced to extract local and global features to enhance the model's ability to capture complex patterns.

[0675] Feature vector after multi-head attention processing Through a set of trainable "atomic" mapping layers a m Perform nonlinear feature mapping:

[0676]

[0677] Among them, a m It is an atomic-level feature representation with a size of d2×1; M a and b a are the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension of the atomic-level feature.

[0678] All atomic-level features are integrated into a global feature representation through pooling operations:

[0679]

[0680] in, is the global feature representation extracted by the neural atomic bag, with size d2×1; k is the number of atomic mapping layers.

[0681] S7.3.3 The feature vector extracted by multi-head attention mechanism and neural atomic bag The severity level of CIPN is predicted through the classification head.

[0682] The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities:

[0683]

[0684] Among them, W c is the weight matrix of the classification head, with size C×d2; b c is the bias vector of the classification head, with size C×1; is the predicted result of the CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels.

[0685] S7.3.4 Model training and updating

[0686] The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is:

[0687]

[0688] Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c.

[0689] S7.3.5 Repeat training until convergence

[0690] Repeat steps S7.3.1 to S7.3.4 to gradually optimize the model parameters until the performance on the validation set is no longer significantly improved. Finally, the trained MANBA model can be used to predict the CIPN severity level of new test subjects.

[0691] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0692] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A CIPN information fusion method based on multimodal information, characterized in that: include: Step S1: Evaluate the subjective questionnaire of the person to be tested; Step S2: evaluating walking speed and gait based on the walking video of the person to be tested; Step S3: assessing the fine motor skills of the finger of the person to be tested; Step S4: evaluating the nerve conduction velocity of the person to be tested; Step S5: assessing the brain function of the person to be tested; Step S6: evaluating the brain structure of the person to be tested; Step S7: Information fusion is performed based on the subjective questionnaire, walking speed and gait, finger fine motor skills, nerve conduction velocity, brain function and brain structure.

2. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S1: Step S1.1: Design a self-report scale that includes surveys on sensory symptoms, motor symptoms, autonomic symptoms, and impact on quality of life; Step S1.2: Data collection and preprocessing: Collect self-report questionnaire data and normalize all data; where x i is the i-th data in the self-report questionnaire data of the tested person, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data; Step S1.3: Indicator weight setting: Based on the opinions of clinical experts, the importance weight of each self-report questionnaire question was determined: Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the i-th indicator; Step S1.3: Calculation of comprehensive score: The weighted average method was used to calculate the comprehensive CIPN score, and the formula was as follows: Among them, w i is the importance weight of the i-th indicator, x i is the ith data in the CIPN20 questionnaire data, s1 is the score of the person being tested on this scale; M is the total number of indicators.

3. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S2: Step S2.1: Video preprocessing Extract K frames of images {T1,...,T K }; Step S2.2: Gait feature extraction Use the correlation between consecutive frames to perform background subtraction, remove the static background, and retain the moving human body parts, specifically: Step S2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image Diff between the two frames t : Diff t =|T t -T t-1 | Step S2.2.2: Use hard thresholding to transform the difference image Diff t Binarization, pixels greater than the threshold I are foreground: Among them, F t is the foreground image of frame t after removing the background; Use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the foreground image video {(A1,B1),...,(A K ,B K )}; where (A K ,B K ) are the position coordinates of the left toe and the right toe in the Kth frame respectively; Step S2.3: Gait cycle identification: Calculate the distance between toe positions in two consecutive frames: Among them, D m is the distance between the toes of the mth frame and the m+1th frame; Calculate the walking speed of the subject in the video: Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed.

4. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S3: The user's fine motor skills can be assessed by randomly displaying click points on the phone screen and gradually shortening the time interval between the points, recording the user's reaction time. The specific steps are as follows: Step S3.1: define the screen as an area with a width of C and a height of D; Step S3.2: At each time interval t n At the end, a click point appears at a random screen location; Step S3.3: Reaction time calculation: (Δt) n =t click -t start Among them, t start is the time when the click point appears, t click is the click time of the click point, (Δt) n is the nth reaction rate; Step S3.4: After each click, shorten the time interval between dot appearances: t n+1 =α×t n Among them, α is the shortening coefficient, which is greater than 0 and less than 1; t n is the nth time interval; Step S3.5: Calculate the average reaction time of the user and record a sequence of multiple reaction times {(Δt)1,...,(Δt) n1 }; Among them, s3 is the response time of the person to be tested, and n1 is the total number of time intervals.

5. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S4: Nerve conduction velocity is calculated as: Among them, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end.

6. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S5: Step S5.1: Data acquisition: The N5-channel cerebral blood oxygen signal NIRS device was used to collect NIRS data on the frontal lobe function of N test subjects; each channel recorded the changes in the concentrations of oxyhemoglobin HbO and deoxyhemoglobin HbR. The number of data recorded in each channel of the test subject was N6, and the signal in the jth channel of the i-th test subject was: Among them, HbO(i) j is the oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, is the N6th oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR(i) j is the deoxyhemoglobin signal in the jth channel of the i-th person to be tested, is the N6th deoxyhemoglobin signal in the jth channel of the i-th person to be tested; Step S5.2: Data denoising: Use a bandpass filter to remove high-frequency noise and low-frequency drift in the signal to obtain the denoised signal, that is, HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high ) HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high ) Among them, HbO_filtered(i) j is the filtered oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR_filtered(i) j is the filtered deoxyhemoglobin signal in the jth channel of the i-th person to be tested, BF is the bandpass filter, f low ,f high is the cut-off frequency; Step S5.3: Brain function feature extraction Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested: Among them, N5 is the number of data recorded by the channel; Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested: Extract the total blood volume change s5(i) of the i-th person to be tested: s5=ΔHbO(i)+ΔHbR(i) Among them, mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of a signal.

7. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S6: Step S6.1: Data collection: Perform magnetic resonance imaging (MRI) on N persons to be tested and set scanning parameters; Step S6.2: Structural feature extraction: Use the automated tool FreeSurfer to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i ; Step S6.3: Calculation of brain structural indicators: s6=GMV i +CT i +CSA i 。 8. The CIPN information fusion method based on multimodal information according to claim 1, characterized in that: In step S7: Step S7.1: Collect the evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested: P={p1,...,p l …,p N }, Y={y1,...,y l …,y N }, Among them, p l ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6 } T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, y l is the CIPN severity level of the lth person under test; Step S7.2: Data standardization, the specific construction method is as follows: Step S7.2.1: Divide the data of the N persons to be tested into a training set, a validation set and a test set according to a preset ratio; Step S7.2.2: Normalize all the above data using the following formula: Among them, s ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested; s minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature; Step S7.3: Use the Mamba model pair to build a multi-classification model: Step S7.3.1: For each person i in the data set P, the feature vector p i ={sn i1 ,sn i2 ,sn i3 ,sn i4 ,sn i5 ,sn i6 } T Embed it and convert it into a high-dimensional feature representation h i ; -Use linear transformation to map the original normalized features to a high-dimensional embedding space: h i =W e ·p i +b e Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, with size d×1; d is the size of the embedding dimension; Apply multi-head attention mechanism to capture each normalized feature s ij The relationship between them is enhanced, and the feature representation is enhanced; For the embedded feature vector h i Compute the query Q, key K, and value V: Q=W q ·h i ,K=W k ·h i ,V=W v ·h i Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d; Calculate the self-attention score Attention(Q,K,V) Where: d1 is the dimension of the key vector, used to scale the dot product result; For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel: MultiHead(Q,K,V)=Concat(head1,…,head h )·W o Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix, with size d×d; Step S7.3.2: Neural Bag of Atoms: NBA feature extraction After the multi-head attention mechanism, NBA is introduced to extract local and global features. The feature vector after multi-head attention processing Through a set of trainable atomic mapping layers a m Perform nonlinear feature mapping: Among them, a m It is an atomic-level feature representation with a size of d2×1; W a and b a are the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension size of the atomic-level feature; All atomic-level features are integrated into a global feature representation through pooling operations: in, is the global feature representation extracted by the neural atomic bag, with a size of d2×1; k is the number of atomic mapping layers; MaxPooling is the maximum pooling layer; Step S7.3.3: Extract the feature vector after multi-head attention mechanism and neural atomic bag: The CIPN severity level is predicted through the classification head; The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities: Among them, W c is the weight matrix of the classification head, with size C×d2; b c is the bias vector of the classification head, with size C×1; is the predicted result of CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels; Step S7.3.4: Model training and updating: The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is: Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c; C is the severity level of CIPN; Step S7.3.5: Repeat training until convergence Repeat steps S7.3.1 to S7.3.4 to gradually optimize the model parameters until the performance improvement efficiency on the validation set is lower than the preset standard.

9. A CIPN information fusion system based on multimodal information, characterized in that: include: Module M1: Subjective questionnaire to evaluate the person to be tested; Module M2: Evaluate walking speed and gait based on the walking video of the person to be tested; Module M3: Assess the fine motor skills of the test subjects; Module M4: Assess the nerve conduction velocity of the person being tested; Module M5: Assess the brain function of the person being tested; Module M6: Assess the brain structure of the person being tested; Module M7: Information fusion based on subjective questionnaires, walking speed and gait, finger fine motor skills, nerve conduction velocity, brain function and brain structure.

10. The CIPN information fusion system based on multimodal information according to claim 9, characterized in that: In the module M1: Module M1.1: Design a self-report scale that includes surveys on sensory symptoms, motor symptoms, autonomic symptoms, and the impact on quality of life; Module M1.2: Data collection and preprocessing: Collect self-report questionnaire data and normalize all data; where x i is the i-th data in the self-report questionnaire data of the tested person, a is the minimum value of the questionnaire data, and b is the maximum value of the questionnaire data; Module M1.3: Indicator weight setting: Based on the opinions of clinical experts, the importance weight of each self-report questionnaire question was determined: Among them, (w i ) j is the importance weight set by the jth clinical expert for the ith indicator, ranging from 1 to 10, N is the number of experts, and w i is the importance weight of the i-th indicator; Module M1.3: Comprehensive score calculation: The weighted average method was used to calculate the comprehensive CIPN score, and the formula was as follows: Among them, w i is the importance weight of the i-th indicator, x i is the ith data in the CIPN20 questionnaire data, s1 is the score of the person to be tested on this scale; M is the total number of indicators; In the module M2: Module M2.1: Video Preprocessing Extract K frames of images {T1,...,T K }; Module M2.2: Gait feature extraction Use the correlation between consecutive frames to perform background subtraction, remove the static background, and retain the moving human body parts, specifically: Module M2.2.1: Read the current frame image T t With the previous frame image T t-1 , and calculate the difference image Diff between the two frames t : Diff t =|T t -T t-1 | Module M2.2.2: Use hard thresholding to convert the difference image Diff t Binarization, pixels greater than the threshold I are foreground: Among them, F t is the foreground image of frame t after removing the background; Use the emplaceAndPop function in the trained pose estimation model OpenPose to detect the toe position of each frame in the foreground image video {(A1,B1),...,(A K ,B K )}; where (A K ,B K ) are the position coordinates of the left toe and the right toe in the Kth frame respectively; Module M2.3: Gait cycle recognition: Calculate the distance between toe positions in two consecutive frames: Among them, D m is the distance between the toes of the mth frame and the m+1th frame; Calculate the walking speed of the subject in the video: Where s2 is the walking speed of the person to be tested, T is the number of seconds between the 1st frame and the Kth frame, and V is the walking speed; In the module M3: The user's fine motor skills can be assessed by randomly displaying click points on the phone screen and gradually shortening the time interval between the points, recording the user's reaction time. The specific steps are as follows: Module M3.1: Define the screen as an area with a width of C and a height of D; Module M3.2: At each time interval t n At the end, a click point appears at a random screen location; Module M3.3: Reaction time calculation: (Δt) n =t click -t start Among them, t start is the time when the click point appears, t click is the click time of the click point, (Δt) n is the nth reaction rate; Module M3.4: After each click, shorten the time interval between dots: t n+1 =α×t n Among them, α is the shortening coefficient, which is greater than 0 and less than 1; t n is the nth time interval; Module M3.5: Calculate the average reaction time of the user and record a sequence of multiple reaction times {(Δt)1,...,(Δt) n1 }; Among them, s3 is the response time of the person to be tested, and n1 is the total number of time intervals; In the module M4: Nerve conduction velocity is calculated as: Wherein, s4 is the nerve conduction velocity of the person to be tested, D is the distance between the stimulation points, ΔL=L1-L2 is the time when the electrical signal is detected between the distal end and the proximal end, L2 is the time point when the electrical signal is detected at the distal end, and L1 is the time point when the electrical signal is detected at the proximal end; In the module M5: Module M5.1: Data Collection: The N5-channel cerebral blood oxygen signal NIRS device was used to collect NIRS data on the frontal lobe function of N test subjects; each channel recorded the changes in the concentrations of oxyhemoglobin HbO and deoxyhemoglobin HbR. The number of data recorded in each channel of the test subject was N6, and the signal in the jth channel of the i-th test subject was: Among them, HbO(i) j is the oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, is the N6th oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR(i) j is the deoxyhemoglobin signal in the jth channel of the i-th person to be tested, is the N6th deoxyhemoglobin signal in the jth channel of the i-th person to be tested; Module M5.2: Data Denoising: Use a bandpass filter to remove high-frequency noise and low-frequency drift in the signal to obtain the denoised signal, that is, HbO_filtered(i) j =BF(HbO(i) j ,f low ,f high ) HbR_filtered(i) j =BF(HbR(i) j ,f low ,f high ) Among them, HbO_filtered(i) j is the filtered oxygenated hemoglobin signal in the jth channel of the i-th person to be tested, HbR_filtered(i) j is the filtered deoxyhemoglobin signal in the jth channel of the i-th person to be tested, BF is the bandpass filter, f low ,f high is the cut-off frequency; Module M5.3: Brain Function Feature Extraction Extract the average oxygenated hemoglobin concentration change of all channels of the i-th person to be tested: Among them, N5 is the number of data recorded by the channel; Extract the average deoxyhemoglobin concentration change of all channels of the i-th person to be tested: Extract the total blood volume change s5(i) of the i-th person to be tested: s5=ΔHbO(i)+ΔHbR(i) Among them, mean(HbO_filtered(i) j ) to obtain HbO_filtered(i) j The average value of different channels of the signal, mean(HbR_filtered(i) j ) to obtain HbR_filtered(i) j The average value of different channels of the signal; In the module M6: Module M6.1: Data Collection: Perform magnetic resonance imaging (MRI) on N persons to be tested and set scanning parameters; Module M6.2: Structural feature extraction: Use the automated tool FreeSurfer to segment brain tissue into gray matter, white matter, and cerebrospinal fluid; use the FreeSurfer toolkit to calculate the gray matter volume GMV of the frontal lobe of the i-th person to be tested i , calculate the average cortical thickness CT of the prefrontal region i , calculate the cortical surface area CSA of the prefrontal region i ; Module M6.3: Calculation of brain structural indicators: s6=GMV i +CT i +CSA i In the module M7: Module M7.1: Collect the evaluation indicators of N persons to be tested to form a data set P and the CIPN severity level Y of the persons to be tested: P={p1,...,p l …,p N }, Y={y1,...,y l …,y N }, Among them, p l ={s l1 ,s l2 ,s l3 ,s l4 ,s l5 ,s l6 } T is the evaluation index of the lth person to be tested, Y is the CIPN severity level data matrix of N people to be tested, and its size is 5×N, y l is the CIPN severity level of the lth person under test; Module M7.2: Data standardization, the specific construction method is as follows: Module M7.2.1: Divide the data of the N test subjects into training set, validation set and test set according to the preset ratio; Module M7.2.2: Normalize all the above data using the following formula: Among them, s ij is the jth feature of the i-th person to be tested, i∈{1,2,3,...,N}, j∈{1,2,3,4}, sn ij is the jth normalized feature of the i-th person to be tested; s minj is the minimum value of the jth feature, s maxj is the maximum value of the jth feature; Module M7.3: Building multi-classification models using Mamba model pairs: Module M7.3.1: For each person i in the data set P, the feature vector p i ={sn i1 ,sn i2 ,sn i3 ,sn i4 ,sn i5 ,sn i6 } T Embed it and convert it into a high-dimensional feature representation h i ; -Use linear transformation to map the original normalized features to a high-dimensional embedding space: h i =W e ·p i +b e Among them, h i is the embedded feature vector with a size of d×1; W e is the weight matrix of the embedding layer, with a size of d×6; b e is the bias vector of the embedding layer, with size d×1; d is the size of the embedding dimension; Apply multi-head attention mechanism to capture each normalized feature s ij The relationship between them is enhanced, and the feature representation is enhanced; For the embedded feature vector h i Compute the query Q, key K, and value V: Q=W q ·h i ,K=W k ·h i ,V=W v ·h i Where Q is the query vector, size d×1; K is the key vector, size d×1; V is the value vector, size d×1; W q , W k and W v are the weight matrices for query, key, and value, respectively, all with size d×d; Calculate the self-attention score Attention(Q,K,V) Where: d1 is the dimension of the key vector, used to scale the dot product result; For the multi-head attention mechanism, the above operation is extended to multiple heads to calculate multiple attention scores in parallel: MultiHead(Q,K,V)=Concat(head1,…,head h )·W o Among them, head i represents the calculation result of the i-th attention head; W o is the output weight matrix, with size d×d; Module M7.3.2: Neural Bag of Atoms: NBA Feature Extraction After the multi-head attention mechanism, NBA is introduced to extract local and global features. The feature vector after multi-head attention processing Through a set of trainable atomic mapping layers a m Perform nonlinear feature mapping: Among them, a m It is an atomic-level feature representation with a size of d2×1; W a and b a are the weight matrix and bias vector of the mth atomic mapping layer respectively; σ is the ReLU nonlinear activation function; d2 is the dimension size of the atomic-level feature; All atomic-level features are integrated into a global feature representation through pooling operations: in, is the global feature representation extracted by the neural atomic bag, with a size of d2×1; k is the number of atomic mapping layers; MaxPooling is the maximum pooling layer; Module M7.3.3: Extract the feature vector after multi-head attention mechanism and neural atomic bag: The CIPN severity level is predicted through the classification head; The classification head is usually one or more fully connected layers, and uses the Softmax function to output category probabilities: Among them, W c is the weight matrix of the classification head, with size C×d2; b c is the bias vector of the classification head, with size C×1; is the predicted result of CIPN severity level of the i-th person to be tested, with a size of C×1; C is the number of CIPN severity levels; Module M7.3.4: Model training and updating: The model is trained using the cross entropy loss function, and the model parameters are updated through the back propagation algorithm. The cross entropy loss function formula is: Among them, y ic is the true label of the i-th person to be tested in category c; is the predicted probability of the i-th person to be tested in category c; C is the severity level of CIPN; Module M7.3.5: Repeat training until convergence Repeatedly trigger modules M7.3.1 to M7.3.4 to gradually optimize the model parameters until the performance improvement efficiency on the validation set is lower than the preset standard.

Images