Multi-mode dual-path fusion network structure based on data level fusion

Abstract

The invention discloses a multi-modal dual-path fusion network structure based on data level fusion, and relates to the technical field of dual-path fusion network structures, comprising the steps of data set construction and preprocessing, selection and processing of a plurality of data sets, and image enhancement and normalization; constructing a network structure, constructing a dual-path network, pre-training on different data sets by using a mainstream backbone network, optimizing training, splicing feature layers, and performing classified training; and performing data fusion and classification, performing horizontal fusion on the double-path features, determining the weight by adopting a weighting strategy, and inputting the fused features into a full connection layer for classification according to a softmax function. According to the method, the ophthalmic disease diagnosis accuracy and model robustness are improved, the generalization ability is improved, network training is optimized, and medical intelligent diagnosis development is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal dual-path fusion network structures, and in particular to a multimodal dual-path fusion network structure based on data level fusion. Background Art

[0002] In today's digital age, data diversity and complexity are growing, and multimodal data (such as images, text, audio, etc.) are widely present in various fields. How to effectively process and fuse multimodal data and mine the rich information contained in it has become a key challenge faced by many fields.

[0003] In the medical field, especially in the diagnosis of ophthalmic diseases, fundus images contain a wealth of information about eye health. However, fundus image data of different modalities (such as images acquired by different fields of view and different imaging devices) differ in feature expression and distribution. Traditional single-modal analysis methods are unable to fully utilize the complementarity of these data, resulting in limited diagnostic accuracy. At the same time, existing datasets vary in size, diversity, and annotation quality, making model training prone to overfitting or underfitting, and insufficient generalization capabilities.

[0004] The development of deep learning technology has provided a new way to process multimodal data, but traditional models such as convolutional neural networks (CNN) have many limitations when dealing with multimodal fusion problems. On the one hand, most traditional models are mainly designed for single-modal data, lack an effective integration mechanism in multimodal fusion, and cannot give full play to the synergy of multimodal information. On the other hand, as the complexity of the model increases, the difficulty of training increases, and problems such as gradient disappearance are prone to occur, affecting the optimization and performance improvement of the model.

[0005] In terms of data fusion methods, existing fusion strategies such as decision-level fusion and intermediate fusion cannot effectively integrate multimodal data at low-level features in some scenarios, limiting the model's deep understanding and utilization of multimodal information. In addition, data distribution differences between different data sets (such as non-independent and identically distributed problems) seriously affect the performance of the model when applied across data sets, and existing domain adaptation algorithms are insufficient in processing feature distance relationships between categories and within domains, making it difficult to meet actual needs.

[0006] In summary, there is an urgent need for an innovative multimodal fusion network structure and method that can effectively integrate multimodal data, adapt to the characteristics of different data sets, and improve the accuracy, robustness, and generalization ability of the model to promote the development of multimodal data processing technology in fields such as medical diagnosis. Summary of the invention

[0007] The present invention proposes a multi-modal dual-path fusion network structure based on data level fusion to solve the problems mentioned in the above-mentioned prior art.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: a multi-modal dual-path fusion network structure based on data level fusion, comprising the following steps and components:

[0009] Dataset construction and preprocessing steps: DeepDR, Aptos, and EyePacs datasets were selected. The DeepDR dataset contains data from more than 1,000 patients, provides paired images centered on the optic disc and macula, and consists of 1,200 training images and 400 test images, with five types of classification labels; the Aptos dataset has 3,662 training images and 1,928 test images, with five types of classification labels; the EyePacs dataset is the largest fundus dataset, containing 35,129 images. Images with the same hash value but inconsistent labels in the Aptos dataset were deleted, and only one image with the same hash value and label was retained. The EyePacs dataset was sampled in equal proportion for pre-training;

[0010] The image is enhanced, including affine transformation, color space transformation, flipping, and normalized using image sizes of 384×384, 512×512, and 800×800. The formula is: where x norm is the normalized data, x is the original data, μ is the mean of the corresponding modal data, and σ is the standard deviation, so that the data are in a similar numerical range;

[0011] Steps for building the network structure: Build a dual-path feature extraction network, one for images centered on the optic disc and the other for images centered on the macula. Use mainstream backbone networks including VGG, ResNet, Inception or EfficientNet. Pre-train on the EyePacs dataset. Pre-training uses a scale iterative training method, using 384×384, 512×512 and 800×800 images in turn. The batch size is adjusted with the image size to select the optimal model.

[0012] After pre-training, optimization training is performed on the DeepDR dataset using a step-by-step iterative optimization method. The image sizes are 384×384, 512×512, and 800×800, respectively. The network model parameters are obtained, and the feature layers of the two networks are concatenated and input into the classification network for classification training. The best model of the previous stage is used for progressive training in the order of image size.

[0013] In the feature extraction network, a convolutional neural network CNN is used for image data, including the use of a 3×3 convolution kernel, a step size of 1, and a padding of 1 for convolution operations. Selection and filtering are performed through pooling layers, including mean pooling, maximum pooling, or pyramid pooling. The ReLU activation function is introduced to increase nonlinear expression. In the recurrent neural network RNN, including the long short-term memory network LSTM, the number of hidden layer units is set to 128, and the update gate calculation formula is z t =σ(W z ·[h t-1 ,x t ]+b z ), where z t is the update gate output, W z To update the gate weight matrix, h t-1 is the hidden state at the previous moment, x t is the current input, b z is the bias vector, σ is the sigmoid function;

[0014] Data fusion and classification steps: The features extracted from the two paths are horizontally fused using a weighted fusion strategy. The fusion formula is F fusion =αF s +βF t +γF a , F fusion Where is the fused feature, α, β, γ are weight coefficients, and α+β+γ=1. The weight value is determined by training optimization. s ,F t ,F a For different characteristics;

[0015] The fused features are input into the fully connected layer for classification. The number of neurons in the fully connected layer is determined according to the specific classification task. The softmax function is used to calculate the classification probability. The formula is: where p i is the predicted probability of the i-th category, z i is the output of the fully connected layer neurons, C is the total number of categories, and the final classification result is determined based on the predicted probability.

[0016] Furthermore, the rotation angle step size in the image enhancement technology is 1 degree, the scaling step size is 0.05, and the shearing angle step size is 5 degrees.

[0017] Furthermore, the number of training iterations for each scale in the pre-trained classification network is 50, so that the model can fully learn data features at different scales.

[0018] Furthermore, the initial values ​​of the weight coefficients α, β, and γ in the weighted fusion strategy are set to 0.4, 0.3, and 0.3, respectively, and are optimized and adjusted by the gradient descent algorithm during the training process.

[0019] Furthermore, the fully connected layer adopts Dropout technology with a random dropout probability of 0.2 to prevent overfitting and improve the generalization ability of the model.

[0020] Furthermore, a training method for a multi-modal dual-path fusion network mechanism based on the data level fusion comprises the following steps:

[0021] Data preparation steps: Prepare training data according to the data set construction and preprocessing steps, divide the data set into training set, validation set and test set with a ratio of 7:2:1, and use stratified sampling method for division;

[0022] Model construction steps: construct the multimodal dual-path fusion network structure and initialize network parameters, including weight matrices and bias vectors of convolutional neural networks and recurrent neural networks;

[0023] Training steps: Input the training set data into the network and calculate the loss value according to the loss function. The cross entropy loss function is used, and the formula is Where L is the loss value, y i is the true label, p i is the predicted probability, C is the total number of categories, and the network parameters are updated by back propagation using the gradient descent algorithm. The gradient descent algorithm uses the adaptive moment estimation Adam optimizer, whose hyperparameters β1=0.9, β2=0.999, ∈=10 -8 ,The early stopping method is used during the training process. When the validation set loss does not decrease for three consecutive epochs, the training is stopped to prevent overfitting;

[0024] Evaluation step: Use the test set data to evaluate the trained network and calculate the accuracy, recall and F1 value evaluation indicators. The accuracy formula is: Where I(y i =p i ) is the indicator function, when y i =p i is 1, otherwise it is 0, N is the number of test set samples, and the F1 value formula is Precision is the accuracy rate, Recall is the recall rate, and the network structure or training parameters are adjusted according to the evaluation results to optimize the network performance.

[0025] Furthermore, the batch size of each training in the training step is 32, balancing the utilization of computing resources and the speed of model convergence.

[0026] Furthermore, the patience value in the early stopping method is set to 3 to avoid over-training that leads to a decrease in the generalization ability of the model.

[0027] Furthermore, in the model building step, the network parameters are initialized using a random initialization method, and the value range is [-0.1, 0.1].

[0028] Furthermore, the evaluation step also calculates the precision-recall rate PR curve and the area under the curve AUC indicator.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] First, improve diagnostic accuracy. Through a carefully designed multimodal dual-path fusion network structure, the information of fundus images of different fields of view is fully integrated to achieve collaborative representation of multimodal data, providing a more comprehensive and accurate basis for the diagnosis of ophthalmic diseases such as diabetic retinopathy (DR), significantly improving the accuracy of diagnosis.

[0031] Second, enhance the robustness of the model. The improved domain adaptation algorithm effectively copes with the distribution differences between data sets, enabling the model to maintain stable performance under different data distributions, reducing diagnostic errors caused by data changes, and enhancing the model's adaptability to complex real-world scenarios.

[0032] Third, improve generalization ability. The selection and enhancement of diverse data sets, as well as reasonable network training strategies, enable the model to learn common features from a large amount of different data, avoid overfitting, and thus accurately predict on new and unseen data. It is applicable to a wider range of fundus image data and provides strong support for clinical applications.

[0033] Fourth, optimize network training. First, use the optimal model in single-modal training to initialize network parameters, reduce the difficulty of dual-channel fusion training, improve training efficiency, and save computing resources and time costs. At the same time, progressive training methods and reasonable parameter settings ensure that the network model can be gradually optimized to achieve better performance.

[0034] Fifth, promote the development of intelligent medical diagnosis. This patent provides new technical means and ideas for the intelligent diagnosis of ophthalmic diseases, which helps to improve the efficiency and quality of medical diagnosis, promote the development of the medical field towards intelligence and precision, and is of great significance to improving patients' medical services. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic block diagram of the multi-modal dual-path fusion network structure steps based on data level fusion proposed by the present invention;

[0036] Figure 2 A schematic block diagram of the sub-steps of data set construction and preprocessing proposed by the present invention;

[0037] Figure 3 A schematic block diagram of the sub-steps of building the network structure proposed by the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0040] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, and it can be the internal connection of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0041] Reference Figure 1-3 : A multimodal dual-path fusion network structure based on data level fusion, including the following steps and components:

[0042] Dataset construction and preprocessing steps:

[0043] The DeepDR, Aptos, and EyePacs datasets were selected. The DeepDR dataset contains data from more than 1,000 patients, provides paired images centered on the optic disc and macula, and consists of 1,200 training images and 400 test images, with five types of classification labels; the Aptos dataset has 3,662 training images and 1,928 test images, with five types of classification labels; the EyePacs dataset is the largest fundus dataset, containing 35,129 images. Images with the same hash value but inconsistent labels in the Aptos dataset were deleted, and only one image with the same hash value and label was retained. The EyePacs dataset was sampled in equal proportion for pre-training.

[0044] The image is enhanced, including affine transformation (rotation angle -180 to 180 degrees, scaling ratio 0.5 to 1.5, shearing -50 to 50 degrees), color space transformation (brightness 0.5 to 2, contrast 0.5 to 1.2, saturation 0.8 to 1.2), flip (horizontal and vertical flip probability are 0.5) and rotation (0 to 360 degrees). The image size is 384×384, 512×512 and 800×800, and the image is normalized. The formula is: where x norm is the normalized data, x is the original data, μ is the mean of the corresponding modal data, and σ is the standard deviation, so that the data are in a similar numerical range.

[0045] Steps to build the network structure:

[0046] A dual-path feature extraction network was constructed, one for images centered on the optic disc and the other for images centered on the macula. A mainstream backbone network (such as VGG, ResNet, Inception, or EfficientNet) was used, and pre-trained on the EyePacs dataset. The pre-training adopted a scaled iterative training method, using 384×384, 512×512, and 800×800 size images in turn. The batch size was adjusted with the image size, and the optimal model was selected.

[0047] After pre-training, optimization training was performed on the DeepDR dataset using a step-by-step iterative optimization method. The image sizes were 384×384, 512×512, and 800×800, respectively, to obtain the network model parameters. The feature layers of the two networks were concatenated and input into the classification network for classification training. The best model of the previous stage was used for progressive training in the same order of image sizes.

[0048] In the feature extraction network, a convolutional neural network (CNN) is used for image data, such as using a 3×3 convolution kernel, a step size of 1, and a padding of 1 for convolution operations. Selection and filtering are performed through pooling layers (such as mean pooling, maximum pooling, or pyramid pooling). An activation function (such as ReLU) is introduced to increase nonlinear expression. In a recurrent neural network (RNN), such as a long short-term memory network (LSTM), the number of hidden layer units is set to 128, and the update gate calculation formula is z t =σ(W z ·[h t-1 ,x t ]+b z ), where z t is the update gate output, W z To update the gate weight matrix, h t-1 is the hidden state at the previous moment, x t is the current input, b z is the bias vector, and σ is the sigmoid function.

[0049] Data fusion and classification steps:

[0050] The features extracted from the two paths are horizontally fused using a weighted fusion strategy. The fusion formula is: fusion =αF s +βF t +γF a ,F fusion Where is the fused feature, α, β, γ are weight coefficients, and α+β+γ=1. The weight value is determined by training optimization. s ,F t ,F a For different characteristics.

[0051] The fused features are input into the fully connected layer for classification. The number of neurons in the fully connected layer is determined according to the specific classification task. The softmax function is used to calculate the classification probability. The formula is: where p i is the predicted probability of the i-th category, z i is the output of the fully connected layer neurons, C is the total number of categories, and the final classification result is determined based on the predicted probability.

[0052] In image enhancement technology, the rotation angle step is set to 1 degree. This means that when performing rotation enhancement operations on an image, the image will be rotated in the range of -180 degrees to 180 degrees at intervals of 1 degree. Such a fine step setting can produce a rich variety of rotation angles, ensuring that the image is fully transformed in different directions, thereby increasing the diversity of the data and allowing the model to learn the feature invariance of the image under various rotation states. The scaling step is 0.05. From the scaling range of 0.5 to 1.5, the model can learn the feature changes of the image at different scaling levels, which helps to improve the model's adaptability to images of different sizes. The shearing angle step is 5 degrees. Within the shearing range of -50 degrees to 50 degrees, the image is distorted to varying degrees, further enriching the form of image changes. These step settings work together to effectively expand the data set without introducing too much noise or destroying the key information of the image, so that the model can learn more representative features from data with multiple angles, scales, and deformations, and enhance the generalization ability of the model.

[0053] The number of training iterations for each scale in the pre-trained classification network is set to 50. In the pre-training stage, the model is first trained using images of size 384×384. During these 50 iterations, the model calculates the prediction results through forward propagation, compares them with the true labels to obtain the loss value, and then uses the back propagation algorithm to adjust the network parameters according to the loss value. After multiple iterations, the model gradually learns the data features in the images of this scale, such as the vascular structure and lesion features in the fundus images. Then, the model is trained using images of size 512×512. After 50 iterations, the model further adapts to the feature representation of larger images and learns richer detail information. Finally, the model is trained 50 times using images of size 800×800, so that the model can understand the data features in a larger field of view and capture more macroscopic image patterns and feature relationships. Through 50 full training iterations at each scale, the model can comprehensively and deeply learn the intrinsic features and laws of data at different scales, laying a solid foundation for subsequent optimization training and accurate classification.

[0054] In the present invention, the initial values ​​of the weight coefficients , , in the weighted fusion strategy are set to 0.4, 0.3, and 0.3, respectively, and are optimized and adjusted by the gradient descent algorithm during the training process to improve the fusion effect.

[0055] The fully connected layer uses Dropout technology with a random dropout probability of 0.2. During the neuron transmission process of the fully connected layer, each neuron has a probability of 0.2 to be randomly dropped in each training iteration. For example, assuming that the fully connected layer has 1000 neurons, then in one training iteration, about 200 neurons will be randomly set to 0 and will not participate in the forward and backward propagation processes. The purpose of this is to prevent excessive dependence between neurons and avoid overfitting of the model. During the training process, due to the random dropout of some neurons, the network structure changes slightly at each iteration, so that the model will not over-learn specific patterns in the training data. This prompts neurons to learn more universal and robust features, thereby improving the model's generalization ability for unseen data. In the test phase, all neurons participate in the calculation, but the weights of the neurons have been optimized through the Dropout technology during the training process, so that the model can more accurately classify or predict new data, enhancing the reliability and effectiveness of the model in practical applications.

[0056] In the present invention, a training method for a multimodal dual-path fusion network based on data level fusion comprises the following steps:

[0057] Data preparation steps: Prepare training data according to the data set construction and preprocessing steps, and divide the data set into training set, validation set and test set in a ratio of 7:2:1. The division adopts stratified sampling method to ensure that the data categories in each set are evenly distributed.

[0058] Model construction steps: construct the multimodal dual-path fusion network structure and initialize the network parameters, including the weight matrix and bias vector of the convolutional neural network and the recurrent neural network.

[0059] Training step: Input the training set data into the network and calculate the loss value according to the loss function. The cross entropy loss function is used, and the formula is Where L is the loss value, y i is the true label, p i is the predicted probability, and C is the total number of categories. The network parameters are updated by back propagation using the gradient descent algorithm. The gradient descent algorithm uses the adaptive moment estimation (Adam) optimizer, and its hyperparameters are β1 = 0.9, β2 = 0.9999, ∈ = 10 -8 ,The early stopping method is used in the training process. ,The training is stopped when the validation set loss no longer decreases for ,three epochs in a row, to prevent overfitting.

[0060] Evaluation step: Use the test set data to evaluate the trained network and calculate evaluation indicators such as accuracy, recall, and F1 value. The accuracy formula is: Where I(y i =p i ) is the indicator function, when yi =p i is 1, otherwise it is 0, N is the number of test set samples, and the F1 value formula is Precision is the accuracy rate, Recall is the recall rate, and the network structure or training parameters are adjusted according to the evaluation results to further optimize the network performance.

[0061] In the training step, the batch size of each training is set to 32. The choice of batch size is determined after fully considering the balance between computing resource utilization and model convergence speed. Smaller batch sizes, such as 16 or smaller, will increase the computing resource overhead because the model parameters need to be updated more frequently, but may enable the model to capture the local features of the data faster during training, but may also lead to unstable convergence. Larger batch sizes, such as 64 or larger, although they can reduce the frequency of parameter updates and reduce computing resource consumption, may slow down the model convergence speed and even fall into a local optimal solution. After multiple experimental comparisons, when the batch size is 32, under the existing hardware conditions (such as specific GPU models, memory capacity, etc.), it can not only ensure the effective utilization of computing resources and complete the training process within an acceptable time range, but also allow the model to converge at a relatively stable and faster speed, ensuring that the model can fully learn the feature information in the data during training, gradually optimize the model parameters, and improve the accuracy and generalization ability of the model.

[0062] The patience value in the early stopping method is set to 3. The early stopping method is an important means to prevent the model from overtraining. During the training process, the performance of the model on the validation set will change as the number of training rounds (epochs) increases. When the validation set loss stops decreasing for several consecutive epochs, it indicates that the model may have begun to overfit, and continued training may cause the model's generalization ability to decrease. By setting the patience value to 3, it means that the training process will stop when the validation set loss has not improved for three consecutive epochs. This can prevent the model from overtraining in the overfitting area and wasting computing resources while maintaining the generalization performance of the model on unseen data. In actual training, after each epoch, the validation set loss will be calculated and compared with the previous record. If it does not decrease for three consecutive times, the early stopping mechanism will be triggered, the current optimal model parameters will be saved, and the training process will end, thereby ensuring that the model can fully learn the data features during the training process and will not lose its generalization ability due to overtraining.

[0063] In the model building step, the network parameters are initialized using a random initialization method with a value range of [-0.1, 0.1]. Random initialization is a key operation to provide a good starting point for network training. In a neural network, the initial value of the parameter will affect the training process and final performance of the model. If the initial value is set improperly, the model may fall into a local optimal solution at the beginning of training, or the training process may become very slow. By randomly initializing the parameters in the range of [-0.1, 0.1], the symmetry of the model can be broken, so that different neurons have different initial states at the beginning of training, thereby avoiding all neurons from learning the same features at the beginning of training. Specifically, the convolution kernel weights and biases in the convolutional neural network, as well as the weight matrix and bias vector in the recurrent neural network, are all randomly selected within this range. This random initialization method provides a diverse starting point for the model in the subsequent training process, allowing the model to start learning data features under different initial conditions, gradually adjust the parameters through the back-propagation algorithm, converge toward the optimal solution, and improve the learning ability and generalization performance of the model.

[0064] In the evaluation step, in addition to calculating common indicators such as accuracy and recall, indicators such as the area under the precision-recall curve (PR curve) (AUC) can also be calculated to more comprehensively evaluate the model performance. Accuracy and recall measure the prediction effect of the model from different perspectives, but they may have limitations in some cases and cannot fully reflect the performance of the model at different thresholds. The PR curve can show the change in the accuracy of the model at different recall levels. By calculating the AUC under the PR curve, the overall performance of the model under different threshold settings can be comprehensively evaluated. The value range of AUC is between 0 and 1. The larger the AUC value, the stronger the model's ability to distinguish between positive and negative examples. When calculating AUC, first sort the samples according to the model's predicted probability, then calculate the precision and recall at different thresholds, draw the PR curve, and finally calculate the area under the curve through mathematical methods such as integration. This indicator can more comprehensively evaluate the performance of the model when dealing with unbalanced data sets or when different prediction performance requirements are placed on different categories. It can provide richer information for model optimization and improvement, and help researchers better understand the strengths and weaknesses of the model, so as to take targeted measures to further improve model performance.

[0065] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-modal dual-path fusion network structure based on data level fusion, characterized in that: The following steps and components are included: Dataset construction and preprocessing steps: DeepDR, Aptos, and EyePacs datasets were selected. The DeepDR dataset contains data from more than 1,000 patients, provides paired images centered on the optic disc and macula, and consists of 1,200 training images and 400 test images, with five types of classification labels; the Aptos dataset has 3,662 training images and 1,928 test images, with five types of classification labels; the EyePacs dataset is the largest fundus dataset, containing 35,129 images. Images with the same hash value but inconsistent labels in the Aptos dataset were deleted, and only one image with the same hash value and label was retained. The EyePacs dataset was sampled in equal proportion for pre-training; The image is enhanced, including affine transformation, color space transformation, flipping, and normalized using image sizes of 384×384, 512×512, and 800×800. The formula is: where x norm is the normalized data, x is the original data, μ is the mean of the corresponding modal data, and σ is the standard deviation, so that the data are in a similar numerical range; Steps for building the network structure: Build a dual-path feature extraction network, one for images centered on the optic disc and the other for images centered on the macula. Use mainstream backbone networks including VGG, ResNet, Inception or EfficientNet. Pre-train on the EyePacs dataset. Pre-training uses a scale iterative training method, using 384×384, 512×512 and 800×800 images in turn. The batch size is adjusted with the image size to select the optimal model. After pre-training, optimization training is performed on the DeepDR dataset using a step-by-step iterative optimization method. The image sizes are 384×384, 512×512, and 800×800, respectively. The network model parameters are obtained, and the feature layers of the two networks are concatenated and input into the classification network for classification training. The best model of the previous stage is used for progressive training in the order of image size. In the feature extraction network, a convolutional neural network CNN is used for image data, including the use of a 3×3 convolution kernel, a step size of 1, and a padding of 1 for convolution operations. Selection and filtering are performed through pooling layers, including mean pooling, maximum pooling, or pyramid pooling. The ReLU activation function is introduced to increase nonlinear expression. In the recurrent neural network RNN, including the long short-term memory network LSTM, the number of hidden layer units is set to 128, and the update gate calculation formula is z t =σ(W z ·[h t-1 , x t ]+b z ), where z t is the update gate output, W z To update the gate weight matrix, h t-1 is the hidden state at the previous moment, x t is the current input, b z is the bias vector, σ is the sigmoid function; Data fusion and classification steps: The features extracted from the two paths are horizontally fused using a weighted fusion strategy. The fusion formula is F fusion =αF s +βF t +γF a , F fusion Where is the fused feature, α, β, γ are weight coefficients, and α+β+γ=1. The weight value is determined by training optimization. s , F t , F a For different characteristics; The fused features are input into the fully connected layer for classification. The number of neurons in the fully connected layer is determined according to the specific classification task. The softmax function is used to calculate the classification probability. The formula is: where p i is the predicted probability of the i-th category, z i is the output of the fully connected layer neurons, C is the total number of categories, and the final classification result is determined based on the predicted probability.

2. The multi-modal dual-path fusion network structure based on data level fusion according to claim 1 is characterized in that: The rotation angle step size in the image enhancement technology is 1 degree, the scaling step size is 0.05, and the shear angle step size is 5 degrees.

3. The multi-modal dual-path fusion network structure based on data level fusion according to claim 1 is characterized in that: The number of training iterations for each scale in the pre-trained classification network is 50, so that the model can fully learn data features at different scales.

4. The multi-modal dual-path fusion network structure based on data level fusion according to claim 1 is characterized in that: The initial values ​​of the weight coefficients α, β, and γ in the weighted fusion strategy are set to 0.4, 0.3, and 0.3, respectively, and are optimized and adjusted by the gradient descent algorithm during the training process.

5. The multi-modal dual-path fusion network structure based on data level fusion according to claim 1 is characterized in that: The fully connected layer adopts the Dropout technology with a random dropout probability of 0.2 to prevent overfitting and improve the generalization ability of the model.

6. A training method for a multi-modal dual-path fusion network mechanism based on the data level fusion according to any one of claims 1 to 5, characterized in that: The following steps are involved: Data preparation steps: Prepare training data according to the data set construction and preprocessing steps, divide the data set into training set, validation set and test set with a ratio of 7:2:1, and use stratified sampling method for division; Model construction steps: construct the multimodal dual-path fusion network structure and initialize network parameters, including weight matrices and bias vectors of convolutional neural networks and recurrent neural networks; Training steps: Input the training set data into the network and calculate the loss value according to the loss function. The cross entropy loss function is used, and the formula is Where L is the loss value, y i is the true label, p i is the predicted probability, C is the total number of categories, and the network parameters are updated by back propagation using the gradient descent algorithm. The gradient descent algorithm uses the adaptive moment estimation Adam optimizer, whose hyperparameters β1=0.9, β2=0.999, ∈=10 -8 ,The early stopping method is used during the training process. When the validation set loss does not decrease for three consecutive epochs, the training is stopped to prevent overfitting; Evaluation step: Use the test set data to evaluate the trained network and calculate the accuracy, recall and F1 value evaluation indicators. The accuracy formula is: Where I(y i =p i ) is the indicator function, when y i =p i is 1, otherwise it is 0, N is the number of test set samples, and the F1 value formula is Precision is the accuracy rate, Recall is the recall rate, and the network structure or training parameters are adjusted according to the evaluation results to optimize the network performance.

7. The training method of a multimodal dual-path fusion network based on data level fusion according to claim 6 is characterized in that: The batch size of each training in the training step is 32, balancing the utilization of computing resources and the speed of model convergence.

8. The training method of a multimodal dual-path fusion network based on data level fusion according to claim 6 is characterized in that: The patience value in the early stopping method is set to 3 to avoid overtraining that may lead to a decrease in the generalization ability of the model.

9. The training method of a multimodal dual-path fusion network based on data level fusion according to claim 6, characterized in that: In the model building step, the network parameters are initialized using a random initialization method, with a value range of [-0.1, 0.1].

10. The training method of a multimodal dual-path fusion network based on data level fusion according to claim 6, characterized in that: The evaluation step also calculates the precision-recall rate PR curve and the area under the curve AUC indicator.

Images