A multimodal dual-path fusion network structure based on horizontal data fusion

Abstract

This invention discloses an image processing method based on a multimodal dual-path fusion network with horizontal data fusion, relating to the technical field of dual-path fusion network structures. The method includes: dataset construction and preprocessing, selecting and processing multiple datasets for image enhancement and normalization; network structure construction, building a dual-path network, pre-training and optimizing the main backbone network on different datasets, then concatenating feature layers and performing classification training; and data fusion and classification, performing horizontal fusion of dual-path features, using a weighted strategy to determine weights, and inputting the fused features into a fully connected layer for classification using the softmax function. This invention improves the accuracy and robustness of ophthalmic disease diagnosis, enhances generalization ability, optimizes network training, and promotes the development of intelligent medical diagnosis.

Description

Technical Field

[0001] This invention relates to the field of multimodal dual-path fusion network structure technology, and in particular to an image processing method based on a multimodal dual-path fusion network with horizontal data fusion. Background Technology

[0002] In today's digital age, the diversity and complexity of data are increasing, with multimodal data (such as images, text, and audio) widely existing in various fields. How to effectively process and integrate multimodal data and mine the rich information it contains has become a key challenge for many fields.

[0003] In the medical field, especially in the diagnosis of ophthalmic diseases, fundus images contain a wealth of information about eye health. However, fundus image data from different modalities (such as images acquired from different fields of view and different imaging devices) differ in feature representation and distribution. Traditional single-modal analysis methods struggle to fully utilize the complementarity of these data, resulting in limited diagnostic accuracy. Furthermore, existing datasets vary significantly in size, diversity, and annotation quality, making model training prone to overfitting or underfitting, leading to insufficient generalization ability.

[0004] The development of deep learning technology has provided new avenues for multimodal data processing, but traditional models such as convolutional neural networks (CNNs) have many limitations when dealing with multimodal fusion problems. On the one hand, most traditional models are designed primarily for single-modal data and lack effective integration mechanisms when fusing multimodal data, failing to fully leverage the synergistic effects of multimodal information. On the other hand, as model complexity increases, training becomes more difficult, and problems such as gradient vanishing are prone to occur, affecting model optimization and performance improvement.

[0005] Regarding data fusion methods, existing fusion strategies such as decision-level fusion and intermediate fusion cannot effectively integrate multimodal data at low-level features in certain scenarios, limiting the model's deep understanding and utilization of multimodal information. Furthermore, differences in data distribution between different datasets (such as the non-independent and identically distributed problem) severely impact the model's performance when applied across datasets, while existing domain adaptation algorithms are insufficient in handling feature distance relationships between and within categories, making it difficult to meet practical needs.

[0006] In conclusion, there is an urgent need for an innovative multimodal fusion network structure and method that can effectively integrate multimodal data, adapt to the characteristics of different datasets, and improve the accuracy, robustness, and generalization ability of the model, so as to promote the development of multimodal data processing technology in fields such as medical diagnosis. Summary of the Invention

[0007] This invention proposes an image processing method based on a multimodal dual-path fusion network with horizontal data fusion to solve the problems mentioned in the prior art.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: an image processing method based on a multimodal dual-path fusion network with horizontal data fusion, comprising the following steps and components:

[0009] Dataset Construction and Preprocessing Steps: The DeepDR, Aptos, and EyePacs datasets were selected. The DeepDR dataset contains data from over 1000 patients, providing paired images centered on the optic disc and macula, consisting of 1200 training images and 400 test images, with five-class hierarchical labels. The Aptos dataset contains 3662 training images and 1928 test images, with five-class hierarchical labels. The EyePacs dataset is the largest fundus dataset, containing 35129 images. Images with the same hash value but inconsistent labels in the Aptos dataset were deleted, and only one image with both the hash value and label was retained. The EyePacs dataset was proportionally sampled for pre-training.

[0010] Image enhancement processing is performed, including affine transformation, color space transformation, and flipping. Image sizes of 384×384, 512×512, and 800×800 are used. Image normalization is then applied using the following formula: ,in For the normalized data, This is the original data. This represents the mean of the corresponding modal data. The standard deviation is used to keep the data within a similar range of values.

[0011] Network architecture construction steps: Construct a dual-path feature extraction network, one path for images centered on the optic disc and the other for images centered on the macula. Mainstream backbone networks such as VGG, ResNet, Inception, or EfficientNet are adopted. Pre-training is performed on the EyePacs dataset. The pre-training adopts the scale iterative training method, using images of 384×384, 512×512, and 800×800 sizes in turn. The batch size is adjusted according to the image size, and the optimal model is selected.

[0012] After pre-training, optimization training is performed on the DeepDR dataset using a step-by-step iterative optimization method. The image sizes are 384×384, 512×512 and 800×800 respectively to obtain the network model parameters. The feature layers of the two networks are concatenated and input into the classification network for classification training. The best model from the previous stage is used for progressive training according to the image size order.

[0013] In the feature extraction network, a Convolutional Neural Network (CNN) is used for image data. This includes using a 3×3 convolutional kernel with a stride of 1 and padding of 1 for convolution operations. Pooling layers, including mean pooling, max pooling, or pyramid pooling, are used for selection and filtering. The ReLU activation function is introduced to increase non-linear expression. In the Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM) network is included. Assuming 128 hidden units, the update gate calculation formula is... ,in To update the gate output, To update the gate weight matrix, This is the hidden state from the previous moment. For the current input, For bias vectors, It is the sigmoid function;

[0014] Data fusion and classification steps: The features extracted from the two sources are horizontally fused using a weighted fusion strategy. The fusion formula is as follows: , Here, represents the fused features, and α, β, and γ are weight coefficients, where α + β + γ = 1. The weight values ​​are determined through training and optimization. For different characteristics;

[0015] The fused features are input into a fully connected layer for classification. The number of neurons in the fully connected layer is determined based on the specific classification task. The softmax function is used to calculate the classification probability, and the formula is as follows: ,in Let be the predicted probability of class i. is the output of the fully connected layer neurons, C is the total number of categories, and the final classification result is determined based on the predicted probability.

[0016] Furthermore, the rotation angle step size in the image enhancement technology is 1 degree, the scaling step size is 0.05, and the shearing angle step size is 5 degrees.

[0017] Furthermore, the pre-trained classification network undergoes 50 training iterations at each scale, enabling the model to fully learn data features at different scales.

[0018] Furthermore, the initial values ​​of the weight coefficients α, β, and γ in the weighted fusion strategy are set to 0.4, 0.3, and 0.3, respectively, and are optimized and adjusted during the training process using the gradient descent algorithm.

[0019] Furthermore, the fully connected layer employs Dropout technology with a random dropout probability of 0.2 to prevent overfitting and improve the model's generalization ability.

[0020] Furthermore, a training method for a multimodal dual-path fusion network mechanism based on the aforementioned data horizontal fusion includes the following steps:

[0021] Data preparation steps: Prepare training data according to the dataset construction and preprocessing steps, and divide the dataset into training set, validation set and test set in a ratio of 7:2:1. The division adopts stratified sampling method.

[0022] Model construction steps: Construct the multimodal dual-path fusion network structure and initialize the network parameters, including the weight matrix and bias vector of the convolutional neural network and the recurrent neural network;

[0023] Training steps: Input the training set data into the network, calculate the loss value according to the loss function, using the cross-entropy loss function, the formula is as follows: Where L is the loss value, For real labels, To predict probabilities, where C is the total number of classes, the network parameters are updated using backpropagation with the gradient descent algorithm. The gradient descent algorithm employs the Adam optimizer with adaptive moment estimation, and its hyperparameters... , , During training, an early stopping method is used: training is stopped when the validation set loss no longer decreases for three consecutive epochs to prevent overfitting.

[0024] Evaluation steps: Evaluate the trained network using test set data, calculating accuracy, recall, and F1 score as evaluation metrics. The accuracy formula is as follows: ,in For indicator functions, when The value is 1 if it is positive and 0 otherwise, where N is the number of samples in the test set, and the F1 score formula is: Precision is the network's accuracy, and Recall is its recall. The network structure or training parameters are adjusted based on the evaluation results to optimize network performance.

[0025] Furthermore, the batch size for each training step is 32, balancing computational resource utilization and model convergence speed.

[0026] Furthermore, the patience value in the early stopping method is set to 3 to avoid overtraining that could lead to a decrease in the model's generalization ability.

[0027] Furthermore, in the model construction step, the network parameters are initialized using a random initialization method, with values ​​ranging from -0.1 to 0.1.

[0028] Furthermore, the evaluation step also calculates the precision-recall (PR) curve and the area under the curve (AUC) metric.

[0029] Compared with existing technologies, the beneficial effects of this invention are:

[0030] Firstly, it improves diagnostic accuracy. Through a carefully designed multimodal dual-path fusion network structure, it fully integrates information from fundus images in different visual fields, achieving collaborative representation of multimodal data. This provides a more comprehensive and accurate basis for the diagnosis of ophthalmic diseases such as diabetic retinopathy (DR), significantly improving the accuracy of diagnosis.

[0031] Secondly, it enhances model robustness. The improved domain adaptation algorithm effectively addresses the distributional differences between datasets, enabling the model to maintain stable performance under different data distributions, reducing diagnostic errors caused by data changes, and enhancing the model's adaptability to complex real-world scenarios.

[0032] Third, it enhances generalization ability. The selection and augmentation of diverse datasets, along with reasonable network training strategies, enable the model to learn general features from a large number of different data, avoiding overfitting. This allows for accurate predictions even on new and unseen data, making it applicable to a wider range of fundus image data and providing strong support for clinical applications.

[0033] Fourth, optimize network training. First, initialize network parameters using the optimal model from single-modal training to reduce the difficulty of dual-path fusion training, improve training efficiency, and save computational resources and time. Simultaneously, a progressive training approach and reasonable parameter settings ensure that the network model can be gradually optimized to achieve better performance.

[0034] Fifth, it promotes the development of intelligent medical diagnosis. This patent provides new technical means and ideas for the intelligent diagnosis of ophthalmic diseases, which helps to improve the efficiency and quality of medical diagnosis, promotes the development of the medical field towards intelligence and precision, and is of great significance to improving patients' medical services. Attached Figure Description

[0035] Figure 1 This is a schematic block diagram of the steps of the image processing method based on a multimodal dual-path fusion network with horizontal data fusion proposed in this invention.

[0036] Figure 2 This is a schematic block diagram illustrating the sub-steps of dataset construction and preprocessing proposed in this invention;

[0037] Figure 3 This is a schematic block diagram illustrating the sub-steps of building the network structure proposed in this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0040] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified. Furthermore, the terms "installed," "connected," and "linked" should be interpreted broadly; for example, they may refer to a fixed connection, a detachable connection, or an integral connection; they may refer to a mechanical connection or an electrical connection; they may refer to a direct connection or an indirect connection through an intermediate medium; and they may refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances. The invention will now be described in further detail with reference to the accompanying drawings.

[0041] Reference Figure 1-3 An image processing method based on a multimodal dual-path fusion network with horizontal data fusion, comprising the following steps and components:

[0042] Dataset construction and preprocessing steps:

[0043] The DeepDR, Aptos, and EyePacs datasets were selected. The DeepDR dataset contains data from over 1000 patients, providing paired images centered on the optic disc and macula, consisting of 1200 training images and 400 test images with five-class hierarchical labels. The Aptos dataset contains 3662 training images and 1928 test images with five-class hierarchical labels. The EyePacs dataset is the largest fundus dataset, containing 35129 images. Images with the same hash value but inconsistent labels in the Aptos dataset were removed; only one image was retained for images with both the same hash value and label. The EyePacs dataset was proportionally sampled for pre-training.

[0044] Image enhancement processes are performed, including affine transformation (rotation angle -180 to 180 degrees, scaling ratio 0.5 to 1.5, cropping -50 to 50 degrees), color space transformation (brightness 0.5 to 2, contrast 0.5 to 1.2, saturation 0.8 to 1.2), flipping (horizontal and vertical flip probabilities are both 0.5), and rotation (0 to 360 degrees). Image sizes of 384×384, 512×512, and 800×800 are used. Image normalization is performed using the following formula: ,in For the normalized data, This is the original data. This represents the mean of the corresponding modal data. The standard deviation is used to keep the data within a similar range of values.

[0045] Network architecture setup steps:

[0046] A dual-path feature extraction network was constructed, with one path targeting images centered on the optic disc and the other targeting images centered on the macula. Mainstream backbone networks (such as VGG, ResNet, Inception, or EfficientNet) were adopted, and pre-trained on the EyePacs dataset using a scale-iterative training method. Images of sizes 384×384, 512×512, and 800×800 were used sequentially, with the batch size adjusted according to the image size. The optimal model was then selected.

[0047] After pre-training, optimization training was performed on the DeepDR dataset using a stepwise iterative optimization approach. The image sizes were 384×384, 512×512, and 800×800 in sequence to obtain the network model parameters. The feature layers of the two networks were concatenated and input into the classification network for classification training. The same image size order was used for progressive training with the best model from the previous stage.

[0048] In feature extraction networks, convolutional neural networks (CNNs) are used for image data. For example, a 3×3 convolution kernel with a stride of 1 and padding of 1 is used for convolution operations. Pooling layers (such as mean pooling, max pooling, or pyramid pooling) are used for selection and filtering. Activation functions (such as ReLU) are introduced to increase non-linear expression. In recurrent neural networks (RNNs), such as Long Short-Term Memory (LSTM) networks, assuming 128 hidden layer units, the update gate calculation formula is... ,in To update the gate output, To update the gate weight matrix, This is the hidden state from the previous moment. For the current input, For bias vectors, This is the sigmoid function.

[0049] Data fusion and classification steps:

[0050] The features extracted from the two paths are horizontally fused using a weighted fusion strategy. The fusion formula is as follows: , Here, represents the fused features, and α, β, and γ are weight coefficients, where α + β + γ = 1. The weight values ​​are determined through training and optimization. For different characteristics.

[0051] The fused features are input into a fully connected layer for classification. The number of neurons in the fully connected layer is determined based on the specific classification task. The softmax function is used to calculate the classification probability, and the formula is as follows: ,in Let be the predicted probability of class i. is the output of the fully connected layer neurons, C is the total number of categories, and the final classification result is determined based on the predicted probability.

[0052] In image enhancement techniques, the rotation angle step size is set to 1 degree. This means that when performing rotation enhancement on an image, the image will rotate in increments of 1 degree, ranging from -180 degrees to 180 degrees. This fine step size setting generates a rich variety of rotation angles, ensuring that the image is fully transformed in different directions, thereby increasing data diversity and allowing the model to learn the feature invariance of the image under various rotational states. The scaling step size is 0.05, ranging from 0.5 to 1.5. The model can learn the feature changes of the image at different scaling degrees, helping to improve the model's adaptability to images of different sizes. The shearing angle step size is 5 degrees, ranging from -50 degrees to 50 degrees, causing different degrees of distortion and deformation in the image, further enriching the variations in the image. These step size settings work together to effectively expand the dataset without introducing excessive noise or destroying key image information, enabling the model to learn more representative features from data with multiple angles, scales, and degrees of deformation, thus enhancing the model's generalization ability.

[0053] In the pre-training classification network, the number of training iterations at each scale was set to 50. During the pre-training phase, the model was first trained using 384×384 images. During these 50 iterations, the model calculated the prediction results through forward propagation, compared them with the true labels to obtain the loss value, and then adjusted the network parameters based on the loss value using the backpropagation algorithm. After multiple iterations, the model gradually learned the data features in the images at this scale, such as vascular structures and lesion features in fundus images. Next, 512×512 images were used for training, again with 50 iterations, allowing the model to further adapt to the feature representation of larger-sized images and learn richer details. Finally, 800×800 images were used for 50 training iterations, enabling the model to understand data features within a larger field of view and capture more macroscopic image patterns and feature relationships. Through 50 thorough training iterations at each scale, the model can comprehensively and deeply learn the intrinsic features and patterns of data at different scales, laying a solid foundation for subsequent optimization training and accurate classification.

[0054] In this invention, the initial values ​​of the weight coefficients , , and in the weighted fusion strategy are set to 0.4, 0.3, and 0.3, respectively. During the training process, they are optimized and adjusted using the gradient descent algorithm to improve the fusion effect.

[0055] The fully connected layer employs Dropout technology with a random dropout probability of 0.2. During neuron propagation in the fully connected layer, each neuron has a 0.2 probability of being randomly dropped in each training iteration. For example, assuming a fully connected layer has 1000 neurons, approximately 200 neurons will be randomly set to 0 in a single training iteration, not participating in the forward or backward propagation process. This is done to prevent excessive dependence between neurons and avoid overfitting. During training, due to the random dropout of some neurons, the network structure undergoes slight changes in each iteration, preventing the model from overlearning specific patterns in the training data. This encourages neurons to learn more general and robust features, thereby improving the model's ability to generalize to unseen data. In the testing phase, all neurons participate in computation, but their weights have already been optimized during training using Dropout technology, enabling the model to classify or predict new data more accurately, enhancing the model's reliability and effectiveness in real-world applications.

[0056] This invention discloses a training method for a multimodal dual-path fusion network based on horizontal data fusion, comprising the following steps:

[0057] Data preparation steps: Prepare training data according to the dataset construction and preprocessing steps, and divide the dataset into training set, validation set and test set in a ratio of 7:2:1. The division adopts stratified sampling method to ensure that the data categories are evenly distributed in each set.

[0058] Model construction steps: Construct the multimodal dual-path fusion network structure and initialize the network parameters, including the weight matrix and bias vector of the convolutional neural network and the recurrent neural network.

[0059] Training steps: Input the training set data into the network and calculate the loss value according to the loss function. The cross-entropy loss function is used, and the formula is as follows: Where L is the loss value, For real labels, To predict probabilities, C represents the total number of classes. The network parameters are updated via backpropagation using the gradient descent algorithm, which employs the Adaptive Moments Estimation (Adam) optimizer. Its hyperparameters... , , During training, an early stopping method is used. Training is stopped when the validation set loss no longer decreases for three consecutive epochs to prevent overfitting.

[0060] Evaluation steps: Evaluate the trained network using test set data, calculating evaluation metrics such as accuracy, recall, and F1 score. The accuracy formula is as follows: ,in For indicator functions, when The value is 1 if it is positive and 0 otherwise, where N is the number of samples in the test set, and the F1 score formula is: Precision is the network's accuracy, and Recall is its recall. The network structure or training parameters are adjusted based on the evaluation results to further optimize the network performance.

[0061] In the training process, the batch size for each training iteration is set to 32. The choice of batch size was determined after carefully considering the balance between computational resource utilization and model convergence speed. Smaller batch sizes, such as 16 or smaller, increase computational overhead because more frequent model parameter updates are required, but may allow the model to capture local features of the data more quickly during training; however, this could also lead to unstable convergence. Larger batch sizes, such as 64 or larger, while reducing the frequency of parameter updates and lowering computational resource consumption, may slow down model convergence or even cause it to get stuck in local optima. After multiple comparative experiments, a batch size of 32, under existing hardware conditions (such as a specific GPU model and memory capacity), ensures efficient utilization of computational resources, allowing the training process to be completed within an acceptable timeframe, while also enabling the model to converge at a relatively stable and fast speed. This ensures that the model can fully learn the feature information in the data during training, gradually optimize model parameters, and improve the model's accuracy and generalization ability.

[0062] The patience value in the early stopping mechanism is set to 3. Early stopping is an important means of preventing model overtraining. During training, the model's performance on the validation set changes with each training epoch. When the validation set loss stops decreasing for several consecutive epochs, it indicates that the model may have begun to overfit, and continued training may lead to a decline in the model's generalization ability. By setting the patience value to 3, the training process will stop when the validation set loss does not improve for three consecutive epochs. This avoids overtraining the model in the overfitting region, wasting computational resources, while maintaining the model's generalization performance on unseen data. In actual training, after each epoch, the validation set loss is calculated and compared with previous records. If it does not decrease for three consecutive epochs, the early stopping mechanism is triggered, the current optimal model parameters are saved, and the training process ends. This ensures that the model can fully learn the data features during training without losing its generalization ability due to overtraining.

[0063] In the model building step, network parameters are initialized using a random initialization method, with values ​​ranging from [-0.1, 0.1]. Random initialization is a crucial operation for providing a good starting point for network training. In neural networks, the initial values ​​of parameters affect the model's training process and final performance. Improper initialization can cause the model to get stuck in a local optimum early in training, or make the training process very slow. By randomly initializing parameters within the range of [-0.1, 0.1], the symmetry of the model can be broken, allowing different neurons to have different initial states at the start of training, thus preventing all neurons from learning the same features in the early stages of training. Specifically, parameters such as the convolution kernel weights and biases in convolutional neural networks, and the weight matrix and bias vectors in recurrent neural networks, are all randomly selected within this range. This random initialization method provides diverse starting points for the model in subsequent training, enabling the model to begin learning data features under different initial conditions, gradually adjusting parameters through backpropagation, converging towards the optimal solution, and improving the model's learning ability and generalization performance.

[0064] In the evaluation process, in addition to calculating common metrics such as precision and recall, the area under the precision-recall curve (PR curve) (AUC) can also be calculated to more comprehensively assess model performance. Precision and recall measure the predictive effectiveness of a model from different perspectives, but they may have limitations in some cases and cannot fully reflect the model's performance at different thresholds. The PR curve, on the other hand, can show how the precision changes under different recall levels. By calculating the AUC under the PR curve, the overall performance of the model under different threshold settings can be comprehensively evaluated. The AUC value ranges from 0 to 1; the larger the AUC value, the stronger the model's ability to distinguish between positive and negative examples. When calculating the AUC, the samples are first sorted according to the model's predicted probabilities, then the precision and recall at different thresholds are calculated, the PR curve is plotted, and finally, the area under the curve is calculated using mathematical methods such as integration. This metric can more comprehensively evaluate the model's performance when dealing with imbalanced datasets or when different prediction performance requirements apply to different categories. It provides richer information for model optimization and improvement, helps researchers better understand the model's strengths and weaknesses, and thus take targeted measures to further improve model performance.

[0065] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An image processing method based on a multimodal dual-path fusion network with horizontal data fusion, characterized in that, Includes the following steps and components: Dataset Construction and Preprocessing Steps: The DeepDR, Aptos, and EyePacs datasets were selected. The DeepDR dataset contains data from over 1000 patients, providing paired images centered on the optic disc and macula, consisting of 1200 training images and 400 test images, with five-class hierarchical labels. The Aptos dataset contains 3662 training images and 1928 test images, with five-class hierarchical labels. The EyePacs dataset is the largest fundus dataset, containing 35129 images. Images with the same hash value but inconsistent labels in the Aptos dataset were deleted, and only one image with both the hash value and label was retained. The EyePacs dataset was proportionally sampled for pre-training. Image enhancement processing is performed, including affine transformation, color space transformation, and flipping. Image sizes of 384×384, 512×512, and 800×800 are used. Image normalization is then applied using the following formula: ,in For the normalized data, The original data, This represents the mean of the corresponding modal data. The standard deviation is used to keep the data within a similar range of values. Network architecture construction steps: Construct a dual-path feature extraction network, one path for images centered on the optic disc and the other for images centered on the macula. Mainstream backbone networks such as VGG, ResNet, Inception, or EfficientNet are adopted. Pre-training is performed on the EyePacs dataset. The pre-training adopts the scale iterative training method, using images of 384×384, 512×512, and 800×800 sizes in turn. The batch size is adjusted according to the image size, and the optimal model is selected. After pre-training, optimization training is performed on the DeepDR dataset using a step-by-step iterative optimization method. The image sizes are 384×384, 512×512 and 800×800 respectively to obtain the network model parameters. The feature layers of the two networks are concatenated and input into the classification network for classification training. The best model from the previous stage is used for progressive training according to the image size order. In the feature extraction network, a Convolutional Neural Network (CNN) is used for image data. This includes using a 3×3 convolutional kernel with a stride of 1 and padding of 1 for convolution operations. Pooling layers, including mean pooling, max pooling, or pyramid pooling, are used for selection and filtering. The ReLU activation function is introduced to increase non-linear expression. In the Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM) network is included. Assuming 128 hidden units, the update gate calculation formula is... ,in To update the gate output, To update the gate weight matrix, This is the hidden state from the previous moment. For the current input, For bias vectors, It is the sigmoid function; Data fusion and classification steps: The features extracted from the two sources are horizontally fused using a weighted fusion strategy. The fusion formula is as follows: , Here, represents the fused features, and α, β, and γ are weight coefficients, where α + β + γ = 1. The weight values ​​are determined through training and optimization. For different characteristics; The fused features are input into a fully connected layer for classification. The number of neurons in the fully connected layer is determined based on the specific classification task. The softmax function is used to calculate the classification probability, and the formula is as follows: ,in Let be the predicted probability of class i. is the output of the fully connected layer neurons, C is the total number of categories, and the final classification result is determined based on the predicted probability.

2. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 1, characterized in that, The image enhancement technique described above uses a rotation angle step of 1 degree, a scaling step of 0.05, and a shearing angle step of 5 degrees.

3. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 1, characterized in that, The pre-trained classification network performs 50 training iterations at each scale, allowing the model to fully learn data features at different scales.

4. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 1, characterized in that, The initial values ​​of the weight coefficients α, β, and γ in the weighted fusion strategy are set to 0.4, 0.3, and 0.3, respectively, and are optimized and adjusted during training using the gradient descent algorithm.

5. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 1, characterized in that, The fully connected layer employs Dropout technology with a random dropout probability of 0.2 to prevent overfitting and improve the model's generalization ability.

6. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 1, characterized in that, Also includes: Data preparation steps: Prepare training data according to the dataset construction and preprocessing steps, and divide the dataset into training set, validation set and test set in a ratio of 7:2:

1. The division adopts stratified sampling method. Model construction steps: Construct the multimodal dual-path fusion network structure and initialize the network parameters, including the weight matrix and bias vector of the convolutional neural network and the recurrent neural network; Training steps: Input the training set data into the network, calculate the loss value according to the loss function, using the cross-entropy loss function, the formula is as follows: Where L is the loss value, For real labels, To predict probabilities, where C is the total number of classes, the network parameters are updated using backpropagation with the gradient descent algorithm. The gradient descent algorithm employs the Adam optimizer with adaptive moment estimation, and its hyperparameters... , , During training, an early stopping method is used: training is stopped when the validation set loss no longer decreases for three consecutive epochs to prevent overfitting. Evaluation steps: Evaluate the trained network using test set data, calculating accuracy, recall, and F1 score as evaluation metrics. The accuracy formula is as follows: ,in For indicator functions, when The value is 1 if it is positive and 0 otherwise, where N is the number of samples in the test set, and the F1 score formula is: Precision is the network's accuracy, and Recall is its recall. The network structure or training parameters are adjusted based on the evaluation results to optimize network performance.

7. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 6, characterized in that, The batch size for each training step is 32, balancing computational resource utilization and model convergence speed.

8. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 6, characterized in that, The patience value in the early stopping method is set to 3 to avoid overtraining that could lead to a decrease in the model's generalization ability.

9. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 6, characterized in that, In the model construction step, the network parameters are initialized using a random initialization method, with values ​​ranging from -0.1 to 0.

1.

10. The image processing method based on a multimodal dual-path fusion network with horizontal data fusion according to claim 6, characterized in that, The evaluation steps also include calculating the precision-recall (PR) curve and the area under the curve (AUC).

Images