Strawberry disease multi-modal visual question-answering method based on improved LLaVA

Abstract

The invention discloses a strawberry disease multi-modal visual question and answer method based on improved LLaVA, and belongs to the technical field of artificial intelligence, and the method comprises the steps: obtaining a plurality of strawberry disease plant images, and constructing a strawberry disease visual question and answer data set based on the strawberry disease plant images; the LLaVA model is improved; performing fine tuning by using an improved LLaVA model to obtain a strawberry disease visual question and answer model; performing question answering according to the strawberry disease visual question answering model. According to the method, the LLaVA model is improved, so that the recognition capability of strawberry disease plants is improved, and the accuracy of diagnosis and decision making of the model on strawberry diseases is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multimodal visual question answering method for strawberry diseases based on an improved LLaVA. Background Technology

[0002] In recent years, with the development of computer vision and natural language processing technologies, visual question answering (VQA) technology based on multimodal large models has gradually attracted attention in the field of agricultural pest and disease detection. Traditional strawberry pest and disease detection mainly relies on manual inspection, which suffers from low efficiency, strong subjectivity, and difficulty in large-scale application. Among existing technologies, deep learning-based object detection algorithms (such as the YOLO series) have made some progress in pest and disease identification, but they mostly focus on single image classification or localization, lacking the ability to interact with users through natural language, and cannot directly provide farmers with decision support such as prevention and control suggestions. Summary of the Invention

[0003] The purpose of this invention is to provide a multimodal visual question answering method for strawberry diseases based on an improved LLaVA, in order to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, this invention provides a multimodal visual question-answering method for strawberry diseases based on an improved LLaVA, comprising the following steps:

[0005] S1. Obtain images of strawberry plants with various diseases and construct a visual question-and-answer dataset for strawberry diseases based on the images of strawberry plants with diseases.

[0006] S2. Improve the LLaVA model;

[0007] S3. Based on the strawberry disease visual question answering dataset, the improved LLaVA model was fine-tuned to obtain the strawberry disease visual question answering model.

[0008] S4. Conduct a question-and-answer session based on the visual question-and-answer model for strawberry diseases.

[0009] Preferably, the strawberry disease visual question-answering dataset includes closed-set question-answering and open-set question-answering for images of strawberry diseased plants, wherein the closed-set question-answering is accuracy question-answering.

[0010] Preferably, the improvement of the LLaVA model in step S2 includes: retaining the visual encoder, visual-language connector and large language model in the LLaVA model, and introducing a dual-view pruning module and a dual-view enhancement module. The visual-language connector is located between the visual encoder and the large language model, the dual-view pruning module is set before the visual encoder, and the dual-view enhancement module is set after the visual encoder.

[0011] Preferably, fine-tuning the improved LLaVA model in step S3 includes:

[0012] S31. Divide the strawberry disease visual question-and-answer dataset into two training sets according to the proportions.

[0013] S32. Freeze the weights of the visual encoder and the large language model, select a training set for training, and align the strawberry diseased plant images with text features.

[0014] S33. Unfreeze the weights of the large language model, use another training set for instruction tuning, fine-tune the improved LLaVA model based on preset instructions, and optimize the model's reasoning and decision-making ability in the strawberry disease task.

[0015] Preferably, step S4 specifically includes:

[0016] S41. Input images of diseased strawberry plants into the strawberry disease visual question-and-answer model;

[0017] S42, the dual-view cropping module segments high-resolution strawberry diseased plant images into sub-images from local and global views. Local view cropping segments the high-resolution image into non-overlapping sub-images according to the preset resolution of the visual encoder. Global view cropping uses a specific step size mode to interleave sampling pixels and generate sub-images containing global layout information to capture the spatial relationship between objects.

[0018] S43. The visual encoder extracts features from each sub-image. The sub-image features are recombined based on 2D position priors to generate a comprehensive set of high-resolution local and global features.

[0019] S44. The dual-view enhancement module promotes the interaction between local and global features through two symmetrical processes: global enhancement and local enhancement. Global enhancement injects local detail features into global features through a cross-attention mechanism; local enhancement integrates global contextual information into local features, making the two types of features complementary to each other.

[0020] S45. The refined visual markers and the text markers of the instructions are concatenated and input into the large language model to generate answers by marker one by one.

[0021] Preferably, the specific processes for local view cropping and global view cropping in step S42 include:

[0022] Local perspective cropping:

[0023] Given high-resolution images of diseased strawberry plants First, determine the size (W) of the given high-resolution image of a diseased strawberry plant. HR ×H HR ) and the size expected by the pre-trained visual encoder (W)LR ×H LR The relationship between ) is expressed by the formula:

[0024]

[0025] in, This represents the floor function, which rounds down to the nearest integer. and These correspond to the number of local sub-images along the width and height of the high-resolution image, respectively;

[0026] High-resolution image I is divided into Local sub-images The size of each local sub-image is W. LR ×H LR For each sub-image in The formulas for calculating row and column positions are:

[0027]

[0028] Where mod represents the modulo operation, c loc Indicates column position; r loc Indicates the row position;

[0029] Each local sub-image The bounding box formula is:

[0030]

[0031] Global view cropping:

[0032] Determine the size (W) of the input high-resolution image of a diseased strawberry plant. HR ×H HR ) and the size (W) used by the pre-trained visual encoder LR ×H LR The direct relationship is expressed by the formula:

[0033]

[0034] In the formula, These correspond to the number of global sub-images along the width and height of the high-resolution image, respectively;

[0035] For the i-th row and j-th column of the global sub-image, the set of pixel indices Defined as:

[0036]

[0037] Where (x, y) corresponds to the pixel index within the high-resolution image. Let m represent the set of natural numbers, where m represents the set of natural numbers between 0 and the width of the sub-image, and n represents the set of natural numbers between 0 and the height of the sub-image.

[0038] at last, Each pixel (u, v) in the sub-image is mapped back to the high-resolution image I, using the formula:

[0039]

[0040] Preferably, step S44 specifically includes the following process:

[0041] To perform global view enhancement, firstly, crop both local and global view features from the global perspective. The formula is as follows:

[0042]

[0043] in, This indicates a global perspective clipping operation. and These represent the global and local sub-features generated by cropping from this global perspective, respectively.

[0044] Perform a cross-attention operation between the corresponding local and global sub-features, injecting local sub-feature information into the global sub-features, as shown in the formula:

[0045]

[0046] in, It is a learnable embedding matrix. It is an attention mapping between local sub-features and global sub-features. It is the i-th global enhancer feature;

[0047] Finally, enhance the sub-feature set Combining to form global enhanced features

[0048] To enhance local perspective, firstly, local and global perspective features are cropped into sub-features using a local cropping operation, as shown in the formula:

[0049]

[0050] in, This represents the sub-features obtained from a partial perspective clipping operation. and These represent the global and local sub-features generated from this local cropping, respectively.

[0051] A cross-attention mechanism is employed to promote interaction between local and global sub-features, leveraging global context to enhance local features. The formula is as follows:

[0052]

[0053] in, These are learnable embedding matrices for querying, key-value transformation, and attention maps. Capture the attention mapping between local and global sub-features. It is the i-th obtained local enhancer feature;

[0054] Finally, the local enhancement sub-feature set Aggregated to form local enhancement features

[0055] Feature fusion is performed by first using two independent embedding layers to reduce the dimensionality of the features, as shown in the formula:

[0056]

[0057] in, These are the learnable projection matrices for global and local features, respectively;

[0058] The embedded global and local features are concatenated along the channel dimension to form a dual-enhanced feature, as shown in the formula:

[0059]

[0060] Where [·;·] represents cascading operations along the channel dimension.

[0061] Therefore, the present invention employs the above-mentioned multimodal visual question-answering method for strawberry diseases based on improved LLaVA, which has the following beneficial effects:

[0062] (1) By constructing a dedicated visual question-and-answer dataset for strawberry diseases, the model can learn the unique features and diagnostic logic of strawberry diseases in depth, which greatly improves the accuracy of diagnosis and the reliability of decision-making for strawberry diseases, and provides more professional technical support for disease diagnosis in actual planting scenarios.

[0063] (2) The dual-view cropping and dual-view enhancement methods were used to improve the model resolution limitation, significantly enhance the model’s ability to capture and identify small lesions on strawberry plants, enable the model to distinguish similar diseases, and further improve the feasibility of early intervention for diseases.

[0064] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0065] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0066] Figure 2This is a structural diagram of the improved LLaVA model according to an embodiment of the present invention;

[0067] Figure 3 This is a schematic diagram of the dual-view cropping module structure according to an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of the dual-view enhancement module structure according to an embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can be arranged and designed in various different configurations. In the description of the present invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicating orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationships commonly used when the product is in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention.

[0070] Example

[0071] Reference Figure 1-4 This invention provides a multimodal visual question-answering method for strawberry diseases based on an improved LLaVA, the steps of which include:

[0072] S1. Obtain images of strawberry plants affected by seven diseases (strawberry angular leaf spot, strawberry anthracnose, flower blight, leaf spot, strawberry fruit powdery mildew, strawberry leaf powdery mildew, and gray mold). Construct a strawberry disease visual question-and-answer dataset based on these images. The dataset includes closed-set questions and answers, which are accuracy-based questions including multiple choice, true / false, and short answer questions, such as "Is the strawberry in the picture infected with powdery mildew?" and open-set questions and answers, including plant symptoms, disease diagnosis, and control methods, such as "What disease does the strawberry in the picture have, and how can this disease be controlled?".

[0073] It is important to note that when using the strawberry disease visual question-and-answer dataset, a combination of manual annotation and GPT-assisted annotation should be used to ensure the accuracy of the question-and-answer pairs. For professional agricultural knowledge, the Chinese Academy of Agricultural Sciences Pest and Disease Database should be consulted.

[0074] S2. Improve the LLaVA model. Specifically:

[0075] Based on LLaVA 1.5 (based on Vicuna-7B), the visual encoder, visual-language connector and large language model in the LLaVA model are retained, and a dual-view pruning module and a dual-view enhancement module are introduced. The visual-language connector is located between the visual encoder and the large language model, the dual-view pruning module is set before the visual encoder, and the dual-view enhancement module is set after the visual encoder.

[0076] S3. Based on the strawberry disease visual question-and-answer dataset, the improved LLaVA model was fine-tuned to obtain the strawberry disease visual question-and-answer model. The fine-tuning process included:

[0077] S31. Divide the strawberry disease visual question-and-answer dataset into two training sets according to the proportions.

[0078] S32. Freeze the weights of the visual encoder and the large language model, select a training set for training, and align the strawberry diseased plant images with text features.

[0079] S33. Unfreeze the weights of the large language model, use another training set for instruction tuning, fine-tune the improved LLaVA model based on preset instructions, and optimize the model's reasoning and decision-making ability in the strawberry disease task.

[0080] S4. Conduct question-and-answer sessions based on the strawberry disease visual question-and-answer model, specifically including:

[0081] S41. Input images of diseased strawberry plants into the strawberry disease visual question-and-answer model.

[0082] The S42 dual-view cropping module segments high-resolution strawberry diseased plant images into sub-images from local and global perspectives, balancing detail preservation with global context capture. Local view cropping segments the high-resolution image into non-overlapping sub-images according to the visual encoder's preset resolution, ensuring each sub-image contains continuous local details (such as lesion areas on strawberry leaves). By preserving pixel-level continuous information, it lays the foundation for fine feature extraction. Global view cropping uses a specific step-size pattern to interleave pixel sampling, generating sub-images containing global layout information. This captures the spatial relationships between objects (such as the distribution of lesions on leaves), avoiding global context breaks caused by local cropping. The sub-image resolution is 336×336.

[0083] S43. The visual encoder extracts features from each sub-image. The sub-image features are recombined based on 2D position priors to generate a comprehensive set of high-resolution local and global features.

[0084] S44. The dual-view enhancement module promotes the interaction between local and global features through two symmetrical processes: global enhancement and local enhancement. Global enhancement injects local detail features into global features through a cross-attention mechanism; while local enhancement integrates global contextual information into local features, making the two types of features complementary to each other.

[0085] S45. Reduce the number of visual tags by using an average pooling layer, concatenate the refined visual tags with the text tags of the instructions, and input them into the large language model to generate answers by tagging them one by one.

[0086] Specifically, the local view cropping and global view cropping processes in step S42 include:

[0087] Local perspective cropping:

[0088] Local viewpoint cropping aims to systematically extract smaller regions from high-resolution images while preserving detailed and continuous visual information. This method ensures that each sub-image retains the high-resolution details required for accurate analysis and representation. By strategically segmenting high-resolution images into local sub-images, the complex details of the original image are effectively preserved, allowing the model to observe minute details such as lesions more clearly and reducing the probability of hallucinations or misclassification as other diseases. The process includes:

[0089] Given high-resolution images of diseased strawberry plants First, determine the size (W) of the given high-resolution image of a diseased strawberry plant. HR ×H HR ) and the size expected by the pre-trained visual encoder (W) LR ×H LR The relationship between ) is expressed by the formula:

[0090]

[0091]

[0092] in, This represents the floor function, which rounds down to the nearest integer. and These correspond to the number of local sub-images along the width and height of the high-resolution image, respectively;

[0093] Therefore, the high-resolution image I is divided into Local sub-images The size of each local sub-image is W. LR ×H LR For each sub-image in The row and column positions are determined using the following formula:

[0094]

[0095] Where mod represents the modulo operation, c loc Indicates column position; r loc Indicates the row position;

[0096] Each local sub-image The bounding box formula is:

[0097]

[0098] This process ensures that each local sub-image It contains continuous and detailed visual information from high-resolution images, thus maintaining the integrity and quality of the images for local lesion analysis.

[0099] Global view cropping:

[0100] Global perspective cropping aims to capture broader contextual information from high-resolution images, primarily information such as leaf vein extension and lesion distribution, to preserve spatial relationships between objects. This method ensures that the model maintains an understanding of both micro and macro details in the extracted sub-images, thereby promoting the integration of comprehensive global contextual information and preventing the model from focusing only on local lesion features while ignoring the overall information of the leaf or fruit, which is detrimental to disease diagnosis. The process includes:

[0101] Determine the size (W) of the input high-resolution image of a diseased strawberry plant. HR ×H HR ) and the size (W) used by the pre-trained visual encoder LR ×H LR The direct relationship is expressed by the formula:

[0102]

[0103] In the formula, These correspond to the number of global sub-images along the width and height of the high-resolution image, respectively;

[0104] For the i-th row and j-th column of the global sub-image, the set of pixel indices Defined as:

[0105]

[0106] Where (x, y) corresponds to the pixel index within the high-resolution image. Let m represent the set of natural numbers, where m represents the set of natural numbers between 0 and the width of the sub-image, and n represents the set of natural numbers between 0 and the height of the sub-image.

[0107] at last, Each pixel (u, v) in the sub-image is mapped back to the high-resolution image I, using the formula:

[0108]

[0109] Global view cropping effectively divides strawberry virus leaf images into sub-images, which encapsulate the global view by interleaving pixels from different regions.

[0110] Specifically, step S44 includes the following process:

[0111] (1) Global view enhancement: First, crop features from both the local and global views from the global view, using the following formula:

[0112]

[0113] in, This indicates a global perspective clipping operation. and These represent the global and local sub-features generated by cropping from this global perspective, respectively.

[0114] Perform a cross-attention operation between the corresponding local and global sub-features, injecting local sub-feature information into the global sub-features, as shown in the formula:

[0115]

[0116] in, It is a learnable embedding matrix. It is an attention mapping between local sub-features and global sub-features. It is the i-th global enhancer feature;

[0117] Finally, enhance the sub-feature set Combining to form global enhanced features This process ensures that the resulting feature representations capture rich multi-scale information by effectively combining local details with global context, thereby enhancing the overall robustness and expressiveness of the model.

[0118] (2) Perform local perspective enhancement: First, crop the local and global perspective features into sub-features using a local cropping operation, as shown in the formula:

[0119]

[0120] in, This represents the sub-features obtained from a partial perspective clipping operation. and These represent the global and local sub-features generated from this local cropping, respectively.

[0121] A cross-attention mechanism is employed to promote interaction between local and global sub-features, leveraging global context to enhance local features. The formula is as follows:

[0122]

[0123] in, These are learnable embedding matrices for querying, key-value transformation, and attention maps. Capture the attention mapping between local and global sub-features. It is the i-th obtained local enhancer feature.

[0124] Finally, the local enhancement sub-feature set Aggregated to form local enhancement features This process ensures that local features are enriched by supplemented global contextual information, thereby providing a more comprehensive and robust representation of the original high-resolution image.

[0125] (3) Feature fusion: First, two independent embedding layers are used to reduce the dimensionality of the features, as shown in the formula:

[0126]

[0127] in, These are the learnable projection matrices for global and local features, respectively;

[0128] The embedded global and local features are concatenated along the channel dimension to form a dual-enhanced feature, as shown in the formula:

[0129]

[0130] Where [·;·] represents cascading operations along the channel dimension.

[0131] Example 1

[0132] To verify the accuracy of the fine-tuned and improved LLaVA model, it was compared with other models. The comparison results are shown in Table 1. ScienceQA-img was used to test the model's ability to interpret and generate accurate context-aware responses; OKVQA tested the model's ability to generate responses based on prior knowledge and visual understanding; SEEDBench comprehensively evaluated the performance of multimodal large-scale language models in various multimodal tasks; MMBench evaluated the model's multimodal capabilities by providing a structured and comprehensive testing environment; MMBench-CN, the Chinese version of MMBench-Bench, tested the model's multilingual and cross-cultural adaptability; LLaVA-Bench-in-the-wild and MM-VET evaluated the model's ability to participate in visual question answering in various tasks and tested the model's applicability in the real world; POPE tested the hallucination phenomenon that the model is prone to produce. As shown in Table 1, the improved model has higher accuracy.

[0133] Table 1 shows the comparison results with other models.

[0134]

[0135]

[0136] Example 2

[0137] To verify the model's adaptability, a comparative experiment was conducted for the specific scenario of visual question answering about strawberry diseases. The comparative experiment is shown in Table 2. As can be seen from Table 2, the improved model has a higher accuracy.

[0138] Table 2. Comparison of test results of the improved model on the strawberry disease task.

[0139]

[0140] Therefore, the present invention adopts the above-mentioned multimodal visual question answering method for strawberry diseases based on improved LLaVA. By improving the LLaVA model, the ability to identify strawberry diseased plants is improved, and the accuracy of the model in diagnosing and making decisions about strawberry diseases is increased.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal visual question-answering method for strawberry diseases based on improved LLaVA, characterized by the following steps: include: S1. Obtain images of strawberry plants with various diseases and construct a visual question-and-answer dataset for strawberry diseases based on the images of strawberry plants with diseases. S2. Improve the LLaVA model; S3. Based on the strawberry disease visual question answering dataset, the improved LLaVA model was fine-tuned to obtain the strawberry disease visual question answering model. S4. Conduct a question-and-answer session based on the visual question-and-answer model for strawberry diseases.

2. The strawberry disease multimodal visual question answering method based on improved LLaVA according to claim 1, characterized in that: The strawberry disease visual question-answering dataset contains closed-set question-answering and open-set question-answering for images of strawberry diseased plants, wherein the closed-set question-answering is accuracy-based.

3. The strawberry disease multimodal visual question answering method based on improved LLaVA according to claim 1, characterized in that, Step S2 improves the LLaVA model by retaining the visual encoder, visual-language connector, and large language model in the LLaVA model, and introducing a dual-view pruning module and a dual-view enhancement module. The visual-language connector is located between the visual encoder and the large language model, the dual-view pruning module is set before the visual encoder, and the dual-view enhancement module is set after the visual encoder.

4. The strawberry disease multimodal visual question answering method based on improved LLaVA according to claim 3, characterized in that: Step S3 involves fine-tuning the improved LLaVA model, including: S31. Divide the strawberry disease visual question-and-answer dataset into two training sets according to the proportions. S32. Freeze the weights of the visual encoder and the large language model, select a training set for training, and align the strawberry diseased plant images with text features. S33. Unfreeze the weights of the large language model, use another training set for instruction tuning, fine-tune the improved LLaVA model based on preset instructions, and optimize the model's reasoning and decision-making ability in the strawberry disease task.

5. A multimodal visual question-answering method for strawberry diseases based on improved LLaVA according to claim 4, characterized in that, Step S4 specifically includes: S41. Input images of diseased strawberry plants into the strawberry disease visual question-and-answer model; S42, the dual-view cropping module segments high-resolution strawberry diseased plant images into sub-images from local and global views. Local view cropping segments the high-resolution image into non-overlapping sub-images according to the preset resolution of the visual encoder. Global view cropping uses a specific step size mode to interleave sampling pixels and generate sub-images containing global layout information to capture the spatial relationship between objects. S43. The visual encoder extracts features from each sub-image. The sub-image features are recombined based on 2D position priors to generate a comprehensive set of high-resolution local and global features. S44. The dual-view enhancement module promotes the interaction between local and global features through two symmetrical processes: global enhancement and local enhancement. Global enhancement injects local detail features into global features through a cross-attention mechanism; local enhancement integrates global contextual information into local features, making the two types of features complementary to each other. S45. The refined visual markers and the text markers of the instructions are concatenated and input into the large language model to generate answers by marker one by one.

6. The strawberry disease multimodal visual question answering method based on improved LLaVA according to claim 5, characterized in that: The specific processes for local and global view cropping in step S42 include: Local perspective cropping: Given high-resolution images of diseased strawberry plants First, determine the size (W) of the given high-resolution image of a diseased strawberry plant. HR ×H HR ) and the size expected by the pre-trained visual encoder (W) LR ×H LR The relationship between ) is expressed by the formula: in, This represents the floor function, which rounds down to the nearest integer. and These correspond to the number of local sub-images along the width and height of the high-resolution image, respectively; High-resolution image I is divided into Local sub-images The size of each local sub-image is W. LR ×H LR For each sub-image in The formulas for calculating row and column positions are: Where mod represents the modulo operation, c loc Indicates column position; r loc Indicates the row position; Each local sub-image The bounding box formula is: Global view cropping: Determine the size (W) of the input high-resolution image of a diseased strawberry plant. HR ×H HR ) and the size (W) used by the pre-trained visual encoder LR ×H LR The direct relationship is expressed by the formula: In the formula, These correspond to the number of global sub-images along the width and height of the high-resolution image, respectively; For the i-th row and j-th column of the global sub-image, the set of pixel indices Defined as: Where (x, y) corresponds to the pixel index within the high-resolution image. Let m represent the set of natural numbers, where m represents the set of natural numbers between 0 and the width of the sub-image, and n represents the set of natural numbers between 0 and the height of the sub-image. at last, Each pixel (u, v) in the sub-image is mapped back to the high-resolution image I, using the formula:

7. A multimodal visual question-answering method for strawberry diseases based on improved LLaVA according to claim 6, characterized in that, Step S44 specifically includes the following process: To perform global view enhancement, firstly, crop both local and global view features from the global perspective. The formula is as follows: in, This indicates a global perspective clipping operation. and These represent the global and local sub-features generated by cropping from this global perspective, respectively. Perform a cross-attention operation between the corresponding local and global sub-features, injecting local sub-feature information into the global sub-features, as shown in the formula: in, It is a learnable embedding matrix. It is an attention mapping between local sub-features and global sub-features. It is the i-th global enhancer feature; Finally, enhance the sub-feature set Combining to form global enhanced features To enhance local perspective, firstly, local and global perspective features are cropped into sub-features using a local cropping operation, as shown in the formula: in, This represents the sub-features obtained from a partial perspective clipping operation. and These represent the global and local sub-features generated from this local cropping, respectively. A cross-attention mechanism is employed to promote interaction between local and global sub-features, leveraging global context to enhance local features. The formula is as follows: in, These are learnable embedding matrices for querying, key-value transformation, and attention maps. Capture the attention mapping between local and global sub-features. It is the i-th obtained local enhancer feature; Finally, the local enhancement sub-feature set Aggregated to form local enhancement features Feature fusion is performed by first using two independent embedding layers to reduce the dimensionality of the features, as shown in the formula: in, These are the learnable projection matrices for global and local features, respectively; The embedded global and local features are concatenated along the channel dimension to form a double-enhanced feature, as shown in the formula: Where [·;·] represents cascading operations along the channel dimension.

Images