Visual language detection method and device for detecting insects and pests based on open vocabularies

Abstract

The invention relates to a visual language detection method for detecting insects and pests based on open vocabularies, and the method comprises the steps: taking a to-be-detected image and the text description of the to-be-detected image as input, and extracting visual features and text features; and constructing a multi-modal feature fusion module, mapping the visual features and the text features to a unified semantic space, generating language-perceived visual features and image-perceived text features, and aligning the language-perceived visual features and the image-perceived text features in the unified semantic space by using a comparative learning mechanism. The system is designed for effectively detecting pests in the real world, and the key requirement of an adaptive pest recognition system is met; the multi-modal feature fusion module effectively integrates a visual language mode, and the performance of open vocabulary insect pest detection is remarkably improved by enhancing feature representation; regional cues show excellent generalization ability in an open vocabulary scene, thereby enabling more robust and flexible pest detection in different environments.

Description

Technical Field

[0001] The present invention relates to the technical field of intersection of computer vision and natural language processing, and in particular to a visual language detection method and device for detecting insect pests based on open vocabulary. Background Art

[0002] Currently, insect pests pose a significant threat to food production and food security. Their diverse species, widespread distribution, severe damage potential, and high outbreak frequency all pose significant risks to food security. The ongoing impacts of global climate change have exacerbated this problem, leading to an expansion of affected areas, increased severity of crop pests and diseases, and significant agricultural losses. Efficient and accurate detection methods are urgently needed to enhance pest and disease detection and early warning systems.

[0003] Existing methods, developed through advances in deep neural networks, typically rely on predefined pest and disease categories and feature libraries for identification. However, traditional object detection is inherently limited by its reliance on predefined categories, requiring additional annotation and retraining processes to identify new or previously unencountered species. These limitations result in limited system flexibility, poor adaptability, susceptibility to overfitting, and insufficient generalization capabilities. Summary of the Invention

[0004] To address the inherent limitations of existing technologies in adaptability and generalization when faced with novel or previously unencountered pest species, the primary objective of the present invention is to provide a visual language architecture method for insect pest detection using an open vocabulary that demonstrates excellent generalization capabilities in open vocabulary scenarios, thereby enabling more robust and flexible pest detection in different environments.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a visual language architecture method for detecting insect pests using an open vocabulary, the method comprising the following steps in sequence:

[0006] (1) The image to be detected and its text description are taken as input, and the region prompt mechanism is used to give a region image and a text prompt matching the region image in the image to be detected. The image to be detected and the region image are extracted from the visual features through the YOLOv8 backbone network, and the text description of the image to be detected and the text prompt of the region image are used to obtain text features using the CLIP text encoder;

[0007] (2) constructing a multi-modal feature fusion module, taking the visual feature and the text feature as inputs of the multi-modal feature fusion module, integrating to obtain a visual language representation, i.e., mapping the visual feature and the text feature to a unified semantic space, generating a language-aware visual feature and an image-aware text feature, the language-aware visual feature including a high-dimensional feature vector of each object in an image and a prediction box of each object, and the image-aware text feature being a high-dimensional feature vector of each noun phrase;

[0008] (3) using a contrastive learning mechanism to align the language-aware visual feature and the image-aware text feature in the unified semantic space, i.e., minimizing the distance of a similar visual-text pair and maximizing the distance of a dissimilar visual-text pair, thereby realizing alignment, and classifying objects with noun phrase labels; the similar visual-text pair being an object and a noun phrase pair in the image.

[0009] In step (1), the region prompting mechanism specifically includes the following sequential steps:

[0010] (1a) generating a candidate region: using a SAM visual segmentation base model to segment the to-be-detected image I to generate a binary mask of N candidate regions:

[0011] ;

[0012] wherein, represents the binary mask of the i-th candidate region, and the binary mask is a matrix with the same size as the image, wherein each pixel value is 0 or 1, indicating whether the pixel belongs to the candidate region; N represents the number of generated candidate regions; represents a visual segmentation base model, which is used to segment the to-be-detected image I and generate a binary mask of the candidate region;

[0013] (1b) converting the mask into a bounding box:

[0014] ;

[0015] wherein, represents the bounding box of the i-th candidate region; represents a function for converting the binary mask into the bounding box ;

[0016] (1c) extracting object nouns in the text by an N-gram algorithm to form a noun group T, i.e., performing preprocessing operations of tokenization, stop word removal, and special symbol removal on the to-be-detected image I to obtain a tokenized text sequence;

[0017] (1d) generating a preliminary region-text pair by a GLIP-L model:

[0018] ;

[0019] wherein, is the confidence score of GLIP-L, and the one with higher confidence score is selected as the preliminary region-text pair; represents the object noun corresponding to the i-th candidate region; GLIP-L represents a visual language model for matching the to-be-detected image I with the noun group T, and outputting the preliminary region-text pair and its confidence score;

[0020] (1e) According to Crop the image block according to the coordinates:

[0021] ;

[0022] wherein, represents a cropping function for cropping the corresponding image block from the to-be-detected image I according to the given bounding box ; Ri represents the cropped image block;

[0023] (1f) Center the image block and scale it to 224x224 resolution:

[0024] ;

[0025] wherein, represents the image block after centering and scaling processing; represents a function for adjusting the image to 224x224 resolution; represents a function for centering the image block;

[0026] (1g) Calculate the CLIP score:

[0027] ;

[0028] wherein, represents the similarity score calculated by the CLIP model for the given image block and the text label ;

[0029] (1h) Sort and filter:

[0030] ;

[0031] wherein, represents the set of finally filtered region-text pairs; represents a function for selecting the top K elements from the sorted list; represents the CLIP score of all N candidate region-text pairs descending order sorting;

[0032] descending order sorting, retaining the top K high-score region-text pairs; selecting the required region-text pair as the text prompt. descending order sorting, retaining the top K high-score region-text pairs; selecting the required region-text pair as the text prompt.

[0033] In step (2), the specific steps of constructing the multi-modal feature fusion module are as follows:

[0034] (2a) Integrate text information into multi-scale image features: the YOLOv8 network is divided into three parts: backbone network, neck and head. In the bottleneck module of the neck, a bidirectional channel from top to bottom and from bottom to top is adopted to integrate text information into multi-scale visual features {C3, C4, C5}, C3, C4, C5 representing the 3rd, 4th and 5th layers of visual features in the multi-scale visual features, respectively, thereby forming a text-guided visual feature pyramid {P3, P4, P5}, P3, P4, P5 representing the 3rd, 4th and 5th layers of the feature pyramid, respectively; given the text feature vector W and the visual feature vector , , where H represents the height of the visual feature map, D represents the depth of the visual feature map, i.e., the feature dimension, , and i represents the hierarchical index of the visual feature pyramid; in the last bottleneck module, an attention mechanism using a maximum sigmoid function is introduced, and the text feature is fused into the visual feature using the following formula:

[0035] ;

[0036] , where is the language-aware visual feature, and δ represents the sigmoid activation function; C represents the number of text feature vectors; j represents the index of the text feature vector.

[0037] (2b) Integrate visual information into text features: to enhance text features with visual information, update text features by aggregating visual features; that is, apply a multi-head attention mechanism to multi-scale features to extract 3x3 size visual blocks, a total of 27 visual block tokens , and the visual feature is fused into the text feature using the following formula:

[0038] ;

[0039] , where is the image-aware text feature, and MHA is the multi-head attention mechanism.

[0040] In step (3), the contrast learning mechanism specifically includes the following steps in sequence:

[0041] (3a) Calculate the cosine similarity between each visual feature and each text feature to obtain a similarity matrix S;

[0042] (3b) Apply the softmax function to each row and each column of the similarity matrix S to normalize, so that the value of each element in the similarity matrix S is between (0, 1) and the sum is 1; the softmax function is applied to each row as follows:

[0043] ;

[0044] wherein, S p represents the p-th row element of S, and p is the row index; each element in S represents the similarity between the p-th visual feature and the q-th text feature, represents the sum of the exponential function values of all elements in the p-th row, and n is the total number of columns, i.e. the number of text features, represents the probability value of the q-th element after the p-th row element is processed by the softmax function; thus, the normalized matrix of visual features to text features is obtained, and the normalized matrix of text features to visual features is obtained in the same way;

[0045] (3c) Calculate the cross-entropy loss for each visual-text pair that is completely matched correctly, and average all cross-entropy losses to obtain the total loss; the smaller the total loss value, the better the matching degree of the model to the visual-text pair.

[0046] Another object of the present application is to provide an electronic device comprising:

[0047] a processor; and

[0048] a memory having computer program instructions stored therein, the computer program instructions, when executed by the processor, causing the processor to perform the visual language architecture method for detecting insect pests with an open vocabulary as described above.

[0049] The present application also provides a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions, when executed by a processor, causing the processor to perform the visual language architecture method for detecting insect pests with an open vocabulary as described above.

[0050] From the above technical solutions, the beneficial effects of the present application are: first, the present application proposes a novel visual language detection method for detecting insect pests based on an open vocabulary, which is designed for effective real-world pest detection and solves the key requirements of an adaptive pest recognition system; second, the present application develops an innovative multi-modal feature fusion module that effectively integrates visual language modalities, significantly improving the performance of open vocabulary insect pest detection through enhanced feature representation; third, the present application proposes a new method of region prompting, which demonstrates excellent generalization ability in the open vocabulary scenario, thereby enabling more robust and flexible pest detection in different environments. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 A flowchart of the method of the present application;

[0052] Figure 2 A visual result diagram of the present application on the IP102 dataset. DETAILED DESCRIPTION

[0053] As Figure 1 shown, a visual language architecture method for detecting insect pests with an open vocabulary includes the following sequential steps:

[0054] (1) The image to be detected and the text description of the image to be detected are taken as inputs, and a region prompting mechanism is used to give a certain region image in the image to be detected and a text prompt matching the certain region image. The image to be detected and the certain region image are extracted by a YOLOv8 backbone network to obtain visual features, while the text description of the image to be detected and the text prompt of the certain region image are obtained by a CLIP text encoder to obtain text features;

[0055] (2) A multi-modal feature fusion module is constructed, and the visual features and the text features are taken as inputs of the multi-modal feature fusion module to obtain a visual language representation, i.e., the visual features and the text features are mapped to a unified semantic space to generate language-aware visual features and image-aware text features. The language-aware visual features include high-dimensional feature vectors of objects in the image and predicted bounding boxes of the objects, and the image-aware text features are high-dimensional feature vectors of noun phrases;

[0056] (3) A contrastive learning mechanism is used to align the language-aware visual features and the image-aware text features in the unified semantic space, i.e., to minimize the distance of similar visual-text pairs and maximize the distance of dissimilar visual-text pairs, thereby achieving alignment and classifying objects with noun phrase labels. The similar visual-text pair is an object in the image and a noun phrase pair.

[0057] In step (1), the region prompting mechanism specifically includes the following sequential steps:

[0058] (1a) Generating candidate regions: using the SAM visual segmentation base model to segment the image to be detected I, generating N binary masks of candidate regions:

[0059] ;

[0060] wherein, represents the binary mask of the i-th candidate region, and the binary mask is a matrix with the same size as the image, wherein each pixel value is 0 or 1, indicating whether the pixel belongs to the candidate region; N represents the number of generated candidate regions; represents a visual segmentation base model for segmenting the image to be detected I and generating a binary mask of candidate regions;

[0061] (1b) Convert the mask to a bounding box:

[0062] ;

[0063] wherein, represents the bounding box of the i-th candidate region; represents a function for converting a binary mask to a bounding box ;

[0064] (1c) Extract the object nouns in the text by the N-gram algorithm to form a noun group T, i.e. the preprocessing operation of tokenization, stop word removal, and special symbol removal on the image to be detected I, to obtain the tokenized text sequence;

[0065] (1d) Generating preliminary region-text pairs by GLIP-L model:

[0066] ;

[0067] wherein, is the confidence score containing GLIP-L, and the one with high confidence score is selected as the preliminary region-text pair; represents the object noun corresponding to the i-th candidate region; GLIP-L represents a visual language model for matching the image to be detected I with the noun group T, and outputting the preliminary region-text pair and its confidence score;

[0068] (1e) According to crop the image block:

[0069] ;

[0070] wherein, represents a cropping function for cropping an image block according to a given bounding box Crop the corresponding image patch from the image to be detected I; Ri denotes the cropped image patch;

[0071] (1f) Centered image patch while scaling to 224x224 resolution:

[0072] ;

[0073] wherein, denotes the centered and scaled image patch; denotes a function that resizes an image to 224x224 resolution; denotes a function that centers an image patch;

[0074] (1g) Calculate CLIP score:

[0075] ;

[0076] wherein, denotes the similarity score computed by the CLIP model for a given image patch and text label ;

[0077] (1h) Rank and filter:

[0078] ;

[0079] wherein, denotes the final set of region-text pairs that pass the filter; denotes a function that selects the top K elements from a ranked list; denotes the CLIP score for all N candidate region-text pairs in descending order;

[0080] in descending order, keeping the top K high-scoring region-text pairs; select the desired region-text pair as the text prompt. Region prompts are used as learning representations to guide text features towards specific pest characteristics, resulting in more accurate identification. In real-world deployment scenarios, the present invention allows users to specify regions of interest to provide enhanced flexibility, which are then processed to generate corresponding region-text pairs.

[0081] In step (2), the specific steps for constructing the multi-modal feature fusion module are as follows:

[0082]

[0083] ​(2a) Integrate text information into multi-scale image features: The YOLOv8 network is divided into three parts: backbone network, neck and head. In the bottleneck module of the neck, a bidirectional channel from top to bottom and from bottom to top is adopted to integrate text information into multi-scale visual features {C3, C4, C5}, where C3, C4, C5 represent the 3rd, 4th and 5th layers of visual features in the multi-scale visual features, respectively, thereby forming a text-guided visual feature pyramid {P3, P4, P5}, where P3, P4, P5 represent the 3rd, 4th and 5th layers of the feature pyramid, respectively; given a text feature vector W and a visual feature vector , where H represents the height of the visual feature map, D represents the depth of the visual feature map, i.e., the feature dimension, represents the hierarchical index of the visual feature pyramid; in the last bottleneck module, an attention mechanism that maximizes the sigmoid function is introduced, and the text features are fused into the visual features using the following formula:

[0084] ;

[0085] wherein, is the language-aware visual feature, and δ represents the sigmoid activation function; C represents the number of text feature vectors; j represents the index of the text feature vector;

[0086] (2b) Integrate visual information into text features: To enhance the text features with visual information, the text features are updated by aggregating the visual features; that is, a multi-head attention mechanism is applied to the multi-scale features to extract 3x3 size visual blocks, a total of 27 visual block tokens are obtained, and the visual features are fused into the text features using the following formula:

[0087] ;

[0088] wherein, is the image-aware text feature, and MHA is the multi-head attention mechanism.

[0089] In step (3), the contrast learning mechanism specifically includes the following steps in sequence:

[0090] (3a) Calculate the cosine similarity between each visual feature and each text feature to obtain a similarity matrix S;

[0091] (3b) Apply the softmax function to each row and each column of the similarity matrix S for normalization, so that the value of each element in the similarity matrix S is between (0, 1) and the sum is 1; the softmax function is applied to each row as follows:

[0092] ;

[0093] wherein, S p represents the pth row element of S, p is the row index; each element in S represents the similarity between the pth visual feature and the qth text feature, represents the sum of the exponential function values of all elements in the pth row, n is the total number of columns, that is, the number of text features, represents the probability value of the qth element after the pth row element is processed by the softmax function; in this way, the normalized matrix of visual features to text features is obtained, and similarly, the normalized matrix of text features to visual features is obtained;

[0094] (3c) Calculate the cross-entropy loss for each visual-text pair that is completely matched correctly, and average all cross-entropy losses to obtain the total loss; the smaller the total loss value, the better the matching of the model to the visual-text pair.

[0095] Figure 2 The detection capability of the present application on various insect species is demonstrated, and each image shows the detected pests and their corresponding confidence scores, which illustrates the accuracy and robustness of the present application in identifying different pest species in the IP102 dataset.

[0096] In summary, the present application is designed for effective real-world pest detection, and solves the key requirements of an adaptive pest recognition system; the multi-modal feature fusion module of the present application effectively integrates visual and linguistic modalities, significantly improving the performance of open-vocabulary insect pest detection by enhancing feature representation; the region proposal of the present application demonstrates excellent generalization ability in the open-vocabulary scenario, thereby enabling more robust and flexible pest detection in different environments.

[0097] The above shows and describes the basic principles, main features and advantages of the present application. It should be understood by those skilled in the art that the present application is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection claimed by the present application is defined by the appended claims and their equivalents.

Claims

1. A visual language detection method for detecting insect pests based on an open vocabulary, characterized by: The method comprises the following steps in sequence: (1) The image to be detected and its text description are taken as input, and the region prompt mechanism is used to give a region image and a text prompt matching the region image in the image to be detected. The image to be detected and the region image are extracted from the visual features through the YOLOv8 backbone network, and the text description of the image to be detected and the text prompt of the region image are used to obtain text features using the CLIP text encoder; (2) Construct a multimodal feature fusion module, take visual features and text features as input to the multimodal feature fusion module, and integrate them to obtain visual language representation, that is, map visual features and text features into a unified semantic space to generate language-aware visual features and image-aware text features. The language-aware visual features include high-dimensional feature vectors of each object in the image and the predicted boxes of each object, and the image-aware text features are high-dimensional feature vectors of each noun phrase. (3) Using a contrastive learning mechanism, the language-perceived visual features and the image-perceived text features are aligned in a unified semantic space, that is, the distance between similar visual-text pairs is minimized and the distance between dissimilar visual-text pairs is maximized, thereby achieving alignment and classifying objects labeled with noun phrases; the similar visual-text pairs are pairs of objects and noun phrases in the image.

2. The open vocabulary-based visual language detection method for insect pests according to claim 1, characterized in that: In step (1), the region prompt mechanism specifically includes the following steps in order: (1a) Generate candidate regions: Use the SAM visual segmentation basic model to segment the image I to be detected and generate binary masks of N candidate regions: ； Where, Represents the binary mask of the i-th candidate region. The binary mask is a matrix of the same size as the image, where each pixel value is 0 or 1, indicating whether the pixel belongs to the candidate region; N represents the number of generated candidate regions; Represents the basic model of visual segmentation, which is used to segment the image to be detected I and generate binary masks of candidate regions; (1b) Convert the mask to a bounding box: ； Where, represents the bounding box of the i-th candidate region; Represents a function for converting a binary mask Convert to Bounding Box ; (1c) Using the N-gram algorithm, object nouns are extracted from the text to form a noun group T. This involves performing preprocessing operations on the image I to be detected, including word segmentation, removal of stop words, and removal of special symbols, to obtain a text sequence after word segmentation. (1d) Generate preliminary region-text pairs through the GLIP-L model: ； in, For the confidence scores of GLIP-L, the ones with high confidence scores are selected as preliminary region-text pairs; represents the object noun corresponding to the i-th candidate region; GLIP-L represents a visual language model that matches the image to be detected I with the noun group T and outputs preliminary region-text pairs and their confidence scores; (1e) According to Coordinate cropping of image blocks: ； Where, Represents a clipping function for a given bounding box Cut out the corresponding image block from the image to be detected I; Ri Represents the cropped image block; (1f) The centered image patch is scaled to 224x224 resolution: ； Where, Represents the image block after centering and scaling; Represents a function for resizing an image to a resolution of 224×224; represents a function used to center an image block; (1g) Calculate CLIP score: ； Where, Represents the CLIP model for a given image patch and text labels Calculated similarity score; (1h) Sorting and screening: ； Where, Represents the set of region-text pairs that are finally filtered out; represents a function that selects the first K elements from a sorted list; Represents the CLIP score for all N candidate region-text pairs Sort in descending order; according to Sort in descending order and retain the top K high-scoring region-text pairs; select the required region-text pairs as text prompts.

3. The open vocabulary-based visual language detection method for insect pests according to claim 1, characterized in that: In step (2), the specific steps of constructing the multimodal feature fusion module are as follows: (2a) Integrate text information into multi-scale image features: The YOLOv8 network is divided into three parts: backbone network, neck and head. In the bottleneck module of the neck, a top-down and bottom-up bidirectional path is used to integrate text information into the multi-scale visual features {C3, C4, C5}. C3, C4, C5 represent the third, fourth and fifth layers of visual features in the multi-scale visual features, thereby forming a text-guided visual feature pyramid {P3, P4, P5}. P3, P4, P5 represent the third, fourth and fifth layers of the feature pyramid, respectively. Given a text feature vector W and a visual feature vector , , where H represents the height of the visual feature map, and D represents the depth of the visual feature map, i.e., the feature dimension. Represents the hierarchical index of the visual feature pyramid; the attention mechanism of maximizing the sigmoid function is introduced in the last bottleneck module, and the text features are integrated into the visual features using the following formula: ； in, is the visual feature of language perception, δ represents the sigmoid activation function; C represents the number of text feature vectors; j represents the index of the text feature vector; (2b) Integrate visual information into text features: To enhance text features with visual information, the text features are updated by aggregating visual features; that is, a multi-head attention mechanism is applied to multi-scale features to extract 3×3 visual blocks, resulting in a total of 27 visual block tokens. , use the following formula to integrate visual features into text features: ； in, It is the text feature of image perception, and MHA is the multi-head attention mechanism.

4. The open vocabulary-based visual language detection method for insect pests according to claim 1, characterized in that: In step (3), the contrastive learning mechanism specifically includes the following steps in order: (3a) Calculate the cosine similarity between each visual feature and each text feature to obtain a similarity matrix S; (3b) Apply the softmax function to each row and column of the similarity matrix S to normalize it so that the value of each element in the similarity matrix S is between (0, 1) and the sum is 1; the softmax function applies the following formula to each row: ； Among them, S p Represents the p-th row element of S, where p is the row index; each element in S represents the similarity between the p-th visual feature and the q-th text feature, represents the sum of the exponential function values ​​of all elements in the pth row, n is the total number of columns, that is, the number of text features, Represents the probability value of the qth element after applying the softmax function to the elements of the pth row; this gives the normalized matrix from visual features to text features. Similarly, the normalized matrix from text features to visual features is obtained. (3c) Calculate the cross entropy loss for each visual-text pair in which the visual feature and text feature are completely matched correctly, average all cross entropy losses to obtain the total loss; the smaller the total loss value, the better the model matches the visual-text pair.

5. An electronic device comprising: processor; as well as A memory having computer program instructions stored therein, wherein the computer program instructions, when executed by the processor, cause the processor to execute the visual language detection method for detecting insect pests based on open vocabulary according to any one of claims 1 to 4.

6. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is caused to execute the visual language detection method for detecting insect pests based on open vocabulary according to any one of claims 1 to 4.

Images