Agricultural image text anaphora positioning method based on semantic space alignment

Abstract

The invention discloses an agricultural image text anaphora positioning method based on semantic space alignment, and belongs to the technical field of intelligent agriculture, and the method comprises the steps: collecting agricultural images and agricultural target text language description data, constructing an agricultural image text anaphora positioning data set, constructing an agricultural image text anaphora positioning model, and constructing an agricultural image text anaphora positioning model; and extracting image features and text features of the data, further enhancing the image features, and decoding the text features and the enhanced image features by using a target decoding module to obtain coordinates of an agricultural entity target described by a text language. More accurate and robust target positioning can be realized in the agricultural image, and the method is especially suitable for complex agricultural scenes such as dense growth of similar crops, similar backgrounds, small target identification of orchards by unmanned aerial vehicles and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of smart agriculture, computer vision and multi-modal image processing, and particularly relates to an agricultural image text reference positioning method based on semantic space alignment. BACKGROUND

[0002] The agricultural image text reference positioning aims to identify and position the agricultural target entity described by natural language in the agricultural image. With the rapid development of Internet of Things smart agriculture and agricultural unmanned monitoring technology, the technology can play an important value in the application of automatic picking based on language instructions, disease and pest monitoring, etc. Existing agricultural positioning methods are mostly focused on identifying crops that are similar in appearance to predefined categories, and are difficult to adapt to the positioning needs of flexible targets. In recent years, multi-modal learning combining computer vision and natural language processing has promoted the development of visual text reference positioning tasks, which can locate target objects according to language description. However, in the agricultural scene, crops usually grow in a dense state, and the visual features between instances of the same or different categories are small, and the target and background are highly similar in color, texture, etc. This leads to strong spatial proximity between objects and low distinguishability, which makes it difficult for existing reference positioning methods designed for natural images to accurately capture subtle distinguishing features in the agricultural scene, and the positioning robustness is insufficient. Therefore, in the agricultural image text reference positioning, how to enhance the alignment ability of semantic and spatial features, highlight the subtle distinguishing features of the text related target, and effectively estimate and filter the number of real targets, has become a key technical challenge to be solved. SUMMARY

[0003] In order to solve the above technical problems, the present application provides an agricultural image text reference positioning method based on semantic space alignment.

[0004] In order to achieve the above purpose, the agricultural image text reference positioning method based on semantic space alignment designed by the present application has the special features that the method comprises the following steps: S1: Collecting agricultural image and agricultural target text language description data, constructing an agricultural image text reference positioning dataset, and dividing the training set, the validation set and the test set according to the ratio of 6:2:2, which is distributed for subsequent model training, validation and evaluation; S2: Constructing an agricultural image text reference positioning model for obtaining the coordinates of the agricultural entity target described by the text language, the model structure comprises five parts: a multi-modal feature extraction module, a fine-grained feature enhancement module based on semantic space alignment guidance, a query weight initialization module, a cross-modal reference counting module and a target decoding module; S21: Input the agricultural image and agricultural target text language description data from step S1 into the multimodal feature extraction module for feature extraction to obtain the initial feature image and initial feature text; the multimodal feature extraction module includes an image feature extraction submodule and a text feature extraction submodule; Preferably, in step S21, the multimodal feature extraction module includes an image feature extraction submodule and a text feature extraction submodule: Image Feature Extraction Submodule: In the agricultural image input image feature extraction submodule, local features are obtained through a ResNet50 backbone network. These local features are flattened to obtain a spatial sequence, which is then input into a 6-layer Transformer encoder for global context modeling to obtain the initial feature image. ,in For feature dimension, This represents the total number of spatial locations. The text feature extraction submodule performs word segmentation on the agricultural target text language description data, and then uses the text feature extraction submodule, which is composed of a pre-trained BERT-Base model, to take the last hidden state as the encoding result, thus obtaining the initial feature text of the preliminary encoding. ,in The length of the text sequence; S22: Input the initial feature image and initial feature text into the fine-grained feature enhancement module guided by semantic space alignment, respectively, to obtain the image feature sequence. Then, the target augmentation feature map is obtained. The fine-grained feature enhancement module guided by semantic space alignment consists of two parts: a semantic space cross-modal alignment submodule and a visual target enhancement submodule. Preferably, in step S22, the fine-grained feature enhancement module based on semantic space alignment guidance comprises two parts: a semantic space cross-modal alignment submodule and a visual target enhancement submodule. Spatial cross-modal alignment submodule: This module aligns the initial feature image... Initial feature text In the input semantic space cross-modal alignment submodule, the initial feature image Through two Convolutional layers model the spatial structure, resulting in enhanced feature images. and Simultaneously, the initial feature text The first semantic projection feature is obtained through a linear transformation layer. Then the initial feature text The second semantic projection feature is obtained through another linear layer. ,Will and The similarity matrix is ​​obtained by multiplying the transposes of the features. This similarity matrix is ​​then normalized using the Softmax function to obtain a normalized similarity matrix. Finally, the normalized similarity matrix is ​​compared with the second semantic projection feature. Multiplying yields semantic-level alignment features, which are then combined with... Further concatenation along the channel dimension yields concatenated features, which are then dimensionality-reduced using a linear projection layer to obtain a semantically aligned image feature sequence. ; The visual target enhancement module will use the feature sequence output by the semantic space cross-modal alignment module. With primary image features The image features are input into the visual target enhancement module, where alignment features are used to perform weighted enhancement. This process suppresses background interference while highlighting the salient features of the target object, resulting in a target enhancement feature map. The specific implementation steps are as follows: Sa1: Initial feature image Input linear projection matrix Simultaneously, the image feature sequence output by the semantic space cross-modal alignment module is... Input linear projection matrix The projected features were obtained respectively. and ; Sa2: Based on the projection features described in Sa1 above, an enhancement coefficient matrix is ​​generated using a Gaussian weighting function. The formula is as follows:

[0005] in, and For learnable parameters, Represents the dot product similarity function. This represents an exponential function used to calculate Gaussian weighted sums and their dot product similarity. ; Sa3: For the initial feature image Perform layer normalization processing and then use the initial feature image after layer normalization. With enhancement coefficient matrix Perform element-wise multiplication, and compare the result of the element-wise multiplication with the initial feature image. After performing residual connections, the target enhanced image is obtained by performing layer normalization again. The calculation formula is as follows:

[0006] in, The layer normalization function is represented. Represents the initial feature image Values ​​after layer normalization; S23: Input the initial feature image and initial feature text from step S21 into the query weight initialization module to obtain the initial query. The query weight initialization module includes a cross-modal interaction unit, a weight scoring network, and a query generator. Preferably, the query weight initialization module in step S23 initializes the initial feature image extracted by the multimodal feature extraction module. and initial feature text As input, the text importance is first sorted based on cross-modal interaction to obtain a weight score matrix of different text tokens. Subsequently based on From the initial feature text Select before ( Typically, the most important text tokens (the largest number of targets in the dataset) are represented, and then after linear transformation and residual connection, they are fed into a self-attention mechanism to handle and suppress interactions between queries, resulting in the initial query. The specific operating steps are as follows: Sb1: Initial feature image and initial feature text As input, the initial feature text is first calculated. With the initial feature image The product of the transpose matrices; then divide the product of the transpose matrices by the scaling factor. Assuming the feature dimension is used, the result is normalized using the Softmax function; subsequently, the normalized matrix is ​​compared with the initial feature image. Perform matrix multiplication; finally, combine the multiplication result with the initial feature text. By adding elements one by one, we obtain the text features that incorporate visual semantics. The formula for the entire process is as follows:

[0007] in, This represents the Softmax function. Indicates transpose; Sb2: Obtain the text features that incorporate visual semantics from step Sb1. The data are processed by max pooling and average pooling operations along the channel dimension, and then the pooling results are input into two multilayer perceptrons with shared parameters to obtain two feature vectors. and ; Will and Element-wise summation is performed, and the summation results are then passed through a feedforward neural network and a Softmax function for normalization to obtain the final text channel importance weight score matrix. ; Sb3: Based on the importance weight matrix From the initial feature text Select the most important ones Each token has a characteristic, denoted as . ,Will The transformation is performed through a linear projection layer containing two linear layers with ReLU activation functions, and then compared with the original... Features are performed using residual connections. Finally, the results are input into a multi-head self-attention layer for query interaction and suppression, resulting in the initial query. ; S24: Enhance the target feature map The Query, serving as the attention mechanism, inputs the initial feature image into the cross-modal referential counting module to obtain a classification result, namely the predicted number of referential targets. The cross-modal reference counting module consists of a series of intra-modal multi-head attention layers, cross-modal multi-head attention layers, and a feedforward network classifier. Preferably, in step S24, the cross-modal reference counting module mentioned in step S2 is implemented by a series of an intramodal multi-head attention layer, a cross-modal multi-head attention layer, and a feedforward network classifier. The specific execution steps of the cross-modal reference counting module are as follows: Sc1: Enhance the feature image The Query, as an attention mechanism, is the initial feature image. Use the initial feature image as the key and value. The input is fed into an intramodal multi-head attention layer for computation to obtain a self-enhanced feature map. ; Sc2: The self-enhanced feature map output from step Sc1 above. As the key and value of the cross-modal attention mechanism, the initial feature text As a query, cross-modal attention interactions are performed to obtain refined multimodal fusion features. ; Sc3: The multimodal fusion features described in step Sc2 above. With initial feature text Element-wise addition is performed, and the result is passed through a feedforward neural network consisting of three fully connected layers to obtain a classification result, namely the predicted number of referential targets. The formula for the entire process is as follows:

[0008] in This represents a feedforward neural network that ultimately yields a classification result. S25: Initialize the query As an attention mechanism, Query enhances the feature map input to the target decoding module, based on the number of targets it refers to. Filter out the top with the highest confidence The target decoding module consists of two cascaded multi-head attention layers and two parallel feedforward networks. Preferably, the target decoding module in step S25 is implemented by two cascaded multi-head attention layers and two parallel feedforward networks, and the specific implementation steps are as follows: Sd1: Initialize the query output by the query weight initialization module. Query, as an attention mechanism, will enhance the feature map. As the Key and Value of the attention mechanism, they are input into the first multi-head attention layer for interaction, resulting in an enhanced query based on visual context. ; Sd2: The visual context-based enhanced query output from step Sd1 above. As an attention mechanism, Query takes the initial feature text... The Key and Value, serving as the attention mechanism, are input into a second multi-head attention layer for interaction, resulting in an enhanced query based on the text context. ; Sd3: The enhanced query based on text context output from step Sd2 above. Simultaneously inputting two parallel feedforward neural networks The first one predict (Indicating the number of the largest targets in the current dataset) the confidence score of each box in the target boxes; the second Used for regression prediction Initial coordinate parameters of the target bounding boxes ( ),in and These are the initial x and y coordinates of the center point of the target bounding box. and These are the width and height of the target bounding box, respectively; Sd4: Target quantity predicted by the cross-modal indexing module The result obtained from step Sd3 above Filter the prediction boxes to select the top ones with the highest confidence. The initial coordinates of the agricultural entity target described in the text language are obtained from the bounding boxes.

[0009] S3: Train the agricultural image text localization model. Input the agricultural image text localization data into the model, calculate the total loss value of the model, perform backpropagation, optimize the weight parameters of the model through a predetermined optimizer, and obtain the final agricultural image text localization model after multiple rounds of training. Preferably, in step S3, the total loss function value of the model is calculated. Including target bounding box regression loss Confidence loss of predicted target boxes and the predicted loss of the target count. The model's total loss function consists of three parts. The calculation formula is:

[0010] in, , and These are the weighting hyperparameters used to balance the various losses; Preferably, the bounding box regression loss Loss due to L1 and generalized IoU loss Together they constitute a whole, and their calculation is based on the coordinates of the real bounding box. With the predicted bounding box coordinates The Hungarian matching strategy was used to determine the correspondence between the predicted bounding boxes and the ground truth bounding boxes. Preferably, the prediction confidence loss The binary cross-entropy loss is applied to the confidence score of each predicted bounding box. Preferably, the target counting loss refers to... The target quantity prediction loss value, calculated by the Focal Loss function to optimize the cross-modal affixation counting module, is as follows:

[0011] in, For category weights, To focus parameters, This represents the model's predicted probability of the true class. The calculation formula is as follows:

[0012] in, It is a category The true label, Indicate category The predicted probability; The present invention further proposes a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the above-mentioned method for locating textual references in agricultural images based on semantic space alignment.

[0013] Compared with existing technologies, this invention proposes a novel method for text-based localization of agricultural images. This method introduces a fine-grained feature enhancement module guided by semantic space alignment, effectively highlighting the differences between the target and similar background features. Simultaneously, by initializing the query during the decoding stage using a query weight initialization module, the model possesses prior perception of the target before decoding, thereby effectively mitigating the degradation problem of network weights during the decoding stage and improving the model's prediction accuracy. Furthermore, to accurately filter the number of real targets, this invention designs a cross-modal localization module, effectively utilizing the important cue constraints in both text and visual modalities to predict the number of bounding boxes, reducing redundant detection. The synergistic effect of these modules significantly improves the accuracy and robustness of text-based localization of agricultural images, making it particularly suitable for complex agricultural scenarios with numerous similar crops or pests, and for locating small targets like orchard fruits from a UAV perspective. Attached Figure Description

[0014] Figure 1 This is a flowchart of the agricultural image text referencing localization method based on semantic space alignment according to the present invention.

[0015] Figure 2 This is a framework diagram of the agricultural image text reference localization model based on semantic space alignment proposed in this invention.

[0016] Figure 3 This is a block diagram of the fine-grained feature enhancement module based on semantic space alignment guidance.

[0017] Figure 4 The block diagram for initializing the query weight module.

[0018] Figure 5 This is a block diagram of the cross-modal counting module. Detailed Implementation

[0019] The technical solution of the present invention will be described in more detail and completely below with reference to the accompanying drawings and specific embodiments; like Figure 1 As shown, the text reference localization method for agricultural images based on semantic space alignment proposed in this invention includes the following steps: S1: Collect agricultural images and agricultural target text description data, construct an agricultural image text referencing and localization dataset, and divide it into training set, validation set and test set in a 6:2:2 ratio for subsequent model training, validation and evaluation; S2: Construct an agricultural image text-based localization model to obtain the coordinates of agricultural entities described in the text. The model structure includes five parts: a multimodal feature extraction module, a fine-grained feature enhancement module guided by semantic space alignment, a query weight initialization module, a cross-modal reference counting module, and a target decoding module. The overall framework of the model is as follows: Figure 2 As shown; Specifically, the multimodal feature extraction module is divided into an image feature extraction submodule and a text feature extraction submodule; the fine-grained feature enhancement module based on semantic space alignment guidance is implemented by concatenating a semantic space cross-modal alignment submodule and a visual target enhancement submodule; the query weight initialization module is implemented by linear projection, multilayer perceptron (MLP), feedforward network (FFN), and multi-head self-attention mechanism; the cross-modal algebraic index counting module is implemented by concatenating an intramodal multi-head attention layer, a cross-modal multi-head attention layer, and a feedforward network classifier; the target decoding module consists of two cascaded multi-head attention layers and two parallel feedforward networks. The specific processing steps of the agricultural image text referencing localization model based on semantic space alignment proposed in this invention are as follows: S21: Input the agricultural image and agricultural target text language description data from step S1 into the multimodal feature extraction module for feature extraction to obtain the initial feature image and initial feature text; the multimodal feature extraction module includes an image feature extraction submodule and a text feature extraction submodule; For the input image (in This indicates the number of image channels; for a typical RGB image, it is 3. and (representing the height and width of the input image respectively) and text description First, the multimodal feature extraction module processes the data to obtain the initial feature image with preliminary encoding. and initial feature text ; The image feature extraction submodule consists of a ResNet50 backbone network and a 6-layer Transformer encoder. In the agricultural image input image feature extraction submodule, local features are obtained through the ResNet50 backbone network, flattened to obtain a spatial sequence, and then input into the 6-layer Transformer encoder for global context modeling to obtain the initial feature image. ,in For feature dimension, This represents the total number of spatial locations. The text feature extraction submodule consists of a pre-trained BERT model. It performs word segmentation on the agricultural target text language description data, and then uses the text feature extraction submodule, which is composed of a pre-trained BERT-Base model, to extract the last hidden state as the encoding result, thus obtaining the initial encoded feature text. ,in The length of the text sequence; S22: Input the initial feature image and initial feature text into the fine-grained feature enhancement module guided by semantic space alignment, respectively, to obtain the image feature sequence. Then, the target augmentation feature map is obtained. The fine-grained feature enhancement module guided by semantic space alignment consists of two parts: a semantic space cross-modal alignment submodule and a visual target enhancement submodule. Based on the extracted initial feature image and initial feature text A fine-grained feature enhancement module based on semantic space alignment is used to enhance the fine-grained discriminative features of different agricultural targets, such as... Figure 3 As shown, the fine-grained feature enhancement module based on semantic space alignment guidance consists of a semantic space cross-modal alignment module and a visual target enhancement module connected in series, with the initial feature image as input. and initial feature text First, the semantic space cross-modal alignment module models the correlation between the target text description and the visual representation, resulting in a semantic space aligned feature sequence. The feature sequence is then fed into the visual target enhancement module to enhance the visual features of the target, and finally outputs an enhanced feature map. ; The semantic space cross-modal alignment module consists of convolutional units, matrix multiplication, linear transformation layers, and feature concatenation and projection units, which align the initial feature image... Initial feature text In the input semantic space cross-modal alignment submodule, the initial feature image Through two Convolutional layers model the spatial structure, resulting in enhanced feature images. and Simultaneously, the initial feature text The first semantic projection feature is obtained through a linear transformation layer. Then the initial feature text The second semantic projection feature is obtained through another linear layer. ,Will and The similarity matrix is ​​obtained by multiplying the transposes of the features. This similarity matrix is ​​then normalized using the Softmax function to obtain a normalized similarity matrix. Finally, the normalized similarity matrix is ​​compared with the second semantic projection feature. Multiplying yields semantic-level alignment features, which are then combined with... Further concatenation along the channel dimension yields concatenated features, which are then dimensionality-reduced using a linear projection layer to obtain a semantically aligned image feature sequence. ; The visual target enhancement module consists of linear projection, dot product similarity calculation, and normalization operations, based on the feature sequence obtained from the aforementioned semantic space cross-modal alignment module. The specific operating steps are as follows: Sa1: Initial feature image Input linear projection matrix Simultaneously, the image feature sequence output by the semantic space cross-modal alignment module is... Input linear projection matrix The projected features were obtained respectively. and ; Sa2: Based on the projection features described in Sa1 above, an enhancement coefficient matrix is ​​generated using a Gaussian weighting function. The formula is as follows:

[0020] in, and For learnable parameters, Represents the dot product similarity function. This represents an exponential function used to calculate Gaussian weighted sums and their dot product similarity. ; Sa3: For the initial feature image Perform layer normalization processing and then use the initial feature image after layer normalization. With enhancement coefficient matrix Perform element-wise multiplication, and compare the result of the element-wise multiplication with the initial feature image. After performing residual connections, the target enhanced image is obtained by performing layer normalization again. The calculation formula is as follows:

[0021] in, The layer normalization function is represented. Represents the initial feature image Values ​​after layer normalization; S23: Input the initial feature image and initial feature text from step S21 into the query weight initialization module to obtain the initial query. The query weight initialization module includes a cross-modal interaction unit, a weight scoring network, and a query generator. The initial feature image obtained based on the multimodal feature extraction module and initial feature text The query weight initialization module generates an initial query vector with prior knowledge of the target for the decoder. This module calculates the importance weights of text tokens through cross-modal interaction and selects key token features to initialize the query. The query weight initialization module consists of a cross-modal interaction unit, a weight scoring network, and a query generator. Figure 4 As shown, the specific operation steps are as follows: Sb1: Initial feature image and initial feature text As input, the initial feature text is first calculated. With the initial feature image The product of the transpose matrices; then divide the product of the transpose matrices by the scaling factor. Assuming the feature dimension is used, the result is normalized using the Softmax function; subsequently, the normalized matrix is ​​compared with the initial feature image. Perform matrix multiplication; finally, combine the multiplication result with the initial feature text. By adding elements one by one, we obtain the text features that incorporate visual semantics. The formula for the entire process is as follows:

[0022] in, This represents the Softmax function. Indicates transpose; Sb2: Obtain the text features that incorporate visual semantics from step Sb1. The data are processed by max pooling and average pooling operations along the channel dimension, and then the pooling results are input into two multilayer perceptrons with shared parameters to obtain two feature vectors. and ;Will and Element-wise summation is performed, and the summation results are then passed through a feedforward neural network and a Softmax function for normalization to obtain the final text channel importance weight score matrix. ; Sb3: Based on the importance weight matrix From the initial feature text Select the most important ones Each token has a characteristic, denoted as . ,Will The transformation is performed through a linear projection layer containing two linear layers with ReLU activation functions, and then compared with the original... Features are performed using residual connections. Finally, the results are input into a multi-head self-attention layer for query interaction and suppression, resulting in the initial query. ; S24: Enhance the target feature map The Query, serving as the attention mechanism, inputs the initial feature image into the cross-modal referential counting module to obtain a classification result, namely the predicted number of referential targets. The cross-modal reference counting module consists of a series of intra-modal multi-head attention layers, cross-modal multi-head attention layers, and a feedforward network classifier. Based on enhanced feature maps Initial feature image and initial feature text The cross-modal referencing counting module is used to predict the number of targets referred to by text descriptions in an image, and the target count is output. This module achieves accurate counting by fusing enhanced feature maps, initial feature images, and initial feature text through a two-level attention mechanism of intramodal self-enhancement and cross-modal interaction. The cross-modal reference counting module is implemented by cascading an intramodal multi-head attention layer, a cross-modal multi-head attention layer, and a feedforward network (FFN) classifier, as shown below. Figure 5 As shown, for the input, the specific processing steps of the cross-modal reference counting module are as follows: Sc1: Enhance the feature image The Query, as an attention mechanism, is the initial feature image. Use the initial feature image as the key and value. The input is fed into an intramodal multi-head attention layer for computation to obtain a self-enhanced feature map. ; Sc2: The self-enhanced feature map output from step Sc1 above. As the key and value of the cross-modal attention mechanism, the initial feature text As a query, cross-modal attention interactions are performed to obtain refined multimodal fusion features. ; Sc3: The multimodal fusion features described in step Sc2 above. With initial feature text Element-wise addition is performed, and the result is passed through a feedforward neural network consisting of three fully connected layers to obtain a classification result, namely the predicted number of referential targets. The formula for the entire process is as follows:

[0023] in This represents a feedforward neural network that ultimately yields a classification result.

[0024] S25: Initialize the query As an attention mechanism, Query enhances the feature map input to the target decoding module, based on the number of targets it refers to. Filter out the top with the highest confidence The target decoding module consists of two cascaded multi-head attention layers and two parallel feedforward networks. Based on initial query Enhanced feature maps and text features The target decoding module generates the final predicted bounding boxes. This module gradually fuses visual and textual information through a two-level attention mechanism and uses a parallel feedforward network to simultaneously predict the bounding box coordinates and confidence scores. The target decoding module consists of two cascaded multi-head attention layers and two parallel feedforward networks (FFNs). The specific processing steps are as follows: Sd1: Initialize the query output by the query weight initialization module. Query, as an attention mechanism, will enhance the feature map. As the Key and Value of the attention mechanism, they are input into the first multi-head attention layer for interaction, resulting in an enhanced query based on visual context. ; Sd2: The visual context-based enhanced query output from step Sd1 above. As an attention mechanism, Query takes the initial feature text... The Key and Value, serving as the attention mechanism, are input into a second multi-head attention layer for interaction, resulting in an enhanced query based on the text context. ; Sd3: The enhanced query based on text context output from step Sd2 above. Simultaneously inputting two parallel feedforward neural networks The first one predict (Indicating the number of the largest targets in the current dataset) the confidence score of each box in the target boxes; the second Used for regression prediction Initial coordinate parameters of the target bounding boxes ( ),in and These are the initial x and y coordinates of the center point of the target bounding box. and These are the width and height of the target bounding box, respectively; Sd4: Target quantity predicted by the cross-modal indexing module The result obtained from step Sd3 above Filter the prediction boxes to select the top ones with the highest confidence. The initial coordinates of the agricultural entity target described in the text language are obtained from the bounding boxes. S3: Train the agricultural image-text localization model. Input agricultural images and agricultural target text descriptions into the model, calculate the total loss value of the model, perform backpropagation, optimize the model's weight parameters through a predetermined optimizer, and obtain the final agricultural image-text localization model after multiple rounds of training. The processing steps are as follows: (1) Total loss value Includes bounding box regression loss Predicted confidence loss and refer to the target count loss The model's total loss function consists of three parts. The calculation formula is:

[0025] in, , and These are the weighting hyperparameters used to balance the various losses; (2) Calculate the bounding box regression loss using L1 loss and generalized IoU loss. Its calculation is based on the coordinates of the actual bounding box. With predicted bounding box coordinates The Hungarian matching strategy was used to determine the correspondence between the predicted bounding boxes and the ground truth bounding boxes. (3) Calculate the prediction confidence loss using binary cross-entropy loss. This applies to the confidence score of each predicted bounding box; (4) Use the Focal Loss function to calculate the target counting loss. The target quantity prediction loss value used to optimize the cross-modal index counting module is calculated as follows:

[0026] in, For category weights, To focus parameters, This represents the model's predicted probability of the true class. The calculation formula is as follows:

[0027] in, It is a category The true label, Indicate category The predicted probability is set in this invention. , ; The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0028] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit its scope of protection. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that any modifications, substitutions, or equivalent variations to the technical content without departing from the core idea of ​​the present invention should be considered to fall within the scope of protection of the present invention, and the scope of protection of the present invention should be determined by the claims.

Claims

1. A method for locating agricultural image text references based on semantic space alignment, characterized in that: The method comprises the following steps: S1: Collecting agricultural image and agricultural target text language description data, constructing an agricultural image text reference positioning dataset, and dividing the dataset into a training set, a validation set and a test set according to a ratio of 6:2:2, which are distributed for subsequent model training, validation and evaluation; S2: Constructing an agricultural image text reference positioning model for obtaining the coordinates of an agricultural entity target described by text language, the model structure comprising five parts: a multi-modal feature extraction module, a fine-grained feature enhancement module based on semantic space alignment guidance, a query weight initialization module, a cross-modal reference counting module and a target decoding module; S21: Inputting the agricultural image and agricultural target text language description data in step S1 into the multi-modal feature extraction module for feature extraction to obtain an initial feature image and an initial feature text; The multi-modal feature extraction module comprises an image feature extraction submodule and a text feature extraction submodule; S22: input the initial feature image and the initial feature text into the fine-grained feature enhancement module based on semantic space alignment guidance respectively, first obtain the image feature sequence , and then obtain the target enhanced feature image ; The fine-grained feature enhancement module based on semantic space alignment guidance comprises a semantic space cross-modal alignment submodule and a visual target enhancement submodule; S23: input the initial feature image and the initial feature text in the S21 step into a query weight initialization module to obtain an initialized query The query weight initialization module comprises a cross-modal interaction unit, a weight scoring network, and a query generator. S24: inputting the target enhanced feature map into the cross-modal reference counting module to obtain a classification result, i.e., a predicted number of reference targets The initial feature map is input into the cross-modal reference counting module as a query of the attention mechanism to obtain a classification result, i.e., a predicted number of reference targets The cross-modal reference counting module is composed of a series connection of an intra-modal multi-head attention layer, a cross-modal multi-head attention layer, and a feedforward network classifier. S25: initialize query Query as attention mechanism, input the enhanced feature map into the target decoding module, according to the number of reference targets Filter out the top Box with the highest confidence, get the initial coordinates of the agricultural entity target described by the text language; the target decoding module is composed of two cascaded multi-head attention layers and two parallel feedforward networks; S3: Training the agricultural image text reference positioning model, inputting the agricultural image and agricultural target text language description data into the model, calculating the total loss value of the model, performing reverse gradient propagation, optimizing the weight parameters of the model through a predetermined optimizer, and obtaining the final agricultural image text reference positioning model after multiple rounds of training; S4: Based on the trained agricultural image text reference positioning model, inputting the agricultural image to be positioned and the agricultural target text language description data to obtain the final coordinates of the agricultural entity target described by text language. 2.The method of claim 1, wherein: In step S21, the agricultural image text reference positioning data is input into the multi-modal feature extraction module to obtain an initial feature image and an initial feature text, and the specific implementation steps are as follows: S21.1: In the image feature extraction submodule of the agricultural image, local features are obtained through the ResNet50 backbone network, the local features are flattened to obtain a spatial sequence, and the spatial sequence is input into a 6-layer Transformer encoder for global context modeling to obtain an initial feature image wherein is the feature dimension, is the total number of spatial positions; S21.2: The agricultural target text language description data is segmented, and a text feature extraction submodule composed of a pre-trained BERT-Base model is used to take the last layer hidden state as the encoding result to obtain the initial feature text of the preliminary encoding wherein is the text sequence length. 3.The method of claim 1, wherein: In step S22, the image feature sequence is obtained The specific implementation steps are as follows: the initial feature image , the initial feature text In the input semantic space cross-modal alignment sub-module, the initial feature image is subjected to spatial structure modeling through two convolutional layers to obtain enhanced feature images and ; meanwhile, the initial feature text is subjected to a linear transformation layer to obtain a first semantic projection feature ; then the initial feature text is subjected to another linear layer to obtain a second semantic projection feature , the matrix product between the transpose of is obtained to obtain a similarity matrix, the similarity matrix is normalized through a Softmax function to obtain a normalized similarity matrix, and the normalized similarity matrix is multiplied by the second semantic projection feature to obtain an alignment feature at the semantic level, and the alignment feature at the semantic level is further concatenated with in the channel dimension to obtain a concatenated feature, and the concatenated feature is subjected to a linear projection layer for dimension reduction to obtain a semantic space aligned image feature sequence .

4. The method of claim 1, wherein: In step S22, the initial feature image is... Image feature sequences The visual target enhancement submodule is used to obtain the target enhancement feature map. The specific implementation steps are as follows: Sa1: input the initial feature image input linear projection matrix , and the image feature sequence output by the semantic space cross-modal alignment module input linear projection matrix , respectively, to obtain projected features and ; Sa2: generating an enhancement coefficient matrix using a Gaussian weighting function based on the projected features described in Sa1 , as follows: ; wherein, with are learnable parameters, denotes a dot product similarity function, denotes an exponential function for calculating a Gaussian weighting, calculating its dot product similarity ; Sa3: the initial feature image layer normalization processing is performed, and the initial feature image and the enhancement coefficient matrix element-wise multiplication is performed, and the element-wise multiplication result residual connection is performed, and finally the target enhancement image is obtained through layer normalization operation again The calculation formula is as follows: ; wherein, represents a layer normalization function, represents an initial feature image a value that is processed by the layer normalization.

5. The method of claim 1, wherein: In step S23, the initial feature image and the initial feature text are input into the query weight initialization module to obtain an initialized query The specific operation steps are as follows. Sb1 : the initial feature image and initial feature text As input, first compute the initial feature text and the product between the transpose of the initial feature image and the initial feature text; The product between the transposed matrices is then divided by the scaling factor for the feature dimension, the result is normalized by applying the Softmax function; The normalized matrix is then multiplied with the initial feature image The multiplication result is finally added with the initial feature text The element-wise addition is performed to obtain the text feature fused with visual semantics The formula of the whole process is shown as follows: ; wherein denotes a Softmax function, denotes a transpose; Sb2: the fused visual semantic text feature obtained in step Sb1 respectively, and then input the pooled results into two parameter-shared multi-layer perceptrons respectively to obtain two feature vectors and ; element-wise addition is performed on and , and then the addition result is sequentially normalized by a feedforward neural network and a Softmax function to obtain a final text channel importance weight score matrix ; Sb3: according to the importance weight matrix , the top token features with the highest importance are selected from the initial feature text , denoted as , and transformed through a linear projection layer containing two linear layers and a ReLU activation function, and connected in residual with the original feature , and finally, the result is input into the multi-head self-attention layer for interaction and suppression between queries to obtain the initialized query .

6. The method of claim 1, wherein: In step S24, the initial feature image is input into the cross-modal reference counting module to obtain a classification result, and the specific operation steps are as follows: Sc1: the enhanced feature image as Query of the attention mechanism, the initial feature image as Key and Value, the initial feature image is input to the intra-modal multi-head attention layer for calculation to obtain the self-enhanced feature ; Sc2: the output of the above step Sc1 is self-enhanced feature map As Key and Value of cross-modal attention mechanism, the initial feature text As Query, cross-modal attention interaction is performed to obtain refined multi-modal fusion features ; Sc3: the multi-modal fusion features described in step Sc2 above with the initial feature text element-wise addition, and the added results are passed through a feedforward neural network composed of three fully connected layers to obtain a classification result, i.e., the predicted number of reference targets The entire process is represented by the following formula: ; wherein represents a feedforward neural network, resulting in a classification.

7. The method of claim 1, wherein: In step S25, the enhanced feature image is input into the target decoding module to obtain the initial coordinates of the agricultural entity target described by text language, and the processing steps are as follows: Sd1: initialize the query weight output by the query weight initialization module Query as attention mechanism, enhance the feature map Key and Value as attention mechanism, input into the first multi-head attention layer to interact, get the enhanced query based on visual context ; Sd2: the enhanced query based on visual context outputted from the above step Sd1 Query as the attention mechanism, input the initial feature text Key and Value as the attention mechanism, input to the second multi-head attention layer for interaction, to obtain the enhanced query based on the text context ; Sd3: the text context-based enhanced query outputted by the above step Sd2 two parallel feedforward neural networks are simultaneously input : the first prediction confidence of each of the (representing the number of maximum targets in the current data set) target boxes; the second for regression prediction initial coordinate parameters of the target box (x0, y0, w, h) , wherein and are the initial horizontal and vertical coordinates of the center point of the target bounding box, respectively, and are the width and height of the target bounding box, respectively; Sd4: The target number predicted by the cross-modal reference counting module From the step Sd3 above Select the top boxes with the highest confidence from the predicted boxes to obtain the initial coordinates of the agricultural entity target described by the text language.

8. The method of claim 1, wherein: In step S3, the agricultural image and agricultural target text language description data are input into the model, and the total loss value of the model is calculated, and the total loss calculation formula is as follows: ； The total loss function is composed of a bounding box regression loss , a prediction confidence loss , and a referent count loss . wherein, hyperparameters balancing the various losses; the bounding box regression loss is calculated using an L1 loss function and a generalized IoU loss function, which is calculated based on real bounding box coordinates and predicted bounding box coordinates and a Hungarian matching strategy is used to determine the correspondence between the predicted boxes and the real boxes; the prediction confidence loss computed using a binary cross-entropy loss function; The target count loss The target count loss is calculated using the Focal Loss function, and the calculation formula is as follows: ; wherein, is a class weight, is a focus parameter, denotes the predicted probability of the model for the true class, The calculation formula of is as follows: ； wherein, is a true label of the class , denotes a predicted probability of the class .

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the method of any one of claims 1-8.

Images