A method and system for generating agricultural scene graphs based on dual attention perception

Abstract

This invention belongs to the field of agricultural technology based on image processing applications. It discloses a dual-attention-based method for generating agricultural scene graphs, comprising the following steps: performing object detection, candidate box feature extraction, and object recognition on the input image to obtain the visual features, class label features, positional features, and union box visual features of the candidate boxes; fusing the visual features, positional features, and semantic features of objects to more accurately identify object categories; introducing a dual-attention network, using relational attention and object attention modules to aggregate volume context features to obtain relational representations, thereby identifying the relationships between objects and obtaining the scene graph. The dual-attention-based agricultural scene graph generation method proposed in this invention can guide the selection of useful contextual information for fusion, ignore noise information, accurately identify the relationships between objects, and greatly improve the effectiveness and generalization of agricultural scene graphs.

Description

Technical Field

[0001] This invention belongs to the field of image processing application technology, specifically relating to a method and system for generating agricultural scene images based on dual attention perception. Background Technology

[0002] Agricultural scene graph generation technology identifies objects and relationships between them in a given agricultural image. For example, it identifies farmers, farmland, drones, etc., and the pairwise relationships between them, forming relation triples such as <farmer, standing, farmland> and <drone, irrigation, farmland>, which can be used for agricultural scene monitoring. However, existing scene graph generation methods extract relation triple features by exploring the information interactions between objects, neglecting the guiding role of predicate semantics in contextual information fusion. This makes it difficult to obtain highly discriminative relation representations, resulting in low model prediction accuracy, poor generalization performance, and low information content and utility in the generated agricultural scene graphs. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides a dual-attention perception method for generating agricultural scene graphs, which solves the problems of lack of discriminativeness in relation representation and low accuracy in relation category identification.

[0004] To achieve the above objectives, the technical solution adopted by this invention is: a method for generating agricultural scene graphs based on dual attention perception, comprising the following steps:

[0005] The system performs target detection, candidate box feature extraction, and target recognition on the input agricultural image to obtain the visual features of the candidate boxes, the class label features of the candidate boxes, the positional features of the candidate boxes, and the visual features of the union box.

[0006] By fusing visual features, positional features, and class label features of objects, object categories can be identified more accurately.

[0007] A dual-attention network is introduced, which combines relational attention and object attention modules to aggregate body context features to obtain relational representations, thereby identifying the relationships between objects and obtaining a scene graph.

[0008] The specific steps for object detection, candidate bounding box feature extraction, and object recognition in agricultural images are as follows:

[0009] For the input agricultural image, the Faster R-CNN network is first used to locate candidate boxes. For each candidate bounding box, the object detection network obtains a visual feature, a class label feature, and a location feature;

[0010] Visual features of candidate boxes v i After the backbone network obtains the image feature map, it selects the candidate bounding box b. iMap the bounding box (v) onto the feature map, and then use ROI pooling or ROI alignment to obtain a bounding box feature of uniform size. i ;

[0011] Class label features p of candidate boxes i The pre-trained GloVe model transforms discrete candidate box category labels into 200-dimensional continuous word embedding features to represent the category information of the candidate boxes.

[0012] Positional features of candidate boxes i The candidate box is obtained by concatenating its two center coordinates, four position coordinates, and three area features, and then performing a linear transformation.

[0013] The positional features of the candidate bounding boxes are as follows:

[0014] Among them, (x i1 ,y i1 ,x i2 ,y i2 ) represents the position coordinates of the candidate box, and w and h represent the width and height of the candidate box, respectively.

[0015] When extracting visual features from a union box, for two matching candidate boxes b i and b j Calculate the union box position, then map the union box onto the image feature map, and use ROI pooling or ROI alignment to obtain a visual feature u of the union box of uniform size. ij .

[0016] The method of fusing visual features, positional features, and class label features of objects to more accurately identify object categories is as follows:

[0017] For candidate box b i The visual features, class label features, and location features are fused to obtain multimodal features. Then, a linear layer is used to reduce the dimensionality of the multimodal representation to obtain an initial target feature x of 512 dimensions. i ;

[0018] The initial target features are fed into the encoder formed by the attention module to model the feature interactions between targets. The self-attention calculates the similarity between each target feature and other target features, uses the similarity to weight the features, calculates the attention in multiple subspaces, and then obtains the context target features through the feedforward neural network.

[0019] Predict the category label of an object using contextual target features.

[0020] The relationship attention and object attention modules aggregate object context features to obtain relationship representations, thereby identifying relationships between objects and generating scene graphs, as detailed below:

[0021] The relational attention module uses the visual features of the union box as query features and the predicate embeddings as key and value features. It queries the predicate embedding features through the visual features of the union box to obtain the semantic representation of the predicate.

[0022] The predicate semantic features and the updated target features are used as input to the object attention module, and the output is contextual features related to the predicate.

[0023] By merging the contextual features, subject object features, and object object features obtained from object attention, a relation representation is obtained;

[0024] The relationship between paired objects is predicted using the relationship characterization to obtain a scene graph.

[0025] The target features are updated according to the following rules to obtain the updated target features.

[0026] y i =FC(Concat(v i ,x i E O (c i ))),

[0027] Among them, E O It is the target word embedding matrix, c i The predicted object category label.

[0028] In addition, the present invention also provides a computer device, including a processor and a memory, the memory being used to store a computer executable program, the processor reading the computer executable program from the memory and executing it, and the processor executing the computer executable program can realize the dual attention perception agricultural scene graph generation method described herein.

[0029] Simultaneously, a computer-readable storage medium may be provided, in which a computer program is stored, which, when executed by a processor, can implement the dual-attention perception agricultural scene graph generation method described herein.

[0030] Compared with the prior art, the present invention has at least the following beneficial effects:

[0031] The proposed dual-attention perception method for generating agricultural scene graphs utilizes a relational attention mechanism to explore the feature interaction between relation triples and predicate embeddings, which can transform complex visual content into compact semantic representations. Object attention uses semantic representations to guide the aggregation of object context information, which can filter out irrelevant noise features and obtain highly discriminative relational representations, greatly improving the accuracy and generalization of agricultural relation recognition. Attached Figure Description

[0032] Figure 1 This is a method for generating agricultural scene graphs using dual attention perception.

[0033] Figure 2 The diagram shows the attention modules: (a) Relational attention module, (b) Object attention module. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] Based on the same technical concept, the present invention provides an agricultural scene graph generation system with dual attention perception, including an object detection module, an object recognition module, and a relationship recognition module;

[0036] The object detection module performs target detection, candidate box feature extraction, and target recognition on agricultural images, obtaining the visual features of candidate boxes, the class label features of candidate boxes, the positional features of candidate boxes, and the visual features of the union of pairs of objects.

[0037] The object recognition module integrates the visual features, positional features, and class label features of objects to more accurately identify object categories;

[0038] The relationship recognition module introduces a dual attention network, which combines the relation attention and object attention modules to aggregate the body context features to obtain relationship representations, thereby recognizing the relationships between objects and obtaining a scene graph.

[0039] Specifically, such as Figure 1 As shown, the dual-attention perception agricultural scene graph generation system proposed in this invention consists of three modules:

[0040] 1) Object detection module: Utilizes Faster R-CNN to detect objects in agricultural images and extracts the visual features, label features, location features, and visual features of the union boxes between each pair of objects;

[0041] 2) Object recognition module: It fuses the visual features, positional features and label features of objects, feeds the resulting multimodal features into the encoder for contextual information fusion, and then uses a fully connected network (FC) to identify the object category;

[0042] 3) Relationship recognition module: A dual attention network is introduced. The relation attention module and object attention module are used to aggregate the body context to obtain the relationship representation. Then, a fully connected network is used to identify the relationship between the subject object and the object object to obtain the scene graph.

[0043] The dual-attention perception method for generating agricultural scene graphs proposed in this invention can guide the selection of useful contextual information for fusion and ignore noise information, which greatly improves the accuracy and generalization of agricultural relationship recognition.

[0044] Specific implementation of the invention:

[0045] 1) Object detection

[0046] Given an agricultural image, first use the Faster R-CNN network to locate candidate bounding boxes. For each candidate bounding box, the object detection network obtains a visual feature, a class label feature, and a location feature. Furthermore, for two matched candidate bounding boxes b... i and b j The object detection network extracts visual features from the union box.

[0047] Visual features of candidate boxes v i After the backbone network obtains the image feature map, it selects the candidate bounding box b. i Mapping onto a feature map, and then using ROI pooling or ROI alignment to obtain visual features v of candidate boxes of uniform size. i .

[0048] Class label features p of candidate boxes i The pre-trained GloVe model transforms discrete candidate box category labels into 200-dimensional continuous word embedding features to represent the category information of the candidate boxes.

[0049] Positional features of candidate boxes i To comprehensively encode the spatial location information of candidate boxes, the two center coordinates, four position coordinates, and three area features of the candidate boxes are concatenated, and a linear transformation is performed to obtain the position features of the candidate boxes.

[0050]

[0051] Among them, (x i1 ,y i1 ,x i2 ,y i2 ) represents the position coordinates of the candidate box, and w and h represent the width and height of the candidate box, respectively.

[0052] Union box visual features u ij For the two matching candidate boxes b i and b j The union box positions are calculated, and then the union boxes are mapped onto the image feature map. ROI pooling or ROI alignment is used to obtain visual features of the union boxes of uniform size. ij .

[0053] 2) Object recognition

[0054] Multimodal feature fusion: for candidate box b i Integrating its visual features v i Class label features p i and location features l i To obtain multimodal features, a linear layer is then used to reduce the dimensionality of the multimodal representation to obtain an initial target feature x of 512 dimensions. i ,

[0055] x i =FC(Concat(v i p i , l i ))

[0056] The initial set of target features for all candidate bounding boxes in image I is:

[0057] Contextual Feature Interaction: The initial target features X are fed into the attention module to model the feature interactions between targets. Self-attention calculates the similarity between each target feature and other target features, and uses the similarity to weight the features. To learn richer contextual information, attention is calculated in multiple subspaces, and then the contextual target features are obtained through a feedforward neural network.

[0058] X' = ​​FFN(MultiHead(X,X,X)),

[0059] MultiHead(X,X,X)=Concat(head1,...,head h W O ,

[0060]

[0061] Among them, Q i (X), K i (X),V i (X) represent the query feature, key feature, and value feature obtained by performing a linear transformation on X in the i-th subspace, respectively. For scaling factor, head iFor the attention of the i-th subspace, W O Let FFN be a linear mapping matrix, representing a two-layer feedforward neural network.

[0062] Object classification: Predicting the category label of an object using contextual target features.

[0063] c i =argmax(softmax(FC(x′) i ))).

[0064] 3) Relationship identification

[0065] Relational attention: such as Figure 2 As shown in (a), the relational attention module transforms the visual features of the union box into predicate semantic features through a query-aggregation process based on multi-head self-attention. The predicate embedding matrix is ​​defined as E. r Then the relational attention will be the union of the visual features u of the bounding box. ij As query features, predicate embeddings serve as both key and value features. By querying the predicate embedding features through the union box visual features, we obtain the semantic representation of the predicates.

[0066]

[0067] MultiHead(u ij E r E r = Concat(head1, ..., head) h W O ,

[0068]

[0069] Object attention: such as Figure 2 As shown in (b), the object attention module will output the predicate semantic features of the relation attention module. and updated target features As input, the output is contextual features related to the predicate. First, update the target features.

[0070] y i =FC(Concat(v i x i E O (c i ))),

[0071] Among them, E OThis is the target word embedding matrix. Then, the predicate semantic features are mapped to query features, and the updated target features are mapped to key and value features. The similarity between the query features and key features is calculated to weight the value features, thereby achieving the fusion of contextual features related to the predicate semantics.

[0072]

[0073] Then, input feature mapping and attention computation are performed in multiple subspaces, and the outputs of all subspaces are merged and linearly transformed to obtain context features.

[0074]

[0075]

[0076] Relationship representation: By merging the contextual features obtained from object attention and the features of paired object i and object j, a relationship representation is obtained.

[0077]

[0078] Relation Classification: Predicting the Relationship Between Subject and Object Objects Using Relation Representations ij ,

[0079] r ij =argmax(Softmax(FC(f ij ))).

[0080] Simultaneously, a dual-attention perception agricultural scene graph generation system is provided, including an object detection module, an object recognition module, and a relationship recognition module;

[0081] The object detection module performs object detection, bounding box feature extraction, and object recognition on agricultural images;

[0082] The object recognition module integrates the visual features, positional features, and semantic features of objects to more accurately identify object categories;

[0083] The relationship recognition module introduces a dual attention network, which combines the relation attention and object attention modules to aggregate the body context features to obtain relationship representations, thereby recognizing the relationships between objects and obtaining a scene graph.

[0084] In addition, the present invention can also provide a computer device, including a processor and a memory, wherein the memory is used to store a computer executable program, the processor reads part or all of the computer executable program from the memory and executes it, and the processor can realize the dual attention perception agricultural scene map generation method of the present invention when executing part or all of the computer executable program.

[0085] On the other hand, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the dual-attention perception agricultural scene graph generation method described in the present invention.

[0086] The computer device may be a laptop, a desktop computer, or a workstation.

[0087] The processor can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or an off-the-shelf programmable gate array (FPGA).

[0088] The memory described in this invention can be an internal storage unit of a laptop, desktop computer, or workstation, such as memory or hard disk; or it can be an external storage unit, such as a portable hard disk or flash memory card.

[0089] Computer-readable storage media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. Random access memory can include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

Claims

1. A method for generating agricultural scene graphs using dual attention perception, characterized in that, Includes the following steps: Target detection, candidate box feature extraction, and target recognition are performed on agricultural images to obtain the visual features of candidate boxes, the class label features of candidate boxes, the positional features of candidate boxes, and the visual features of the union of pairs of objects. By fusing visual features, positional features, and class label features of objects, object categories can be identified more accurately; specifically as follows: For candidate boxes The visual features, class label features, and location features are fused to obtain multimodal features. Then, a linear layer is used to reduce the dimensionality of the multimodal representation to obtain an initial target feature of 512 dimensions. ; The initial target features are fed into the encoder formed by the attention module to model the feature interactions between targets. The self-attention calculates the similarity between each target feature and other target features, uses the similarity to weight the features, calculates the attention in multiple subspaces, and then obtains the context target features through the feedforward neural network. Predict the category label of an object using contextual target features; A dual-attention network is introduced, which combines volume context features through relational attention and object attention modules to obtain relation representations, and then predicts the relationships between objects to obtain a scene graph. The specific steps for obtaining relation representations and then performing relation recognition through the combination of relational attention and object attention modules with volume context features are as follows: The relational attention module uses the visual features of the union box as query features and the predicate embeddings as key and value features. It queries the predicate embedding features through the visual features of the union box to obtain the semantic representation of the predicate. The predicate semantic features and the updated target features are used as input to the object attention module, and the output is contextual features related to the predicate. By merging the contextual features, subject object features, and object object features obtained from object attention, a relation representation is obtained; The relationship characterization is used to predict the relationship between paired objects; The target features are updated according to the following rules to obtain the updated target features. in, It is the target word embedding matrix. For the predicted object category label, Visual features of candidate boxes.

2. The method for generating agricultural scene graphs using dual attention perception according to claim 1, characterized in that, The specific steps for object detection, candidate bounding box feature extraction, and object recognition in agricultural images are as follows: For the input agricultural image, the Faster R-CNN network is first used to locate candidate boxes. For each candidate box, the object detection network obtains a visual feature, a class label feature, and a position feature; Visual features of candidate boxes After the backbone network obtains the image feature map, it sets the candidate boxes... Mapping onto a feature map, and then using ROI pooling or ROI alignment to obtain visual features of candidate boxes of uniform size. ; Class label features of candidate boxes The pre-trained GloVe model transforms discrete candidate box category labels into 200-dimensional continuous word embedding features to represent the category information of the candidate boxes. Location features of candidate boxes The candidate box is obtained by concatenating its two center coordinates, four position coordinates, and three area features, and then performing a linear transformation.

3. The method for generating agricultural scene graphs using dual attention perception according to claim 2, characterized in that, The positional features of the candidate bounding boxes are as follows: in, These are the position coordinates of the candidate box. and These are the width and height of the candidate box, respectively.

4. The method for generating agricultural scene graphs using dual attention perception according to claim 1, characterized in that, When extracting visual features from a union box, for two matching candidate boxes... and Calculate the union box position, then map the union box onto the image feature map, and use ROI pooling or ROI alignment to obtain visual features of the union box of uniform size. .

5. A dual-attention perception agricultural scene graph generation system, characterized in that, The method for generating agricultural scene graphs with dual attention perception as described in any one of claims 1 to 4 includes an object detection module, an object recognition module, and a relationship recognition module. The object detection module performs target detection, candidate box feature extraction, and target recognition on agricultural images, obtaining the visual features of candidate boxes, the class label features of candidate boxes, the positional features of candidate boxes, and the visual features of the union of pairs of objects. The object recognition module integrates the visual features, positional features, and semantic features of objects to more accurately identify object categories; The relationship recognition module introduces a dual attention network, which combines the relation attention and object attention modules to aggregate the body context features to obtain relationship representations, thereby recognizing the relationships between objects and obtaining a scene graph.

6. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer-executable program, the processor reading the computer-executable program from the memory and executing it, and the processor executing the computer-executable program being able to implement the dual-attention perception agricultural scene graph generation method of any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, enables the dual-attention perception method for generating agricultural scene graphs as described in any one of claims 1 to 4.

Images