A rule supervision method based on fine-grained image text matching

Abstract

The application discloses a rule supervision method based on fine-grained image text matching. First, the rule text is converted into abstract semantic representation, then the abstract semantic representation form text vector and natural language text vector are combined into a new text node vector, then the image features of the image to be supervised are extracted and the text vector and the image features are fused, and then it is judged whether the image and the fine-grained text match and logical splicing is performed to complete the rule supervision. The method aims to input rule text and image to be supervised, and output whether the image conforms to the supervision result of the rule text. When the image and the whole sentence rule text do not match, the application method can obtain the specific reason for the inconsistency from the relationship between the fine-grained rule text and the image. In addition, the application method can avoid the problems of low efficiency and incorrect judgment caused by traditional manual supervision.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and natural language processing, and relates to a rule-based regulatory method based on fine-grained image-text matching. Background Technology

[0002] Traditional rule-based regulatory methods primarily rely on human inspection, manually comparing the image to be regulated with the rule text to identify examples that do not comply with the regulations. This approach is not only inefficient but also prone to errors during manual judgment.

[0003] Currently, the main method for monitoring rule text and images to be monitored is to match the image with the entire rule text. This method can only determine the relationship between the image to be monitored and the rule text, but cannot identify the specific contradictions when the rules are not followed. Summary of the Invention

[0004] To address the aforementioned technical deficiencies in existing technologies, this invention provides a rule-based regulatory method for fine-grained image-text matching, which achieves rule regulation by determining whether an image and fine-grained text match and then logically concatenating them.

[0005] Specifically, this invention represents rules in text form and completes rule supervision by judging whether the image and fine-grained text are aligned and performing logical splicing.

[0006] A rule-based regulatory method based on fine-grained image-text matching aims to take as input rule text and an image to be regulated, and output whether the image conforms to the rule text. The method includes the following steps:

[0007] (1) Rule text processing. The rule text is abstracted into concept nodes and semantic relations, and represented in the form of AMR (Abstract Semantic Representation). The concept nodes and semantic relations are converted into vector form through the AMR language model, and the rule text in natural language form is converted into vector form through the natural language model. The two vectors are combined into a new vector as the text node vector.

[0008] (2) Image processing. Image features are extracted using ViT-B / 16 and the image is converted into an image vector.

[0009] (3) Image and text feature fusion. Image and text features are fused based on cross-attention. By defining the source of the query matrix Q as text information and the source of the key matrix K and value matrix V as image information, the correlation between image information and text information is obtained.

[0010] (4) Classification and Judgment. The fusion features of each node are classified and judged. The method of this invention divides the relationship between image and text nodes into three categories, representing that the image to be supervised conforms to the rule text, the image to be supervised does not conform to the rule text, and the image to be supervised and the rule text are unrelated.

[0011] (5) Logical concatenation and output of results. By logically combining the judgment results of each node, the result of whether the image to be monitored conforms to the rule text is obtained. When there is a discrepancy between the image and the entire rule text, the method of the present invention can derive the specific reasons for the discrepancy from the relationship between the fine-grained rule text and the image.

[0012] In step (1), the rule text is abstracted into concept nodes and semantic relations, and the rule text vectors in AMR form and natural language form are combined, specifically including:

[0013] An AMR parser converts the rule text into AMR form, abstracting the rule text into concept nodes and semantic relations, dividing it into multiple text nodes. Two language models are defined: one takes the AMR-form rule text as input and outputs a vector of AMR-form rule text; the other takes the natural language-form rule text as input and outputs a vector of natural language-form rule text. The outputs of the two language models are combined to form the final text node vector.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] This invention's rule-based supervision method, based on fine-grained image-text matching, avoids the inefficiencies and misjudgments inherent in traditional manual supervision. Furthermore, methods that match images with entire rule texts can only determine the relationship between the image and the rule text, but cannot pinpoint specific points of discrepancy when rules are not met. This invention's method, through fine-grained text segmentation and analysis of each text node, can locate specific points of discrepancy when they occur. Attached Figure Description

[0016] Figure 1 A flowchart of a rule-based regulatory method based on fine-grained image-text matching;

[0017] Figure 2 This is a schematic diagram of the AMR representation of rule text. Detailed Implementation

[0018] To describe the present invention in more detail, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] This example aims to implement rule-based supervision based on fine-grained image-text matching according to the present invention. The method includes steps such as rule text processing, image processing, feature fusion, classification judgment, and logical concatenation to output the result, as follows: Figure 1 As shown, the specific implementation process is as follows:

[0020] (1) Rule Text Processing. This example uses an AMR parser to abstract the rule text into text semantics and text relation nodes, such as... Figure 2 As shown. Then, the concept nodes and semantic relationships are converted into vector form using an AMR language model, and the rule text in natural language form is converted into vector form using a natural language model. Finally, the two vectors are combined into a new vector as the text node vector:

[0021] A i =AMR_BertEmbedding(a i )

[0022] T i = BertEmbedding(t i )

[0023] W i =A i +T i

[0024] Where a i Let t represent the i-th text semantic and text relation node. i A represents the corresponding i-th natural language text field. i and T i They represent a respectively i After AMR language model and t i The vector W after transformation by the natural language model i This represents the combined text node vector. The language model is initialized based on the first six layers of the BERT model.

[0025] (2) Image Processing. This example extracts image features using a 12-layer ViT-B / 16 layer. The input image I is converted into a series of image vectors:

[0026] V=ViTEmbedding(I)={v1,v2,…,v n}

[0027] Where I represents the input image of ViT, V represents the series of vectors into which the image is converted, and v i This represents each field that makes up the vector.

[0028] (3) Image-text feature fusion. This example uses the last six layers of BERT as the basis for the multimodal fusion encoder. In each layer of the multimodal encoder, image features and text features are fused through cross-attention.

[0029] Cross-attention is an improvement upon self-attention. Self-attention allows the model to notice the correlations between different parts of the input data. For an input text vector W, self-attention constructs a query matrix Q, a key matrix K, and a value matrix V through linear transformations:

[0030] Q = ω q ·W

[0031] K = ω k ·W

[0032] V=ω v ·W

[0033] Where ω q ω k ω v These are the weight parameters that need to be learned. The query matrix Q, key matrix K, and value matrix V are all obtained by multiplying the input vector W by the corresponding weights.

[0034] The correlation between the two input vectors can be calculated using the obtained query matrix Q and key matrix K.

[0035] A = K T ·Q

[0036] Here, A represents the correlation between the two input vectors. To ensure that the probability distribution of the output weights sums to 1, a softmax operation is then performed on A. When the vector dimension d... k When the vectors are large, the value of the vector dot product can easily become very large, while the gradient of the softmax function becomes very small. Therefore, it is necessary to divide by before performing softmax. Then, the processed A' and V are multiplied together to calculate the Attention:

[0037]

[0038] Cross-attention modifies the input of self-attention, so that only the query matrix Q comes from its own output, while the key matrix K and value matrix V come from other outputs.

[0039] In this example, in order to fuse information from the two modalities of image and text, the source of the query matrix Q is defined as text information, and the source of the key matrix K and value matrix V is defined as image information, thereby obtaining the correlation between image information and text information.

[0040] (4) Classification and Judgment. In this example, the relationship between image and text nodes is divided into three categories: Entailment, Contradiction, and Neutral, which respectively represent that the image to be monitored conforms to the rule text, the image to be monitored does not conform to the rule text, and the image to be monitored and the rule text are unrelated.

[0041] This example uses a fully connected layer, a ReLU activation function layer, and a three-class classification network to determine the relationship between image and text nodes after fusing the features. The model training uses cross-entropy loss, defined as:

[0042]

[0043] Where y i p represents the true value. i This represents the predicted value, and N represents the number of text nodes.

[0044] (5) Logical concatenation and output of results. This example uses logical combination of the judgment results of each node to obtain the result of whether the image to be monitored conforms to the rule text.

[0045] When the relationship between all text nodes and the image is implicit, it can be concluded that the image to be monitored conforms to the whole sentence rule text:

[0046]

[0047] When there is a contradiction between the relationship between text nodes and images, it can be concluded that the image to be monitored does not conform to the rules of the entire sentence:

[0048]

[0049] When there exists a neutral relationship between a text node and an image, and no contradictory relationship exists between them, it can be concluded that the relationship between the image to be monitored and the entire rule text is uncertain.

[0050]

[0051] Here, label(n) represents the relationship between text node n and the image, label(s) represents the relationship between the entire text rule s and the image, entailment represents the implication label, contradiction represents the contradiction label, and neutral represents the neutral label. After obtaining the relationships between all nodes of the rule text and the image, the relationship between the rule text and the image to be monitored can be derived through the above logic. When a discrepancy occurs between the image and the entire rule text, analyzing the relationship between text node n and the image can pinpoint the specific cause of the discrepancy.

[0052] Finally, it should be noted that the above examples are only used to illustrate and not limit the technical solutions of the present invention. Although the present invention has been described in detail with reference to the above examples, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the present invention without departing from the spirit and scope of the present invention. Any modifications or partial substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A rule-based regulatory method based on fine-grained image-text matching, characterized in that, Includes the following steps: (1) The rule text is abstracted into concept nodes and semantic relations. The concept nodes and semantic relations are converted into vector form through the AMR language model. The rule text in natural language form is converted into vector form through the natural language model. The two vectors are combined into text node vectors, specifically including: The AMR parser converts the rule text into AMR form, which involves abstracting the rule text into concept nodes and semantic relations, dividing it into multiple text nodes, and defining two language models. One AMR language model takes the rule text in AMR form as input and outputs a vector of rule text in AMR form. The other natural language model takes the rule text in natural language form as input and outputs a vector of rule text in natural language form. The outputs of the two language models are combined to form the final text node vector. (2) Extracting image features from the images to be monitored; (3) Based on the cross-attention mechanism, the image features obtained in step (2) and the text node vectors obtained in step (1) are fused to obtain the correlation between image information and text information, which is the fused feature node; (4) Classify and judge the fusion feature nodes obtained in step (3) to obtain the image matching results of each text node; (5) Logically concatenate the image matching results of each text node obtained in step (4) and output the result; When the relationship between all text nodes and images is implicit, it can be concluded that the image to be monitored conforms to the entire sentence rule text: When there is a contradiction between the relationship between text nodes and images, it is concluded that the image to be monitored does not conform to the rules of the entire sentence: When there exists a neutral relationship between text nodes and images, and no contradictory relationship exists between text nodes and images, it can be concluded that the relationship between the image to be monitored and the entire rule text is uncertain. in This represents the relationship between the text node n and the image. This indicates the relationship between the entire text rule s and the image. entailment represents the implication label, contradiction represents the contradiction label, and neutral represents the neutral label.

2. The rule-based regulatory method based on fine-grained image-text matching as described in claim 1, characterized in that, In step (2), image features of the image to be monitored are extracted using ViT-B / 16.

3. The rule-based regulatory method based on fine-grained image-text matching as described in claim 1, characterized in that, In step (5), the output result is the relationship between the whole rule text and the image to be monitored.

Images