Neurological reasoning-assisted visual language interpretable learning method and system

Abstract

The invention relates to the technical field of multi-modal model reasoning, in particular to a visual language interpretable learning method and system assisted by neural logic reasoning. The method comprises the following steps: carrying out vector representation on an image-text pair; performing relation matrix representation on the first-order logic; performing logic combination and multi-hop reasoning based on the first-order logic and the representation of the image-text pair; based on logic combination and multi-hop reasoning, a logic reasoning micro attention network fused with a visual language model is constructed. According to the method, the LogicVLM model is designed, so that a microframework combining neural inductive learning and logical reasoning can be realized. And further learning first-order logic and logic combination from the input visual text semantic concept, constructing a path of a tree structure to execute multi-hop reasoning, and completing a complex visual language reasoning task. In an experiment, compared with a traditional visual language model, the model training speed and the reasoning speed of the method are basically kept unchanged, and various tasks are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal model reasoning, and in particular to a method and system for interpretable learning of visual language assisted by neural logic reasoning. Background Art

[0002] In recent years, various large models have sprung up like mushrooms after rain, and the capabilities of artificial intelligence have been greatly improved. Multimodal models are more in line with real-world application scenarios, and related technologies have therefore become a hot topic for many teams in scientific research and industry to compete for research. At present, although existing multimodal models (such as CLIP, VILT, etc.) can complete tasks such as image-text matching well, they have significant defects in tasks that require deep logical understanding and reasoning. Although Deepseek-R1 and GPT-4o with more complex MOE architectures have good reasoning capabilities, they still cannot correctly answer simple tasks such as comparing numbers. This is because the rigorous reasoning process has not been completed in the underlying technology. The visual language models of existing technologies usually only rely on pure data sub-symbolic methods. Summary of the Invention

[0003] To address the technical problem that existing visual language models typically rely solely on pure data sub-symbol methods and fail to implement a rigorous reasoning process in the underlying technology, the present invention provides a method and system for interpretable learning of visual language assisted by neural logic reasoning. The technical solution is as follows:

[0004] On the one hand, a method for interpretable learning of visual language assisted by neural logic reasoning is provided, characterized in that the method includes:

[0005] S1. Obtain a dataset for training, and represent the image-text pairs in the dataset as vectors; and represent the first-order logic as a relational matrix.

[0006] S2, based on first-order logic and image-text pair representation, performs logical combination and multi-hop reasoning;

[0007] S3. Based on logical combination and multi-hop reasoning, a logical reasoning differentiable attention network is constructed that is integrated with the visual language model.

[0008] Optionally, in S1, vector representation of the image-text pair includes:

[0009] The vector representation of the image-text pair is performed according to the following formulas (1), (2) and (3):

[0010]

[0011]

[0012] Among them, t CLS and v CLS To facilitate the special notation of fine-tuning tasks, t represents text; v represents image; z0 represents the image-text pair vector group input by the model; t i and v i Represent text and visual vector embeddings respectively, T represents word embedding matrix; V represents linear projection embedding of image; T pos and V pos denote the learnable text and visual position embeddings, t type and v type denote learnable textual and visual type embeddings, respectively.

[0013] Optionally, in S1, a relation matrix representation of the first-order logic is performed, including:

[0014] For the first-order logic part, the entity is represented as a node object entity x∈X, where X represents the set of all possible regions X; the predicate is represented as the adjacency matrix M k , representing the relationship between two entity variables x and x′; then the first-order logic relationship matrix is expressed as shown in the following formula (4):

[0015]

[0016] Among them, ψ k represents the kth first-order logic relation; σ(·) represents the sigmoid activation function, The product representation of multiple relation matrices representing multi-hop reasoning; k is the sequence number of the first-order logic representation; represents a set of unary predicates; Represents a set of binary predicates.

[0017] Optionally, in S2, logical combination and multi-hop reasoning are performed based on first-order logic and the representation of image-text pairs, including:

[0018] By combining first-order logic statements, the symbolic reasoning space is expanded to form a multi-layered tree-structured reasoning space.

[0019] Introducing basic logical operations, and only considering logical AND and logical NOT, the logical operations are replaced by mathematical approximations, as shown in the following formula (5):

[0020]

[0021] Among them, \(p,q\in[0,1]\) represents the primitive statement of first-order logic;

[0022] Based on the first-order logic combination containing four basic logical operations, a logical combination representation is constructed, as shown in the following formula (6):

[0023]

[0024] Among them, the formula set f:f∈F l Each element in is called a function, so that f∈[0,1]; in the tree-like logical combination space: F0 represents the initial formula set, which only contains the original statement Ψ; F l-1 Represents a set of functions of level l-1, for F l-1 Perform a logical NOT operation to obtain its negated set representation Next level F l Each formula in F l-1 and Logical AND of the functions in ;

[0025] By setting the memory limit c to indicate the maximum number of combinations that can be tracked at each level, logical combinations of the next level logic statements with large probability are prioritized, that is, each level F l To be in Search for c logical AND combinations on the formulas in , so that the c-th formula of the l-th level is as shown in the following formula (7):

[0026]

[0027] Optionally, in S3, based on logical combination and multi-hop reasoning, a logical reasoning differentiable attention network integrated with the visual language model is constructed, including:

[0028] Define the learnable embeddings of all predicates as And define the dummy parameters X and X′ as embedding Therefore, the entire logical attention generation module is expressed as follows:

[0029]

[0030] The logical attention network consists of three multi-head attention modules in a stacked structure MA as a logical parameter generator; each attention module simulates predicate search, original statement search and logical combination formula search respectively.

[0031] Optionally, predicate search is used to select the best predicate, obtain operator path attention vectors and aggregate outputs;

[0032] The original statement search is used to output all possible predicates based on the predicate search, combined with the input entities, and combined into a basic unit that can be used for more complex logical reasoning;

[0033] Logical combination formula search is used to generate complex logical formulas.

[0034] On the other hand, a neural logic reasoning-assisted visual language interpretable learning system is provided, which is applied to the neural logic reasoning-assisted visual language interpretable learning method, and the system includes:

[0035] Embedding representation module, used to represent image-text pairs as vectors and first-order logic as relational matrices;

[0036] Combinatorial reasoning module, which is used to perform logical combination and multi-hop reasoning based on first-order logic and image-text pair representation;

[0037] The attention network building module is used to build a logical reasoning differentiable attention network integrated with the visual language model based on logical combination and multi-hop reasoning.

[0038] Optionally, the embedding representation module is further used to perform vector representation of the image-text pair according to the following formulas (1), (2), and (3):

[0039]

[0040] Among them, t CLS and v CLS To facilitate the special notation of fine-tuning tasks, t represents text; v represents image; z0 represents the image-text pair vector group input by the model; t i and v i Represent text and visual vector embeddings respectively, T represents word embedding matrix; V represents linear projection embedding of image; T pos and V pos denote the learnable text and visual position embeddings, t type and v type denote learnable textual and visual type embeddings, respectively.

[0041] On the other hand, a visual language interpretable learning device assisted by neural logical reasoning is provided, and the visual language interpretable learning device assisted by neural logical reasoning includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned neural logical reasoning assisted visual language interpretable learning methods is implemented.

[0042] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned neural logic reasoning-assisted visual language interpretable learning methods.

[0043] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0044] In the embodiment of the present invention, the advantages of symbolic reasoning, such as strong reasoning ability and high interpretability, are used to improve the problem of poor understanding and reasoning ability of the visual language model. By integrating differentiable logic networks with multimodal coding technology, the structured reasoning ability of the visual language model is significantly improved. Its core innovation lies in converting the first-order logic system (entity, predicate, logical formula) into differentiable operations (such as logical and p∧q=p*q, non- ), combined with a tree-like multi-hop reasoning mechanism to construct complex logical paths, and introduced a dynamic pruning strategy (based on memory constraints c) to optimize computational efficiency, addressing the problem of traditional models relying on data statistics and lacking symbolic reasoning. A multi-head attention module generates predicate embeddings and logical parameters, achieving a deep fusion of symbolic rules and neural networks, enhancing cross-modal alignment capabilities. Testing has shown progress in traditional visual language tasks. Significant results have been achieved in tasks such as visual language question answering and visual language reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0046] Figure 1 A flowchart of a method for interpretable learning of visual language assisted by neural logic reasoning provided by an embodiment of the present invention;

[0047] Figure 2 A block diagram of a visual language interpretable learning system assisted by neural logic reasoning provided by an embodiment of the present invention;

[0048] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0050] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0051] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0052] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0053] The embodiment of the present invention provides a method for interpretable learning of visual language assisted by neural logic reasoning, which can be implemented by a device for interpretable learning of visual language assisted by neural logic reasoning, and the device for interpretable learning of visual language assisted by neural logic reasoning can be a terminal or a server. Figure 1 The flowchart of the neural logic reasoning-assisted visual language interpretable learning method is shown in Figure 1 As shown, the neural logic reasoning-assisted visual language interpretable learning method proposed in the present invention may include the following steps:

[0054] S1. Obtain a dataset for training, and vectorize the image-text pairs in the dataset; represent the first-order logic using a relational matrix; wherein, the first-order logic (including entities, predicates, and formulas)

[0055] In a feasible implementation, the datasets used for training are Microsoft COCO (MSCOCO) and Visual Genome (VG).

[0056] In a feasible implementation, in S1, vector representation of the image-text pair includes:

[0057] The vector representation of the image-text pair is performed according to the following formulas (1), (2) and (3):

[0058]

[0059] Among them, t CLS and v CLS To facilitate the special notation of fine-tuning tasks, t represents text; v represents image; z0 represents the image-text pair vector group input by the model; t i and v i Represent text and visual vector embeddings respectively, T represents word embedding matrix; V represents linear projection embedding of image; T pos and V pos denote the learnable text and visual position embeddings, respectively, and v type and v type denote learnable textual and visual type embeddings, respectively.

[0060] In a feasible implementation, in S1, the first-order logic is represented by a relational matrix, including:

[0061] For the first-order logic part, the entity is represented as a node object entity x∈X, for example, given a Patch(v i ) or token(t i ) can be an entity x, and the set of all possible regions is X. A predicate is a function that maps an entity to 0 or 1. A first-order logic (FOL) formula is composed of logical operations ∧ (and), ∨ (or), (non)combines the predicate elements. Further, we represent the predicate as the adjacency matrix M k , represents the relationship between two entity variables x and x′. and binary predicates The expression of is shown in the following formula (4):

[0062]

[0063] Among them, ψ k represents the kth first-order logic relation; σ(·) represents the sigmoid activation function, The product representation of multiple relation matrices representing multi-hop reasoning; k is the sequence number of the first-order logic representation; represents a set of unary predicates; Represents a set of binary predicates.

[0064] S2, based on first-order logic and image-text pair representation, performs logical combination and multi-hop reasoning;

[0065] In one feasible implementation, in S2, logical combination and multi-hop reasoning are performed based on first-order logic and image-text pair representation, including:

[0066] By combining first-order logic statements, the symbolic reasoning space is expanded, forming a multi-layered tree-structured reasoning space. This allows the model to combine multiple first-order logic statements to form long paths and perform complex multi-hop reasoning. Specifically, through multiple consecutive reasoning steps ("hops"), a relational reasoning path is constructed to answer complex queries or derive conclusions.

[0067] To perform logical combination, basic logical operations need to be introduced. That is, these basic logical operations (AND, OR, NOT and imply), among which ∨ and It can also be expressed by the following formula:

[0068]

[0069] use Therefore, we only need to consider logical AND and logical NOT, and replace the logical operations with mathematical approximations, as shown in the following formula (5):

[0070]

[0071] Among them, \(p,q\in[0,1]\) represents the primitive statement of first-order logic;

[0072] Based on the first-order logic combination containing four basic logical operations, a logical combination representation is constructed, as shown in the following formula (6):

[0073]

[0074] Among them, the formula set f:f∈F l Each element in is called a function, so that f∈[0,1]; in the above formula (6), a tree-like logical combination space is defined: F0 represents the initial formula set, which only contains the original statement Ψ; F l-1 Represents a set of functions of level l-1, for F l-1 Perform a logical NOT operation to obtain its negated set representation Next level F l Each formula in F l-1 and Logical AND of the functions in ;

[0075] However, if we construct the logical combination space in this way, enumerating all possible combinations at each level will result in excessive computational cost. Therefore, we set a memory limit c to indicate the maximum number of combinations that can be tracked at each level, and give priority to logical combination of the next level logic statements with high probability, that is, each level F l To be in Search for c logical AND combinations on the formulas in , so that the c-th formula of the l-th level is as shown in the following formula (7):

[0076]

[0077] For example, for Ψ = {ψ1, ψ2}C = 2, a possible level sequence is F0 = {ψ1, ψ2}, F1 = {ψ1*ψ2, (1-ψ2)*ψ1}, F2 = {(ψ1*ψ2)*((1-ψ2)*ψ1, …. In order to collect the rules of all levels, the final level L is the union of all previous sets, that is, F L =F0∪…∪F L-1 However, trivial rules like ψ1*(1-ψ1) that are true regardless of the input are not explicitly prohibited by the formulation, but are mitigated by introducing non-existent queries during training.

[0078] S3. Based on logical combination and multi-hop reasoning, a logical reasoning differentiable attention network is constructed that is integrated with the visual language model.

[0079] In a feasible implementation, in S3, based on logical combination and multi-hop reasoning, a logical reasoning differentiable attention network integrated with the visual language model is constructed, including:

[0080] The logical reasoning process is parameterized by the attention network. The steps to achieve these processes are called predicate search, original statement search, and logical combination formula search. These steps can be approximately learned to achieve the attention parameter matrix of these steps. S ψ ,S′ ψ ,S f ,S′ f and s0. In order to generate such data-independent rules, we define the learnable embeddings of all predicates as And define the dummy parameters X and X′ as embedding Therefore, the entire logical attention generation module is expressed as follows:

[0081]

[0082] The network consists of three multi-head attention modules in a stacked structure MA as a logical parameter generator; each attention module simulates predicate search, original statement search and logical combination formula search respectively.

[0083] In one feasible implementation, predicate search is used to select the best predicate, obtain the operator path attention vector and aggregate output;

[0084] The original statement search is used to output all possible predicates based on the predicate search, combined with the input entities, and combined into a basic unit that can be used for more complex logical reasoning;

[0085] Logical combination formula search is used to generate complex logical formulas.

[0086] In one possible implementation, process P1: predicate search, for selecting the best predicate P * Variable predicate embedding matrix H and image text representation embedding e x and e′ x As input, we define a learnable predicate query embedding Use the MultHeadAttn module to search for operators and finally obtain the operator path attention vector and aggregate output The calculation formula is expressed as:

[0087]

[0088]

[0089] Process P2: original statement search;

[0090] After getting the predicate search output all possible predicates After that, the input entities are combined into a basic unit (original sentence) that can be used for more complex logical reasoning. The validity of the sentence is selected and evaluated through the attention mechanism. The query Q is defined by two parameters of the sentence. ψ and Use the MultiHeadAttn module to generate the attention matrix S for each parameter ψ and Finally, the generated attention values are aggregated through the FeedForward module to obtain the final result of the sentence evaluation O ψ .

[0091]

[0092] Process P3: Logical combination formula search

[0093] Responsible for generating complex logical formulas. In this process, U ψ As the initial input, define the learnable query Q at level l f,l and Generate the corresponding attention S through the MultiHeadAttn module f,l and And use FeedForward to aggregate the output of the previous layer, and finally get the embedding of the current layer formula O f,l .

[0094] Finally, the final output attention s0 is generated through the MultiHeadAttn module.

[0095]

[0096] Here, the e in the formula group + and e - It is a learnable embedding used to represent the information introduced by the original state information and the logical operator "not". Indicates the positive and negative states at the l-1 layer, using two MultiHeadAttn calculation logic and parameters as well as the compatibility between the previous formulas. And through a FeedForward aggregation l-level formula embedding u f,t Finally, the following formula is used to calculate the output attention:

[0097] u o ,s o =MultiHeadAttn(q o ,Vo ).

[0098] After the constructed network is implemented through PyTorch, the parameters inherited from the BILP neural network part are frozen, and the logic network part is pre-trained for secondary logic. The pre-training is performed on the Microsoft COCO (MSCOCO) and Visual Genome (VG) datasets.

[0099] We compare the trained model with state-of-the-art image-text retrieval methods, as shown in Table 1. These methods were fine-tuned on the COCO and Flickr30K datasets. We can see that LogicVLM achieves some improvement over other models.

[0100] Table 1. Comparative experiment of image and text retrieval

[0101]

[0102] The Vision-Language Question Answering (VQA) task requires a model to generate an answer given an image and a question described in natural language. In this paper, we adopt a generative approach, adapted for open-ended VQA. Following this approach, we fine-tune LogicVLM on the training and validation sets of VQAv2, while retaining 1,000 validation images and their associated questions for internal validation. The VQA model is fine-tuned using the LM loss using the true answers as targets. The results are shown in Table 2. Using 14 million images, pass outperforms ALBEF by 2.27% and BLIP by 0.64% on the test set.

[0103] Visual Language Reasoning (NLVR2) is a task set that, given a pair of images and a sentence, determines whether the sentence correctly describes the image. In order to be able to reason about two images simultaneously, two cross-attention layers are used in each Transformer block in the image-based text encoder to process the two input images separately, and then the CA (cross-attention) outputs are merged and provided to FFN for processing. The two cross-attention layers are initialized with the same pre-trained weight values. The Merge layer performs simple average pooling on the first 6 layers of the encoder and linearly connects layers 6-12, followed by linear projection. An MLP classifier is applied to the output embedding of the [Encode] tag. As shown in Table 2, LogicVLM outperforms all existing methods and the BLIP model, except that ALBEF performs an additional custom pre-training step.

[0104] Table 2. Experiments on visual-language question answering and reasoning tasks

[0105]

[0106] In the embodiment of the present invention, the advantages of symbolic reasoning, such as strong reasoning ability and high interpretability, are used to improve the problem of poor understanding and reasoning ability of the visual language model. By integrating differentiable logic networks with multimodal coding technology, the structured reasoning ability of the visual language model is significantly improved. Its core innovation lies in converting the first-order logic system (entity, predicate, logical formula) into differentiable operations (such as logical and p∧q=p*q, non- ), combined with a tree-like multi-hop reasoning mechanism to construct complex logical paths, and introduced a dynamic pruning strategy (based on memory constraints c) to optimize computational efficiency, addressing the problem of traditional models relying on data statistics and lacking symbolic reasoning. A multi-head attention module generates predicate embeddings and logical parameters, achieving a deep fusion of symbolic rules and neural networks, enhancing cross-modal alignment capabilities. Testing has shown progress in traditional visual language tasks. Significant results have been achieved in tasks such as visual language question answering and visual language reasoning.

[0107] Figure 3 1 is a block diagram of a neural logic reasoning-assisted visual language interpretable learning system 300 according to an exemplary embodiment. The system 300 is used for a neural logic reasoning-assisted visual language interpretable learning method. Figure 3 The system includes an embedding representation module 310, a combination reasoning module 320, and an attention network construction module 330.

[0108] The embedding representation module 310 is used to obtain a dataset for training, and to represent the image-text pairs in the dataset as vectors; and to represent the first-order logic as a relational matrix;

[0109] Combinatorial reasoning module 320, for performing logical combination and multi-hop reasoning based on first-order logic and image-text pair representation;

[0110] The attention network construction module 330 is used to construct a logical reasoning differentiable attention network integrated with the visual language model based on logical combination and multi-hop reasoning.

[0111] Optionally, the embedding representation module 310 is further configured to perform vector representation of the image-text pair according to the following formulas (1), (2), and (3):

[0112]

[0113] Among them, t CLS and v CLS To facilitate the special notation of fine-tuning tasks, t represents text; v represents image; z0 represents the image-text pair vector group input by the model; t i and v iRepresent text and visual vector embeddings respectively, T represents word embedding matrix; V represents linear projection embedding of image; T pos and V pos denote the learnable text and visual position embeddings, t type and v type denote learnable textual and visual type embeddings, respectively.

[0114] Optionally, the embedding representation module 310 is further configured to represent the entity as a node object entity x∈X for the first-order logic part, where X represents the set X of all possible regions; and to represent the predicate as an adjacency matrix M k , representing the relationship between two entity variables x and x′; then the first-order logic relationship matrix is expressed as shown in the following formula (4):

[0115]

[0116] Among them, ψ k represents the kth first-order logic relation; σ(·) represents the sigmoid activation function, The product representation of multiple relation matrices representing multi-hop reasoning; k is the sequence number of the first-order logic representation; represents a set of unary predicates; Represents a set of binary predicates.

[0117] Optionally, a combination reasoning module 320 is used to expand the symbolic reasoning space by combining first-order logic statements to form a multi-layer tree-structured reasoning space;

[0118] Introducing basic logical operations, and only considering logical AND and logical NOT, the logical operations are replaced by mathematical approximations, as shown in the following formula (5):

[0119]

[0120] Among them, \(p,q\in[0,1]\) represents the primitive statement of first-order logic;

[0121] Based on the first-order logic combination containing four basic logical operations, a logical combination representation is constructed, as shown in the following formula (6):

[0122]

[0123] Among them, the formula set f:f∈F l Each element in is called a function, so that f∈[0,1]; in the tree-like logical combination space: F0 represents the initial formula set, which only contains the original statement Ψ; F l-1 Represents a set of functions of level l-1, for F l-1Perform a logical NOT operation to obtain its negated set representation Next level F l Each formula in F l-1 and Logical AND of the functions in ;

[0124] By setting the memory limit c to indicate the maximum number of combinations that can be tracked at each level, logical combinations of the next level logic statements with large probability are prioritized, that is, each level F l To be in Search for c logical AND combinations on the formulas in , so that the c-th formula of the l-th level is as shown in the following formula (7):

[0125]

[0126] Optionally, an attention network construction module 330 is used to define the learnable embeddings of all predicates as And define the dummy parameters X and X′ as embedding Therefore, the entire logical attention generation module is expressed as follows:

[0127]

[0128] The logical attention network consists of three multi-head attention modules in a stacked structure MA as a logical parameter generator; each attention module simulates predicate search, original statement search and logical combination formula search respectively.

[0129] Optionally, predicate search is used to select the best predicate, obtain operator path attention vectors and aggregate outputs;

[0130] The original statement search is used to output all possible predicates based on the predicate search, combined with the input entities, and combined into a basic unit that can be used for more complex logical reasoning;

[0131] Logical combination formula search is used to generate complex logical formulas.

[0132] In the embodiment of the present invention, the advantages of symbolic reasoning, such as strong reasoning ability and high interpretability, are used to improve the problem of poor understanding and reasoning ability of the visual language model. By integrating differentiable logic networks with multimodal coding technology, the structured reasoning ability of the visual language model is significantly improved. Its core innovation lies in converting the first-order logic system (entity, predicate, logical formula) into differentiable operations (such as logical and p∧q=p*q, non- ), combined with a tree-like multi-hop reasoning mechanism to construct complex logical paths, and introduced a dynamic pruning strategy (based on memory constraints c) to optimize computational efficiency, addressing the problem of traditional models relying on data statistics and lacking symbolic reasoning. A multi-head attention module generates predicate embeddings and logical parameters, achieving a deep fusion of symbolic rules and neural networks, enhancing cross-modal alignment capabilities. Testing has shown progress in traditional visual language tasks. Significant results have been achieved in tasks such as visual language question answering and visual language reasoning.

[0133] Figure 3 is a structural diagram of a neural logic reasoning-assisted visual language interpretable learning device provided by an embodiment of the present invention, such as Figure 3 As shown, the neural logic reasoning-assisted visual language interpretable learning device may include the above Figure 2 The visual language interpretable learning system assisted by neural logic reasoning is shown. Optionally, the visual language interpretable learning device 410 assisted by neural logic reasoning may include a first processor 2001 .

[0134] Optionally, the neural logic reasoning-assisted visual language interpretable learning device 410 may further include a memory 2002 .

[0135] The first processor 2001 and the memory 2002 may be connected via a communication bus.

[0136] The following combination Figure 3 The components of the neural logic reasoning-assisted visual language interpretable learning device 410 are described in detail:

[0137] The first processor 2001 is the control center of the neural logic reasoning-assisted visual language interpretable learning device 410 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (GPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0138] Optionally, the first processor 2001 can perform various functions of the visual language interpretable learning device 410 assisted by neural logic reasoning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0139] In a specific implementation, as an embodiment, the first processor 2001 may include one or more GPUs, such as Figure 3 GPU0 and GPU1 are shown in .

[0140] In a specific implementation, as an embodiment, the neural logic reasoning-assisted visual language interpretable learning device 410 may also include multiple processors, such as Figure 3 1 and 2. The first processor 2001 and the second processor 2003 are shown in FIG. Each of these processors can be a single-core processor (single-GPU) or a multi-core processor (multi-GPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0141] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0142] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the interface circuit ( Figure 3 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0143] It should be noted that Figure 3 The structure of the neural logic reasoning-assisted visual language interpretable learning device 410 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0144] In addition, the technical effects of the visual language interpretable learning device 410 assisted by neural logic reasoning can refer to the technical effects of the visual language interpretable learning method assisted by neural logic reasoning described in the above method embodiment, and will not be repeated here.

[0145] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (GPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0146] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0147] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensor. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0148] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0149] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0150] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0151] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0153] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0154] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for interpretable learning of visual language assisted by neural logic reasoning, characterized in that: The method comprises: S1. Obtain a dataset for training, and represent the image-text pairs in the dataset as vectors; and represent the first-order logic as a relational matrix. S2, based on first-order logic and image-text pair representation, performs logical combination and multi-hop reasoning; S3. Based on logical combination and multi-hop reasoning, a logical reasoning attention network is constructed that is integrated with the visual language model.

2. The neural logic reasoning-assisted visual language interpretable learning method according to claim 1 is characterized in that: In S1, the image-text pair is represented by a vector, including: The vector representation of the image-text pair is performed according to the following formulas (1), (2) and (3): Among them, t CLS and v CLS To facilitate the special notation of fine-tuning tasks, t represents text; v represents image; z0 represents the image-text pair vector group input by the model; t i and v i Represent text and visual vector embeddings respectively, T represents word embedding matrix; V represents linear projection embedding of image; T pos and V pos denote the learnable text and visual position embeddings, t type and v type denote learnable textual and visual type embeddings, respectively.

3. The neural logic reasoning-assisted visual language interpretable learning method according to claim 1 is characterized in that: In said S1, the first-order logic is represented by a relational matrix, including: For the first-order logic part, the entity is represented as a node object entity x∈X, where X represents the set of all possible regions X; the predicate is represented as the adjacency matrix M k , representing the relationship between two entity variables x and x′; then the first-order logic relationship matrix is expressed as shown in the following formula (4): Among them, ψ k represents the kth first-order logic relation; σ(·) represents the sigmoid activation function, The product representation of multiple relation matrices representing multi-hop reasoning; k is the sequence number of the first-order logic representation; represents a set of unary predicates; Represents a set of binary predicates.

4. The neural logic reasoning-assisted visual language interpretable learning method according to claim 2, characterized in that: In S2, based on first-order logic and the representation of image-text pairs, logical combination and multi-hop reasoning are performed, including: By combining first-order logic statements, the symbolic reasoning space is expanded to form a multi-layered tree-structured reasoning space. Introducing basic logical operations, and only considering logical AND and logical NOT, the logical operations are replaced by mathematical approximations, as shown in the following formula (5): Among them, \(p,q\in[0,1]\) represents the primitive statement of first-order logic; Based on the first-order logic combination containing four basic logical operations, a logical combination representation is constructed, as shown in the following formula (6): F0=Ψ, Among them, the formula set f:f∈F l Each element in is called a function, so that f∈[0,1]; in the tree-like logical combination space: F0 represents the initial formula set, which only contains the original statement Ψ; F l-1 Represents a set of functions of level l-1, for F l-1 Perform a logical NOT operation to obtain its negated set representation Next level F l Each formula in F l-1 and Logical AND of the functions in ; By setting the memory limit c to indicate the maximum number of combinations that can be tracked at each level, logical combinations of the next level logic statements with large probability are prioritized, that is, each level F l To be in Search for c logical AND combinations on the formulas in , so that the c-th formula of the l-th level is as shown in the following formula (7):

5. The neural logic reasoning-assisted visual language interpretable learning method according to claim 2, characterized in that: In S3, based on logical combination and multi-hop reasoning, a logical reasoning attention network integrated with the visual language model is constructed, including: Define the learnable embeddings of all predicates as and replace the dummy parameters X and X ′ Defined as embedded Therefore, the entire logical attention generation module is expressed as follows: The logical attention network consists of three multi-head attention generation modules with a stacked structure MA as a logical parameter generator; each attention module simulates predicate search, original statement search and logical combination formula search respectively.

6. The neural logic reasoning-assisted visual language interpretable learning method according to claim 4 is characterized in that: The predicate search is used to select the best predicate, obtain the operator path attention vector and the aggregate output; The original sentence search is used to output all possible predicates based on the predicate search, and combine them with the input entities to form a basic unit that can be used for more complex logical reasoning; The logic combination formula search is used to generate complex logic formulas.

7. A neural logic reasoning-assisted visual language interpretable learning system, wherein the neural logic reasoning-assisted visual language interpretable learning system is used to implement the neural logic reasoning-assisted visual language interpretable learning method according to any one of claims 1 to 6, characterized in that: The system comprises: The embedding representation module is used to obtain the dataset for training, and to represent the image-text pairs in the dataset as vectors; and to represent the first-order logic as a relational matrix; Combinatorial reasoning module, which is used to perform logical combination and multi-hop reasoning based on first-order logic and image-text pair representation; The attention network building module is used to build a logical reasoning differentiable attention network integrated with the visual language model based on logical combination and multi-hop reasoning.

8. The neural logic reasoning-assisted visual language interpretable learning system according to claim 7, characterized in that: The embedding representation module is also used to perform vector representation of image-text pairs according to the following formulas (1), (2), and (3): Among them, t CLS and v CLS To facilitate the special notation of fine-tuning tasks, t represents text; v represents image; z0 represents the image-text pair vector group input by the model; t i and v i Represent text and visual vector embeddings respectively, T represents word embedding matrix; V represents linear projection embedding of image; T pos and V pos denote the learnable text and visual position embeddings, t type and v type denote learnable textual and visual type embeddings, respectively.

9. A device for interpretable learning of visual language assisted by neural logic reasoning, comprising: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, any one of the methods of visual language interpretable learning assisted by neural logic reasoning as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium storing at least one instruction, wherein the at least one instruction is loaded and executed by a processor to implement any one of the methods for interpretable visual language learning assisted by neural logic reasoning as described in any one of claims 1 to 6.

Images