Knowledge enhanced multi-modal neural symbolic semantic communication method and device

Abstract

This invention discloses a knowledge-enhanced multimodal neural symbolic semantic communication method and apparatus, relating to the interdisciplinary field of artificial intelligence. The method includes: extracting shared task slots using a multimodal backbone; generating observation evidence and retrieval anchors through dual readout; filtering positive examples using a knowledge graph and lightweight compatibility scoring to obtain a slot support subset; generating slot-level knowledge recovery input containing evidence and anchor representations through normalized encoding; dynamically expanding the entity pool and calculating conditional distribution based on this input; obtaining knowledge tokens through expert product fusion and sparse reading; and using the recovery input and knowledge tokens for multi-dimensional uncertainty quantification, weighted fusion, and slot swapping to generate a knowledge-enhanced semantic prefix, driving a large language model to autoregressively output the answer. This invention can significantly improve the overall performance of knowledge-intensive tasks and is suitable for cutting-edge semantic communication scenarios such as edge intelligence and unmanned platforms.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence interdisciplinary technology, and in particular to a knowledge-enhanced multimodal neural symbolic semantic communication method and apparatus. Background Technology

[0002] With the rapid development of artificial intelligence, computer vision, multimodal large language models, and edge intelligent systems, mobile terminals, unmanned platforms, intelligent sensing devices, and embodied intelligent systems are placing higher demands on complex visual scene understanding, natural language interaction, and cross-modal reasoning capabilities. In these application scenarios, multimodal information such as images, text, and question commands typically needs to be transmitted between the sending and receiving ends via wireless links, with visual question answering, semantic understanding, task decision-making, or collaborative reasoning performed at the receiving end. Traditional communication systems primarily aim for bit-level high-fidelity transmission and signal reconstruction, usually requiring the transmission of raw images, video data, or high-dimensional continuous features. When wireless links suffer from channel noise, fading, bandwidth limitations, and terminal power consumption constraints, these methods struggle to simultaneously meet the requirements of low latency, high spectral efficiency, and robustness for complex task reasoning. Semantic communication, by extracting and transmitting semantic information most relevant to downstream tasks, offers a new solution for reducing transmission burden and improving task performance. Some methods have attempted to incorporate knowledge graphs, external memories, or large language models into multimodal reasoning tasks. However, most methods still employ question-level or global-level knowledge retrieval, which easily introduces a large amount of irrelevant knowledge and lacks fine-grained knowledge retrieval constraints based on recovery semantics. Other methods use multimodal large language models as a unified reasoning backend, but do not explicitly address the coordination issues between sender-side task evidence construction, wireless channel transmission, receiver-side knowledge activation, and knowledge injection reliability adjustment. Therefore, there is an urgent need to propose a novel knowledge-enhanced multimodal semantic communication scheme that enables the sender to simultaneously retain instance-level observation information supporting task decisions and receiver-side knowledge retrieval directions. This scheme allows the receiver to perform local, controllable, and high-confidence knowledge activation under impaired signal conditions and adaptively adjust the knowledge injection intensity based on channel recovery quality, knowledge confidence, and cross-branch consistency. This, in turn, improves the reasoning performance, robustness, and unit symbol semantic efficiency of complex multimodal tasks under impaired wireless links. Summary of the Invention

[0003] To address the shortcomings of existing technologies, such as the lack of clear structural semantic interpretation of continuous features, making it difficult for the receiver to determine whether partially recovered semantics correspond to key task evidence even when partially recovered, and the tendency for semantic recovery to deviate under impaired wireless channels, hindering the receiver's stable retrieval of external common sense, factual knowledge, or background information, and the lack of mechanisms for collaborative reasoning between image observation cues, question text, and external knowledge for knowledge-intensive visual question answering and cross-modal decision-making tasks, this invention provides a knowledge-enhanced multimodal neural symbolic semantic communication method and apparatus. The technical solution is as follows: On the one hand, a knowledge-enhanced multimodal neural symbolic semantic communication method is provided, which is implemented by a knowledge-enhanced multimodal neural symbolic semantic communication device, and includes: S1: Obtain the image to be transmitted and the question text corresponding to the image. Use a pre-trained multimodal large language model backbone to extract features from the input image and question text and obtain the shared task slot of the sending end. Process the shared task slot through two readout heads to obtain the observation evidence that constitutes the output of the observation branch and the retrieval anchor point that constitutes the output of the retrieval branch. S2: Introduce knowledge graphs and candidate facts, use a frozen text encoder to encode the question text and candidate facts in the local candidate domain and calculate cosine similarity to obtain a lightweight compatibility score, and filter the training positive examples through semantic compatibility threshold constraints. Obtain the slot support subset based on the similarity between the retrieval anchor and the candidate fact representation in the training positive examples. S3: The observation evidence and retrieval anchor point are independently normalized, and then concatenated based on the slot index of the shared task slot and input into the channel encoder to map the first restored semantic representation of the corresponding observation evidence and the second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input of the i-th shared task slot. S4: Use an entity linker to link the question text to obtain a question entity set and construct a basic local entity pool on the knowledge graph accordingly. Based on the slot-level knowledge recovery input, dynamically expand the basic local entity pool to obtain slot candidate fact representations. Based on the slot-level knowledge recovery input, perform condition distribution calculation on the slot candidate set of the i-th shared task slot to obtain the observation evidence condition distribution and retrieval anchor point condition distribution. S5: The expert product method is used to fuse the observation evidence condition distribution and the retrieval anchor point condition distribution to obtain a joint evidence distribution. Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation on the candidate fact representation of the slot to obtain the knowledge token of the i-th slot. The knowledge tokens corresponding to the slots that failed to form an effective candidate set during the reasoning stage are serialized and combined to obtain a task-related knowledge token sequence. S6: Based on the slot-level knowledge recovery input and task-related knowledge token sequence, perform multi-dimensional uncertainty quantification and determine branch weights. After weighted fusion and inter-slot information exchange, obtain knowledge-enhanced semantic representation. Map the knowledge-enhanced semantic representation to continuous semantic prefixes and concatenate it with text prompts. Input it into a multimodal large language model to obtain the target answer sequence in an autoregressive manner.

[0004] Preferably, step S1 involves acquiring the image to be transmitted and the corresponding question text, using a pre-trained multimodal large language model backbone to extract features from the input image and question text, obtaining the shared task slots at the sending end, and processing the shared task slots through two readout heads to obtain the observation evidence constituting the observation branch output and the retrieval anchor points constituting the retrieval branch output, including: S11: Obtain the image to be transmitted and the question text corresponding to the image. Use a pre-trained multimodal large language model backbone to extract features from the image to be transmitted and the question text to obtain the image visual sequence and the question text sequence. The input image includes the visual information to be analyzed, and the question text includes the query instructions input by the user. S12: Based on the question text sequence, a task summary vector is obtained by mask mean pooling of valid text tokens. The mask mean pooling includes mean aggregation of text tokens only at non-filled positions. S13: Based on the task summary vector, by constructing an adaptive additive bias for the visual tokens and modulating it in the cross-attention scoring process, visual token attention weights with enhanced task relevance are obtained. The adaptive additive bias is used to apply task-related differential weight offsets to each visual token during the attention scoring stage. S14: The learnable task anchor and image visual sequence are input into a single-layer cross-attention mechanism. The image visual sequence is aggregated using visual token attention weights enhanced with task relevance to obtain the shared task slot at the sending end. Then, it is processed by two readout heads to obtain observation evidence and retrieval anchor. The observation branch outputs observation evidence to provide instance observation information to support the final decision, and the retrieval branch outputs retrieval anchor to provide directional information for subsequent knowledge invocation.

[0005] Preferably, in step S2, the introduction of a knowledge graph and candidate facts involves using a frozen text encoder to encode the question text and candidate facts within a local candidate domain and calculate cosine similarity to obtain a lightweight compatibility score. This score is then filtered through a semantic compatibility threshold constraint to obtain training examples. A slot support subset is obtained based on the similarity between the retrieval anchor point and the candidate fact representations in the training examples, including: S21: Obtain the answer text, normalize the answer text and match it with aliases in the knowledge graph to obtain entity alias matching results. The normalization process includes standardization operations on the answer text. S22: A frozen text encoder is used to encode the textual representations of the question text and candidate facts and calculate the cosine similarity within the local candidate domain to obtain a lightweight compatibility score. The parameters of the frozen text encoder remain unchanged during training. S23: Based on the entity alias matching results and the lightweight compatibility score, and after filtering by the semantic compatibility threshold constraint, training positive examples are obtained. The training positive examples include a single candidate fact that meets the conditions or a sample-level set of multiple positive examples consisting of multiple candidate facts that meet the conditions. S24: Based on the similarity between the retrieval anchor and the candidate fact representations in the sample-level multi-positive example set, and according to the slot index of the shared task slot at the sending end, select the top M most compatible supporting facts from the sample-level multi-positive example set for each task slot to obtain a slot support subset. The slot support subset constitutes a slot-level weak supervision signal. The slot-level weak supervision signal is used to enhance the alignment between the retrieval anchor and local knowledge facts during the training phase. The selection operation of the top M supporting facts does not directly participate in gradient backpropagation.

[0006] Preferably, in step S3, the observation evidence and retrieval anchor point are independently normalized, then concatenated based on the slot index of the shared task slot and input into the channel encoder to map into a first restored semantic representation of the corresponding observation evidence and a second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input of the i-th shared task slot, including: S31: Using an intra-branch normalization method, the first semantic representation corresponding to the observation evidence output by the observation branch and the second semantic representation corresponding to the retrieval anchor point output by the retrieval branch are processed to obtain the corresponding first normalized semantic representation and second normalized semantic representation. The intra-branch normalization method includes performing normalization operations independently within the observation branch and the retrieval branch respectively. S32: Based on the slot index of the shared task slot, the first normalized semantic representation and the second normalized semantic representation are concatenated to obtain a unified transmission semantic representation. This unified transmission semantic representation is then input into the channel encoder for mapping to obtain a complex symbol vector. After transmission through the physical channel, it is converted into the semantic representation recovered by the receiving end. The slot index is used to ensure the consistency of the slot index between the sending end and the receiving end. S33: Based on the slot index order consistent with the shared slot order, the semantic representation recovered by the receiving end is split to obtain a first recovered semantic representation and a second recovered semantic representation. The first recovered semantic representation corresponds to the observation evidence recovered by the observation branch, and the second recovered semantic representation corresponds to the retrieval anchor point recovered by the retrieval branch. S34: The observation evidence representation of the i-th slot in the first restored semantic representation and the corresponding retrieval anchor representation in the second restored semantic representation are used as the slot-level knowledge recovery input for the i-th shared task slot. The observation evidence representation of the i-th slot is used for subsequent candidate domain construction and filtering, and the corresponding retrieval anchor representation is used for subsequent slot candidate set selection and conditional distribution calculation.

[0007] Preferably, in step S4, the entity linker is used to link the question text to obtain a question entity set, and a basic local entity pool is constructed on the knowledge graph accordingly. The basic local entity pool is then dynamically expanded based on the slot-level knowledge recovery input to obtain slot candidate fact representations. Finally, based on the slot-level knowledge recovery input, conditional distribution calculations are performed on the slot candidate set for the i-th shared task slot to obtain the observation evidence conditional distribution and the retrieval anchor point conditional distribution, including: S41: Recover the input based on the slot-level knowledge, dynamically expand the basic local entity pool to obtain an expanded candidate domain, wherein the dynamic expansion includes adaptively expanding the coverage of the basic local entity pool based on the observation evidence recovered from the observation branch; S42: Using a representation construction method, the candidate facts in the extended candidate domain are processed to obtain the slot candidate fact representation. The representation construction method includes adding the embedding representations of the head entity, relation and tail entity after semantic adapter transformation, adding a bias term, and performing layer normalization operation. S43: Recover the input based on the slot-level knowledge, filter the extended candidate domain, and obtain a slot candidate set for the i-th shared task slot. The filtering includes sorting the slot candidate facts based on the cosine similarity between the recovered observation evidence or the recovered retrieval anchor and the slot candidate fact representation, and taking the top N candidate facts. During the training phase, the slot candidate set also includes training positive examples that simultaneously satisfy entity alias matching and semantic compatibility threshold constraints. S44: Based on the slot-level knowledge recovery input, perform conditional distribution calculation on the slot candidate set of the i-th shared task slot to obtain the observation evidence conditional distribution corresponding to the observation branch and the retrieval anchor point conditional distribution corresponding to the retrieval branch. The conditional distribution calculation uses a temperature parameter to control the smoothness of the conditional distribution. During the training phase, the calculation of the retrieval anchor point conditional distribution is further constrained by the slot-level weak supervision signal. The slot-level weak supervision signal uses the supporting facts in the slot support subset as positive sample labels for the corresponding slot, enabling the retrieval anchor point to learn a stable knowledge invocation direction.

[0008] Preferably, in step S5, the expert product method is used to fuse the distribution of observed evidence conditions and the distribution of retrieval anchor conditions to obtain a joint evidence distribution. Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation of the candidate fact representations of slots to obtain the knowledge token for the i-th slot. The knowledge tokens corresponding to the slots that failed to form a valid candidate set during the reasoning stage are serialized and combined to obtain a task-related knowledge token sequence, including: S51: Using the expert product method, the observation evidence condition distribution corresponding to the observation branch and the retrieval anchor point condition distribution corresponding to the retrieval branch are fused to obtain a joint evidence distribution. The retrieval anchor point condition distribution is used to provide retrieval priors, and the observation evidence condition distribution is used for local correction. S52: Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation on the candidate fact representation of the slot to obtain the knowledge token of the i-th slot. The sparse read operation includes using a sparse maximization function, and the sparse read operation introduces a minimal constant to prevent numerical instability and a sparse read temperature parameter to control the degree of read concentration. S53: Degenerate the knowledge tokens corresponding to the slots that failed to form a valid candidate set during the inference stage to obtain a predefined empty knowledge vector. The empty knowledge vector is used to retain the output of the observation branch in subsequent fusion. S54: Serialize and combine the predefined empty knowledge vectors corresponding to all slots to obtain a task-related knowledge token sequence. The length of the task-related knowledge token sequence is the same as the number of shared task slots.

[0009] Preferably, step S6 involves performing multi-dimensional uncertainty quantification and determining branch weights based on the slot-level knowledge recovery input and task-related knowledge token sequence, obtaining a knowledge-enhanced semantic representation through weighted fusion and inter-slot information exchange, mapping the knowledge-enhanced semantic representation to continuous semantic prefixes, concatenating it with text prompts, and inputting it into a multimodal large language model to obtain the target answer sequence in an autoregressive manner, including: S61: Using a cross-branch shared projection matrix, the recovered observation evidence and the task-related knowledge token sequence in the first recovered semantic representation are mapped to a unified feature space to obtain fused features; uncertainty quantification is performed based on the average deviation between the recovered observation evidence and high-probability candidate facts to obtain observation dispersion; confidence is evaluated based on the concentration of knowledge tokens under the joint evidence distribution to obtain knowledge dispersion. The cross-branch shared projection matrix is ​​used to align the representations of the observation branch and the retrieval branch to the same feature space. S62: By calculating the probability overlap between the observation branch and the retrieval branch on the same slot candidate set, a cross-branch consistency coefficient is obtained; the validity of fact support is quantified based on the highest consensus similarity of the dual-path evidence to obtain the candidate sufficiency; the weight of the observation branch is determined based on the observation dispersion, and the weight of the retrieval branch is determined based on the joint gating of the cross-branch consistency coefficient and the candidate sufficiency; after normalizing the weights of the two branches, the fusion features are weighted and fused to obtain the fusion representation of the task slot, and the slot index is embedded and injected into each fused task slot to obtain an initial interaction representation containing structured identity information; S63: Employing multi-head self-attention and feedforward networks, controlled information exchange is performed between slots on the initial interaction representation to restore cross-slot dependencies and obtain the final knowledge-enhanced semantic representation. This representation is then mapped to a continuous semantic prefix aligned with the word embedding space of the large language model through a multilayer perceptron. This prefix is ​​then concatenated with text prompts consisting of task instructions and question texts and input into the multimodal large language model to generate the target answer sequence in an autoregressive manner.

[0010] On the other hand, a knowledge-enhanced multimodal neural symbolic semantic communication device is provided, which is applied to the knowledge-enhanced multimodal neural symbolic semantic communication method. The device includes: Feature extraction module: used to acquire the image to be transmitted and the question text corresponding to the image. It uses a pre-trained multimodal large language model backbone to extract features from the input image and question text and acquire the shared task slots of the sending end. It processes the shared task slots through two readout heads to obtain the observation evidence that constitutes the output of the observation branch and the retrieval anchor points that constitute the output of the retrieval branch. Training Positive Examples Module: This module is used to introduce knowledge graphs and candidate facts. It employs a frozen text encoder to encode the question text and candidate facts within a local candidate domain and calculates cosine similarity to obtain a lightweight compatibility score. The module then filters the results through semantic compatibility threshold constraints to obtain training positive examples. Based on the similarity between the retrieval anchor and the candidate fact representations in the training positive examples, a slot support subset is obtained. Channel encoder module: It is used to independently normalize the observation evidence and retrieval anchor point, and then concatenate them based on the slot index of the shared task slot and input them into the channel encoder to map the first restored semantic representation of the corresponding observation evidence and the second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input of the i-th shared task slot. Entity Linker Module: Used to link entities in the question text to obtain a set of question entities and construct a basic local entity pool on the knowledge graph accordingly. Based on the slot-level knowledge recovery input, the basic local entity pool is dynamically expanded to obtain slot candidate fact representations. Based on the slot-level knowledge recovery input, the condition distribution is calculated on the slot candidate set of the i-th shared task slot to obtain the observation evidence condition distribution and the retrieval anchor condition distribution. The sparse reading module is used to fuse the observation evidence condition distribution and the retrieval anchor point condition distribution using the expert product method to obtain a joint evidence distribution. Based on the joint evidence distribution, the sparse reading operation is used to perform a weighted summation on the candidate fact representation of the slot to obtain the knowledge token of the i-th slot. The knowledge tokens corresponding to the slots that failed to form a valid candidate set during the reasoning stage are serialized and combined to obtain a task-related knowledge token sequence. The knowledge-enhanced semantic module is used to recover the input and task-related knowledge token sequence based on the slot-level knowledge, perform multi-dimensional uncertainty quantification and determine branch weights, obtain the knowledge-enhanced semantic representation through weighted fusion and inter-slot information exchange, map the knowledge-enhanced semantic representation into continuous semantic prefixes and concatenate them with text prompts, input them into a multimodal large language model to obtain the target answer sequence in an autoregressive manner.

[0011] On the other hand, a knowledge-enhanced multimodal neural symbolic semantic communication device is provided, the knowledge-enhanced multimodal neural symbolic semantic communication device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the method described in any one of the above-described knowledge-enhanced multimodal neural symbolic semantic communication methods is implemented.

[0012] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following: High-efficiency semantic compression and decoupling of transmit and receive features are achieved. By extracting observation evidence and retrieval anchor points through shared task slots, dual-path semantics are effectively eliminated, not only improving semantic fidelity under constrained symbol budgets but also decoupling instance observation preservation and knowledge retrieval guidance at the physical level. This fundamentally avoids mutual interference between the two types of information in traditional single-path black-box transmission, making knowledge retrieval at the receiving end more stable and controllable. Precise knowledge activation and robust semantic correction are achieved in complex channel environments. At the receiving end, global retrieval is abandoned, and instead, local candidate domains are dynamically constructed based on problem priors and recovery evidence, and slot-level activation is performed, significantly reducing computational overhead and irrelevant knowledge interference. Simultaneously, an innovative uncertainty-aware adaptive fusion mechanism is introduced, dynamically adjusting the fusion gating based on the dispersion and consistency of multi-path evidence. When faced with severe channel fading or noise, the system can adaptively suppress the injection of erroneous knowledge and only participate in error correction when the knowledge support is sufficient, which significantly enhances the system's noise robustness. It also significantly improves the end-to-end inference accuracy and overall reliability of the system. By introducing a powerful multimodal large language model as the core of inference, the system maps knowledge-enhanced semantics into continuous prefixes and inputs them into the model, making full use of the technical advantages of large models in cross-modal understanding and logic generation, thereby improving the overall performance of knowledge-intensive tasks and making it suitable for cutting-edge semantic communication scenarios such as edge intelligence and unmanned platforms. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a knowledge-enhanced multimodal neural symbolic semantic communication method provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a knowledge-enhanced multimodal neural-symbolic semantic communication system provided in an embodiment of the present invention; Figure 3 This is a block diagram of a knowledge-enhanced multimodal neural symbolic semantic communication device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a knowledge-enhanced multimodal neural symbolic semantic communication device provided in an embodiment of the present invention. Detailed Implementation

[0015] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0016] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0017] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0018] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0019] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0020] This invention provides a knowledge-enhanced multimodal neural symbolic semantic communication method, which can be implemented by a knowledge-enhanced multimodal neural symbolic semantic communication device, which can be a terminal or a server. Figure 1 The flowchart shown is for a knowledge-enhanced multimodal neural symbolic semantic communication method. The processing flow of this method may include the following steps: Preferably, the process involves acquiring the image to be transmitted and the corresponding question text, using a pre-trained multimodal large language model backbone to extract features from the input image and question text, obtaining the shared task slots at the sending end, and processing the shared task slots through two readout heads to obtain observation evidence constituting the output of the observation branch and retrieval anchor points constituting the output of the retrieval branch, including: The system acquires the image to be transmitted and the corresponding question text. Using a pre-trained multimodal large language model backbone, it extracts features from the image to be transmitted and the question text to obtain the image visual sequence and the question text sequence. The input image includes the visual information to be analyzed, and the question text includes the query command input by the user. Based on the question text sequence, a task summary vector is obtained by mask mean pooling of valid text tokens. The mask mean pooling includes mean aggregation of text tokens only at non-filled positions. Based on the task summary vector, an adaptive additive bias is constructed for the visual tokens and modulated by introducing it into the cross-attention scoring process to obtain task-related enhanced visual token attention weights. The adaptive additive bias is used to apply task-related differential weight offsets to each visual token during the attention scoring stage. The learnable task anchors and image visual sequences are input into a single-layer cross-attention mechanism. The image visual sequences are aggregated using visual token attention weights enhanced with task relevance to obtain shared task slots at the sending end. The sequences are then processed by two readout heads to obtain observation evidence and retrieval anchors. The observation branch outputs observation evidence to provide instance observation information that supports the final decision, while the retrieval branch outputs retrieval anchors to provide directional information for subsequent knowledge invocation.

[0021] In some embodiments, the image visual sequence is extracted at the sending end using a pre-trained multimodal large language model backbone. and question text sequence ,in and These represent the number of visual tokens and the length of the text sequence, respectively. To define the dimensions of the shared semantic space, the valid text tokens are then subjected to masked mean pooling to obtain the task summary vector. . use An adaptive additive bias is constructed for each visual token and introduced into the cross-attention scoring process:

[0022] in, For the sigmoid function, and For learnable weights, This is a numerically stable term. This bias, as an additive modulation term under problem conditions, is added to the attention scoring of visual tokens in the slots to suppress visual tokens weakly correlated with the problem and enhance the selectivity of cross-attention for task-relevant regions.

[0023] Introduction A learnable task anchor An alternative to unstructured global compressed representations is used for initial queries of shared task slots. Subsequently, a task summary vector is generated. Assuming global conditional constraints, slot-level task units are extracted from the visual sequence using a single-layer cross-attention mechanism:

[0024] Among them, slot query , It is a vector of all 1s. For learnable mappings, Indicates cross attention. Broadcast to all slots according to the query dimension. This results in... It provides a shared task slot for the sending end and a unified task-level semantic interface for subsequent dual-evidence reading.

[0025] Based on shared task slots Observational evidence and retrieval anchor points are generated using two readout heads:

[0026] in, , These are learnable parameters. The two branches share the same slot index but have different responsibilities: Retain instance observation information that supports the final decision; Used to preserve the direction of subsequent knowledge invocation.

[0027] Preferably, a knowledge graph and candidate facts are introduced. A frozen text encoder is used to encode the question text and candidate facts within a local candidate domain and calculate cosine similarity to obtain a lightweight compatibility score. Training examples are then selected through semantic compatibility threshold constraints. A slot support subset is obtained based on the similarity between the retrieval anchor and the candidate fact representations in the training examples, including: Obtain the answer text, normalize the answer text and match it with aliases in the knowledge graph to obtain entity alias matching results. The normalization process includes standardization operations on the answer text. A frozen text encoder is used to encode the textual representations of the question text and candidate facts and calculate cosine similarity within a local candidate domain to obtain a lightweight compatibility score. The parameters of the frozen text encoder remain unchanged during training. Based on the entity alias matching results and the lightweight compatibility score, and after filtering by the semantic compatibility threshold constraint, training positive examples are obtained. The training positive examples include a single candidate fact that meets the conditions or a sample-level set of multiple positive examples consisting of multiple candidate facts that meet the conditions. Based on the similarity between the retrieval anchor and the candidate fact representations in the sample-level multi-positive example set, and according to the slot index of the shared task slot at the sending end, the top M most compatible supporting facts are selected from the sample-level multi-positive example set for each task slot to obtain a slot support subset. The slot support subset constitutes a slot-level weak supervision signal. The slot-level weak supervision signal is used to enhance the alignment between the retrieval anchor and local knowledge facts during the training phase. The selection operation of the top M supporting facts does not directly participate in gradient backpropagation.

[0028] In some embodiments, for each training sample First, check the answer. Normalize the data and match it with aliases in the knowledge graph. Let the matching result be denoted as . Subsequently, only in the local candidate domain Internal screening supports the facts. Specifically, a frozen text encoder is used to encode the textual representations of the question text and candidate facts separately, and the cosine similarity is used as the basis for the selection. As a lightweight compatibility score, only candidate facts that simultaneously satisfy both entity alias matching and semantic compatibility threshold constraints are retained as training positive examples.

[0029] in This represents a Boolean matching function. If multiple facts satisfy the condition, they are retained as a set of multiple positive examples; if... If so, then no explicit supervision of the retrieval branch is applied to that sample.

[0030] After obtaining a sample-level set of supporting facts Subsequently, this paper further decomposes it into a slot support subset based on the shared task slots. This refines the sample-level weak supervision into slot-level weak supervision signals. Specifically, for the first... Each task slot, based on the search anchor point With candidate fact representation Similarity, from Select the most compatible one Each supporting fact constitutes a subset of slot support.

[0031] Instead of additional manual annotation, the sample-level set of multiple positive examples is structurally allocated according to shared task slots. The Top-M operation described above is only used to construct slot-level weak supervision signals and does not directly participate in gradient backpropagation.

[0032] Preferably, the observation evidence and retrieval anchor points are independently normalized, then concatenated based on the slot index of the shared task slot and input into the channel encoder to map into a first restored semantic representation of the corresponding observation evidence and a second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input for the i-th shared task slot, including: The intra-branch normalization method is used to process the first semantic representation corresponding to the observation evidence output by the observation branch and the second semantic representation corresponding to the retrieval anchor point output by the retrieval branch to obtain the corresponding first normalized semantic representation and second normalized semantic representation. The intra-branch normalization method includes performing normalization operations independently within the observation branch and the retrieval branch respectively. Based on the slot index of the shared task slot, the first normalized semantic representation and the second normalized semantic representation are concatenated to obtain a unified transmission semantic representation. This unified transmission semantic representation is then input into the channel encoder for mapping to obtain a complex symbol vector. After transmission through the physical channel, it is converted into the semantic representation recovered by the receiving end. The slot index is used to ensure the consistency of the slot index between the sending end and the receiving end. Based on the slot index order consistent with the shared slot order, the semantic representation recovered by the receiving end is split to obtain a first recovered semantic representation and a second recovered semantic representation. The first recovered semantic representation corresponds to the observation evidence recovered by the observation branch, and the second recovered semantic representation corresponds to the retrieval anchor point recovered by the retrieval branch. The observation evidence representation of the i-th slot in the first restored semantic representation and the corresponding retrieval anchor representation in the second restored semantic representation are used as the slot-level knowledge recovery input for the i-th shared task slot. The observation evidence representation of the i-th slot is used for subsequent candidate domain construction and filtering, and the corresponding retrieval anchor representation is used for subsequent slot candidate set selection and conditional distribution calculation.

[0033] In some embodiments, for E o and E r Perform normalization within the branch and concatenate them according to the shared slot order to form a unified transmission semantic representation E = [E o E r This represents the complex symbol vector mapped by the channel encoder. The receiving end retrieves the recovered semantic representation based on the same slot index order, and completes the transmission. Mid-splitting and and will the Each slot represents and As input for subsequent slot-level knowledge recovery.

[0034] Preferably, an entity linker is used to link the question text to obtain a question entity set, and a basic local entity pool is constructed on the knowledge graph accordingly. The basic local entity pool is then dynamically expanded based on slot-level knowledge recovery input to obtain slot candidate fact representations. Finally, based on the slot-level knowledge recovery input, conditional distribution calculations are performed on the slot candidate set for the i-th shared task slot to obtain the observation evidence conditional distribution and the retrieval anchor point conditional distribution, including: The input is recovered based on the slot-level knowledge, and the basic local entity pool is dynamically expanded to obtain an expanded candidate domain. The dynamic expansion includes adaptively expanding the coverage of the basic local entity pool based on the observation evidence recovered from the observation branch. A representation construction method is used to process the candidate facts in the extended candidate domain to obtain the slot candidate fact representation. The representation construction method includes adding the embedding representations of the head entity, relation and tail entity after semantic adapter transformation, adding a bias term, and performing layer normalization operation. The input is recovered based on the slot-level knowledge, and the extended candidate domain is filtered to obtain a slot candidate set for the i-th shared task slot. The filtering includes sorting the slot candidate facts based on the cosine similarity between the recovered observation evidence or the recovered retrieval anchor and the slot candidate fact representation, and taking the top N candidate facts. During the training phase, the slot candidate set also includes training positive examples that simultaneously satisfy entity alias matching and semantic compatibility threshold constraints. Based on the slot-level knowledge recovery input, conditional distribution calculation is performed on the slot candidate set of the i-th shared task slot to obtain the observation evidence conditional distribution corresponding to the observation branch and the retrieval anchor point conditional distribution corresponding to the retrieval branch. The conditional distribution calculation uses a temperature parameter to control the smoothness of the conditional distribution. During the training phase, the calculation of the retrieval anchor point conditional distribution is further constrained by the slot-level weak supervision signal. The slot-level weak supervision signal uses the supporting facts in the slot support subset as positive sample labels for the corresponding slot, enabling the retrieval anchor point to learn a stable knowledge invocation direction.

[0035] In some embodiments, an extended candidate domain is constructed: a set of question entities is obtained using an entity linker, and then... in a knowledge graph. The basic local entity pool is constructed above; subsequently, based on the restored observational evidence... The basic local entity pool is dynamically expanded to obtain an expanded candidate domain. Secondly, construct a slot candidate fact representation: for the extended candidate domain Candidate facts Its slot candidate fact indicates satisfy:

[0036] in, , , These are the embedding representations of the head entity, relation, and tail entity, respectively. For semantic adapters, For bias terms, Presentation layer normalization operation; finally, select the slot candidate set: for the first... A shared task slot, based on the recovered observational evidence. and the retrieved anchor points From the extended candidate domain Candidate slots were selected from the pool. The selection formula satisfies:

[0037] in, Indicates the restored observational evidence Or search anchor Slot candidate fact representation Cosine similarity; This represents a training positive example that simultaneously satisfies the entity alias matching and semantic compatibility threshold constraints.

[0038] Preferably, the joint evidence distribution is obtained by fusing the observation evidence condition distribution and the retrieval anchor point condition distribution using the expert product method. Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation of the slot candidate fact representations to obtain the knowledge token for the i-th slot. The knowledge tokens corresponding to the slots that failed to form a valid candidate set during the reasoning stage are serialized and combined to obtain a task-related knowledge token sequence, including: The expert product method is used to fuse the observation evidence condition distribution corresponding to the observation branch and the retrieval anchor point condition distribution corresponding to the retrieval branch to obtain a joint evidence distribution. The retrieval anchor point condition distribution is used to provide retrieval priors, and the observation evidence condition distribution is used for local correction. Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation on the candidate fact representations of the slots to obtain the knowledge token for the i-th slot. The sparse read operation includes using a sparse maximization function, and the sparse read operation introduces a minimal constant to prevent numerical instability and a sparse read temperature parameter to control the degree of read concentration. The knowledge tokens corresponding to slots that failed to form a valid candidate set during the inference stage are degraded to obtain a predefined empty knowledge vector. The empty knowledge vector is used to retain the output of the observation branch in subsequent fusion. The predefined empty knowledge vectors corresponding to all slots are serialized and combined to obtain a task-related knowledge token sequence. The length of the task-related knowledge token sequence is the same as the number of shared task slots.

[0039] In some embodiments, for the first Each shared task slot contains restored observational evidence and restored retrieval anchor points in the slot candidate set. The conditional distributions on are as follows:

[0040] in To retrieve the temperature related to monitoring and tank matching, and considering that the retrieval anchor point and observational evidence respectively bear the responsibility of retrieval prior and local correction, the Product of Experts (PoE) is applied to the two conditional distributions to form a joint evidence distribution:

[0041] Finally, in the joint evidence distribution Perform a sparse read and then perform a weighted summation on the candidate fact representations to obtain the first... Knowledge token for each slot:

[0042] in To sparsely read temperature parameters, To prevent the use of extremely small constants with unstable values, sparsemax is employed to concentrate knowledge invocation for each task slot on a small number of high-confidence facts, thus maintaining the locality, interpretability, and discriminability of knowledge retrieval. If a slot fails to form a valid candidate set during the inference phase, its knowledge token is degenerated into a predefined empty knowledge vector, and the observation branch output is primarily retained in subsequent fusion. Combining all slots yields the task-related knowledge token sequence for the receiver. .

[0043] In some embodiments, the recovered observation evidence is obtained by utilizing a projection matrix shared across branches. First, knowledge tokens and local candidate facts are mapped to a unified feature fusion space; second, the observation dispersion is calculated based on the average deviation between the recovered observation evidence and the high-probability candidate facts.

[0044] To characterize the uncertainty of semantic impairment in observations, and to calculate knowledge dispersion based on the degree of concentration of knowledge tokens under the joint evidence distribution:

[0045] To measure the confidence level of external knowledge activation; simultaneously, to calculate the probability overlap between the observation branch and the retrieval branch on the same candidate set, i.e., the cross-branch consistency coefficient:

[0046] The sufficiency of candidates used to quantify the validity of factual support is calculated based on the highest consensus similarity of the two-way evidence.

[0047] Finally, the weights of the observation branches are determined based on the above indicators. Knowledge branch weights jointly gated by consistency and sufficiency :

[0048] in, Hyperparameters for controlling the decay rate. It is determined solely by the degree of concentration of the recovered observational evidence in the local candidate domain; This is further affected by cross-branch consistency. With candidate sufficiency Joint gating.

[0049] Normalizing each of the above branches by their sum, we obtain the first... The merged representation of each task slot:

[0050] This automatically suppresses unreliable knowledge injection when the channel deteriorates.

[0051] Preferably, based on the slot-level knowledge recovery input and task-related knowledge token sequence, multi-dimensional uncertainty quantification is performed and branch weights are determined. After weighted fusion and inter-slot information exchange, a knowledge-enhanced semantic representation is obtained. This knowledge-enhanced semantic representation is mapped to continuous semantic prefixes and concatenated with text prompts. The result is then input into a multimodal large language model to obtain the target answer sequence in an autoregressive manner, including: A cross-branch shared projection matrix is ​​used to perform unified feature space mapping on the recovered observation evidence and the task-related knowledge token sequence in the first recovered semantic representation to obtain fused features; uncertainty quantification is performed based on the average deviation between the recovered observation evidence and high-probability candidate facts to obtain observation dispersion; confidence is evaluated based on the concentration of knowledge tokens under the joint evidence distribution to obtain knowledge dispersion. The cross-branch shared projection matrix is ​​used to align the representations of the observation branch and the retrieval branch to the same feature space. The cross-branch consistency coefficient is obtained by calculating the probability overlap between the observation branch and the retrieval branch on the same slot candidate set; the fact support validity is quantified based on the highest consensus similarity of the dual-path evidence to obtain the candidate sufficiency; the weight of the observation branch is determined based on the observation dispersion, and the weight of the retrieval branch is determined based on the joint gating of the cross-branch consistency coefficient and the candidate sufficiency; the weights of the two branches are normalized and then the fusion features are weighted and fused to obtain the fusion representation of the task slots, and the slot index is embedded and injected into each fused task slot to obtain the initial interaction representation containing structured identity information. A multi-head self-attention and feedforward network is used to perform controlled information exchange between slots on the initial interaction representation, restore cross-slot dependencies, and obtain the final knowledge-enhanced semantic representation. The representation is then mapped to a continuous semantic prefix aligned with the word embedding space of the large language model through a multilayer perceptron. This prefix is ​​then concatenated with text prompts consisting of task instructions and question texts and input into the multimodal large language model to generate the target answer sequence in an autoregressive manner.

[0052] In some embodiments, a slot index embedding for maintaining structured identity is injected into each merged task slot. The system constructs an initial interaction representation, and then utilizes multi-head self-attention and a feedforward network to perform controlled information exchange between slots to recover cross-slot dependencies, resulting in a final knowledge-enhanced semantic representation. :

[0053]

[0054] in, It is an intermediate state of semantic representation, MHA Indicates bullish self-attention, FFN This represents a slot-by-slot feedforward network.

[0055] Subsequently, the knowledge is enhanced into a semantic representation using a multilayer perceptron. Mapped as continuous semantic prefixes aligned with the word embedding space of a large language model (MLLM).

[0056] T m = MLP(H)

[0057] and semantic prefix T m Together with text prompts consisting of task instructions and question text, they form a unified conditional context for the multimodal large language model. Finally, the model is built using an autoregressive approach to model and output the target answer sequence. .

[0058] The above is an introduction to the method embodiments. The following describes the solution described in this application through device embodiments.

[0059] Figure 3 This is a block diagram illustrating a knowledge-enhanced multimodal neural symbolic semantic communication device according to an exemplary embodiment. The device is used in a knowledge-enhanced multimodal neural symbolic semantic communication method. (Refer to...) Figure 3 The device includes a feature extraction module 310, a training positive example module 320, a channel encoder module 330, an entity linker module 340, a sparse reading module 350, and a knowledge-enhanced semantic module 360.

[0060] Feature extraction module 310: used to acquire the image to be transmitted and the question text corresponding to the image, use a pre-trained multimodal large language model backbone to extract features from the input image and question text and acquire the shared task slot of the sending end, and process the shared task slot through two readout heads to obtain the observation evidence that constitutes the output of the observation branch and the retrieval anchor point that constitutes the output of the retrieval branch. Training Positive Example Module 320: This module is used to introduce knowledge graphs and candidate facts. It employs a frozen text encoder to encode the question text and candidate facts within a local candidate domain and calculates cosine similarity to obtain a lightweight compatibility score. Training positive examples are then selected through semantic compatibility threshold constraints. Slot support subsets are obtained based on the similarity between the retrieval anchor and the candidate fact representations in the training positive examples. Channel encoder module 330: It is used to independently normalize the observation evidence and the retrieval anchor point, and then concatenate them based on the slot index of the shared task slot and input them into the channel encoder to map the first restored semantic representation of the corresponding observation evidence and the second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input of the i-th shared task slot. Entity Linker Module 340: Used to link entities in the question text using an entity linker to obtain a set of question entities and construct a basic local entity pool on the knowledge graph accordingly. Based on the slot-level knowledge recovery input, the basic local entity pool is dynamically expanded to obtain a slot candidate fact representation. Based on the slot-level knowledge recovery input, the condition distribution is calculated on the slot candidate set of the i-th shared task slot to obtain the observation evidence condition distribution and the retrieval anchor condition distribution. Sparse reading module 350: It is used to fuse the observation evidence condition distribution and the retrieval anchor point condition distribution using the expert product method to obtain a joint evidence distribution, and to use sparse reading operation to perform weighted summation on the slot candidate fact representation to obtain the knowledge token of the i-th slot based on the joint evidence distribution, and to serialize and combine the knowledge tokens corresponding to the slots that failed to form an effective candidate set during the reasoning stage to obtain a task-related knowledge token sequence. The knowledge-enhanced semantic module 360 ​​is used to perform multi-dimensional uncertainty quantification and determine branch weights based on the slot-level knowledge recovery input and task-related knowledge token sequence. After weighted fusion and inter-slot information exchange, a knowledge-enhanced semantic representation is obtained. The knowledge-enhanced semantic representation is mapped to a continuous semantic prefix and then concatenated with the text prompt. The input is used to obtain the target answer sequence in an autoregressive manner from a multimodal large language model.

[0061] A knowledge-enhanced multimodal neural symbolic semantic communication device, comprising: a processor; and a memory storing computer-readable instructions, wherein when executed by the processor, the computer-readable instructions implement the method described in any one of the above-described knowledge-enhanced multimodal neural symbolic semantic communication methods.

[0062] Figure 4 This is a schematic diagram of the structure of a knowledge-enhanced multimodal neural symbolic semantic communication device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the knowledge-enhanced multimodal neural symbolic semantic communication device may include the above-mentioned Figure 3 The illustrated knowledge-enhanced multimodal neural symbolic semantic communication device. Optionally, the knowledge-enhanced multimodal neural symbolic semantic communication device 410 may include a first processor 2001.

[0063] Optionally, the knowledge-enhanced multimodal neural symbolic semantic communication device 410 may also include a memory 2002 and a transceiver 2003.

[0064] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0065] The following is combined Figure 4 A detailed description of each component of the knowledge-enhanced multimodal neural symbolic semantic communication device 410 is provided below: The first processor 2001 is the control center of the knowledge-enhanced multimodal neural symbolic semantic communication device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0066] Optionally, the first processor 2001 can perform various functions of the knowledge-enhanced multimodal neural symbolic semantic communication device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0067] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0068] In a specific implementation, as one example, the knowledge-enhanced multimodal neural symbolic semantic communication device 410 may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0069] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0070] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be connected via the interface circuit of the knowledge-enhanced multimodal neural symbolic semantic communication device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0071] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0072] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4(Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0073] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the knowledge-enhanced multimodal neural symbolic semantic communication device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0074] It should be noted that, Figure 4 The structure of the knowledge-enhanced multimodal neural symbolic semantic communication device 410 shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0075] Furthermore, the technical effects of the knowledge-enhanced multimodal neural symbolic semantic communication device 410 can be referred to the technical effects of the knowledge-enhanced multimodal neural symbolic semantic communication method described in the above method embodiments, and will not be repeated here.

[0076] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0077] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0078] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0079] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0080] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0081] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0082] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A knowledge-enhanced multimodal neural symbolic semantic communication method, characterized in that, The method includes: S1: Obtain the image to be transmitted and the question text corresponding to the image. Use a pre-trained multimodal large language model backbone to extract features from the input image and question text and obtain the shared task slot of the sending end. Process the shared task slot through two readout heads to obtain the observation evidence that constitutes the output of the observation branch and the retrieval anchor point that constitutes the output of the retrieval branch. S2: Introduce knowledge graphs and candidate facts, use a frozen text encoder to encode the question text and candidate facts in the local candidate domain and calculate cosine similarity to obtain a lightweight compatibility score, and filter the training positive examples through semantic compatibility threshold constraints. Obtain the slot support subset based on the similarity between the retrieval anchor and the candidate fact representation in the training positive examples. S3: The observation evidence and retrieval anchor point are independently normalized, and then concatenated based on the slot index of the shared task slot and input into the channel encoder to map the first restored semantic representation of the corresponding observation evidence and the second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input of the i-th shared task slot. S4: Use an entity linker to link the question text to obtain a question entity set and construct a basic local entity pool on the knowledge graph accordingly. Based on the slot-level knowledge recovery input, dynamically expand the basic local entity pool to obtain slot candidate fact representations. Based on the slot-level knowledge recovery input, perform condition distribution calculation on the slot candidate set of the i-th shared task slot to obtain the observation evidence condition distribution and retrieval anchor point condition distribution. S5: The expert product method is used to fuse the observation evidence condition distribution and the retrieval anchor point condition distribution to obtain a joint evidence distribution. Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation on the candidate fact representation of the slot to obtain the knowledge token of the i-th slot. The knowledge tokens corresponding to the slots that failed to form an effective candidate set during the reasoning stage are serialized and combined to obtain a task-related knowledge token sequence. S6: Based on the slot-level knowledge recovery input and task-related knowledge token sequence, perform multi-dimensional uncertainty quantification and determine branch weights. After weighted fusion and inter-slot information exchange, obtain knowledge-enhanced semantic representation. Map the knowledge-enhanced semantic representation to continuous semantic prefixes and concatenate it with text prompts. Input it into a multimodal large language model to obtain the target answer sequence in an autoregressive manner.

2. The knowledge-enhanced multimodal neural symbolic semantic communication method according to claim 1, characterized in that, The step S1 involves acquiring the image to be transmitted and the corresponding question text. A pre-trained multimodal large language model backbone is used to extract features from the input image and question text, and the shared task slots at the sending end are obtained. Two readout heads process the shared task slots to obtain observation evidence constituting the observation branch output and retrieval anchor points constituting the retrieval branch output, including: S11: Obtain the image to be transmitted and the question text corresponding to the image. Use a pre-trained multimodal large language model backbone to extract features from the image to be transmitted and the question text to obtain the image visual sequence and the question text sequence. The input image includes the visual information to be analyzed, and the question text includes the query instructions input by the user. S12: Based on the question text sequence, a task summary vector is obtained by mask mean pooling of valid text tokens. The mask mean pooling includes mean aggregation of text tokens only at non-filled positions. S13: Based on the task summary vector, by constructing an adaptive additive bias for the visual tokens and modulating it in the cross-attention scoring process, visual token attention weights with enhanced task relevance are obtained. The adaptive additive bias is used to apply task-related differential weight offsets to each visual token during the attention scoring stage. S14: The learnable task anchor and image visual sequence are input into a single-layer cross-attention mechanism. The image visual sequence is aggregated using visual token attention weights enhanced with task relevance to obtain the shared task slot at the sending end. Then, it is processed by two readout heads to obtain observation evidence and retrieval anchor. The observation branch outputs observation evidence to provide instance observation information to support the final decision, and the retrieval branch outputs retrieval anchor to provide directional information for subsequent knowledge invocation.

3. The knowledge-enhanced multimodal neural symbolic semantic communication method according to claim 1, characterized in that, The S2 process introduces a knowledge graph and candidate facts. A frozen text encoder encodes the question text and candidate facts within a local candidate domain and calculates cosine similarity to obtain a lightweight compatibility score. Training examples are then selected through semantic compatibility threshold constraints. A slot support subset is obtained based on the similarity between the retrieval anchor and the candidate fact representations in the training examples, including: S21: Obtain the answer text, normalize the answer text and match it with aliases in the knowledge graph to obtain entity alias matching results. The normalization process includes standardization operations on the answer text. S22: A frozen text encoder is used to encode the textual representations of the question text and candidate facts and calculate the cosine similarity within the local candidate domain to obtain a lightweight compatibility score. The parameters of the frozen text encoder remain unchanged during training. S23: Based on the entity alias matching results and the lightweight compatibility score, and after filtering by the semantic compatibility threshold constraint, training positive examples are obtained. The training positive examples include a single candidate fact that meets the conditions or a sample-level set of multiple positive examples consisting of multiple candidate facts that meet the conditions. S24: Based on the similarity between the retrieval anchor and the candidate fact representations in the sample-level multi-positive example set, and according to the slot index of the shared task slot at the sending end, select the top M most compatible supporting facts from the sample-level multi-positive example set for each task slot to obtain a slot support subset. The slot support subset constitutes a slot-level weak supervision signal. The slot-level weak supervision signal is used to enhance the alignment between the retrieval anchor and local knowledge facts during the training phase. The selection operation of the top M supporting facts does not directly participate in gradient backpropagation.

4. The knowledge-enhanced multimodal neural symbolic semantic communication method according to claim 1, characterized in that, The S3 process involves independently normalizing the observation evidence and retrieval anchor points, then concatenating them based on the slot index of the shared task slot and inputting them into the channel encoder to map them into a first restored semantic representation of the corresponding observation evidence and a second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the corresponding retrieval anchor point representation in the second restored semantic representation are used as the slot-level knowledge recovery input for the i-th shared task slot, including: S31: Using an intra-branch normalization method, the first semantic representation corresponding to the observation evidence output by the observation branch and the second semantic representation corresponding to the retrieval anchor point output by the retrieval branch are processed to obtain the corresponding first normalized semantic representation and second normalized semantic representation. The intra-branch normalization method includes performing normalization operations independently within the observation branch and the retrieval branch respectively. S32: Based on the slot index of the shared task slot, the first normalized semantic representation and the second normalized semantic representation are concatenated to obtain a unified transmission semantic representation. This unified transmission semantic representation is then input into the channel encoder for mapping to obtain a complex symbol vector. After transmission through the physical channel, it is converted into the semantic representation recovered by the receiving end. The slot index is used to ensure the consistency of the slot index between the sending end and the receiving end. S33: Based on the slot index order consistent with the shared slot order, the semantic representation recovered by the receiving end is split to obtain a first recovered semantic representation and a second recovered semantic representation. The first recovered semantic representation corresponds to the observation evidence recovered by the observation branch, and the second recovered semantic representation corresponds to the retrieval anchor point recovered by the retrieval branch. S34: The observation evidence representation of the i-th slot in the first restored semantic representation and the corresponding retrieval anchor representation in the second restored semantic representation are used as the slot-level knowledge recovery input for the i-th shared task slot. The observation evidence representation of the i-th slot is used for subsequent candidate domain construction and filtering, and the corresponding retrieval anchor representation is used for subsequent slot candidate set selection and conditional distribution calculation.

5. The knowledge-enhanced multimodal neural symbolic semantic communication method according to claim 1, characterized in that, The S4 process involves using an entity linker to link entities in the question text to obtain a question entity set, and constructing a basic local entity pool on the knowledge graph accordingly. Based on the slot-level knowledge recovery input, the basic local entity pool is dynamically expanded to obtain slot candidate fact representations. Then, based on the slot-level knowledge recovery input, conditional distribution calculations are performed on the slot candidate set for the i-th shared task slot to obtain the observation evidence conditional distribution and the retrieval anchor point conditional distribution, including: S41: Recover the input based on the slot-level knowledge, dynamically expand the basic local entity pool to obtain an expanded candidate domain, wherein the dynamic expansion includes adaptively expanding the coverage of the basic local entity pool based on the observation evidence recovered from the observation branch; S42: Using a representation construction method, the candidate facts in the extended candidate domain are processed to obtain the slot candidate fact representation. The representation construction method includes adding the embedding representations of the head entity, relation and tail entity after semantic adapter transformation, adding a bias term, and performing layer normalization operation. S43: Recover the input based on the slot-level knowledge, filter the extended candidate domain, and obtain a slot candidate set for the i-th shared task slot. The filtering includes sorting the slot candidate facts based on the cosine similarity between the recovered observation evidence or the recovered retrieval anchor and the slot candidate fact representation, and taking the top N candidate facts. During the training phase, the slot candidate set also includes training positive examples that simultaneously satisfy entity alias matching and semantic compatibility threshold constraints. S44: Based on the slot-level knowledge recovery input, perform conditional distribution calculation on the slot candidate set of the i-th shared task slot to obtain the observation evidence conditional distribution corresponding to the observation branch and the retrieval anchor point conditional distribution corresponding to the retrieval branch. The conditional distribution calculation uses a temperature parameter to control the smoothness of the conditional distribution. During the training phase, the calculation of the retrieval anchor point conditional distribution is further constrained by the slot-level weak supervision signal. The slot-level weak supervision signal uses the supporting facts in the slot support subset as positive sample labels for the corresponding slot, enabling the retrieval anchor point to learn a stable knowledge invocation direction.

6. The knowledge-enhanced multimodal neural symbolic semantic communication method according to claim 1, characterized in that, S5 employs an expert product method to fuse the observed evidence condition distribution and the retrieval anchor point condition distribution to obtain a joint evidence distribution. Based on this joint evidence distribution, a sparse read operation is used to perform a weighted summation of the slot candidate fact representations to obtain the knowledge token for the i-th slot. The knowledge tokens corresponding to the slots that failed to form a valid candidate set during the inference stage are serialized and combined to obtain a task-related knowledge token sequence, including: S51: Using the expert product method, the observation evidence condition distribution corresponding to the observation branch and the retrieval anchor point condition distribution corresponding to the retrieval branch are fused to obtain a joint evidence distribution. The retrieval anchor point condition distribution is used to provide retrieval priors, and the observation evidence condition distribution is used for local correction. S52: Based on the joint evidence distribution, a sparse read operation is used to perform a weighted summation on the candidate fact representation of the slot to obtain the knowledge token of the i-th slot. The sparse read operation includes using a sparse maximization function, and the sparse read operation introduces a minimal constant to prevent numerical instability and a sparse read temperature parameter to control the degree of read concentration. S53: Degenerate the knowledge tokens corresponding to the slots that failed to form a valid candidate set during the inference stage to obtain a predefined empty knowledge vector. The empty knowledge vector is used to retain the output of the observation branch in subsequent fusion. S54: Serialize and combine the predefined empty knowledge vectors corresponding to all slots to obtain a task-related knowledge token sequence. The length of the task-related knowledge token sequence is the same as the number of shared task slots.

7. The knowledge-enhanced multimodal neural symbolic semantic communication method according to claim 1, characterized in that, S6 performs multi-dimensional uncertainty quantification and determines branch weights based on the slot-level knowledge recovery input and task-related knowledge token sequence. After weighted fusion and inter-slot information exchange, a knowledge-enhanced semantic representation is obtained. This knowledge-enhanced semantic representation is mapped to continuous semantic prefixes and concatenated with text prompts. The result is then input into a multimodal large language model to obtain the target answer sequence via autoregression, including: S61: Using a cross-branch shared projection matrix, the recovered observation evidence and the task-related knowledge token sequence in the first recovered semantic representation are mapped to a unified feature space to obtain fused features; uncertainty quantification is performed based on the average deviation between the recovered observation evidence and high-probability candidate facts to obtain observation dispersion; confidence is evaluated based on the concentration of knowledge tokens under the joint evidence distribution to obtain knowledge dispersion. The cross-branch shared projection matrix is ​​used to align the representations of the observation branch and the retrieval branch to the same feature space. S62: By calculating the probability overlap between the observation branch and the retrieval branch on the same slot candidate set, a cross-branch consistency coefficient is obtained; the validity of fact support is quantified based on the highest consensus similarity of the dual-path evidence to obtain the candidate sufficiency; the weight of the observation branch is determined based on the observation dispersion, and the weight of the retrieval branch is determined based on the joint gating of the cross-branch consistency coefficient and the candidate sufficiency; after normalizing the weights of the two branches, the fusion features are weighted and fused to obtain the fusion representation of the task slot, and the slot index is embedded and injected into each fused task slot to obtain an initial interaction representation containing structured identity information; S63: Employing multi-head self-attention and feedforward networks, controlled information exchange is performed between slots on the initial interaction representation to restore cross-slot dependencies and obtain the final knowledge-enhanced semantic representation. This representation is then mapped to a continuous semantic prefix aligned with the word embedding space of the large language model through a multilayer perceptron. This prefix is ​​then concatenated with text prompts consisting of task instructions and question texts and input into the multimodal large language model to generate the target answer sequence in an autoregressive manner.

8. A knowledge-enhanced multimodal neural symbolic semantic communication device, wherein the knowledge-enhanced multimodal neural symbolic semantic communication device is used to implement the knowledge-enhanced multimodal neural symbolic semantic communication method as described in any one of claims 1-7, characterized in that, The device includes: Feature extraction module: used to acquire the image to be transmitted and the question text corresponding to the image. It uses a pre-trained multimodal large language model backbone to extract features from the input image and question text and acquire the shared task slots of the sending end. It processes the shared task slots through two readout heads to obtain the observation evidence that constitutes the output of the observation branch and the retrieval anchor points that constitute the output of the retrieval branch. Training Positive Examples Module: This module is used to introduce knowledge graphs and candidate facts. It employs a frozen text encoder to encode the question text and candidate facts within a local candidate domain and calculates cosine similarity to obtain a lightweight compatibility score. The module then filters the results through semantic compatibility threshold constraints to obtain training positive examples. Based on the similarity between the retrieval anchor and the candidate fact representations in the training positive examples, a slot support subset is obtained. Channel encoder module: It is used to independently normalize the observation evidence and retrieval anchor point, and then concatenate them based on the slot index of the shared task slot and input them into the channel encoder to map the first restored semantic representation of the corresponding observation evidence and the second restored semantic representation of the corresponding retrieval anchor point. The observation evidence representation of the i-th slot in the first restored semantic representation and the retrieval anchor point representation corresponding to the second restored semantic representation are used as the slot-level knowledge recovery input of the i-th shared task slot. Entity Linker Module: Used to link entities in the question text to obtain a set of question entities and construct a basic local entity pool on the knowledge graph accordingly. Based on the slot-level knowledge recovery input, the basic local entity pool is dynamically expanded to obtain slot candidate fact representations. Based on the slot-level knowledge recovery input, the condition distribution is calculated on the slot candidate set of the i-th shared task slot to obtain the observation evidence condition distribution and the retrieval anchor condition distribution. The sparse reading module is used to fuse the observation evidence condition distribution and the retrieval anchor point condition distribution using the expert product method to obtain a joint evidence distribution. Based on the joint evidence distribution, the sparse reading operation is used to perform a weighted summation on the candidate fact representation of the slot to obtain the knowledge token of the i-th slot. The knowledge tokens corresponding to the slots that failed to form a valid candidate set during the reasoning stage are serialized and combined to obtain a task-related knowledge token sequence. The knowledge-enhanced semantic module is used to recover the input and task-related knowledge token sequence based on the slot-level knowledge, perform multi-dimensional uncertainty quantification and determine branch weights, obtain the knowledge-enhanced semantic representation through weighted fusion and inter-slot information exchange, map the knowledge-enhanced semantic representation into continuous semantic prefixes and concatenate them with text prompts, input them into a multimodal large language model to obtain the target answer sequence in an autoregressive manner.

9. A knowledge-enhanced multimodal neural symbolic semantic communication device, characterized in that, The knowledge-enhanced multimodal neural symbolic semantic communication device includes: a processor; and a memory storing computer-readable instructions, which, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images