Multi-modal information alignment application method and system, medium and equipment

Abstract

The invention provides a multi-modal information alignment application method and system, a medium and equipment. The multi-modal information alignment application method comprises the following steps: constructing brain language feature information codes among a plurality of functional modules; utilizing continuous feedback learning to align and unify brain language feature coding information of multiple modules; adjusting competition between long-term memory LTM modules to become weight information of short-term memory STM modules; and outputting STM brain language feature information to an execution LTM unit to realize decision visualization and learning performance evaluation. According to the method, in the brain language information extraction process of the LTM modules, the attention mechanism is combined, and the mutual influence between the LTM modules is fully considered, so that different functional modules generate associated decision evaluation, and it is ensured that information extracted from different modules has uniform and aligned expression; and the weight of the information block is optimized by combining a distributed type Hertz learning algorithm, and the prediction and evaluation capability of the model is improved, so that the competitive decision and collaborative learning performance of the brain-like behavior decision architecture is comprehensively improved, and the brain-like behavior decision system has better adaptability and intelligence.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-like computing, and specifically, to a multimodal information alignment application method and system, and in particular to a multimodal information alignment application method and computer system based on a collaborative attention mechanism. Background Art

[0002] In the field of brain-like computing, the key to achieving truly autonomous robots lies in building behavioral decision-making computing models and architectures that can simulate human consciousness, so that robots have the ability to handle unforeseen events. Brain-like behavioral decision-making architectures, such as models based on Conscious Turing Machines (CTMs), make it possible to achieve this goal. However, current brain-like behavioral decision-making architectures still face many challenges in practical applications.

[0003] First, the collaboration and competition between multiple functional modules rely on the specific information encoding transmitted between modules. Since different modules work based on different algorithms and data sources, the information they generate may be inconsistent and noisy. These inconsistencies not only increase the complexity of information processing, but may also lead to erroneous decision results. Therefore, how to achieve unified encoding of information between multiple functional modules and accurate decision evaluation is an urgent problem to be solved.

[0004] Secondly, traditional methods often have shortcomings in dealing with the mutual influence between multiple functional modules. They often find it difficult to fully consider the interactions and dynamic changes between different modules, resulting in insufficient prediction capabilities and poor decision-making results. In addition, the information encoding capabilities between different modules vary greatly, the coordination method is single and inefficient, and there is a lack of learning and adaptability, making it difficult to adapt to complex and changing environments.

[0005] Therefore, it is necessary to propose a new technical solution to improve the above technical problems. Summary of the invention

[0006] In view of the defects in the prior art, the object of the present invention is to provide a multimodal information alignment application method, system, medium and device.

[0007] According to a multimodal information alignment application method provided by the present invention, the method comprises the following steps:

[0008] Step S1: Building a brain-like behavior decision-making system based on the CTM model, including multiple functional LTM, STM units and a two-way communication network, and building a unified feature information encoding between LTM modules to serve as the information basis for two-way consciousness communication between LTM and STM;

[0009] Step S2: In the process of extracting brain language information from the LTM module, the attention mechanism is combined to consider the mutual influence between LTM modules;

[0010] Step S3: Information alignment is learned through multimodal feedback, so that the information extracted from different LTM modules has a unified and aligned expression;

[0011] Step S4: Combined with the distributed Hebbian-like learning algorithm, the frequency and correlation of information transmission between different LTM modules are calculated to adjust the weight of LTM competition to become STM;

[0012] Step S5: The STM brain language information after the competition is successful will be further broadcast to the execution LTM module for translation and output of the decision result.

[0013] Preferably, in step S1:

[0014] The forms of LTM adopted include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module, and an action LTM module for execution. The constructed feature information encoding includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding.

[0015] Preferably, in step S2:

[0016] LTM adjusts its own brain language information encoding by combining the attention mechanism to align information between LTMs; the attention described includes: the LTM module publishes information blocks Chunk to STM to achieve STM consciousness changes; the LTM module receives information broadcast from STM to form top-down attention, making it sensitive to information to affect the encoder's focus range or area on the input information, thereby affecting the output content of the perceptual encoder.

[0017] Preferably, in step S3:

[0018] Brain speech information alignment combines the LTM-STM dynamics of the brain-like behavior decision-making model. The perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives other LTM brain speech information encoding broadcast by STM. Each LTM compares the vector of its own output brain speech with the vector of other LTM brain speech received from STM to obtain the correspondence between the brain speech vectors output by different LTMs, and realizes brain speech alignment, and combines attention to achieve unified aligned brain speech information encoding.

[0019] Preferably, in step S4:

[0020] A separate Hebbian learning sleep expert algorithm runs in each LTM processor; the LTM module uses the Hebbian learning sleep expert algorithm to modify the strength of the set weights and define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations: if the content in the STM is correct, then p does nothing; if the content in the STM is wrong: if p is correct, then p increases the strength of the information block it generates; if p is wrong: p corrects its error; then, p reduces the strength of the information block it generates;

[0021] In the Hebbian learning sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised method is that the LTM module finds out that it has made an error by discovering the fact that the received STM information block is inconsistent with its own brain language information with confidence. Therefore, each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies its strength based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm is expanded to include a supervisory signal of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt.

[0022] Preferably, in step S5:

[0023] The STM brain language information will be further broadcast to the action execution LTM module to output the decision visualization results or output the natural semantic information encoded in the brain language through the semantic understanding LTM module to reflect explainable general artificial intelligence; the decision visualization includes converting the brain language feature information output by the STM into visual graphics to help users understand the decision logic of the system.

[0024] The present invention also provides a multimodal information alignment application system, the system comprising the following modules:

[0025] Module M1: Building a brain-like behavior decision-making system based on the CTM model, including multiple functional LTM, STM units and a two-way communication network. A unified feature information encoding is built between LTM modules to serve as the information basis for two-way consciousness communication between LTM and STM.

[0026] Module M2: In the process of extracting brain language information from the LTM module, the attention mechanism is combined to consider the mutual influence between LTM modules;

[0027] Module M3: Information alignment is learned through multimodal feedback, and the information extracted from different LTM modules has a unified and aligned expression;

[0028] Module M4: Combined with the distributed Hebbian-like learning algorithm, the frequency and correlation of information transmission between different LTM modules are calculated to adjust the weight of LTM competition to become STM;

[0029] Module M5: After the competition is successful, the STM brain language information will be further broadcast to the execution LTM module for translation and output of the decision result.

[0030] Preferably, in the module M1:

[0031] The forms of LTM used include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module, and an action LTM module for execution. The feature information encoding constructed includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding;

[0032] In the module M2:

[0033] LTM adjusts its own brain language information encoding by combining the attention mechanism to align information between LTMs; the attention described includes: the LTM module publishes information chunks to STM to achieve STM consciousness changes; the LTM module receives information broadcast from STM to form top-down attention, making it sensitive to information to affect the encoder's focus range or area on the input information, thereby affecting the output content of the perceptual encoder;

[0034] In the module M3:

[0035] Brain speech information alignment combines the LTM-STM dynamics of the brain-like behavior decision model. The perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives other LTM brain speech information encoding broadcast by STM. Each LTM compares the vector of its own output brain speech with the vector of other LTM brain speech received from STM to obtain the corresponding relationship between the brain speech vectors output by different LTMs, and realizes brain speech alignment, and combines attention to realize unified and aligned brain speech information encoding;

[0036] In the module M4:

[0037] A separate Hebbian learning sleep expert algorithm runs in each LTM processor; the LTM module uses the Hebbian learning sleep expert algorithm to modify the strength of the set weights and define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations: if the content in the STM is correct, then p does nothing; if the content in the STM is wrong: if p is correct, then p increases the strength of the information block it generates; if p is wrong: p corrects its error; then, p reduces the strength of the information block it generates;

[0038] In the Hebbian learning sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised system is that the LTM module finds out that it has made an error by discovering that the received STM information block contradicts its own brain language information with confidence, so each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies its strength based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm is expanded to include a supervisory signal of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt;

[0039] In the module M5:

[0040] The STM brain language information will be further broadcast to the action execution LTM module to output the decision visualization results or output the natural semantic information encoded in the brain language through the semantic understanding LTM module to reflect explainable general artificial intelligence; the decision visualization includes converting the brain language feature information output by the STM into visual graphics to help users understand the decision logic of the system.

[0041] The present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the multimodal information alignment application method described above are implemented.

[0042] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the multimodal information alignment application method described above when executed by the processor.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. The core of the present invention is that in the process of extracting brain language information from the LTM module, the attention mechanism is combined to fully consider the mutual influence between LTM modules, so that different functional modules produce associated decision evaluations, ensuring that the information extracted from different modules has a unified and aligned expression;

[0045] 2. The present invention combines the distributed "Hebbian-like learning" algorithm to optimize the weights of information blocks and improve the prediction and evaluation capabilities of the model, thereby comprehensively improving the competitive decision-making and collaborative learning performance of the brain-like behavior decision-making architecture, making the brain-like behavior decision-making system more adaptable and intelligent;

[0046] 3. The present invention constructs a unified feature information encoding between multiple modules, combines the attention mechanism and continuous feedback learning to align the brain language information of multiple modules, and uses Hebbian-like learning of LTM module competition weight information to provide a unified and aligned information expression basis for competitive decision-making and collaborative learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0048] Figure 1 A schematic diagram of a multimodal information alignment application method and a computer system in the present invention;

[0049] Figure 2 It is a schematic diagram of the module of the LTM functional module in the present invention based on the pre-training model and online learning practice encoding and decoding;

[0050] Figure 3 It is a schematic diagram of the brain-speech alignment process of the upstream and downstream attention mechanisms of the LTM functional module in the present invention;

[0051] Figure 4 It is a schematic diagram of the process of realizing information alignment through feedback learning of the LTM functional module in the present invention;

[0052] Figure 5 It is a schematic diagram of adjusting the LTM competition to become the STM weight in the present invention;

[0053] Figure 6 It is a schematic diagram of the action LTM output behavior decision in the present invention. DETAILED DESCRIPTION

[0054] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0055] Embodiment 1:

[0056] Reference Figure 1 and Figure 2 According to a multimodal information alignment application method provided by the present invention, the method comprises the following steps:

[0057] Step S1: construct a brain-like behavior decision-making system based on the CTM model, which includes multiple functional LTM, STM units and a two-way communication network. A unified feature information encoding is constructed between the LTM modules to serve as the information basis for the two-way conscious communication between LTM and STM; the forms of LTM adopted include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module and an action LTM module for execution. The constructed feature information encoding includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding.

[0058] Step S2: In the process of extracting brain speech information from the LTM module, the attention mechanism is combined to consider the mutual influence between LTM modules; the LTM adjusts its own brain speech information encoding by combining the attention mechanism to align the information between LTMs; the attention described herein includes: the LTM module publishes the information block Chunk to the STM to achieve the consciousness change of the STM; the LTM module receives the information broadcast from the STM to form top-down attention, making it sensitive to information to affect the encoder's focus range or area on the input information, thereby affecting the output content of the perceptual encoder.

[0059] Step S3: Through multimodal feedback learning information alignment, the information extracted from different LTM modules has a unified and aligned expression; brain language information alignment combines the LTM-STM dynamics of the brain-like behavior decision-making model. The perception information encoding of different LTM modules forms brain language information to participate in STM competition, and at the same time receives other LTM brain language information encodings broadcast by STM. Each LTM compares the vector of its own output brain language with the vector of other LTM brain language received from STM to obtain the correspondence between the brain language vectors output by different LTMs, and realizes brain language alignment, and combines attention to realize unified and aligned brain language information encoding.

[0060] Step S4: Combined with the distributed Hebbian learning algorithm, by calculating the frequency and correlation of information transmission between different LTM modules, adjust the weight of LTM competition to become STM; a separate Hebbian learning sleep expert algorithm is run in each LTM processor; the LTM module uses the Hebbian learning sleep expert algorithm to modify the intensity of the set weights and define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM functional module p performs the following operations: if the content in STM is correct, then p does nothing; if the content in STM is wrong: if p is correct, then p increases the intensity of the information block it generates; if p is wrong: p corrects its error; then, p reduces The strength of the information block it generates; in the Hebbian learning sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised method is that the LTM module finds out that it has made an error by discovering the fact that the received STM information block is inconsistent with its own brain language information with confidence. Therefore, each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies the strength based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm is expanded to include supervisory signals of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt.

[0061] Step S5: The STM brain language information after the competition is successful will be further broadcast to the execution LTM module for translation and output of the decision result; the STM brain language information will be further broadcast to the action execution LTM module to output the decision visualization result or output the natural semantic information encoded in the brain language through the semantic understanding LTM module to reflect explainable general artificial intelligence; the decision visualization includes converting the brain language feature information output by the STM into a visual graphic to help users understand the decision logic of the system.

[0062] The present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the multimodal information alignment application method described above are implemented.

[0063] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the multimodal information alignment application method described above when executed by the processor.

[0064] The present invention also provides a multimodal information alignment application system, which can be implemented by executing the process steps of the multimodal information alignment application method, that is, those skilled in the art can understand the multimodal information alignment application method as a preferred implementation of the multimodal information alignment application system.

[0065] Embodiment 2:

[0066] The present invention also provides a multimodal information alignment application system, the system comprising the following modules:

[0067] Module M1: A brain-like behavior decision-making system is constructed based on the CTM model, which includes multiple functional LTM, STM units and a two-way communication network. A unified feature information encoding is constructed between the LTM modules to serve as the information basis for the two-way consciousness communication between LTM and STM. The LTM forms used include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, an emotion module for generating decision-oriented guidance, a navigation planning module and an action LTM module for execution. The constructed feature information encoding includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding.

[0068] Module M2: In the process of extracting brain speech information from the LTM module, the attention mechanism is combined to consider the mutual influence between LTM modules; LTM adjusts its own brain speech information encoding by combining the attention mechanism to align information between LTMs; the attention described includes: the LTM module publishes information chunks to the STM to achieve STM consciousness changes; the LTM module receives information broadcast from the STM to form top-down attention, making it sensitive to information to affect the encoder's focus range or area on the input information, thereby affecting the output content of the perceptual encoder;

[0069] Module M3: Through multimodal feedback learning information alignment, the information extracted from different LTM modules has a unified and aligned expression; brain speech information alignment combines the LTM-STM dynamics of the brain-like behavior decision model. The perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives other LTM brain speech information encoding broadcast by STM. Each LTM learns the correspondence between the brain speech vectors output by different LTMs by comparing the vector of its own output brain speech with the vector of other LTM brain speech received from STM, and realizes brain speech alignment, and realizes unified and aligned brain speech information encoding in combination with attention;

[0070] Module M4: Combined with the distributed Hebbian-like learning algorithm, by calculating the frequency and correlation of information transmission between different LTM modules, adjust the weight of LTM competition to become STM; a separate Hebbian-like learning sleep expert algorithm runs in each LTM processor; the LTM module uses the Hebbian learning sleep expert algorithm to modify the intensity of the set weights and define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations: if the content in STM is correct, then p does nothing; if the content in STM is wrong: if p is correct, then p increases the intensity of the information block it generates; if p is wrong: p corrects its error; then, p reduces The strength of the information block it generates; in the Hebbian learning sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised system is that the LTM module finds out that it has made an error by discovering that the received STM information block is inconsistent with its own brain language information with confidence, so each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies the strength based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm is expanded to include supervisory signals from the environment feedback of the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt;

[0071] Module M5: The STM brain language information after the competition is successful will be further broadcast to the execution LTM module for translation and output of the decision result; the STM brain language information will be further broadcast to the action execution LTM module to output the decision visualization result or output the natural semantic information encoded in the brain language through the semantic understanding LTM module to reflect explainable general artificial intelligence; the decision visualization includes converting the brain language feature information output by the STM into a visual graphic to help users understand the decision logic of the system.

[0072] Embodiment 3:

[0073] The present invention relates to a multimodal information alignment application method and computer system, which aims to improve the collaborative learning and competitive decision-making performance between multiple functional modules (also known as long-term memory LTM modules) in brain-like behavior decision models and architectures such as consciousness Turing machines (CTMs). It mainly includes 1) establishing a brain-like behavior decision-making system based on multiple functional modules based on models such as CTM, and constructing brain language feature information encoding between multiple functional modules; 2) based on the attention mechanism, using continuous feedback learning to align the brain language feature encoding information of unified multiple modules; 3) adjusting the weight information of the competition between long-term memory LTM modules to become a short-term memory STM module, providing an information expression basis for competitive behavior decision-making; 4) outputting STM brain language feature information to the execution LTM unit to realize decision visualization and learning performance evaluation. The core of the present invention is that in the process of extracting brain language information from the LTM module, the attention mechanism is combined to fully consider the mutual influence between LTM modules, so that different functional modules produce related decision evaluations, ensuring that the information extracted from different modules has a unified and aligned expression; at the same time, the distributed "Hebbian-like learning" algorithm is combined to optimize the weights of information blocks and enhance the model's predictive evaluation capabilities, thereby comprehensively improving the competitive decision-making and collaborative learning performance of the brain-like behavior decision-making architecture, making the brain-like behavior decision-making system more adaptable and intelligent.

[0074] The present invention provides a multimodal information alignment application method and a computer system, which constructs a unified feature information encoding between multiple modules, combines the attention mechanism and continuous feedback learning to align the brain language information of multiple modules, and utilizes the Hebbian-like learning LTM module competition weight information to provide a unified aligned information expression basis for competitive decision-making and collaborative learning.

[0075] A multimodal information alignment application method comprises the following steps:

[0076] Step 1: Construct a brain-like behavior decision-making system based on the CTM model, including multiple functional LTM, STM units and a two-way communication network. A unified feature information encoding is constructed between LTM modules to serve as the information basis for two-way consciousness communication between LTM and STM.

[0077] Step 2: In the process of extracting brain language information from the LTM module, the attention mechanism is combined to fully consider the mutual influence between LTM modules;

[0078] Step 3: Learn information alignment through multimodal feedback to ensure that the information extracted from different LTM modules has a unified and aligned expression;

[0079] Step 4: Combined with the distributed "Hebbian-like learning" algorithm, by calculating the frequency and correlation of information transmission between different LTM modules, adjust the weight of LTM competition to become STM, thereby improving the model's predictive evaluation ability and ensuring the competitive decision-making and collaborative learning performance of the brain-like behavioral decision-making architecture.

[0080] Step 5: After the competition is successful and stable, the STM brain language information will be further broadcast to the execution LTM module for translation and output of the decision result.

[0081] The step 1 specifically includes the following contents:

[0082] The form of LTM used is not limited, including a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module, and an action LTM module for execution. The constructed feature information encoding includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding (such as a large model front end), etc., but all LTM modules are required to have similar encoding behaviors, so that they have high similarity for the same decision-making tasks.

[0083] The step 2 specifically includes the following contents:

[0084] LTM adjusts its own brain language information encoding by combining the attention mechanism to achieve information alignment between LTMs, thereby improving its own evaluation and prediction capabilities. The attention mentioned includes 1) the LTM module publishes information chunks to STM to achieve STM's consciousness change, achieving bottom-up attention; 2) at the same time, the LTM module receives information broadcast from STM to form top-down attention, making it sensitive to specific information to affect the encoder's focus range or area on the input information, thereby affecting the output content of the perceptual encoder and improving the correctness of the encoding.

[0085] The step three specifically includes the following contents:

[0086] Brain speech information alignment fully combines the LTM-STM dynamics of the brain-like behavior decision-making model. The perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives other LTM brain speech information encoding broadcast by STM. Each LTM compares the vector of its own output brain speech with the vector of other LTM brain speech received from STM to obtain the correspondence between the brain speech vectors output by different LTMs, and realizes brain speech alignment, thereby realizing unified aligned brain speech information encoding in combination with the attention in step 2 described in claim 1.

[0087] The step 4 specifically includes the following contents:

[0088] A separate "Hebbian-like learning" sleep expert algorithm runs in each LTM processor. The LTM module uses the "Hebbian learning" sleep expert algorithm to modify the strength of the set weights, which helps to define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations: 1) If the content in the STM is correct, then p does nothing. 2) If the content in the STM is wrong: If p is correct, then p increases the strength of the information block it generates. If p is wrong: p corrects its error; then, p reduces the strength of the information block it generates.

[0089] In the "Hebbian-like learning" sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised method is that the LTM module finds out that it has made an error by discovering that the received STM information block contradicts its own brain language information with a sufficiently high confidence (intensity). Therefore, each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies the intensity based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm can be expanded to include supervisory signals of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt.

[0090] The step five specifically includes the following contents:

[0091] The STM brain language information will be further broadcasted to the action execution LTM module to output the decision visualization results, or the natural semantic information encoded by the brain language can be output through the semantic understanding LTM module to reflect the explainable general artificial intelligence. The decision visualization includes converting the brain language feature information output by the STM into visual graphics, such as decision trees, heat maps, etc., to help users understand the decision logic of the system

[0092] A computer system for multimodal information alignment application mainly includes multiple LTM functional modules, STM modules, and a bidirectional communication network. It is characterized in that there are attention perception units and feedback learning alignment modules between the LTM functional modules, and there is a Hebbian-like learning module in the bidirectional communication network unit.

[0093] The multiple LTM functional modules are not limited in form, and include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module, and an action LTM module for execution, etc.

[0094] The bidirectional communication network is used to transmit information between each LTM module and STM module, ensuring efficient collaboration and information sharing between the modules.

[0095] The attention perception unit between the LTM functional modules is used in the step 2 described in claim 1. During the brain language information encoding process, the LTM module is considered to be sensitive to information in a specific area, thereby affecting the encoder's focus range or area on the input information.

[0096] The feedback learning alignment module is used to align and unify the brain language feature encoding information of multiple LTM modules and STM modules in step three of claim 1.

[0097] The Hebbian-like learning module in the bidirectional communication network unit is used in step 4 of claim 1 to adjust the competition between LTM modules to become the weight information of STM.

[0098] Establishment of brain-like behavioral decision-making system: Figure 1 The multimodal information alignment application method and computer system schematic diagram of the present invention are given in the figure. The computer system is implemented based on the CTM architecture model and can be described by an octet.<STM,LTM,Up Tree,Down Tree,Links,Input,Output,Chunk> . Among them, STM is a short-term memory module; LTM is a long-term memory module; Up Tree is the up-tree connecting LTM to STM; Down Tree is the down-tree connecting STM to LTM; Links is a dark channel for direct communication established between LTMs in the form of associative learning; Input obtains environmental information collected by sensors and sends it to the specified LTM processor, and Output transmits the behavior output, that is, the decision result, to the actuator to act on the environment; Chunk is the information block transmitted by communication between STM and each LTM, and the core communication information is collectively referred to as "brain language".

[0099] Furthermore, the computer system includes a short-term memory module STM and multiple long-term memory modules LTM. The STM here has expanded its memory capacity based on the original CTM architecture model and has a certain number of storage units. Each LTM corresponds to a different functional partition of the brain and realizes different functions. Multiple LTMs transmit multiple "brain language" information to STM through the ascending tree; when the information correlation between two LTMs is high enough, a dark channel will be established to skip the complex competition process and communicate directly;

[0100] Furthermore, the multimodal information alignment application method of the collaborative attention mechanism includes adjusting the brain speech information encoding for the LTM-STM bidirectional attention mechanism, feedback learning alignment of the brain speech information encoding process of the LTM module, and using the Hebbian learning-like sleep expert algorithm to adjust the LTM competition to become the importance weight index of the STM. The LTM module also realizes the online learning of the LTM codec itself based on the brain speech information broadcast by the STM, combining supervised and unsupervised information methods, and improving the system's adaptability to complex environments.

[0101] Attention Mechanism: Figure 2 It is a schematic diagram of a module for implementing encoding and decoding based on a pre-trained model and online learning by the LTM functional module in the information alignment method of the present invention and the computer system, including obtaining information from the outside world, converting it into "brain language" encoded information through a pre-trained model, and aligning the broadcasted brain language information and online learning of its own encoding and decoding structure.

[0102] Further, Figure 3 The present invention is based on the Figure 2 The LTM encoding and decoding module in the STM realizes bottom-up attention perception. Among them, the perceptual LTM generates the information block Chunk that needs to be published to the STM based on the information from the STM and the information observed by itself. The core information includes the brain language element gist, that is, the feature encoding vector output by the encoder, and the importance index weight. By publishing the information block Chunk to the STM to realize the "consciousness" change of the STM, bottom-up attention is realized. At the same time, the perceptual LTM module receives information broadcast from the STM to form top-down attention, making it sensitive to specific information to affect the encoder's attention range or area for the input information, thereby affecting the output content of the perceptual encoder and improving the correctness of the encoding. By combining the attention mechanism, the information correlation between the LTM modules can be fully considered, so that different functional modules can produce related decision evaluations, thereby improving the predictive ability of the brain-like behavior decision architecture.

[0103] Continuous feedback learning brain language information alignment application: Figure 4 The present invention is based on the Figure 2 The LTM encoding and decoding module in the feedback learning realizes the brain speech information alignment process. Among them, the perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives the brain speech information encoding of other LTM broadcast by STM. Each LTM learns the correspondence between the brain speech vectors output by different LTMs by comparing the vector of its own output brain speech with the vector of other LTM brain speech received from STM, and realizes brain speech information alignment.

[0104] Application of Hebbian-like learning weight information alignment: Figure 5The basic competitive communication structure of the present invention for realizing weight information alignment is described, wherein the brain language competition of the multifunctional LTM module forms a short-term memory STM (upward competitive communication), and then the STM broadcast communication is used to realize decision-making and feedback learning (downward tree broadcast communication).

[0105] Furthermore, the present invention uses a distributed "Hebbian learning" sleep expert algorithm to adjust the weight of the information block, that is, a separate "Hebbian learning" sleep expert algorithm is run in each LTM processor. The LTM module uses the "Hebbian learning" sleep expert algorithm to modify the intensity of the set weight, which helps to define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations:

[0106] 1) If the content in the STM is correct, then p does nothing. 2) If the content in the STM is wrong: If p is correct, then p increases the strength of the information block it generates. If p is wrong: p corrects its error; then, p reduces the strength of the information block it generates.

[0107] In the "Hebbian-like learning" sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised method is that the LTM module finds out that it has made an error by discovering that the received STM information block contradicts its own brain language information with a sufficiently high confidence (intensity). Therefore, each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies the intensity based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm can be expanded to include supervisory signals of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt.

[0108] Decision Visualization and Performance Evaluation: Figure 6 The present invention describes the process of outputting action decisions based on the input brain language information state, and combines decision networks such as reinforcement learning to output the final action execution state. At the same time, the brain language feature information output by STM is converted into visual graphics, such as decision trees and heat maps, in combination with the decision visualization module to help users understand the decision logic of the system. At the same time, a variety of evaluation indicators are designed, such as accuracy, recall rate, F1 value, etc., to evaluate the competitive decision-making and collaborative learning performance of the system.

[0109] Those skilled in the art may understand this embodiment as a more specific description of Embodiment 1 and Embodiment 2.

[0110] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0111] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A multimodal information alignment application method, characterized in that: The method comprises the following steps: Step S1: Building a brain-like behavior decision-making system based on the CTM model, including multiple functional LTM, STM units and a two-way communication network, and building a unified feature information encoding between LTM modules to serve as the information basis for two-way consciousness communication between LTM and STM; Step S2: In the process of extracting brain language information from the LTM module, the attention mechanism is combined to consider the mutual influence between LTM modules; Step S3: Information alignment is learned through multimodal feedback, so that the information extracted from different LTM modules has a unified and aligned expression; Step S4: Combined with the distributed Hebbian-like learning algorithm, the frequency and correlation of information transmission between different LTM modules are calculated to adjust the weight of LTM competition to become STM; Step S5: The STM brain language information after the competition is successful will be further broadcast to the execution LTM module for translation and output of the decision result.

2. The multimodal information alignment application method according to claim 1, characterized in that: In step S1: The forms of LTM adopted include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module, and an action LTM module for execution. The constructed feature information encoding includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding.

3. The multimodal information alignment application method according to claim 1, characterized in that: In step S2: LTM adjusts its own brain language information encoding by combining the attention mechanism to align information between LTMs; The attention described therein includes: the LTM module realizes the consciousness change of the STM by publishing the information chunk Chunk to the STM; The LTM module receives information broadcast from the STM and forms a top-down attention, making it sensitive to the information to influence the range or area of ​​the encoder's attention to the input information, thereby affecting the output content of the perceptual encoder.

4. The multimodal information alignment application method according to claim 1, characterized in that: In step S3: Brain speech information alignment combines the LTM-STM dynamics of the brain-like behavior decision-making model. The perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives other LTM brain speech information encoding broadcast by STM. Each LTM compares the vector of its own output brain speech with the vector of other LTM brain speech received from STM to obtain the correspondence between the brain speech vectors output by different LTMs, and realizes brain speech alignment, and combines attention to achieve unified aligned brain speech information encoding.

5. The multimodal information alignment application method according to claim 1, characterized in that: In step S4: A separate Hebbian-like learning sleep expert algorithm runs in each LTM processor; the LTM module uses the Hebbian learning sleep expert algorithm to modify the strength of the set weights and define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations: if the content in the STM is correct, then p does nothing; if the content in the STM is wrong: if p is correct, then p increases the strength of the information block it generates; if p is wrong: p corrects its error; Then, p reduces the strength of the information block it produces; In the Hebbian learning sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised method is that the LTM module finds out that it has made an error by discovering the fact that the received STM information block is inconsistent with its own brain language information with confidence. Therefore, each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies its strength based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm is expanded to include a supervisory signal of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt.

6. The multimodal information alignment application method according to claim 1, characterized in that: In step S5: The STM brain language information will be further broadcast to the action execution LTM module to output the decision visualization results or output the natural semantic information encoded in the brain language through the semantic understanding LTM module to reflect explainable general artificial intelligence; the decision visualization includes converting the brain language feature information output by the STM into visual graphics to help users understand the decision logic of the system.

7. A multimodal information alignment application system, characterized in that: The system includes the following modules: Module M1: Building a brain-like behavior decision-making system based on the CTM model, including multiple functional LTM, STM units and a two-way communication network. A unified feature information encoding is built between LTM modules to serve as the information basis for two-way consciousness communication between LTM and STM. Module M2: In the process of extracting brain language information from the LTM module, the attention mechanism is combined to consider the mutual influence between LTM modules; Module M3: Information alignment is learned through multimodal feedback, and the information extracted from different LTM modules has a unified and aligned expression; Module M4: Combined with the distributed Hebbian-like learning algorithm, the frequency and correlation of information transmission between different LTM modules are calculated to adjust the weight of LTM competition to become STM; Module M5: After the competition is successful, the STM brain language information will be further broadcast to the execution LTM module for translation and output of the decision result.

8. The multimodal information alignment application system according to claim 7, characterized in that: In the module M1: The forms of LTM used include a perception LTM module for receiving sensor input, a storage LTM module for memory playback, a semantic understanding module for translating natural language, a decision-oriented emotion module, a navigation planning module, and an action LTM module for execution. The feature information encoding constructed includes semantic feature information encoding, time / space feature information encoding, or neural network information encoding; In the module M2: LTM adjusts its own brain language information encoding by combining the attention mechanism to align information between LTMs; The attention described includes: the LTM module publishes information chunks to the STM to achieve STM consciousness changes; the LTM module receives information broadcast from the STM to form top-down attention, making it sensitive to information to affect the encoder's focus range or area on the input information, thereby affecting the output content of the perceptual encoder; In the module M3: Brain speech information alignment combines the LTM-STM dynamics of the brain-like behavior decision model. The perception information encoding of different LTM modules forms brain speech information to participate in STM competition, and at the same time receives other LTM brain speech information encoding broadcast by STM. Each LTM learns the correspondence between the brain speech vectors output by different LTMs by comparing the vector of its own output brain speech with the vector of other LTM brain speech received from STM, and realizes brain speech alignment, and realizes unified and aligned brain speech information encoding by combining attention. In the module M4: A separate Hebbian learning sleep expert algorithm runs in each LTM processor; the LTM module uses the Hebbian learning sleep expert algorithm to modify the strength of the set weights and define the dynamic process of competition and feedback. In the sleep expert algorithm, the corresponding LTM function module p performs the following operations: if the content in the STM is correct, then p does nothing; if the content in the STM is wrong: if p is correct, then p increases the strength of the information block it generates; if p is wrong: p corrects its error; then, p reduces the strength of the information block it generates; In the Hebbian learning sleep expert algorithm, the LTM module determines whether it has made an error by combining unsupervised signals and supervised signals: the unsupervised system is that the LTM module finds out that it has made an error by discovering that the received STM information block contradicts its own brain language information with confidence, so each broadcast enables each LTM module to check and judge whether it has made an error, that is, the LTM module modifies its strength based on the feedback from the STM broadcast; at the same time, the feedback information learned in the supervised learning algorithm is expanded to include a supervisory signal of environmental feedback from the input mapping, and the weight information of the information block generated by itself is modified according to the external supervisory signal prompt; In the module M5: The STM brain language information will be further broadcast to the action execution LTM module to output the decision visualization results or output the natural semantic information encoded in the brain language through the semantic understanding LTM module to reflect explainable general artificial intelligence; the decision visualization includes converting the brain language feature information output by the STM into visual graphics to help users understand the decision logic of the system.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multimodal information alignment application method described in any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the multimodal information alignment application method described in any one of claims 1 to 6 are implemented.

Images