A thought chain reasoning method, device, equipment and storage medium

Abstract

The application discloses a kind of thought chain reasoning method, device, equipment and storage medium, it is applied to robot dialogue system in computer technology field, comprising: the original language text of user input obtained, picture information and target common knowledge are carried out feature extraction and obtain first text feature, visual feature and common knowledge feature;First input content determined based on first text feature, visual feature and common knowledge feature is input into large language model, and thought chain principle is obtained;Second text feature is obtained by extracting the feature of target language text determined according to thought chain principle and original language text;Second input content determined based on second text feature, visual feature and common knowledge feature is input into large language model, and the reasoning answer content of large language model reasoning output is obtained.The reasoning of the present application is carried out by introducing visual feature and common knowledge feature into thought chain, and the reasoning ability of large language model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to a thought chain reasoning method and device, equipment and a storage medium. BACKGROUND

[0002] Currently, in the robot dialogue scene, the existing robot dialogue system mainly applies natural language processing (NLP) technology and dialogue technology, and through the pre-training and fine-tuning mode, the understanding of the user input text and the ability to answer questions are realized. These robot dialogue systems usually use machine learning techniques such as neural networks, decision trees, etc. to train and optimize the language model. Through the application of these machine learning techniques, the existing robot dialogue system can understand, analyze and answer the questions raised by the user, so as to better provide investment suggestions for the user. At the same time, with the emergence of large language models such as GPT-3 series, the robot dialogue system has shown unprecedented potential in the dialogue scene. This kind of large language model can be trained through a large amount of corpus, so as to generate high-quality text content. Compared with traditional natural language processing technology, the quality of the text generated by the GPT-3 model has been significantly improved.

[0003] In order to improve the reasoning performance of the large language model, someone has proposed a CoT (Chain of Thought, thought chain) method, which represents a thought mode that can help large-scale language models solve complex arithmetic, common sense and character reasoning tasks. At present, it mainly includes two CoT methods. One is manual-CoT, which adds a series of thought chain description texts as prompt paradigms to the prompt (starting text), thereby significantly enhancing and stimulating the reasoning ability of the large language model. The other is zero-shot-CoT. In this method, only a simple inductive sentence such as "Let's think step by step" needs to be added to the prompt, which can significantly stimulate the reasoning ability of the large language model without constructing specific thought chain prompt examples.

[0004] However, in existing robot dialogue scenarios, only the user's input text is often considered, and other dimensions of information such as pictures, videos, and voice are less considered, which limits the comprehensive understanding and processing ability of the robot dialogue system to user demand, and cannot fully meet the user's demand. In addition, in the existing robot dialogue scenario, there is often a lack of common sense knowledge injection, which cannot perform logical reasoning and continuity reasoning of the thinking chain, which makes the robot dialogue system difficult to handle complex problems and answer multiple related questions, and cannot provide accurate and useful answers. Moreover, in existing CoT methods, only natural language text modal thinking chains are considered, and there are few methods of introducing visual features and common sense knowledge into thinking chain reasoning, which greatly limits the improvement of large language models in reasoning performance. Although there are thinking chain reasoning methods that combine image features, there are still many common sense errors in the reasoning process, which leads to errors in the final reasoning result.

[0005] In summary, the existing robot dialogue system often only relies on text input information when answering user questions, and cannot reason from multiple angles such as images and common sense knowledge, which limits the answering effect of the robot dialogue system on some complex problems and makes it difficult to meet user demand. Therefore, how to provide a solution to the above technical problems is a problem that needs to be solved by those skilled in the art at present. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a thinking chain reasoning method, device, equipment and storage medium, which can improve the reasoning ability of large language models and solve the problem of limited answering effect of the robot dialogue system on some complex problems. The specific scheme is as follows:

[0007] In a first aspect, the present application discloses a thinking chain reasoning method applied to a robot dialogue system, comprising:

[0008] Obtaining the original language text input by the user, and obtaining picture information and target common sense knowledge related to the original language text; the picture information includes a picture or a video;

[0009] Performing feature extraction on the original language text, the target language text, the picture information and the target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features;

[0010] Determining the first input content of the large language model based on the first text features, the visual features and the common sense knowledge features, and inputting the first input content into the large language model to obtain the thinking chain principle generated by the large language model;

[0011] determine a target language text according to the thought chain principle and the original language text, and perform feature extraction on the target language text to obtain a corresponding second text feature;

[0012] determine a second input content of the large language model based on the second text feature, the visual feature, and the common sense knowledge feature, input the second input content into the large language model to perform logical reasoning to obtain an inference answer content related to the original language text of the large language model inference output.

[0013] Optionally, the determining the first input content of the large language model based on the first text feature, the visual feature, and the common sense knowledge feature comprises:

[0014] determining the first text feature, the visual feature, and the common sense knowledge feature as the first input content of the large language model;

[0015] Or, determining a first attention weight between the first text feature and the visual feature, a second attention weight between the visual feature and the common sense knowledge feature, and a third attention weight between the common sense knowledge feature and the first text feature based on an attention mechanism;

[0016] determining a first associated feature between the first text feature and the visual feature, a second associated feature between the visual feature and the common sense knowledge feature, and a third associated feature between the common sense knowledge feature and the first text feature based on the first attention weight, the second attention weight, and the third attention weight;

[0017] performing feature fusion on the first associated feature, the second associated feature, and the third associated feature based on a preset fusion gate mechanism to obtain a corresponding first fused total feature, and determining the first fused total feature as the first input content of the large language model.

[0018] Optionally, the performing feature fusion on the first associated feature, the second associated feature, and the third associated feature based on the preset fusion gate mechanism to obtain the corresponding first fused total feature comprises:

[0019] calculating the first associated feature, the second associated feature, and the third associated feature using a preset nonlinear activation function to obtain a corresponding first gate parameter;

[0020] determining the corresponding first fused total feature based on the first gate parameter, the first associated feature, the second associated feature, and the third associated feature.

[0021] Optionally, the determining the target language text according to the thought chain principle and the original language text comprises:

[0022] performing a connection operation on the thought chain principle and the original language text to obtain a corresponding target language text.

[0023] Optionally, the determining the second input content of the large language model based on the second text feature, the visual feature and the common sense knowledge feature comprises:

[0024] determining the second text feature, the visual feature and the common sense knowledge feature as the second input content of the large language model;

[0025] or, determining new first attention weights between the second text feature and the visual feature, new second attention weights between the visual feature and the common sense knowledge feature, and new third attention weights between the common sense knowledge feature and the second text feature based on an attention mechanism;

[0026] determining new first associated features between the second text feature and the visual feature, new second associated features between the visual feature and the common sense knowledge feature, and new third associated features between the common sense knowledge feature and the second text feature based on the new first attention weights, the new second attention weights and the new third attention weights;

[0027] performing feature fusion on the new first associated features, the new second associated features and the new third associated features based on a preset fusion gate mechanism to obtain a corresponding second fusion total feature, and determining the second fusion total feature as the second input content of the large language model.

[0028] Optionally, the performing feature fusion on the new first associated features, the new second associated features and the new third associated features based on a preset fusion gate mechanism to obtain a corresponding second fusion total feature comprises:

[0029] calculating the new first associated features, the new second associated features and the new third associated features using a preset nonlinear activation function to obtain a corresponding second gate parameter;

[0030] determining a corresponding second fusion total feature based on the second gate parameter, the new first associated features, the new second associated features and the new third associated features.

[0031] Optionally, the performing feature extraction on the original language text, the target language text, the picture information and the target common sense knowledge to obtain a corresponding first text feature, a visual feature and a common sense knowledge feature comprises:

[0032] extracting features of the original language text by using a first preset feature extractor to obtain corresponding first text features;

[0033] extracting features of the picture information by using a second preset feature extractor to obtain corresponding visual features;

[0034] extracting features of the target common sense knowledge by using a third preset feature extractor to obtain corresponding common sense knowledge features;

[0035] Correspondingly, the extracting features of the target language text to obtain corresponding second text features comprises:

[0036] extracting features of the target language text by using the first preset feature extractor to obtain corresponding second text features.

[0037] In a second aspect, the present application discloses a thinking chain reasoning device applied to a robot dialogue system, comprising:

[0038] an acquisition module configured to acquire an original language text input by a user, and acquire picture information and target common sense knowledge related to the original language text; the picture information comprises a picture or a video;

[0039] a first feature extraction module configured to extract features of the original language text, target language text, picture information and target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features;

[0040] a first input content determination module configured to determine a first input content of a large language model based on the first text features, visual features and common sense knowledge features;

[0041] a thinking chain principle generation module configured to input the first input content into the large language model to obtain a thinking chain principle generated by the large language model;

[0042] a target language text determination module configured to determine a target language text according to the thinking chain principle and the original language text;

[0043] a second feature extraction module configured to extract features of the target language text to obtain corresponding second text features;

[0044] a second input content determination module configured to determine a second input content of the large language model based on the second text features, visual features and common sense knowledge features;

[0045] a thought chain reasoning module configured to input the second input content into the large language model to perform logical reasoning to obtain reasoning answer content related to the original language text from the large language model.

[0046] In a third aspect, the present application discloses an electronic device, comprising:

[0047] a memory configured to store a computer program;

[0048] a processor configured to execute the computer program to implement the steps of the thought chain reasoning method disclosed above.

[0049] In a fourth aspect, the present application discloses a computer readable storage medium configured to store a computer program; wherein the computer program is executed by a processor to implement the steps of the thought chain reasoning method disclosed above.

[0050] It can be seen that the present application provides a thought chain reasoning method applied to a robot dialogue system, comprising: obtaining original language text input by a user, and obtaining picture information and target common sense knowledge related to the original language text; the picture information comprises a picture or a video; performing feature extraction on the original language text, the target language text, the picture information and the target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features; determining first input content of a large language model based on the first text features, the visual features and the common sense knowledge features, and inputting the first input content into the large language model to obtain a thought chain principle generated by the large language model; determining a target language text according to the thought chain principle and the original language text, and performing feature extraction on the target language text to obtain corresponding second text features; determining second input content of the large language model based on the second text features, the visual features and the common sense knowledge features, and inputting the second input content into the large language model to perform logical reasoning to obtain reasoning answer content related to the original language text from the large language model. It can be seen that the present application considers introducing visual features and injecting common sense knowledge features to perform reasoning of the thought chain, that is, the robot dialogue system not only relies on language text when answering a user question, but also relies on picture information and common sense knowledge, that is, reasoning from multiple angles of language text, picture information and common sense knowledge can improve the reasoning ability of the large language model, solves the problem that the answering effect of the robot dialogue system on some complex questions is limited, and thus better serves the user. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0052] Figure 1 A flow chart of a thinking chain reasoning method is disclosed in the present application.

[0053] Figure 2 A thinking chain reasoning framework schematic diagram combined with picture information and common sense knowledge is disclosed in the present application.

[0054] Figure 3 A flow chart of a specific thinking chain reasoning method is disclosed in the present application.

[0055] Figure 4 A structure schematic diagram of a thinking chain reasoning device is disclosed in the present application.

[0056] Figure 5 An electronic device structure diagram is disclosed in the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only represent some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0058] At present, in the existing robot dialogue scene, the single-dimensional information input limits the comprehensive understanding and processing ability of the robot dialogue system to the user demand, and the lack of common sense knowledge input makes the robot dialogue system unable to reason from multiple angles such as images and common sense knowledge when processing complex problems and answering multiple related questions, resulting in that the robot dialogue system cannot provide accurate and useful answers and cannot fully meet the user's demand. Therefore, the present application provides a thinking chain reasoning scheme, which can improve the reasoning ability of large language models and solve the problem that the answering effect of the robot dialogue system on some complex problems is limited, so as to better serve the user.

[0059] The embodiment of the present application discloses a thinking chain reasoning method, as shown in Figure 1 The method is applied to a robot dialogue system and includes the following steps.

[0060] Step S11: Obtain the original language text input by the user, and obtain picture information and target common sense knowledge related to the original language text; the picture information includes a picture or a video.

[0061] In this embodiment, the user's questions about a certain field are obtained through user input, for example, the user's questions about the investment field are obtained, that is, in the dialogue scenario, the user can input text to the robot dialogue system to ask questions, for example, the user can input text to the robot dialogue system to ask questions related to the investment field, such as "How is the recent stock market trend?" and "How to invest in stocks?", etc. These questions will be used as input for subsequent steps.

[0062] In this embodiment, the picture information related to the original language text is obtained through picture information acquisition, that is, related pictures or videos are obtained. For example, in the dialogue scenario, the user can upload or provide related pictures or videos, or when the user mentions certain stocks or companies in the question, the robot dialogue system can automatically identify and search for related pictures as part of the large language model input.

[0063] In this embodiment, common sense knowledge can be obtained from different sources. Common sense knowledge represents the knowledge that people acquire in daily life about the world and things, including facts, laws, principles, and reasons, and is an important basis for reasoning, judgment, and decision-making. It should be noted that the sources of common sense knowledge can include but are not limited to Baidu Encyclopedia, Wikipedia, Internet search engines, knowledge graphs, etc., and the obtained common sense knowledge can be converted into some structured data forms such as knowledge graphs or common sense knowledge bases, that is, in the dialogue scenario, common sense knowledge can be stored and represented using structures similar to knowledge graphs or knowledge bases. When the user asks a question, the robot dialogue system can query the knowledge graph or other common sense knowledge base to extract relevant common sense knowledge, thereby better understanding the user's question and demand.

[0064] For example, common sense knowledge can be stored and expressed in a structured form such as a knowledge graph. KG (Knowledge Graph) can represent (E, R), that is, KG = (E, R), where E represents a set of entities, and R represents a set of relationships.

[0065] wherein G={(h, r, t) | h, t E, r R}, the relationship is composed of the head entity h and the tail entity t, that is, r=(h, t), and the entities and relationships in the knowledge graph can be represented by vectors, wherein the entity vector v e represents the characteristics of the entity e, and the relationship vector v r represents the characteristics of the relationship r, the knowledge graph representation is a way of representing and organizing various information and knowledge in the form of a graph so that the computer system can more conveniently and efficiently process and analyze, and then in the dialogue scene, the relevant entities and relationships can be retrieved through the query operation.

[0066] Step S12: feature extraction is performed on the original language text, the target language text, the picture information and the target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features.

[0067] In this embodiment, after obtaining the original language text input by the user and the picture information and target common sense knowledge related to the original language text, feature extraction is performed on the original language text, the target language text, the picture information and the target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features. It can be understood that the original language text, the target language text, the picture information and the target common sense knowledge are converted into a data format required by the robot dialogue system for subsequent processing. Specifically, the original language text is feature-extracted by using a first preset feature extractor to obtain corresponding first text features; the picture information is feature-extracted by using a second preset feature extractor to obtain corresponding visual features; and the target common sense knowledge is feature-extracted by using a third preset feature extractor to obtain corresponding common sense knowledge features.

[0068] For example, the original language text is vectorized and encoded by a language encoder such as a transformer to obtain first text features That is,

[0069]

[0070] That is, the hidden states of the last layer of the encoder of the transformer are used to represent wherein n represents the length of the original language text, d represents the dimension of the hidden layer, represents the original language text input by the user.

[0071] Similarly, the depth features can be extracted from the image by a visual feature extractor, and the depth features are determined as visual features, or the picture information such as the picture information is vectorized and encoded by a visual encoder to obtain visual features H vision That is,

[0072] H vision = VisionEncoder(X vision );

[0073] wherein, VisionEncoder() represents a visual feature obtained by vectorizing encoding the picture, and then through a learnable projection matrix W h , the shape of VisionEncoder(X vision ) is converted into the shape of H , X vision represents picture information.

[0074] It can be understood that the application of natural language processing technologies such as preprocessing and encoding of the original language text input by the user, and the application of visual processing technologies such as feature extraction and recognition of the picture information, the robot dialogue system can use computer vision technology to extract visual features, which can be information such as objects, scenes, emotions, and the like in pictures or videos, such as color, shape, texture, and semantics.

[0075] And the related common sense knowledge obtained is converted into common sense knowledge features H comsense corresponding to the text features and the visual features by using a third preset feature extractor, that is:

[0076] H comsense = ComSenseEncoder(X comsense );

[0077] wherein, X comsense represents related common sense knowledge under the corresponding semantics of the original language text.

[0078] Step S13: determining first input content of the large language model based on the first text features, the visual features, and the common sense knowledge features, and inputting the first input content into the large language model to obtain a thought chain principle generated by the large language model.

[0079] In this embodiment, after feature extraction of the original language text, the target language text, the picture information, and the target common sense knowledge to obtain corresponding first text features, visual features, and common sense knowledge features, the first input content of the large language model is determined based on the first text features, the visual features, and the common sense knowledge features, and the first input content is input into the large language model to obtain a thought chain principle generated by the large language model. It can be understood that the language text input by the user is analyzed and processed by using the large language model technology, and the common sense knowledge features and the visual features are fused to form a complete thought chain.

[0080] In a specific embodiment, the first text feature, the visual feature, and the common sense knowledge feature are determined as the first input content of the large language model. It can be understood that the first text feature, the visual feature, and the common sense knowledge feature are transmitted together as input features to the large language model to improve the quality and accuracy of the dialogue.

[0081] In another specific embodiment, a first attention weight between the first text feature and the visual feature, a second attention weight between the visual feature and the common sense knowledge feature, and a third attention weight between the common sense knowledge feature and the first text feature are determined based on an attention mechanism; first associated features between the first text feature and the visual feature, second associated features between the visual feature and the common sense knowledge feature, and third associated features between the common sense knowledge feature and the first text feature are determined based on the first attention weight, the second attention weight, and the third attention weight; and a corresponding first fused total feature is obtained by feature fusion of the first associated features, the second associated features, and the third associated features based on a preset fusion gating mechanism, and the first fused total feature is determined as the first input content of the large language model. It can be understood that the attention mechanism calculates the corresponding attention weights through direct interaction between the features, considers all possible interactions between the features, and not only the independent performance of the features, and then determines the associated features between the features based on the attention weights between the features, and then fuses the associated features between the features based on the fusion gating mechanism to obtain the fused total feature, which is transmitted to the large language model as the input feature to further improve the quality and accuracy of the dialogue.

[0082] For example, the attention weights between the first text feature, the visual feature, and the common sense knowledge feature are calculated using a Softmax function, i.e.,

[0083]

[0084]

[0085]

[0086] wherein a, b, g represent the first attention weight between the first text feature and the visual feature, the second attention weight between the visual feature and the common sense knowledge feature, and the third attention weight between the common sense knowledge feature and the first text feature, respectively, W α , W β , W γdenote weight matrices of the bilinear transformation corresponding to the text features, the visual features and the common sense knowledge features, respectively, denotes the transpose of the first text features, denotes the transpose of the visual features, denotes the transpose of the common sense knowledge features.

[0087] Then, the associated features between the features are determined based on the attention weights between the features through feature association interaction, that is:

[0088]

[0089]

[0090]

[0091] wherein, M lang-vision , M vision-comsense , M comsense-lang denote the first associated features between the first text features and the visual features, the second associated features between the visual features and the common sense knowledge features, and the third associated features between the common sense knowledge features and the first text features, respectively, denotes the multiplication between elements, denotes the matrix multiplication, W m1 , W m2 , W m3 denote the interaction weight matrices corresponding to the text features, the visual features and the common sense knowledge features, respectively.

[0092] In this embodiment, the feature fusion based on the preset fusion gating mechanism on the first associated features, the second associated features and the third associated features to obtain the corresponding first fused total features can specifically include: using a preset nonlinear activation function to calculate the first associated features, the second associated features and the third associated features to obtain the corresponding first gating parameters; based on the first gating parameters, the first associated features, the second associated features and the third associated features to determine the corresponding first fused total features. It can be understood that using a nonlinear activation function for feature fusion can enhance the expression ability of the large language model. For example, using the hyperbolic tangent activation function to calculate the first associated features, the second associated features and the third associated features to obtain the first gating parameters, that is:

[0093] G=tanh(W g [M lang-vision , M lang-vision , M comsense-lang ]+b g );

[0094] wherein, G denotes the first gating parameter, G can determine which features should be more fused, Wg denotes a weight matrix, b g denotes a bias term, tanh denotes a hyperbolic tangent activation function.

[0095]

[0096] where F fusion denotes the first fused total feature, ° denotes element-wise multiplication.

[0097] Therefore, in this embodiment, the attention mechanism and the gating fusion can determine the effective interaction and fusion between different features.

[0098] Step S14: determining a target language text according to the thought chain principle and the original language text, and performing feature extraction on the target language text to obtain corresponding second text features.

[0099] In this embodiment, after obtaining the thought chain principle generated by the large language model by inputting the first input content into the large language model, the target language text is determined according to the thought chain principle and the original language text. Specifically, a connection operation is performed on the thought chain principle and the original language text to obtain a corresponding target language text. For example, the original language text is concatenated with R to obtain i.e., the target language text is: wherein denotes a concat operation.

[0100] In this embodiment, after determining the target language text according to the thought chain principle and the original language text, the second text features are extracted from the target language text. The first preset feature extractor is used to extract the second text features from the target language text. For example, a language encoder such as transformer is used to encode the target language text to obtain the second text features i.e.,

[0101]

[0102] Step S15: determining the second input content of the large language model based on the second text features, the visual features, and the common sense knowledge features, and inputting the second input content into the large language model to perform logical reasoning to obtain the reasoning answer content related to the original language text of the large language model reasoning output.

[0103] In this embodiment, according to the thought chain principle and the original language text, the target language text is determined, and after feature extraction of the target language text, the corresponding second text feature is obtained, the second input content of the large language model is determined based on the second text feature, the visual feature and the common sense knowledge feature, and the second input content is input into the large language model to perform logical reasoning to obtain the reasoning answer content related to the original language text of the large language model reasoning output. It can be understood that the final output of the large language model is the answer to the user's question and the display of related explanations and pictures after logical reasoning by injecting visual features and common sense knowledge features, that is, the generated related text description is combined with the extracted visual features and entity information to perform logical reasoning and answer to the user's question, and related explanations and pictures are displayed, that is, the robot dialogue system can automatically add related pictures when answering questions to better explain and display.

[0104] In a specific embodiment, the second text feature, the visual feature and the common sense knowledge feature are determined as the second input content of the large language model. It can be understood that the second text feature, the visual feature and the common sense knowledge feature are input features together to the large language model for logical reasoning to improve the quality and accuracy of the dialogue.

[0105] In another specific embodiment, based on the attention mechanism, a new first attention weight between the second text feature and the visual feature, a new second attention weight between the visual feature and the common sense knowledge feature, and a new third attention weight between the common sense knowledge feature and the second text feature are determined; based on the new first attention weight, the new second attention weight and the new third attention weight, a new first associated feature between the second text feature and the visual feature, a new second associated feature between the visual feature and the common sense knowledge feature, and a new third associated feature between the common sense knowledge feature and the second text feature are determined; based on a preset fusion gate mechanism, the new first associated feature, the new second associated feature and the new third associated feature are fused to obtain a corresponding second fusion total feature, and the second fusion total feature is determined as the second input content of the large language model. It can be understood that the second text feature, the visual feature and the common sense knowledge feature are associated by using the attention network, then the associated features are fused with the second text feature to obtain a corresponding second fusion feature, and the second fusion feature is input to the large language model as an input feature to further improve the quality and accuracy of the dialogue.

[0106] For example, the second text feature After that, the attention weights between the second text feature, the visual feature and the common sense knowledge feature are calculated by using the Softmax function, that is:

[0107]

[0108]

[0109]

[0110] wherein, α', β', γ' represent the first attention weight between the second text feature and the visual feature, the second attention weight between the visual feature and the common sense knowledge feature, and the third attention weight between the common sense knowledge feature and the second text feature, respectively, W α , W β , W γ represent the weight matrix of the bilinear transformation corresponding to the text feature, the visual feature and the common sense knowledge feature, respectively, represents the transpose of the second text feature, represents the transpose of the visual feature, represents the transpose of the common sense knowledge feature.

[0111] Then, the associated features between the features are determined based on the attention weights between the features to determine the associated information between the features, that is:

[0112]

[0113]

[0114]

[0115] wherein, M' lang-vision , M' vision-comsense , M' comsense-lang represent the first associated feature between the second text feature and the visual feature, the second associated feature between the visual feature and the common sense knowledge feature, and the third associated feature between the common sense knowledge feature and the second text feature, respectively, represents the multiplication between elements, represents the matrix multiplication, W m1 , W m2 , W m3 represent the interaction weight matrix corresponding to the text feature, the visual feature and the common sense knowledge feature, respectively.

[0116] In this embodiment, the step of fusing the new first associated feature, the new second associated feature, and the new third associated feature based on a preset fusion gating mechanism to obtain the corresponding second fused total feature may specifically include: calculating the corresponding second gating parameters using a preset nonlinear activation function on the new first associated feature, the new second associated feature, and the new third associated feature; and determining the corresponding second fused total feature based on the second gating parameters, the new first associated feature, the new second associated feature, and the new third associated feature. For example, the second gating parameters can be calculated using a hyperbolic tangent activation function on the new first associated feature, the new second associated feature, and the new third associated feature, i.e.:

[0117] G′=tanh(W g [M′ lang-vision M′ lang-vision M′ comsense-lang ]+b g );

[0118] Where G′ represents the second gating parameter, G′ determines which features should be fused more, W g Let b represent the weight matrix. g denoted as the bias term, and tanh is the hyperbolic tangent activation function.

[0119]

[0120] Among them, F′ fusion It is the overall characteristic after the second fusion. This indicates element-wise multiplication.

[0121] As can be seen, in this embodiment of the invention, by considering the introduction of visual features and the injection of common sense knowledge features for reasoning of thought chains, the robot dialogue system, when answering user questions, not only relies on language text, but also on visual information and common sense knowledge. That is, reasoning from multiple perspectives, including language text, visual information, and common sense knowledge, can improve the reasoning ability of the large language model, solve the problem of the limited answering effect of the robot dialogue system on some complex questions, and thus better serve users.

[0122] For example, see Figure 2As shown, the thought chain reasoning combined with visual features and common sense knowledge can be divided into two stages, that is, the first stage is a thought chain generation stage, in the first stage, by utilizing language text, picture information and common sense knowledge, inputting into a large-scale language model, corresponding thought chain rationales are generated;The second stage is that the generated thought chain rationals are inputted into the large-scale language model again combined with the original language text, picture information and common sense knowledge, and corresponding answers are reasoned.

[0123] For example, in the first stage, that is, the reasoning chain generation stage, the data inputted into the model is

[0124]

[0125] Among them, X represents the original language text inputted by the user in the first stage, such as the user's question on a certain field, etc. vision X represents picture information such as pictures, etc. comsense X represents the relevant common sense knowledge under the corresponding semantics of the original language text, and the main target of the first stage is to generate thought chain rationals R=F(X), wherein R represents the generated thought chain rationals.

[0126] In the second stage, the original language text is concatenated with R to obtain That is, the target language text is:

[0127]

[0128] Among them, represents a concat operation.

[0129] Then, the is inputted into the large language model to obtain the final answer A.

[0130] It should be pointed out that, given the language input visual features X vision and common sense knowledge X comsense , the probability of the generated target thought chain text with a length of N can be calculated as:

[0131]

[0132] The application also discloses an embodiment applied to a specific investment advisor scene, see Figure 3As shown, the user can input a question to the robot dialogue system in the form of text, that is, the prompt is: How is the trend of A company today? The robot dialogue system obtains pictures and common sense knowledge related to the prompt, such as common technical indicators of the stock market, macroeconomic data, and industry trends, etc. Based on this information, the large language model generates the corresponding thought chain principle, that is: According to the stock trend chart today, it can be seen that other stock prices as a whole show an upward trend. First, from the macroeconomic perspective, recent macroeconomic data shows that the domestic economy as a whole maintains a steady growth trend, and the employment situation is stable and improving, which injects certain information into the market. In addition, in the securities information service industry, A company has always been in a leading position, and A company has continuously introduced new technologies and new models, enriching its product line and improving user stickiness. Then the large language model generates the corresponding answer based on the thought chain principle, related pictures and common sense knowledge after logical reasoning, that is: A company performed well today with a gain of about 13%. That is, by combining user input text, images and other multi-dimensional information, as well as common sense knowledge, the user's investment demand is reasoned, a thought chain related to the user's investment demand is established, and through continuous iteration and reasoning, the user's precise investment demand is obtained, that is, through reasoning and analysis of the user's multi-dimensional information, more personalized and precise investment suggestions and decision support can be provided to the user to help the user make better investment decisions, and the user's investment demand is accurately reasoned, thereby better meeting the user's investment demand.

[0133] Correspondingly, the embodiment of the present application also discloses a thought chain reasoning device applied to a robot dialogue system, as shown in Figure 4 The device comprises:

[0134] The acquisition module 11 is configured to acquire the original language text input by the user and acquire picture information and target common sense knowledge related to the original language text; the picture information comprises pictures or videos;

[0135] The first feature extraction module 12 is configured to perform feature extraction on the original language text, the target language text, the picture information and the target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features;

[0136] The first input content determination module 13 is configured to determine the first input content of the large language model based on the first text features, the visual features and the common sense knowledge features;

[0137] The thought chain principle generation module 14 is configured to input the first input content into the large language model to obtain the thought chain principle generated by the large language model;

[0138] A target language text determination module 15 is configured to determine a target language text according to the thought chain principle and the original language text;

[0139] A second feature extraction module 16 is configured to perform feature extraction on the target language text to obtain corresponding second text features;

[0140] A second input content determination module 17 is configured to determine second input content of the large language model based on the second text features, the visual features and the common sense knowledge features;

[0141] A thought chain reasoning module 18 is configured to input the second input content into the large language model to perform logical reasoning to obtain reasoning answer content related to the original language text in the large language model reasoning output.

[0142] As can be seen from the above, in the embodiment of the application, the visual features are introduced and the common sense knowledge features are injected for reasoning of the thought chain, that is, the robot dialogue system not only depends on the language text when answering the user's question, but also depends on the picture information and the common sense knowledge, that is, reasoning from multiple angles of the language text, the picture information and the common sense knowledge can improve the reasoning ability of the large language model, and solve the problem that the answering effect of the robot dialogue system on some complex problems is limited, thereby better serving the user.

[0143] In some specific embodiments, the first input content determination module 13 can specifically include:

[0144] A first determination unit is configured to determine the first text features, the visual features and the common sense knowledge features as first input content of the large language model;

[0145] Or, a first attention weight calculation unit is configured to determine, based on an attention mechanism, first attention weights between the first text features and the visual features, second attention weights between the visual features and the common sense knowledge features, and third attention weights between the common sense knowledge features and the first text features;

[0146] A first feature association unit is configured to determine, based on the first attention weights, the second attention weights and the third attention weights, first association features between the first text features and the visual features, second association features between the visual features and the common sense knowledge features, and third association features between the common sense knowledge features and the first text features;

[0147] The first feature fusion unit is configured to perform feature fusion on the first associated feature, the second associated feature and the third associated feature based on a preset fusion gating mechanism to obtain a corresponding first fused total feature, and determine the first fused total feature as the first input content of the large language model.

[0148] In some specific embodiments, the first feature fusion unit can specifically include:

[0149] The first calculation sub-unit is configured to calculate the first associated feature, the second associated feature and the third associated feature by using a preset nonlinear activation function to obtain a corresponding first gating parameter;

[0150] The first feature fusion sub-unit is configured to determine a corresponding first fused total feature based on the first gating parameter, the first associated feature, the second associated feature and the third associated feature.

[0151] In some specific embodiments, the target language text determination module 15 can specifically include:

[0152] The connection unit is configured to perform a connection operation on the thought chain principle and the original language text to obtain a corresponding target language text.

[0153] In some specific embodiments, the second input content determination module 17 can specifically include:

[0154] The second determination unit is configured to determine the second text feature, the visual feature and the common sense knowledge feature as the second input content of the large language model.

[0155] Or, the second attention weight calculation unit is configured to determine, based on an attention mechanism, a new first attention weight between the second text feature and the visual feature, a new second attention weight between the visual feature and the common sense knowledge feature, and a new third attention weight between the common sense knowledge feature and the second text feature.

[0156] The second feature association unit is configured to determine, based on the new first attention weight, the new second attention weight and the new third attention weight, a new first associated feature between the second text feature and the visual feature, a new second associated feature between the visual feature and the common sense knowledge feature, and a new third associated feature between the common sense knowledge feature and the second text feature.

[0157] The second feature fusion unit is configured to perform feature fusion on the new first associated feature, the new second associated feature, and the new third associated feature based on a preset fusion gating mechanism to obtain a corresponding second fused total feature, and determine the second fused total feature as a second input content of the large language model.

[0158] In some specific embodiments, the second feature fusion unit can specifically include:

[0159] The second calculation subunit is configured to calculate the new first associated feature, the new second associated feature, and the new third associated feature by using a preset nonlinear activation function to obtain a corresponding second gating parameter;

[0160] The second feature fusion subunit is configured to determine a corresponding second fused total feature based on the second gating parameter, the new first associated feature, the new second associated feature, and the new third associated feature.

[0161] In some specific embodiments, the first feature extraction module 12 can specifically include:

[0162] The first text feature extraction unit is configured to perform feature extraction on the original language text by using a first preset feature extractor to obtain a corresponding first text feature;

[0163] The visual feature extraction unit is configured to perform feature extraction on the picture information by using a second preset feature extractor to obtain a corresponding visual feature;

[0164] The common sense feature extraction unit is configured to perform feature extraction on the target common sense knowledge by using a third preset feature extractor to obtain a corresponding common sense knowledge feature;

[0165] In some specific embodiments, the second feature extraction module 16 can specifically include:

[0166] The second text feature extraction unit is configured to perform feature extraction on the target language text by using the first preset feature extractor to obtain a corresponding second text feature.

[0167] Further, the embodiments of the present application also provide an electronic device. Figure 5 The electronic device 20 shown in the figure is according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0168] Figure 5A structural schematic diagram of an electronic device 20 is provided in the embodiment of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the thought chain reasoning method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in the embodiment can be specifically an electronic computer.

[0169] In the embodiment, the power supply 23 is configured to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 is capable of creating a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not specifically limited here; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not specifically limited here.

[0170] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0171] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the thought chain reasoning method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0172] Further, the embodiment of the present application further discloses a computer readable storage medium, and the computer readable storage medium stores a computer program. When the computer program is loaded and executed by a processor, the steps of the thought chain reasoning method disclosed in any of the foregoing embodiments are implemented.

[0173] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0174] Finally, it needs to be pointed out that in this article, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0175] The above describes in detail the thought chain reasoning method, device, equipment and storage medium provided by the present application. The principles and implementation manners of the present application are described by applying specific examples. The above example is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method of mind mapping reasoning, characterized by, The application is applied to a robot dialogue system, comprising: obtaining original language text input by a user, and obtaining picture information and target common sense knowledge related to the original language text; the picture information comprises a picture or a video; performing feature extraction on the original language text, the picture information and the target common sense knowledge to obtain corresponding first text features, visual features and common sense knowledge features; determining first input content of a large language model based on the first text features, the visual features and the common sense knowledge features, and inputting the first input content into the large language model to obtain a thinking chain principle generated by the large language model; determining target language text according to the thinking chain principle and the original language text, and performing feature extraction on the target language text to obtain corresponding second text features; determining second input content of the large language model based on the second text features, the visual features and the common sense knowledge features, and inputting the second input content into the large language model to perform logical reasoning to obtain reasoning answer content related to the original language text output by the large language model.

2. The method of claim 1, wherein, The method comprises the following steps: determining the first text features, the visual features and the common sense knowledge features as the first input content of the large language model; or, determining first attention weights between the first text features and the visual features, second attention weights between the visual features and the common sense knowledge features, and third attention weights between the common sense knowledge features and the first text features based on an attention mechanism; determining first associated features between the first text features and the visual features, second associated features between the visual features and the common sense knowledge features, and third associated features between the common sense knowledge features and the first text features based on the first attention weights, the second attention weights and the third attention weights; performing feature fusion on the first associated features, the second associated features and the third associated features based on a preset fusion gate mechanism to obtain corresponding first total features after fusion, and determining the first total features after fusion as the first input content of the large language model.

3. The method of claim 2, wherein, The method comprises the following steps: calculating the first associated features, the second associated features and the third associated features by using a preset nonlinear activation function to obtain corresponding first gate parameters; determining the corresponding first total features after fusion based on the first gate parameters, the first associated features, the second associated features and the third associated features.

4. The method of claim 1, wherein, The method comprises the following steps: performing a connection operation on the thinking chain principle and the original language text to obtain the corresponding target language text.

5. The method of claim 1, wherein, The second input content of the large language model is determined based on the second text feature, the visual feature, and the common sense knowledge feature, including: The second text feature, the visual feature, and the common sense knowledge feature are determined as the second input content of the large language model. Or, based on an attention mechanism, new first attention weights between the second text feature and the visual feature, new second attention weights between the visual feature and the common sense knowledge feature, and new third attention weights between the common sense knowledge feature and the second text feature are determined. Based on the new first attention weights, the new second attention weights, and the new third attention weights, new first associated features between the second text feature and the visual feature, new second associated features between the visual feature and the common sense knowledge feature, and new third associated features between the common sense knowledge feature and the second text feature are determined. Based on a preset fusion gate mechanism, the new first associated features, the new second associated features, and the new third associated features are fused to obtain corresponding second fused total features, and the second fused total features are determined as the second input content of the large language model.

6. The method of claim 5, wherein, The second input content of the large language model is determined based on the second text feature, the visual feature, and the common sense knowledge feature, including: The new first associated features, the new second associated features, and the new third associated features are calculated using a preset nonlinear activation function to obtain corresponding second gate parameters. Based on the second gate parameters, the new first associated features, the new second associated features, and the new third associated features, corresponding second fused total features are determined.

7. The method according to any one of claims 1 to 6, wherein, The first text feature, the visual feature, and the common sense knowledge feature are obtained by feature extraction on the original language text, the picture information, and the target common sense knowledge, including: The first text feature is obtained by feature extraction on the original language text using a first preset feature extractor. The visual feature is obtained by feature extraction on the picture information using a second preset feature extractor. The common sense knowledge feature is obtained by feature extraction on the target common sense knowledge using a third preset feature extractor. Correspondingly, the second text feature is obtained by feature extraction on the target language text, including: The second text feature is obtained by feature extraction on the target language text using the first preset feature extractor.

8. A thought chain reasoning apparatus characterized by comprising: The application is applied to a robot dialogue system, including: An acquisition module is configured to acquire an original language text input by a user, and acquire picture information and target common sense knowledge related to the original language text; the picture information includes a picture or a video. A first feature extraction module is configured to perform feature extraction on the original language text, the picture information, and the target common sense knowledge to obtain a corresponding first text feature, a visual feature, and a common sense knowledge feature. a first input content determination module configured to determine first input content of a large language model based on the first text feature, the visual feature, and the common sense knowledge feature; a thought chain principle generation module configured to input the first input content into the large language model to obtain a thought chain principle generated by the large language model; a target language text determination module configured to determine a target language text according to the thought chain principle and the original language text; a second feature extraction module configured to perform feature extraction on the target language text to obtain corresponding second text features; a second input content determination module configured to determine second input content of the large language model based on the second text features, the visual feature, and the common sense knowledge feature; a thought chain reasoning module configured to input the second input content into the large language model to perform logical reasoning and obtain reasoning answer content related to the original language text output by the large language model.

9. An electronic device, comprising: comprising: a memory configured to save a computer program; a processor configured to execute the computer program to implement the steps of the thought chain reasoning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, for storing a computer program; wherein the computer program is executed by a processor to implement the steps of the thought chain reasoning method according to any one of claims 1 to 7.

Images