A multimodal data dynamic reasoning system and method based on cognitive graph

Abstract

The present invention discloses a multimodal data dynamic reasoning system and method based on cognitive graphs, which relates to the technical field of graph data management. The present invention extracts text, voice, and image data input by users, analyzes the user data, and generates structured interaction records by combining user identification and timestamps; analyzes historical interaction records, detects continuous query events of similar questions by the same user, aggregates negative feedback signals, analyzes the repetition frequency of repeated events among all users, and analyzes and outputs the feedback intensity coefficient of repeated behavior; locates corresponding nodes in the cognitive graph according to the characteristics of repeated behavior, analyzes the attenuation coefficient, activates the associated alternative reasoning paths, and outputs the node set to be updated and the candidate path; adjusts the node weight based on the attenuation coefficient, updates the weight according to the semantic relevance of the candidate path, updates the graph connection relationship, and outputs the optimized cognitive graph. It identifies abnormal situations, adjusts the knowledge graph node weights, and improves the accuracy of analysis.

Description

Technical Field

[0001] The present invention relates to the field of graph data management technology, and specifically to a multimodal data dynamic reasoning system and method based on cognitive graphs. Background Art

[0002] With the continuous development of artificial intelligence technology, knowledge graphs, as the knowledge pillar of the AI ​​field, have attracted widespread attention from academia and industry for their powerful knowledge representation and reasoning capabilities. In recent years, multimodal knowledge graphs have been widely used in fields such as semantic search, question answering, and knowledge management.

[0003] Traditional systems are poorly equipped to handle repeated user questions or negative feedback. For one thing, the system determines question repetitiveness based solely on single conversation records, failing to incorporate the spatiotemporal correlations of historical interactions and common patterns across users. This makes it difficult to distinguish between occasional errors and systemic flaws. Furthermore, the node weights of cognitive graphs rely on manual configuration and lack automated feedback-driven mechanisms, which can lead to inefficient reasoning.

[0004] Therefore, the present invention discloses a multimodal data dynamic reasoning system and method based on cognitive graph to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal data dynamic reasoning system and method based on cognitive graphs to solve the problems raised in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal data dynamic reasoning method based on cognitive graph, the method comprising the following steps:

[0007] S1: Extracts text, voice, and image data input by the user, analyzes the user data, and combines it with the user ID and timestamp to generate a structured interaction record;

[0008] S2: Analyze historical interaction records, detect consecutive query events of the same user for similar questions, collect negative feedback signals, analyze the repetition frequency of repeated events among all users, and analyze and output the feedback intensity coefficient of repeated behavior;

[0009] S3: Locate the corresponding nodes in the cognitive map based on the characteristics of repeated behaviors, analyze the attenuation coefficient, activate the associated alternative reasoning paths, and output the set of nodes to be updated and the candidate paths;

[0010] S4: Adjust node weights based on the decay coefficient, update weights according to the semantic relevance of candidate paths, update graph connectivity, and output the optimized cognitive graph.

[0011] According to the above scheme, S1 includes the following contents:

[0012] S101: Extract the voice stream, image data, and question text collected by the user terminal; based on the pre-trained end-to-end model, convert the voice stream into intermediate text and output the voice recognition confidence Ca; perform detection on the image data based on the preset object detection network to obtain the entity set E = {ei|i∈[1, I]}, where I represents the total number of entities; extract the visual feature vector of each entity, and record the visual feature vector of entity ei as vi;

[0013] S102: Extract the intermediate text converted from the speech stream to generate a speech-text vector q, analyze the feature similarity between each visual feature vector and the text vector, and record the feature similarity between the visual feature vector vᵢ and the speech-text vector q as sim(vᵢ, q), where sim() represents the cosine similarity function. Analyze the attention weight based on the adjustment coefficient τ, where the adjustment coefficient is equal to the product of the basic adjustment value and the speech recognition confidence deviation value. The speech recognition confidence deviation value is equal to the difference between the speech recognition confidence reference value and the speech recognition confidence. The speech recognition confidence reference value is a system preset constant. Record the attention weight of the entity ei as αi, and integrate the attention weights of each entity to generate an attention distribution vector, which is recorded as α=[α1, α2, …, αI]: ;

[0014] Where exp() represents the exponential function with natural numbers as the base; the entity e i The visual feature vector v i and entity category embedding vector embed (e i ) cascade, input two hidden layers, ReLU activated multi-layer perceptron to analyze the fusion feature vector, denoted as fᵢ;

[0015] Among them, in the first hidden layer of the multilayer perceptron, h i （1） =ReLU(W （1） [v i ;embed(e i )]+b （1） ); where h i （1） Represents entity e i The hidden state of the first layer, W （1） represents the weight matrix of the first layer, b （1） represents the bias vector of the first layer;

[0016] In the second hidden layer, fᵢ=W （2） h i （1） +b （2） ;W （2） represents the weight matrix of the second layer, b （2）represents the bias vector of the second layer;

[0017] Combine the text vector, attention weight and fusion feature vector to analyze the joint intent vector, denoted as Q: ;

[0018] S103: Extract the device's current timestamp and time compensation, where the time compensation is equal to the difference between the current timestamp and the last sampling timestamp, and generate a spatiotemporal code st; based on the user's unique identifier u, the joint intention vector, the question text, the spatiotemporal code, and the attention distribution fingerprint hash (A), generate an integrated interaction record R n ={u,Q,T a ,st,hash(A)}, write the interaction records into the distributed history database.

[0019] This application calculates the similarity between the visual feature vector and the speech text vector, combined with a confidence-based dynamic adjustment coefficient, which significantly improves the robustness of low-confidence speech scenarios; attention weights enable the model to focus on visual entities that are strongly related to the speech content and suppress interference from irrelevant entities; the joint intent vector fuses text, attention weights, and multimodal features, combined with timestamp-compensated encoding and attention fingerprints to ensure that the interaction record contains spatiotemporal context, providing a complete data foundation for subsequent reasoning.

[0020] According to the above scheme, S2 includes the following contents:

[0021] S201: Retrieving a historical question text set based on the user, applying the BERT-Base model to each historical question text to generate a historical question text vector, and using the cosine similarity function to analyze the similarity between any two historical question texts of the same user; if the similarity between the two historical question texts is greater than a first similarity threshold, and the corresponding timestamp difference is less than a preset time interval threshold, marking the two historical question texts as duplicate events;

[0022] S202: Extract user actions after repeated events are triggered, and identify the "re-ask" or "negative" keywords that trigger negative feedback; when negative feedback is detected, extract repeated events of all users, analyze the similarity of any two repeated events, and if the similarity of the two repeated events is greater than a second similarity threshold, mark them as similar repeated events, and count the total number of the same similar repeated events, recorded as M; analyze the feedback strength coefficient β based on the total number of the same similar repeated events, β=β1+log(M), where β1 is a system preset constant.

[0023] This application uses dual thresholds to identify repeated events and avoid single misjudgments; triggers negative feedback marking, aggregates similar repeated events across users, and calculates feedback strength using a logarithmic function.

[0024] According to the above solution, S3 includes the following:

[0025] S301: Structural and semantic encoding is performed on each node in the cognitive map, and a graph embedding algorithm is used to generate a node vector library. The cosine similarity function is used to analyze the similarity between the question text vector of the current repeated event and the vectors of each node, and the node with the highest similarity is marked as the main node.

[0026] S302: Analyze the time interval Δt between the current timestamp and the last repeated event trigger timestamp, and analyze the attenuation coefficient ΔD based on the feedback strength coefficient and the time interval, where ΔD = β2 × β × exp(-λΔt); where β2 represents the basic attenuation rate and λ represents the time attenuation factor; the basic attenuation rate and the time attenuation factor are system preset constants;

[0027] S303: Based on the main node, search the cognitive map for a set of g-hop neighbor nodes directly connected to the main node and obtain the corresponding edge weights, where g is a system preset constant; consider each path from the main node to the neighbor node as a candidate path, extract the similarity between the node vector of the end node of each candidate path and the question text vector of the current repeated event, and screen out candidate paths with a similarity greater than a third similarity threshold; calculate the sum of the edge weights of each screened candidate path and record it as the candidate path weight, and select a preset number of candidate paths from the largest to the smallest according to the candidate path weights to generate a final candidate path set, which is recorded as P={P r |r∈[1,R]}, where P r represents the rth final candidate path, and R represents the total number of final candidate paths in the final candidate path set.

[0028] This application quickly locates the main node most relevant to the current problem. The attenuation coefficient integrates feedback strength and time attenuation to implement dynamic adjustment logic of "the more frequent and recent errors, the greater the impact on the graph"; triple filtering ensures path quality, avoids graph search explosion, and focuses on high-value reasoning paths.

[0029] According to the above scheme, S4 includes the following contents:

[0030] S401: Update the weight of the master node based on the decay coefficient: W z *=max(W z -△D×γ,W min ), where γ represents a learning parameter, which is preset based on the node activity, and max() represents the maximum value function; W z Represents the original weight of the master node; W min Indicates the minimum preset weight;

[0031] S402: Extract the similarity between the endpoint node vector of the final candidate path and the question text vector of the current repeated event, analyze the weight increment based on the similarity between the endpoint node vector of the final candidate path and the question text vector of the current repeated event, and record the weight increment of the rth final candidate path as △W r =σ(△D×S r / S sum ); where σ represents the Sigmoid function, S r S represents the similarity between the terminal node vector of the rth final candidate path and the question text vector of the current repeated event; sum Represents the sum of the similarities between the terminal node vector of each final candidate path in the final candidate path set and the question text vector of the current repeated event;

[0032] The weight of each final candidate path is updated, and the updated weight is equal to the sum of the weight of the original final candidate path and the weight increment;

[0033] S403: Extract the newly generated answer and the original answer after the weight update, generate semantic vectors for the newly generated answer and the original answer, and analyze the feature distance between the semantic vectors of the newly generated answer and the original answer; the feature distance adopts Euclidean distance; if the feature distance is less than or equal to the feature distance threshold, automatically confirm that the update is valid; if the feature distance is greater than the feature distance threshold, trigger the manual review process; the feature distance threshold is set based on the historical mean and standard deviation of the feature distance.

[0034] The master node weight update formula of this application is adjusted differently according to the node activity, and the weight is prevented from returning to zero, causing the node to "fail". The candidate paths are incrementally allocated, and paths with high similarity receive more weight, constraining the incremental range to avoid mutations; and a balance is achieved between automated updates and system security.

[0035] Another aspect of the present application provides a multimodal data dynamic reasoning system based on cognitive graphs, which is applied to the above-mentioned multimodal data dynamic reasoning method based on cognitive graphs. The system includes an interaction record generation module, a repetitive behavior feature analysis unit, a knowledge node candidate module, and a dynamic weight update module.

[0036] The interaction record generation module is used to extract text, voice and image data input by the user, analyze the user data and generate a structured interaction record by combining the user identifier and timestamp;

[0037] The repeated behavior feature analysis unit is used to analyze historical interaction records, detect continuous query events of the same user for similar questions, collect negative feedback signals, analyze the repetition frequency of repeated events among all users, and analyze and output the feedback intensity coefficient of repeated behavior;

[0038] The knowledge node candidate module is used to locate the corresponding node in the cognitive map according to the repetitive behavior characteristics, analyze the attenuation coefficient, activate the associated alternative reasoning path, and output the node set to be updated and the candidate path;

[0039] The dynamic weight updating module is used to adjust the node weights based on the attenuation coefficient, update the weights according to the semantic relevance of the candidate paths, update the graph connection relationship, and output the optimized cognitive graph.

[0040] According to the above solution, the interaction record generation module includes a data acquisition and conversion unit and a data fusion unit;

[0041] The data acquisition and conversion unit is used to extract the voice stream, image data and question text collected by the user terminal; based on the pre-trained end-to-end model, convert the voice stream into intermediate text and output the voice recognition confidence; based on the preset object detection network, perform detection on the image data to obtain an entity set; and extract the visual feature vector of each entity;

[0042] The data fusion unit is used to analyze the feature similarity between each visual feature vector and the text vector, analyze the attention weight based on the adjustment coefficient, and analyze the joint intention vector by combining the text vector, attention weight and fused feature vector; extract the current timestamp and time compensation of the device to generate a spatiotemporal code; based on the user's unique identifier, joint intention vector, question text, spatiotemporal code and attention distribution fingerprint, generate an integrated interaction record.

[0043] According to the above solution, the repetitive behavior feature analysis unit includes a repetitive event marking unit and a frequency analysis unit;

[0044] The repeated event marking unit is used to retrieve a historical question text set according to the user, generate a historical question text vector for each historical question text, and use a cosine similarity function to analyze the similarity between any two historical question texts of the same user; if the similarity between the two historical question texts is greater than a first similarity threshold, and the corresponding timestamp difference is less than a preset time interval threshold, the two historical question texts are marked as repeated events;

[0045] The frequency analysis unit is used to extract user operations after repeated events are triggered, and identify "re-ask" or "negative" keywords that trigger negative feedback; when negative feedback is detected, the repeated events of all users are extracted, and the similarity of any two repeated events is analyzed. If the similarity of the two repeated events is greater than a second similarity threshold, they are marked as similar repeated events, and the total number of the same similar repeated events is counted; and the feedback strength coefficient is analyzed based on the total number of the same similar repeated events.

[0046] According to the above solution, the knowledge node candidate module includes a main node analysis unit, a decay coefficient analysis unit and a candidate path analysis unit;

[0047] The main node analysis unit is used to perform structural and semantic encoding on each node in all the node sets in the cognitive map, call the graph embedding algorithm, and generate a node vector library; use the cosine similarity function to analyze the similarity between the question text vector of the current repeated event and each node vector, and mark the node corresponding to the highest similarity as the main node;

[0048] The attenuation coefficient analysis unit is used to analyze the time interval between the current timestamp and the last repeated event triggering timestamp, and analyze the attenuation coefficient based on the feedback strength coefficient and the time interval;

[0049] The candidate path analysis unit is used to retrieve a set of neighbor nodes directly connected to the main node in the cognitive map based on the main node, and obtain corresponding edge weights; regard each path from the main node to the neighbor node as a candidate path, extract the similarity between the node vector of the end node of each candidate path and the question text vector of the current repeated event, and screen out candidate paths with a similarity greater than a third similarity threshold; calculate the sum of the edge weights of each screened candidate path and record it as the candidate path weight, and select a preset number of candidate paths from the largest to the smallest candidate path weight to generate a final candidate path set.

[0050] According to the above solution, the dynamic weight update module includes a weight update unit and an update confirmation unit;

[0051] The weight updating unit is used to update the weight of the main node based on the attenuation coefficient, extract the similarity between the end node vector of the final candidate path and the question text vector of the current repeated event, analyze the weight increment based on the similarity between the end node vector of the final candidate path and the question text vector of the current repeated event, and update the weight of each final candidate path, where the updated weight is equal to the sum of the weight of the original final candidate path and the weight increment;

[0052] The update confirmation unit is used to extract the newly generated answer and the original answer after the weight update, generate semantic vectors for the newly generated answer and the original answer respectively, and analyze the feature distance between the semantic vectors of the newly generated answer and the original answer; if the feature distance is less than or equal to the feature distance threshold, automatically confirm that the update is valid; if the feature distance is greater than the feature distance threshold, trigger the manual review process; the feature distance threshold is set based on the historical mean and standard deviation of the feature distance.

[0053] Compared with the prior art, the present invention has the following advantages: The similarity calculation between the visual feature vector and the speech text vector is combined with a confidence-based dynamic adjustment coefficient, significantly improving the robustness of low-confidence speech scenarios; attention weighting enables the model to focus on visual entities strongly related to the speech content and suppress interference from irrelevant entities; the combined intent vector integrates text, attention weighting, and multimodal features, combined with timestamp-compensated encoding and attention fingerprinting, ensuring that interaction records contain spatiotemporal context, providing a complete data foundation for subsequent reasoning; the present invention identifies repeated events through dual threshold determination to avoid single false positives; triggers negative feedback marking, aggregates similar repeated events across users, and calculates feedback strength using a logarithmic function; the present invention quickly locates the main node most relevant to the current problem, and the attenuation coefficient integrates feedback strength and time decay to achieve dynamic adjustment logic; triple filtering ensures path quality, avoids graph search explosion, and focuses on high-value reasoning paths. The present invention's main node weight update formula is differentially adjusted according to node activity and prevents weights from returning to zero, causing nodes to become "invalid"; candidate paths are incrementally allocated, with paths with high similarity receiving more weight, and the incremental range is constrained to avoid sudden changes; and a balance is achieved between automated updates and system security. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 This is a flow chart of a multimodal data dynamic reasoning method based on cognitive graphs according to the present invention;

[0056] Figure 2 This is a structural diagram of a multimodal data dynamic reasoning system based on cognitive graphs in the present invention. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] See also Figure 1 The present invention provides a technical solution: a multimodal data dynamic reasoning method based on cognitive graph, the method comprising the following steps:

[0059] S1: Extracts text, voice, and image data input by the user, analyzes the user data, and combines it with the user ID and timestamp to generate a structured interaction record;

[0060] In S1, the following are included:

[0061] Example 1: S101: Extracting the voice stream A, image data and question text T collected by the user terminal b In this embodiment, based on the pre-trained end-to-end Transformer ASR model, the speech stream A is converted into the intermediate text T a , and output the speech recognition confidence C a ; Based on the preset YOLOv5 target detection network, perform detection on the image data and obtain the entity set E={e i |i∈[1,I]}, I represents the total number of entities; ResNet-50 is used to extract the visual feature vector of each entity, and the entity e i The visual feature vector is denoted as v i ;

[0062] Example 2: S102: Extract the intermediate text converted from the speech stream. In this embodiment, the BERT-Base model is called to generate a speech text vector q, and the feature similarity between each visual feature vector and the text vector is analyzed. The feature similarity between the visual feature vector vᵢ and the speech text vector q is recorded as sim(vᵢ, q), where sim() represents the cosine similarity function, and the attention weight is analyzed based on the adjustment coefficient τ. The adjustment coefficient is equal to the product of the basic adjustment value and the speech recognition confidence deviation value. The speech recognition confidence deviation value is equal to the difference between the speech recognition confidence reference value and the speech recognition confidence. The speech recognition confidence reference value is a system preset constant. The entity e i The attention weight is denoted as α i , integrating the attention weights of each entity to generate the attention distribution vector, recorded as α=[α1, α2, …, α I ]: ;

[0063] Where exp() represents the exponential function with natural numbers as the base; the entity e i The visual feature vector v i and entity category embedding vector embed (e i ) cascade, input two hidden layers, ReLU activated multi-layer perceptron to analyze the fusion feature vector, denoted as fᵢ;

[0064] Combine the text vector, attention weight and fusion feature vector to analyze the joint intent vector, denoted as Q:

[0065] Example 3: S103: Extract the current timestamp and time compensation of the device. The time compensation is equal to the difference between the current timestamp and the last sampling timestamp. In this embodiment, the PositionalEncoding function is used to generate the spatiotemporal code st; wherein the PositionalEncoding function is sine-cosine coding, and the coding length is consistent with the dimension of the joint intention vector; based on the user unique identifier u, the joint intention vector, the question text, the spatiotemporal code and the attention distribution fingerprint hash (A), the interaction record R is generated and integrated. n ={u,Q,T a ,st,hash(A)}, write the interaction records into the distributed historical database; hash() represents the hash function.

[0066] S2: Analyze historical interaction records, detect consecutive query events of the same user for similar questions, collect negative feedback signals, analyze the repetition frequency of repeated events among all users, and analyze and output the feedback intensity coefficient of repeated behavior;

[0067] In S2, the following are included:

[0068] Example 4: S201: Retrieving a historical question text set based on the user, calling the BERT-Base model for each historical question text to generate a historical question text vector, and using the cosine similarity function to analyze the similarity between any two historical question texts of the same user; if the similarity between the two historical question texts is greater than a first similarity threshold, and the corresponding timestamp difference is less than a preset time interval threshold, marking the two historical question texts as duplicate events;

[0069] S202: Extract user actions after repeated events are triggered, and identify the "re-ask" or "negative" keywords that trigger negative feedback; when negative feedback is detected, extract repeated events of all users, analyze the similarity of any two repeated events, and if the similarity of the two repeated events is greater than a second similarity threshold, mark them as similar repeated events, and count the total number of the same similar repeated events, recorded as M; analyze the feedback strength coefficient β based on the total number of the same similar repeated events, β=β1+log(M), where β1 is a system preset constant.

[0070] Example 5: In this example, there are users u1, u2, u3, and u4;

[0071] The historical question texts of user u1 are {Question1, Question2, Question3}, the historical question texts of user u2 are {Question4, Question5, Question6}, the historical question texts of user u3 are {Question7, Question8, Question9}, and the historical question texts of user u4 are {Question10, Question11, Question12};

[0072] Among them, in the historical question texts of user u1, Question1 and Question2 are repeated events, in the historical question texts of user u2, Question4, Question5 and Question6 are repeated events in pairs, in the historical question texts of user u3, Question8 and Question9 are repeated events, and in the historical question texts of user u3, Question10, Question11 and Question12 are not repeated events;

[0073] Then there are repeated events (Question1, Question2), (Question4, Question5), (Question4, Question6), (Question5, Question6) and (Question8, Question9) in u1, u2, u3, u4;

[0074] If the similarities between the repeated events (Question1, Question2), (Question4, Question5), (Question4, Question6) or (Question5, Question6) are all greater than the second similarity threshold, but the similarities between the repeated event (Question8, Question9) and other repeated events are all less than or equal to the second similarity threshold; then M in this embodiment is 4.

[0075] S3: Locate the corresponding nodes in the cognitive map based on the characteristics of repeated behaviors, analyze the attenuation coefficient, activate the associated alternative reasoning paths, and output the set of nodes to be updated and the candidate paths;

[0076] In S3, include the following:

[0077] S301: Structural and semantic encoding is performed on each node in the cognitive map, and a graph embedding algorithm is used to generate a node vector library. The cosine similarity function is used to analyze the similarity between the question text vector of the current repeated event and the vectors of each node, and the node with the highest similarity is marked as the main node.

[0078] S302: Analyze the time interval Δt between the current timestamp and the last repeated event trigger timestamp, and calculate the attenuation coefficient ΔD based on the feedback strength coefficient and the time interval: ΔD = β2 × β × exp(-λΔt); where β2 represents the basic attenuation rate and λ represents the time attenuation factor; the basic attenuation rate and the time attenuation factor are system preset constants;

[0079] S303: Based on the main node, search the cognitive map for a set of g-hop neighbor nodes directly connected to the main node and obtain the corresponding edge weights, where g is a system preset constant; consider each path from the main node to the neighbor node as a candidate path, extract the similarity between the node vector of the end node of each candidate path and the question text vector of the current repeated event, and screen out candidate paths with a similarity greater than a third similarity threshold; calculate the sum of the edge weights of each screened candidate path and record it as the candidate path weight, and select a preset number of candidate paths from the largest to the smallest according to the candidate path weights to generate a final candidate path set, which is recorded as P={P r |r∈[1,R]}, where P r represents the rth final candidate path, and R represents the total number of final candidate paths in the final candidate path set.

[0080] S4: Adjust node weights based on the decay coefficient, update weights according to the semantic relevance of candidate paths, update graph connectivity, and output the optimized cognitive graph.

[0081] In S4, include the following:

[0082] S401: Update the weight of the master node based on the decay coefficient: W z *=max(W z -△D×γ,W min ), where γ represents the learning parameter, which is preset based on the node activity, and max() represents the maximum value function; W z Represents the original weight of the master node; W min Indicates the minimum preset weight;

[0083] S402: Extract the similarity between the endpoint node vector of the final candidate path and the question text vector of the current repeated event, analyze the weight increment based on the similarity between the endpoint node vector of the final candidate path and the question text vector of the current repeated event, and record the weight increment of the rth final candidate path as △W r =σ(△D×S r / S sum ); where σ represents the Sigmoid function, S r S represents the similarity between the terminal node vector of the rth final candidate path and the question text vector of the current repeated event; sumRepresents the sum of the similarities between the terminal node vector of each final candidate path in the final candidate path set and the question text vector of the current repeated event;

[0084] The weight of each final candidate path is updated, and the updated weight is equal to the sum of the weight of the original final candidate path and the weight increment;

[0085] Example 6: S403: Extract the newly generated answer and the original answer after the weight update. In this example, the BERT-Base model is used to generate semantic vectors for the newly generated answer and the original answer respectively, and the feature distance of the semantic vectors of the newly generated answer and the original answer is analyzed; the feature distance uses Euclidean distance; if the feature distance is less than or equal to the feature distance threshold, the update is automatically confirmed to be valid; if the feature distance is greater than the feature distance threshold, the manual review process is triggered; the feature distance threshold is set based on the historical mean and standard deviation of the feature distance.

[0086] See also Figure 2 , the present invention provides a technical solution: a multimodal data dynamic reasoning system based on cognitive graph, the system includes an interaction record generation module, a repeated behavior feature analysis unit, a knowledge node candidate module and a dynamic weight update module;

[0087] The interaction record generation module is used to extract text, voice and image data input by users, analyze the user data and combine it with user identification and timestamp to generate structured interaction records;

[0088] The repeated behavior feature analysis unit is used to analyze historical interaction records, detect continuous query events of the same user for similar questions, collect negative feedback signals, analyze the repetition frequency of repeated events among all users, and analyze and output the feedback intensity coefficient of repeated behavior;

[0089] The knowledge node candidate module is used to locate the corresponding nodes in the cognitive map according to the characteristics of repeated behaviors, analyze the attenuation coefficient, activate the associated alternative reasoning paths, and output the set of nodes to be updated and the candidate paths;

[0090] The dynamic weight update module is used to adjust the node weights based on the attenuation coefficient, update the weights according to the semantic relevance of the candidate paths, update the graph connection relationship, and output the optimized cognitive graph.

[0091] The interaction record generation module includes a data acquisition and conversion unit and a data fusion unit;

[0092] The data acquisition and conversion unit is used to extract the voice stream, image data, and question text collected by the user terminal; based on the pre-trained end-to-end model, it converts the voice stream into intermediate text and outputs the voice recognition confidence; based on the preset object detection network, it performs detection on the image data to obtain an entity set; and extracts the visual feature vector of each entity;

[0093] The data fusion unit is used to analyze the feature similarity between each visual feature vector and the text vector, analyze the attention weight based on the adjustment coefficient, and analyze the joint intention vector by combining the text vector, attention weight and fused feature vector; extract the current timestamp and time compensation of the device to generate a spatiotemporal code; based on the user's unique identifier, joint intention vector, question text, spatiotemporal code and attention distribution fingerprint, generate an integrated interaction record.

[0094] The repetitive behavior feature analysis unit includes a repetitive event marking unit and a frequency analysis unit;

[0095] The repeated event marking unit is used to retrieve a historical question text set according to the user, generate a historical question text vector for each historical question text, and use a cosine similarity function to analyze the similarity between any two historical question texts of the same user; if the similarity between the two historical question texts is greater than a first similarity threshold, and the corresponding timestamp difference is less than a preset time interval threshold, the two historical question texts are marked as repeated events;

[0096] The frequency analysis unit is used to extract user operations after repeated events are triggered, and identify "re-ask" or "negative" keywords that trigger negative feedback; when negative feedback is detected, the repeated events of all users are extracted, and the similarity of any two repeated events is analyzed. If the similarity of the two repeated events is greater than a second similarity threshold, they are marked as similar repeated events, and the total number of the same similar repeated event is counted; the feedback intensity coefficient is analyzed based on the total number of the same similar repeated event.

[0097] The knowledge node candidate module includes a main node analysis unit, an attenuation coefficient analysis unit, and a candidate path analysis unit;

[0098] The main node analysis unit is used to perform structural and semantic encoding on all nodes in the cognitive map, call the graph embedding algorithm, and generate a node vector library. It uses the cosine similarity function to analyze the similarity between the question text vector of the current repeated event and each node vector, and marks the node with the highest similarity as the main node.

[0099] The attenuation coefficient analysis unit is used to analyze the time interval between the current timestamp and the last repeated event triggering timestamp, and analyze the attenuation coefficient based on the feedback strength coefficient and the time interval;

[0100] The candidate path analysis unit is used to retrieve the set of neighbor nodes directly connected to the main node in the cognitive map based on the main node, and obtain the corresponding edge weights; each path from the main node to the neighbor node is regarded as a candidate path, and the similarity between the node vector of the end node of each candidate path and the question text vector of the current repeated event is extracted, and the candidate paths with similarity greater than the third similarity threshold are screened out; the sum of the edge weights of each screened candidate path is calculated and recorded as the candidate path weight, and a preset number of candidate paths are selected from the candidate path weights in descending order to generate the final candidate path set.

[0101] The dynamic weight update module includes a weight update unit and an update confirmation unit;

[0102] The weight updating unit is used to update the weight of the main node based on the attenuation coefficient, extract the similarity between the terminal node vector of the final candidate path and the question text vector of the current repeated event, analyze the weight increment based on the similarity between the terminal node vector of the final candidate path and the question text vector of the current repeated event, and update the weight of each final candidate path. The updated weight is equal to the sum of the weight of the original final candidate path and the weight increment;

[0103] The update confirmation unit is used to extract the newly generated answer and the original answer after the weight update, generate semantic vectors for the newly generated answer and the original answer respectively, and analyze the feature distance between the semantic vectors of the newly generated answer and the original answer; if the feature distance is less than or equal to the feature distance threshold, the update is automatically confirmed to be valid; if the feature distance is greater than the feature distance threshold, the manual review process is triggered; the feature distance threshold is set based on the historical mean and standard deviation of the feature distance.

[0104] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0105] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A multimodal data dynamic reasoning method based on cognitive graph, characterized by: The method comprises the following steps: S1: Extracts text, voice, and image data input by the user, analyzes the user data, and combines it with the user ID and timestamp to generate a structured interaction record; S2: Analyze historical interaction records, detect consecutive query events for similar questions by the same user, aggregate negative feedback signals, analyze the recurrence frequency of repeated events among all users, and analyze and output the feedback intensity coefficient of repeated behaviors; S3: Locate the corresponding nodes in the cognitive map based on the characteristics of repeated behaviors, analyze the attenuation coefficient, activate the associated alternative reasoning paths, and output the set of nodes to be updated and the candidate paths; S4: Adjust node weights based on the decay coefficient, update weights according to the semantic relevance of candidate paths, update graph connectivity, and output the optimized cognitive graph; In S2, the following are included: S201: Retrieving a historical question text set based on the user, converting each historical question text into a historical question text vector, and using a cosine similarity function to analyze the similarity between any two historical question texts of the same user; if the similarity between the two historical question texts is greater than a first similarity threshold, and the corresponding timestamp difference is less than a preset time interval threshold, marking the two historical question texts as duplicate events; S202: Extract user actions triggered by repeated events and identify "re-ask" or "negative" keywords that trigger negative feedback. When negative feedback is detected, extract repeated events for all users and analyze the similarity between any two repeated events. If the similarity between the two repeated events is greater than a second similarity threshold, mark them as similar repeated events. Count the total number of the same similar repeated events, which is recorded as M. Analyze the feedback intensity coefficient β based on the total number of similar repeated events, β=β1+log(M), where β1 is a system preset constant; In S3, include the following: S301: Structural and semantic encoding is performed on each node in the cognitive map, and a graph embedding algorithm is used to generate a node vector library. The cosine similarity function is used to analyze the similarity between the question text vector of the current repeated event and the vectors of each node, and the node with the highest similarity is marked as the main node. S302: Analyze the time interval Δt between the current timestamp and the last repeated event trigger timestamp, and analyze the attenuation coefficient ΔD based on the feedback strength coefficient and the time interval, where ΔD = β2 × β × exp(-λΔt); where β2 represents the basic attenuation rate and λ represents the time attenuation factor; the basic attenuation rate and the time attenuation factor are system preset constants; S303: Based on the master node, search the cognitive graph for a set of g-hop neighbor nodes directly connected to the master node and obtain the corresponding edge weights, where g is a system preset constant. Each path from the main node to the neighboring node is considered as a candidate path. The similarity between the node vector of the terminal node of each candidate path and the question text vector of the current repeated event is extracted, and the candidate paths with similarity greater than the third similarity threshold are screened out. The sum of the edge weights of each screened candidate path is calculated and recorded as the candidate path weight. A preset number of candidate paths are selected from the largest to the smallest candidate path weights to generate the final candidate path set, which is recorded as P={P r |r∈[1,R]}, where P r represents the rth final candidate path, and R represents the total number of final candidate paths in the final candidate path set.

2. The multimodal data dynamic reasoning method based on cognitive graph according to claim 1 is characterized by: In S1, Includes the following: S101: Extract the voice stream, image data and question text collected by the user terminal; based on the pre-trained end-to-end model, convert the voice stream into intermediate text and output the voice recognition confidence C a ; Based on the preset target detection network, perform detection on the image data and obtain the entity set E={e i |i∈[1,I]}, I represents the total number of entities; extract the visual feature vector of each entity, and put the entity e i The visual feature vector is denoted as v i ; S102: Extract the intermediate text converted from the speech stream, generate a speech text vector q, analyze the feature similarity between each visual feature vector and the text vector, record the feature similarity between the visual feature vector vᵢ and the speech text vector q as sim(vᵢ,q), where sim() represents the cosine similarity function, and analyze the attention weight based on the adjustment coefficient τ, where the adjustment coefficient is equal to the product of the basic adjustment value and the speech recognition confidence deviation value, the speech recognition confidence deviation value is equal to the difference between the speech recognition confidence reference value and the speech recognition confidence, and the speech recognition confidence reference value is a system preset constant; i The attention weight is denoted as α i , integrating the attention weights of each entity to generate the attention distribution vector, recorded as α=[α1, α2, …, α I ]: The entity e i The visual feature vector v i and entity category embedding vector embed (e i ) cascade, input two hidden layers, ReLU activated multi-layer perceptron to analyze the fusion feature vector, denoted as fᵢ; Combine the text vector, attention weight and fusion feature vector to analyze the joint intent vector, denoted as Q; S103: extracting the current timestamp of the device and a time compensation amount, where the time compensation amount is equal to the difference between the current timestamp and the last sampling timestamp, and generating a spatiotemporal code st; Based on the user's unique identifier u, joint intention vector, question text, spatiotemporal encoding and attention distribution fingerprint hash (A), an interaction record R is generated and integrated. n ={u,Q,T a ,st,hash(A)}, write the interaction records into the distributed history database.

3. The multimodal data dynamic reasoning method based on cognitive graph according to claim 2 is characterized by: In S4, include the following: S401: Update the weight of the master node based on the decay coefficient: W z *=max(W z -△D×γ,W min ), where γ represents a learning parameter, which is preset based on the node activity, and max() represents the maximum value function; W z Represents the original weight of the master node; W min Indicates the minimum preset weight; S402: Extract the similarity between the endpoint node vector of the final candidate path and the question text vector of the current repeated event, analyze the weight increment based on the similarity between the endpoint node vector of the final candidate path and the question text vector of the current repeated event, and record the weight increment of the rth final candidate path as △W r =σ(△D×S r / S sum ); where σ represents the Sigmoid function, S r S represents the similarity between the terminal node vector of the rth final candidate path and the question text vector of the current repeated event; sum Represents the sum of the similarities between the terminal node vector of each final candidate path in the final candidate path set and the question text vector of the current repeated event; The weight of each final candidate path is updated, and the updated weight is equal to the sum of the weight of the original final candidate path and the weight increment; S403: Extract the newly generated answer and the original answer after the weight update, generate semantic vectors for the newly generated answer and the original answer respectively, and analyze the feature distance between the semantic vectors of the newly generated answer and the original answer; the feature distance adopts Euclidean distance; if the feature distance is less than or equal to the feature distance threshold, automatically confirm that the update is valid; if the feature distance is greater than the feature distance threshold, trigger the manual review process; the feature distance threshold is set based on the historical mean and standard deviation of the feature distance.

4. A multimodal data dynamic reasoning system based on cognitive graph, wherein the system is applied to implement a multimodal data dynamic reasoning method based on cognitive graph according to any one of claims 1 to 3, characterized in that: The system includes an interaction record generation module, a repeated behavior feature analysis unit, a knowledge node candidate module, and a dynamic weight update module; The interaction record generation module is used to extract text, voice and image data input by the user, analyze the user data and generate a structured interaction record by combining the user identifier and timestamp; The repeated behavior feature analysis unit is used to analyze historical interaction records, detect continuous query events of the same user for similar questions, collect negative feedback signals, analyze the repetition frequency of repeated events among all users, and analyze and output the feedback intensity coefficient of repeated behavior; The knowledge node candidate module is used to locate the corresponding node in the cognitive map according to the repetitive behavior characteristics, analyze the attenuation coefficient, activate the associated alternative reasoning path, and output the node set to be updated and the candidate path; The dynamic weight updating module is used to adjust the node weights based on the attenuation coefficient, update the weights according to the semantic relevance of the candidate paths, update the graph connection relationship, and output the optimized cognitive graph.

5. The multimodal data dynamic reasoning system based on cognitive graph according to claim 4 is characterized by: The interaction record generation module includes a data acquisition and conversion unit and a data fusion unit; The data acquisition and conversion unit is used to extract the voice stream, image data and question text collected by the user terminal; based on the pre-trained end-to-end model, convert the voice stream into intermediate text and output the voice recognition confidence; based on the preset object detection network, perform detection on the image data to obtain an entity set; and extract the visual feature vector of each entity; The data fusion unit is used to analyze the feature similarity between each visual feature vector and the text vector, analyze the attention weight based on the adjustment coefficient, and analyze the joint intention vector by combining the text vector, attention weight and fused feature vector; extract the current timestamp of the device and the time compensation amount to generate a spatiotemporal code; Based on the user's unique identifier, joint intent vector, question text, spatiotemporal encoding and attention distribution fingerprint, an interaction record is generated and integrated.

6. The multimodal data dynamic reasoning system based on cognitive graph according to claim 4, characterized in that: The repetitive behavior feature analysis unit includes a repetitive event marking unit and a frequency analysis unit; The repeated event marking unit is used to retrieve a historical question text set according to the user, generate a historical question text vector from each historical question text, and use a cosine similarity function to analyze the similarity between any two historical question texts of the same user; If the similarity between the two historical question texts is greater than a first similarity threshold, and the corresponding timestamp difference is less than a preset time interval threshold, the two historical question texts are marked as duplicate events; The frequency analysis unit is used to extract user operations after repeated events are triggered, and identify "re-ask" or "negative" keywords that trigger negative feedback; when negative feedback is detected, the repeated events of all users are extracted, and the similarity of any two repeated events is analyzed. If the similarity of the two repeated events is greater than a second similarity threshold, they are marked as similar repeated events, and the total number of the same similar repeated events is counted; and the feedback strength coefficient is analyzed based on the total number of the same similar repeated events.

7. The multimodal data dynamic reasoning system based on cognitive graph according to claim 4, characterized in that: The knowledge node candidate module includes a main node analysis unit, a decay coefficient analysis unit and a candidate path analysis unit; The main node analysis unit is used to perform structural and semantic encoding on each node in all the node sets in the cognitive map, call the graph embedding algorithm, and generate a node vector library; use the cosine similarity function to analyze the similarity between the question text vector of the current repeated event and each node vector, and mark the node corresponding to the highest similarity as the main node; The attenuation coefficient analysis unit is used to analyze the time interval between the current timestamp and the last repeated event triggering timestamp, and analyze the attenuation coefficient based on the feedback strength coefficient and the time interval; The candidate path analysis unit is used to retrieve a set of neighbor nodes directly connected to the main node in the cognitive map based on the main node, and obtain corresponding edge weights; regard each path from the main node to the neighbor node as a candidate path, extract the similarity between the node vector of the end node of each candidate path and the question text vector of the current repeated event, and screen out candidate paths with a similarity greater than a third similarity threshold; calculate the sum of the edge weights of each screened candidate path and record it as the candidate path weight, and select a preset number of candidate paths from the largest to the smallest candidate path weight to generate a final candidate path set.

8. The multimodal data dynamic reasoning system based on cognitive graph according to claim 4 is characterized by: The dynamic weight updating module includes a weight updating unit and an update confirmation unit; The weight updating unit is used to update the weight of the main node based on the attenuation coefficient, extract the similarity between the end node vector of the final candidate path and the question text vector of the current repeated event, analyze the weight increment based on the similarity between the end node vector of the final candidate path and the question text vector of the current repeated event, and update the weight of each final candidate path, where the updated weight is equal to the sum of the weight of the original final candidate path and the weight increment; The update confirmation unit is used to extract the newly generated answer and the original answer after the weight update, generate semantic vectors for the newly generated answer and the original answer respectively, and analyze the feature distance between the semantic vectors of the newly generated answer and the original answer; If the feature distance is less than or equal to the feature distance threshold, the update is automatically confirmed to be valid; if the feature distance is greater than the feature distance threshold, the manual review process is triggered; the feature distance threshold is set based on the historical mean and standard deviation of the feature distance.

Images