Knowledge graph-based large-model intelligent question-answering system optimization method and apparatus, and electronic device

Abstract

The invention discloses a knowledge graph-based large-model intelligent question-answering system optimization method and apparatus, and an electronic device. The method comprises the steps of obtaining newly-added multi-modal data of a target domain; performing consistency check on the newly added multi-modal data and a historical knowledge graph, and updating the historical knowledge graph by adopting a mixed resolution strategy according to a consistency check result to obtain an updated knowledge graph; performing model parameter fine adjustment on the large model based on the updated knowledge graph; and generating a result corresponding to the problem data through the large model after fine adjustment of the model parameters, constructing a reverse optimization link based on the result corresponding to the problem data, and optimizing the large model after fine adjustment of the model parameters.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a method, device and electronic device for optimizing a large-model intelligent question-answering system based on a knowledge graph. Background Art

[0002] A knowledge graph is a structured semantic knowledge base that represents entities, concepts, and their relationships in a graphical form. It can provide rich semantic information and logical reasoning capabilities for question-answering systems. A large-scale intelligent question-answering system that integrates knowledge graphs can help generate more accurate, relevant, and trustworthy content by combining and retrieving relevant information from the knowledge graph.

[0003] However, in related technologies, the collaborative optimization solution of knowledge graphs and large models faces the following technical bottlenecks. At the knowledge graph level, the knowledge graph relies on static rules and single-dimensional confidence assessment, resulting in insufficient conflict resolution capabilities and difficulty in effectively handling boundary knowledge conflicts. At the question-answering generation level, the knowledge prompts corresponding to the knowledge graph are usually static templates, and static templates fail to dynamically couple with the attention mechanism of the large model, which can easily lead to knowledge misuse or semantic deviation, resulting in insufficient reasoning ability of the large model for complex relationships and prone to "hallucination" problems. Summary of the Invention

[0004] The embodiments of the present application provide a method, device and electronic device for optimizing a large-model intelligent question-answering system based on a knowledge graph, which can solve the problems in related technologies such as insufficient conflict resolution capabilities of knowledge graphs that make it difficult to effectively handle boundary knowledge conflicts, and the failure of static templates to dynamically couple with the attention mechanism of large models, which can easily lead to knowledge misuse or semantic deviation, resulting in insufficient reasoning capabilities of large models for complex relationships and the proneness to "hallucination" problems.

[0005] In the first aspect, an embodiment of the present application provides a method for optimizing a large-model intelligent question-answering system based on a knowledge graph, the method comprising: obtaining newly added multimodal data in a target field; performing a consistency check on the newly added multimodal data and the historical knowledge graph, and updating the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph; fine-tuning the model parameters of the large model based on the updated knowledge graph; generating results corresponding to the question data through the large model after fine-tuning the model parameters, constructing a reverse optimization link based on the results corresponding to the question data, and optimizing the large model after fine-tuning the model parameters.

[0006] In the second aspect, an embodiment of the present application provides a large-model intelligent question-answering system optimization device based on a knowledge graph, which includes: an acquisition module for acquiring new multimodal data in a target field; a knowledge graph update module for performing consistency check on the new multimodal data and the historical knowledge graph, and updating the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph; a fine-tuning module for fine-tuning model parameters of the large model based on the updated knowledge graph; an optimization module for generating results corresponding to question data through the large model after fine-tuning the model parameters, constructing a reverse optimization link based on the results corresponding to the question data, and optimizing the large model after fine-tuning the model parameters.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the method described in the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes a program or instructions, which, when executed, implement the steps of the method described in the first aspect.

[0011] In an embodiment of the present application, by acquiring new multimodal data in the target field, a consistency check is performed on the new multimodal data and the historical knowledge graph, and a hybrid resolution strategy is adopted to update the historical knowledge graph according to the result of the consistency check to obtain an updated knowledge graph. Then, based on the updated knowledge graph, the model parameters of the large model are fine-tuned, and the results corresponding to the question data are generated by the large model after the model parameters are fine-tuned. Based on the results corresponding to the question data, a reverse optimization link is constructed, and the large model after the model parameters are fine-tuned is optimized to achieve comprehensive optimization of the large model intelligent question and answer system. That is, through consistency comparison, the historical knowledge graph used to retrieve knowledge in the large model intelligent question and answer system is dynamically updated, which not only ensures that the knowledge graph can be updated in real time and holds the latest knowledge in the target field, but also provides a more accurate basis for the subsequent generation of results corresponding to the question data. In addition, by fine-tuning the model parameters of the large model on the basis of the updated knowledge graph, the large model is made lightweight, thereby improving the efficiency of result generation and reducing resource consumption. And a reverse optimization link is constructed in combination with the result feedback of the large model to optimize the large model after fine-tuning the model parameters, to achieve iterative enhancement of the large model, to ensure the adaptability and logical consistency of the generated results with the target field, and to suppress the illusion of the generated results. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a flow chart of a method for optimizing a large-model intelligent question-answering system based on a knowledge graph provided in an embodiment of the present application; Figure 2 This is a flow chart of another method for optimizing a large-model intelligent question-answering system based on a knowledge graph provided in an embodiment of the present application; Figure 3 This is a structural diagram of a large-model intelligent question-answering system optimization device based on a knowledge graph provided in an embodiment of the present application; Figure 4 This is a structural diagram of a large-model intelligent question-answering system based on a knowledge graph provided in an embodiment of the present application; Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0014] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0015] With the continuous development of artificial intelligence technology, knowledge graphs and the multimodal question-answering systems they drive have become a hot topic in research and application. The relevant multimodal question-answering systems involve the following technologies: 1. Retrieval-Augmented Generation: Retrieval-Augmented Generation (RAG) combines large models with information retrieval techniques. Specifically, when a large model needs to generate text or answer a question, it first retrieves relevant information from a large knowledge base and then uses this retrieved information to guide content generation.

[0016] 2. Knowledge Graph: A knowledge graph is a diagram used to display the development process and structural relationships of knowledge. It describes knowledge and its carriers in a visual form, and mines, analyzes, constructs, maps, and displays knowledge and its interrelationships. It can present various types of knowledge in a structured form, providing strong support for efficient knowledge management and retrieval. The knowledge graph displays the relationships between entities in the form of a graph, which can effectively integrate structured knowledge and provide support for intelligent question-answering, recommendation systems, etc. However, when processing multimodal data, existing knowledge graphs still have difficulty fully integrating multimodal data such as images, audio, and video, and are unable to fully display the associations of complex knowledge.

[0017] 3. The combination of RAG and knowledge graphs has become a key technical path for enhancing the accuracy of knowledge in question-answering systems. Retrieval systems generally convert user queries into contextual strings that blend structured knowledge and unstructured text through three steps: multimodal search, semantic reranking, and knowledge graph retrieval. This allows the LLM to generate factually accurate answers. When creating content within the constraints of the knowledge graph, the large model uses the graph's embedded verification mechanism and path backtracking algorithm to verify in real time that intermediate reasoning conforms to the graph logic. The final output is an answer that maintains both natural language fluency and knowledge reliability, accompanied by an interpretable visual reasoning path.

[0018] 4. Large Language Model: A Large Language Model (LLM) is a deep learning model trained using large amounts of text data. This model can generate natural language text or understand the meaning of language text.

[0019] 5. Convolutional Neural Networks: Convolutional Neural Networks (CNNs) are deep learning models for processing grid-structured data (such as images and audio). They consist of convolutional layers, pooling layers, and fully connected layers, and can automatically extract data features. In image recognition, they can identify objects and their features; in audio processing, they can extract audio features.

[0020] 6. Three-dimensional Convolutional Neural Networks: CNNs are well-suited for static image recognition tasks. Their advantages lie in parameter sharing, which reduces computational complexity, and translation invariance, which enhances generalization. 3D-CNNs extend this to the spatiotemporal dimension, using three-dimensional convolutional kernels to simultaneously extract both spatial features and temporal dynamics from videos or consecutive frames. By stacking 3D convolutional layers, they capture the evolution of motion. Compared to 2D-CNNs, they can simultaneously process both spatial and temporal features, leading to a better understanding of video content.

[0021] 7. Long Short-Term Memory Network: LSTM (Long Short-Term Memory) is a specialized recurrent neural network that addresses the vanishing and exploding gradient problems of traditional recurrent neural networks and is suitable for processing long sequences of data. In natural language processing, it is often used for text feature extraction, capturing long-term dependencies and understanding semantics and grammar.

[0022] 8. Mel-Frequency Cepstral Coefficients: Mel-Frequency Cepstral Coefficients (MFCCs) are a common technique for audio feature extraction. They simulate the human auditory perception of sound frequency by converting the audio signal into the mel-frequency domain and then calculating the cepstral coefficients. They can effectively extract audio features, reflecting information such as timbre and pitch, and are widely used in speech recognition and audio classification.

[0023] In related technologies, the processing of multimodal data presents the following problems: First, multimodal data is not fully integrated, resulting in incomplete coverage of the constructed knowledge graph. Second, simple data concatenation and average pooling cannot fully capture the correlations between multimodal data, resulting in information loss in multimodal data. Finally, a unified feature representation that includes multimodal data is not generated, which limits the ability to reason about complex problems. For example, for a specific risk issue, it is impossible to combine image, video, and text-based multimodal data for a comprehensive risk analysis.

[0024] The large-model intelligent question-answering system combined with the knowledge graph has the following problems: the knowledge prompts of the large-model intelligent question-answering system are usually static templates, which have the problem of knowledge lag. This also makes the large model's reasoning ability for complex relationships insufficient, and it is easy to have the illusion that the generated results are not compatible with the domain or are logically contradictory.

[0025] Below, in combination with the accompanying drawings, the optimization method, device and electronic device of the large-model intelligent question-answering system based on knowledge graph provided by the embodiments of the present application are described in detail through specific embodiments and their application scenarios, so as to at least solve the above-mentioned problems.

[0026] Figure 1 The flowchart of a method for optimizing a large-model intelligent question-answering system based on a knowledge graph provided by an embodiment of the present application is shown. The method can be executed by an electronic device. Figure 1 , the method may include the following steps.

[0027] Step 102: Acquire new multimodal data in the target domain.

[0028] New multimodal data can include documents, images, audio, video, and other data, representing new knowledge in the target domain. Since these new multimodal data may come from different sources, Kafka, a messaging queue, can be used to centrally access these data sources. Structured, semi-structured, and unstructured data streams can be standardized and pre-processed, and metadata (such as source authority, timestamp, and confidence level) can be added to each piece of data.

[0029] Optionally, before updating the newly added multimodal data to the historical knowledge graph, the newly added multimodal data is preprocessed. Since unstructured multimodal data (such as text, voice, image, video) lacks a fixed format, it can be converted into a structured or semi-structured form through natural language processing (NLP) technology, such as word segmentation, denoising, syntactic analysis, named entity recognition (NER), etc., to facilitate subsequent analysis.

[0030] Step 104: Perform a consistency check on the newly added multimodal data and the historical knowledge graph, and update the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph.

[0031] Among them, the historical knowledge graph is constructed based on multi-source multimodal data of the target field that has been collected in history. In order to ensure the timeliness and accuracy of the knowledge in the knowledge graph, the knowledge graph needs to be updated. For newly added multimodal time, when updating it to the historical knowledge graph, first check the consistency between the newly added multimodal data and the historical knowledge graph, for example, whether the newly added multimodal data and the knowledge in the historical knowledge graph have duplicate content, whether there are conflicts, whether the source is credible, etc., and then deal with each problem separately to update the historical knowledge graph. Specifically, the consistency check of the newly added multimodal data and the historical knowledge graph can include: pre-processing the newly added multimodal data to generate one or more fused multimodal feature representations, defining knowledge nodes and edges based on the one or more multimodal feature representations with an associated relationship, and then performing a consistency check on the knowledge nodes and edges with the historical knowledge graph.

[0032] Step 106: Fine-tune the model parameters of the large model based on the updated knowledge graph.

[0033] Among them, in one implementation method, fine-tuning the model parameters of the large model based on the updated knowledge graph can include the following steps: step 1061, encoding the updated knowledge graph into a structured prompt template; step 1062, aligning the model parameters of the large model according to the structured prompt template.

[0034] Among them, by encoding the updated knowledge graph into a structured prompt template, based on the constraints of the structured prompt template, and combining dynamic path pruning, low-rank adaptation technology and other technologies, the model parameters of the large model are aligned to ensure that the large model can generate results corresponding to the problem data according to the structured prompt template, and directionally enhance the sensitivity of the large model to the structured prompt template, so that the results corresponding to the problem data generated by the large model can be more adapted to the knowledge of the target field.

[0035] Step 108 : Generate a result corresponding to the problem data using the large model after fine-tuning the model parameters, build a reverse optimization link based on the result corresponding to the problem data, and optimize the large model after fine-tuning the model parameters.

[0036] Among them, the updated knowledge graph serves as the prior knowledge constraint of the large model generation results. That is to say, a reverse optimization link is constructed based on the feedback of the generation results, and the performance of the large model is evaluated by comparing with the knowledge of the updated knowledge graph. The large model is then optimized to achieve iterative enhancement of the large model and deep collaboration between the knowledge graph drive and the large model capabilities.

[0037] In an embodiment of the present application, by acquiring new multimodal data in the target domain, a consistency check is performed on the new multimodal data and the historical knowledge graph, and a hybrid solution strategy is used to update the historical knowledge graph based on the result of the consistency check to obtain an updated knowledge graph. Then, based on the updated knowledge graph, the model parameters of the large model are fine-tuned, and the results corresponding to the problem data are generated by the large model after the model parameters are fine-tuned. Based on the results corresponding to the problem data, a reverse optimization link is constructed to optimize the large model to achieve iterative enhancement of the large model. That is, through consistency comparison, the historical knowledge graph used to retrieve knowledge in the system is dynamically updated, which not only ensures that the knowledge graph can be updated in real time and holds the latest knowledge in the target domain, but also provides a more accurate basis for the subsequent generation of results corresponding to the problem data. In addition, by fine-tuning the model parameters of the large model based on the updated knowledge graph, the large model is lightweight, thereby improving the result generation efficiency and reducing resource consumption. And the reverse optimization link is constructed in combination with the result feedback of the large model to achieve iterative enhancement of the large model, ensuring the adaptability and logical consistency of the generated results with the target domain, and suppressing the illusion of the generated results.

[0038] In one implementation, the above-mentioned step 1062 of aligning the model parameters of the large model according to the structured prompt template may include the following steps.

[0039] Step 10621: Generate path vectors between the query entity and the target entity based on the knowledge entities in the updated knowledge graph and the relationships between the knowledge entities; wherein the knowledge entities include the query entity, the target entity, and the intermediate entities from the query entity to the target entity.

[0040] Among them, the path vector represents the process of how to find the target entity from the query entity. This process may include multiple paths, and different paths have different characteristics. For example, in order to quickly find the target entity, you can choose a path that includes fewer intermediate entities. For example, in order to comprehensively explore various knowledge entities related to the target entity, you can choose a path that includes more intermediate entities.

[0041] Step 10622: Pruning the large model according to the similarity between the query vector corresponding to the question data and each path vector and a dynamic similarity threshold.

[0042] Among them, similarity is used to measure the degree of match between the target entity obtained by selecting each path query and the current problem data, and this is used as the standard for selecting the path. For example, paths with similarity greater than the dynamic similarity threshold are retained to provide more accurate target entities and intermediate entities for the problem data.

[0043] Step 10623: perform importance evaluation on the attention layer of the large model after pruning to obtain the importance value of each attention head.

[0044] Among them, by evaluating the importance of the attention head of the attention layer in the large model, the contribution of each attention head to the generation result of the large model is measured.

[0045] Step 10624, performing low-rank decomposition processing according to the importance values of the respective attention heads according to the low-rank adaptation strategy corresponding to the importance values, and obtaining a large model whose structure of the model parameters matches the structured prompt template.

[0046] For example, the top 10% of the heads in terms of importance execute a low-rank adaptation strategy with a rank of r=12; the middle 70% of the heads execute a low-rank adaptation strategy with a rank of r=8; and the bottom 20% of the heads execute a low-rank adaptation strategy with a rank of r=4.

[0047] In an embodiment of the present application, by performing dynamic pruning processing on the large model and fine-tuning the low-rank decomposition processing, the structured prompt template corresponding to the updated knowledge graph is aligned with the model parameters of the large model, and the large model is lightweighted through dynamic pruning processing. Compared with the fixed rank allocation in the prior art, the embodiment of the present application takes into account the difference in the importance of attention heads, avoids model parameter redundancy, and reduces the computing power overhead of the large model, thereby improving the result generation efficiency and reducing resource consumption.

[0048] In one implementation, the above-mentioned step 108 generates a result corresponding to the problem data through the large model after fine-tuning the model parameters, builds a reverse optimization link based on the result corresponding to the problem data, and optimizes the large model after fine-tuning the model parameters, which may include the following steps.

[0049] Step 1081: Input the question data into the large model after fine-tuning the model parameters for processing, and generate a result corresponding to the question data according to the structured prompt template.

[0050] Among them, the structured prompt template constrains the results corresponding to the question data generated by the large model to ensure that the results corresponding to the generated question data are more suitable for the updated knowledge graph of the current target field.

[0051] Step 1082: Determine the cross entropy loss function and reward function of the large model after fine-tuning the model parameters based on the results corresponding to the question data and the corresponding multimodal feature representation in the updated knowledge graph.

[0052] Among them, the accuracy of the results generated by the large model is judged by the cross-entropy loss function, and the adaptability and logical consistency of the results generated by the large model to the target field are judged by the reward function, thereby achieving multi-faceted optimization of the large model.

[0053] In one implementation, the cross entropy loss function of the large model can be determined based on the number of results, the results, and the true answers corresponding to the question data in the updated knowledge graph.

[0054] Among them, the cross entropy loss function As shown below:

[0055] Among them, N is the total number of input problem data, that is, the number of results corresponding to the problem data. It is The true answer corresponding to the question data in the updated knowledge graph. It is the result corresponding to the problem data generated by the large model.

[0056] Optionally, in order to prevent the large model from overfitting, L2 regularization technology is used, that is, a regularization term is added to the above cross entropy loss function to limit the model parameters of the large model. As shown below:

[0057] in, is the cross entropy loss function mentioned above, is the regularization coefficient, is the total number of model parameters of the large model; For the large model model parameters.

[0058] Among them, the reward function of the large model can be determined based on the matching degree between the result corresponding to the question data generated by the large model and the multimodal feature representation corresponding to the question data in the updated knowledge graph.

[0059] Specifically, the degree of matching between the result and the multimodal feature representation includes at least one of the following: (1) The degree of relationship matching between the result and the knowledge entity corresponding to the multimodal feature representation.

[0060] The knowledge entities corresponding to the multimodal feature representations are the various multimodal data in the updated knowledge graph. The relationship between the currently generated result and the original multimodal data corresponding to the multimodal feature representation is measured by the degree of relationship matching between the result and the knowledge entities corresponding to the multimodal feature representations. For example, whether the result is related to the original multimodal data corresponding to the multimodal feature representation, what kind of relationship exists, and whether the result can be derived from the original multimodal data corresponding to the multimodal feature representation is determined, and a reward value is assigned based on this.

[0061] (2) The logical matching degree between the structure and the knowledge unit corresponding to the multimodal feature representation.

[0062] The knowledge unit corresponding to the multimodal feature representation is composed of at least one of the multimodal data items associated with the updated knowledge graph. The logical match between the generated result and the knowledge unit corresponding to the multimodal feature representation is used to measure whether the current result is compatible with the knowledge unit corresponding to the multimodal feature representation and whether the logic is consistent. This is used as a basis for assigning a reward value.

[0063] In an embodiment of the present application, the reward function of the large model can be determined based on at least one of the relationship matching degree between the result and the knowledge entity corresponding to the multimodal feature representation, and the logical matching degree between the result and the knowledge unit corresponding to the multimodal feature representation. For example, when the relationship between the result and the knowledge entity corresponding to the multimodal feature representation in the knowledge graph and the logic of the knowledge unit are highly consistent, a high reward value is assigned. When there is a contradiction between the relationship between the result and the knowledge entity corresponding to the multimodal feature representation in the knowledge graph and the logic of the knowledge unit, and when the result deviates from the constraints of the knowledge graph, a low reward value is assigned. This reward mechanism can effectively suppress the hallucination problem in the generated results, improve the accuracy of the results generated by the large model, and make the generated results more suitable for the current target field and consistent with the logic of the target field.

[0064] Step 1083: Optimize the large model after fine-tuning the model parameters according to the cross entropy loss function and the reward function until the preset convergence condition is met to obtain the target large model.

[0065] The preset convergence condition includes a first preset convergence condition corresponding to the cross entropy loss function and a second preset convergence condition corresponding to the reward function. Optimizing the large model thus includes determining first target model parameters when the cross entropy loss function satisfies the first preset convergence condition. The first preset convergence condition may be minimizing the cross entropy loss function, that is, taking the model parameters when the cross entropy loss function is at its minimum as the first target model parameters, so as to optimize the accuracy of the results corresponding to the problem data generated by the large model.

[0066] Optimizing the model parameters of the large model further includes optimizing the first target model parameters obtained through the optimization based on a reward function, and determining second target model parameters when the reward function satisfies a second preset convergence condition. The second target model parameters are the optimized model parameters of the target large model. The second preset convergence condition may be maximizing the reward function, that is, taking the model parameters when the reward function is at its maximum value as the second target model parameters, so as to optimize the adaptability and logical consistency of the results generated by the large model with the target domain.

[0067] In an embodiment of the present application, the problem data is input into the large model after fine-tuning the model parameters for processing, the result corresponding to the problem data is generated according to the structured prompt template, and the cross-entropy loss function and reward function of the large model after fine-tuning the model parameters are determined according to the result corresponding to the problem data and the corresponding multimodal feature representation in the updated knowledge graph. The cross-entropy loss function can judge the accuracy of the result generated by the large model, and the reward function can judge the adaptability and logical consistency of the result generated by the large model to the target domain, so that the model parameters of the large model after fine-tuning the model parameters are optimized according to the cross-entropy loss function and the reward function until the preset convergence conditions are met to obtain the target large model. The optimized target large model has both accuracy and adaptability and logical consistency to the target domain, so that the target large model can generate more accurate and appropriate results for the problem data.

[0068] In one implementation, after obtaining the target large model as described above, the method may further include the following steps.

[0069] Step 110: Obtain candidate hyperparameter combinations of the target large model from a predefined hyperparameter space.

[0070] Among them, the hyperparameter space is the set of all possible hyperparameter combinations of the target large model. Hyperparameters are other parameters pre-set in the large model in addition to the model parameters. They are not obtained through the above-mentioned fine-tuning and optimization, but directly affect the performance of the large model.

[0071] Step 112: Perform performance evaluation on the target large model configured with the candidate hyperparameter combination through cross-validation to determine the evaluation index of the target large model.

[0072] Cross-validation involves dividing the training dataset into K unique subsets, with one subset serving as a validation set and the remaining subsets serving as training sets. Each time, the training set is used to train the target large model configured with the candidate hyperparameter combination, and the validation set is used to evaluate the performance of the target large model and calculate evaluation metrics. These metrics may include accuracy, recall, F1 score, and response time.

[0073] Step 114: Determine a target hyperparameter combination from the candidate hyperparameter combinations according to the evaluation index.

[0074] Among them, the evaluation index is used to evaluate the performance of the target large model configured with the candidate hyperparameter combination. For multiple evaluation indicators, the performance of the target large model can be evaluated by combining multiple evaluation indicators through the average performance index. Calculate the average performance index Metric As shown below: Metric Metric

[0075] in, is the number of cross-validation folds; Metric For the Folding performance indicators.

[0076] In an embodiment of the present application, the performance of the target large model can be further improved by optimizing the hyperparameters of the target large model. A candidate hyperparameter combination of the target large model is obtained from a predefined hyperparameter space, and the performance of the target large model configured with the candidate hyperparameter combination is evaluated through cross-validation to determine the evaluation index of the target large model. The performance of the target large model configured with the candidate hyperparameter combination is quantified by the evaluation index, and then the target hyperparameter combination is determined from the candidate hyperparameter combination according to the evaluation index. For example, a candidate hyperparameter combination that makes the performance of the target large model better is selected as the target hyperparameter combination, so that the performance of the target large model configured with the target hyperparameter combination is further improved.

[0077] In one implementation, the above-mentioned step 104 performs a consistency check on the newly added multimodal data and the historical knowledge graph, and updates the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph, which may include the following steps.

[0078] Step 1041: extract the knowledge entities, knowledge attributes, and relationships between the newly added multimodal data and the historical knowledge graph.

[0079] Among them, optionally, before extracting the knowledge entities, knowledge attributes, and relationships between the newly added multimodal data and the historical knowledge graph, the newly added multimodal data is preprocessed, including denoising processing, standardization processing, etc.

[0080] Step 1042: When the knowledge entity corresponding to the newly added multimodal data is the same as the knowledge entity in the historical knowledge graph, or the knowledge attribute corresponding to the newly added multimodal data is the same as the knowledge attribute in the historical knowledge graph, deduplication processing is performed on the knowledge entity corresponding to the newly added multimodal data or the knowledge attribute corresponding to the newly added multimodal data.

[0081] Step 1043: if the relationship between the newly added multimodal data and the historical knowledge graph conflicts with the basic logic, intercept the newly added multimodal data; Step 1044, when the relationship between the newly added multimodal data and the historical knowledge graph is consistent with the basic logic, the newly added multimodal data is adopted or discarded or manually reviewed according to the confidence score threshold to which the confidence score of the newly added multimodal data belongs.

[0082] In an embodiment of the present application, the consistency check of the newly added multimodal data and the historical knowledge graph may include conflicts in knowledge entities or knowledge attributes and conflicts in the relationship between the newly added multimodal data and the historical knowledge graph. Different conflict resolution strategies are set for the above conflicts. When the knowledge entity corresponding to the newly added multimodal data is the same as the knowledge entity in the historical knowledge graph, or the knowledge attribute corresponding to the newly added multimodal data is the same as the knowledge attribute in the historical knowledge graph, the knowledge entity corresponding to the newly added multimodal data or the knowledge attribute corresponding to the newly added multimodal data is deduplicated. When the relationship between the newly added multimodal data and the historical knowledge graph conflicts with the basic logic, the newly added multimodal data is intercepted. When the relationship between the newly added multimodal data and the historical knowledge graph is consistent with the basic logic, the newly added multimodal data is adopted, discarded, or manually reviewed according to the confidence score threshold to which the confidence score of the newly added multimodal data belongs. This achieves multi-level conflict detection and resolution, efficiently and accurately updates the knowledge graph, and minimizes update conflicts.

[0083] Optionally, for unresolved conflicts, the historical knowledge graph is retained and the conflicts are stored as independent documents for subsequent processing.

[0084] In one implementation, the method may further include the following steps.

[0085] Step 1011: Acquire multimodal data of the target domain.

[0086] Multimodal data in the target domain can include data in multiple modalities from multiple sources. For example, text data can be collected from internal enterprise databases, email systems, and document management systems; image data can be collected from internal enterprise documents and marketing materials; audio data can be collected from enterprise meeting recordings and customer service calls; and video data can be collected from enterprise training videos and product demonstration videos. External data collection includes obtaining text, image, audio, and video data from external sources such as public industry reports, social media, and news websites.

[0087] Optionally, denoising the acquired multimodal data may include: denoising the collected text data, including spell checking and grammar correction of the text data, image denoising using a Gaussian filtering algorithm on the image data, noise suppression using frequency domain filtering on the audio data, and removing video noise using inter-frame filtering technology on the video data.

[0088] Optionally, the denoised multimodal data is standardized, which may include: word segmentation, part-of-speech tagging and named entity recognition for text data, size normalization and color space conversion for image data, resampling and volume normalization for audio data, and frame rate unification and key frame extraction for video data.

[0089] Step 1012: extract multiple features corresponding to the multimodal data respectively through a feature extraction network corresponding to the multimodal data.

[0090] Among them, extracting multiple features corresponding to the multimodal data respectively through the feature extraction network corresponding to the multimodal data specifically includes: using CNN to extract features of the above-mentioned processed image data; using LSTM to extract features of the above-mentioned processed text data; using MFCC to extract features of the above-mentioned processed audio data; and using 3D-CNN to extract features of the above-mentioned processed video data.

[0091] Step 1013: Concatenate the multiple features according to the dimensions corresponding to the features through a multi-head attention mechanism to obtain at least one multimodal feature representation.

[0092] Among them, the multi-head attention mechanism can simultaneously focus on different aspects of different modal data, and splice the above-mentioned extracted features according to the dimensions corresponding to the features through adaptive weighting to generate a unified multimodal feature representation. The features that can be spliced must be relevant or associated in terms of dimensions, so that they can be integrated into a unified multimodal feature representation.

[0093] Step 1014: construct the historical knowledge graph based on at least one of the multimodal feature representations and the relationship between the multimodal features.

[0094] Each node in the historical knowledge graph is a knowledge unit, consisting of one or more multimodal feature representations, a unique identifier, and a preset corresponding question text. Edges between nodes are formed by the relationships between them, which can include similarities, causal relationships, and more. A historical knowledge graph can be constructed based on nodes and edges. More specifically, each node can be traced back to its more fundamental components, namely multimodal data, so each node can also be mapped to specific multimodal data.

[0095] Optionally, an index is established for the historical knowledge graph. The historical knowledge graph is stored in a graph database, and the node and edge data structure of the graph database is used to manage and retrieve knowledge units. The types of indexes may include: full-text index, attribute index, etc.

[0096] In an embodiment of the present application, a knowledge graph is constructed based on multimodal data in a target domain. First, a feature extraction network corresponding to the multimodal data is used to extract multiple features corresponding to the multimodal data. The multiple features are then fused through a multi-head attention mechanism to obtain at least one multimodal feature representation. Based on at least one multimodal feature representation and the relationship between the multimodal features, the knowledge graph is constructed. The knowledge graph can capture the correlation between the various multimodal feature representations and the answer and its source, which helps to understand the problem data and generate results.

[0097] Figure 2 The flowchart of another method for optimizing a large-model intelligent question-answering system based on a knowledge graph provided by an embodiment of the present application is shown. The method can be executed by an electronic device. Figure 2 , the method may include the following steps.

[0098] Step 201: Collect multi-source multi-modal data and perform pre-processing.

[0099] Specifically, cross-source multimodal data collection can be implemented by building heterogeneous data pipelines, covering data in multiple modalities such as unstructured documents, images, audio, and video, and performing unified ETL (Extract, Transform, Load) and standardization processing on the collected data to ensure data consistency.

[0100] In one implementation, step 201 may specifically include the following steps 2011 to 2013.

[0101] Step 2011: collect text, pictures, audio, video and other data from inside or outside the enterprise.

[0102] For example, internal enterprise data collection includes: collecting text data from internal databases, email systems, and document management systems; collecting image data from internal documents and marketing materials; collecting audio data from corporate meeting recordings and customer service calls; and collecting video data from corporate training videos and product demonstration videos; external enterprise data collection includes: obtaining text, image, audio, and video data from external sources such as public industry reports, social media, and news websites.

[0103] Step 2012: De-noising the multimodal data.

[0104] For example, spelling checking and grammar correction are performed on text data, Gaussian filtering algorithm is used to denoise image data, frequency domain filtering is used to suppress noise on audio data, and inter-frame filtering technology is used to remove video noise from video data.

[0105] Step 2013: normalize the denoised multimodal data.

[0106] For example, text data is segmented, part-of-speech tagged, and named entity recognized; image data is size normalized and color space converted; audio data is resampled and volume normalized; and video data is frame-rate unified and key frames extracted.

[0107] Step 202: extract features from the preprocessed multimodal data and fuse them based on a multi-head attention mechanism to generate a multimodal feature representation.

[0108] Feature extraction can be performed based on the model it is adapted for. For example, convolutional neural networks can be used to extract features from preprocessed image data; long short-term memory networks can be used to extract features from preprocessed text data; Mel-frequency cepstral coefficients can be used to extract features from preprocessed audio data; and three-dimensional convolutional networks can be used to extract features from preprocessed video data. These extracted features are then fused using a multi-head attention mechanism to generate a unified multimodal feature representation.

[0109] In one implementation, step 202 may specifically include the following steps 2021 to 2024.

[0110] Step 2021: Multimodal feature extraction and preprocessing. Perform feature extraction on the preprocessed text, audio, video, and image data to obtain feature representations of each modality. , , , , and align the feature sequences by temporal interpolation.

[0111] Step 2022: Feature splicing and projection. Splice each modal feature into a unified feature along the feature dimension. . The query is then generated by sharing the linear transformation matrix ( ),key( ),value( )matrix:

[0112] in, 、 、 is a learnable parameter, is the total dimension of input features, , is the number of attention heads is the model hidden layer dimension. Step 2023: separate projection and attention calculation. Projected to low-dimensional subspaces, each head The calculation of is as follows: Attention score:

[0113] in, is the key vector dimension, scaling factor Used to prevent the dot product result from being too large and causing the gradient to disappear.

[0114] Weighted sum:

[0115] Step 2024: Multi-head output concatenation. The outputs of all heads are concatenated along the feature dimension and linearly transformed to generate the final multimodal feature representation:

[0116] in, is the output projection matrix.

[0117] In one embodiment, the number of multi-head attention heads H=8 and the key vector dimension , query vector dimension , value vector dimension , each modal feature is projected into 8 low-dimensional subspaces, each subspace corresponds to an attention head to capture cross-modal correlations.

[0118] Step 203: construct a knowledge graph of the target domain based on the generated multimodal feature representation.

[0119] The knowledge graph consists of nodes and edges (nodes represent knowledge units, and edges represent the logical connections between knowledge units). Knowledge graphs rely on graph database storage to support efficient knowledge retrieval and management. However, they also require dynamic update capabilities to insert new knowledge units in real time and ensure the integrity of newly inserted knowledge units with the original logic of the graph through consistency checks, enabling flexible expansion and reliable maintenance of the knowledge graph.

[0120] In one implementation, step 203 may specifically include the following steps 2031 to 2035 .

[0121] In step 2031, one or more associated multimodal feature representations are combined into a knowledge unit, which is represented as a node in the knowledge graph. Each node includes a unique identifier, a question text, and its corresponding multimodal feature representation: image nodes must include color and shape attributes, audio nodes must store emotional features, and video nodes must associate keyframes with text nodes.

[0122] Step 2032: Define edges in the knowledge graph based on the association or logical relationship between nodes. Edges represent relationships between knowledge units, including similarity and causal relationships.

[0123] Step 2033: Build a knowledge graph based on nodes and edges.

[0124] Among them, the knowledge graph is stored in a graph database, and the node and edge data structure of the graph database is used to manage and retrieve knowledge units.

[0125] Step 2034: Create an index in the knowledge graph for query.

[0126] Among them, indexes can include full-text indexes, attribute indexes, and geographic indexes.

[0127] Step 2035: Verify the constructed knowledge graph through query testing and consistency checking.

[0128] Step 2036, dynamic update and maintenance of knowledge graph.

[0129] Among them, the dynamic update and maintenance of the knowledge graph adopts a modular pipeline architecture, which ensures the balance between throughput and data consistency in high-concurrency scenarios through multi-source heterogeneous data streaming processing, memory-level write acceleration, conflict layering resolution mechanism and rule dynamic expansion interface.

[0130] In one implementation, step 2036 may specifically include the following steps 20361 to 20364.

[0131] Step 20361: Multi-source data access and standardization. Use the Kafka message queue to centrally access multi-source data. Standardize and pre-process structured, semi-structured, and unstructured data streams, and attach metadata (such as source authority, timestamp, and confidence level) to each piece of data.

[0132] Because unstructured multimodal data (such as text, speech, images, and video) lacks a fixed format, it must be converted into structured or semi-structured forms through natural language processing (NLP) technology before further analysis. To this end, an independent model specifically for preprocessing is introduced, integrating basic steps such as word segmentation, denoising, syntactic analysis, and named entity recognition (NER). It should be noted that moving the NER task to the preprocessing stage allows unstructured data to be converted into semantically rich structured information at an early stage, thereby improving the efficiency and effectiveness of subsequent NLP tasks. For example, NER results can provide richer features for downstream tasks such as classification and summarization, and can also be directly used as graph entity node input.

[0133] Step 20362: Streaming knowledge extraction. Parse the data stream in real time to extract entity, relationship, and attribute triples. This includes entity recognition (part of this work has already been pre-placed in step 20361), relationship extraction, and attribute normalization.

[0134] Step 20363: Optimize incremental update transactions. Graph databases face performance bottlenecks in high-concurrency write scenarios and must strictly enforce ACID properties. To this end, RedisGraph (an in-memory graph database) is used as a buffer layer, combined with asynchronous checksums, Write-Ahead Logging (WAL), and RedLock distributed locks to create a hybrid architecture that balances performance and consistency. RedisGraph achieves μs-level write acceleration, trading space for time. Complex constraint checks are decoupled as asynchronous tasks to ensure a non-blocking main process. Transaction rollbacks are implemented based on WAL logs, and distributed locks ensure atomicity, providing the system with robust fault tolerance.

[0135] Step 20364: Conflict detection and resolution. To achieve data consistency during the dynamic update of the knowledge graph, a multi-layer cascade conflict management framework is built, forming a complete closed loop from real-time monitoring, rule engine interception, confidence scoring decision-making to historical tracing, including: Real-time conflict detection: Based on the lightweight characteristics of RedisGraph memory graph database, a two-layer interception barrier is built during the write phase, including: 1) Attribute conflict interception. The unique constraint index is used to achieve instant deduplication of node primary keys and key attributes. 2) Relationship conflict verification. For bidirectional semantic relationships ("A B)) Establish a reverse link verification mechanism. Write "A B" relationship, automatically triggers reverse query, if no "B A" corresponds to a relationship and the type is marked as requiring a bidirectional constraint, a conflict event log is generated and transaction submission is suspended. Relationship type constraint policies are dynamically loaded through the metadata management module, supporting flexible configuration of one-way / bidirectional validation switches based on business scenarios.

[0136] Hybrid mitigation strategy: Adopting a hierarchical decision-making structure of "hard rules first, soft strategies as a backup" to balance the rigor and flexibility of knowledge graph updates, including: The first layer: the rule engine layer. Predefined rules are used to intercept conflicts that violate basic logic and achieve rapid filtering of hard conflicts. Drools is mainly used to load predefined DRL (Decision Rules Language) rules (sorted by priority), and dynamic configuration of rule priorities is supported. The Rete algorithm is used to quickly match conflict patterns. A short-circuit evaluation strategy is introduced in the rule matching stage. When a rule hits the conflict condition, the process is terminated directly. To prevent performance degradation caused by the expansion of the rule set, a 500ms timeout mechanism is set for single rule execution. The timeout event triggers an alarm and transfers it to the downstream module for processing. For conflicts that are not intercepted, enter the second layer.

[0137] The second layer: confidence scoring decision layer. An interpretable quantitative scoring model is constructed to resolve complex conflicts in the gray area. Scoring dimensions include temporal freshness, data distribution, source authority, context consistency, semantic similarity, etc., including: (1) Real-time scoring: Real-time decisions use static weights to ensure response speed. In one embodiment, after setting weights (time freshness 30%, data distribution 25%, source authority 20%, context consistency 15%, semantic similarity 10%), the update strategy (adoption / discarding / manual review) is decided based on the score and threshold interval. (2) Dynamic weight adaptation: The offline module analyzes historical conflict resolution records every day and optimizes the weight distribution of each dimension. For example, based on the Newton's law of cooling modeling data freshness, update operations within the past 3 days receive a 40% weight of the baseline score, which decays exponentially over time.

[0138] The third layer: the multi-version management layer. This layer retains historical versions of pending conflicts and marks the conflict context. Using the nested model of MongoDB's document-based database, each conflict instance is stored as a separate document containing a snapshot of the original data, a conflict resolution decision tree, operation fingerprints, and microsecond timestamps. Composite indexes are created by pre-aggregating fields (such as the conflict type code) to keep query latency for combined "time range + conflict category" queries under 10ms.

[0139] It's important to note that when a new conflict type is added, the rule template engine automatically generates a DRL rule skeleton. After manual review, it's inserted into the priority queue, and the rule engine hot-loads the new policy without restarting the service. Confidence scoring supports plug-in extensions, with new evaluation factors dynamically injected as JAR packages. The scoring pipeline automatically identifies new dimensions and assigns initial weights. In summary, the aforementioned conflict detection and resolution methods not only avoid the trust issues caused by the "black box model" of graph database updates, but also provide elastic adaptability for complex business scenarios.

[0140] Step 204: Based on step 203, the big model is optimized based on "knowledge graph constraints-big model adaptation-result calibration" to achieve deep collaboration between knowledge graph drive and big model capabilities.

[0141] In one implementation, step 204 may specifically include the following steps 2041 to 2043 .

[0142] Step 2041: Dynamic Knowledge Injection (Preliminary Foundation). First, a structured prompt engine is constructed to encode the input raw knowledge graph relationships into structured prompt templates, providing prior knowledge constraints for the subsequent large model to generate results corresponding to the question data. The engine also provides incremental prompt template generation, triggering local template updates when changes to the graph structure are detected. Next, dynamic path pruning is implemented, aiming to filter the main paths most relevant to the current query through semantic similarity, reducing invalid search paths in the knowledge graph. The TransE model is used to encode multi-hop relationship paths into computable vectors, and pruning is implemented based on dynamic thresholds.

[0143] In one implementation, step 2041 may specifically include the following steps 20411 to 20414.

[0144] Step 20411, path sampling and representation.

[0145] The path is defined as: given a query entity and the target entity ,path Defined as a sequence ,in For relationships, is the intermediate entity. Path embedding generation: Use the TransE model to vectorize the path:

[0146] Input: Embedding vectors of all relations in the path (obtained through TransE pre-training); Output: path vector , representing the global semantics of the path.

[0147] Step 20412, similarity calculation. Query vectorization: convert user query Mapped to vectors via a lightweight text encoder . Similarity metric: Calculate path vector With query vector Cosine similarity of:

[0148] Step 20413, dynamic threshold pruning.

[0149] Threshold initialization: set the initial threshold , the branch paths below the threshold are pruned, and only path.

[0150] Dynamic threshold adjustment strategy: Real-time optimization of threshold based on sliding window statistics. The window is defined as the record of the most recent Similarity distribution of the reasoning task; then update the threshold:

[0151] in is the mean similarity within the window, is the standard deviation, is the attenuation coefficient (the empirical value is 1.5~2.0).

[0152] Step 20414, path importance enhancement.

[0153] Semantic weight correction: add the semantic importance weight of the path itself to the similarity score :

[0154] in, Calculation method: Take the average PageRank value of the entities in the path.

[0155] In this embodiment, the TransE embedding dimension is 128; the sliding window size The value is adjusted dynamically, the interval is [50,200]; the attenuation coefficient is 1.6; semantic weight coefficient Take 0.5.

[0156] Step 2042: Efficiently fine-tune the lightweight parameters of the large model (large model adaptation). Low-rank adaptation (LoRA) technology is used to perform low-rank decomposition of key attention heads, specifically enhancing the large model's sensitivity to structured cues. Integrating low-rank adaptation technology to achieve efficient parameter updates includes: The LoRA module selects attention heads based on gradient magnitude and importance scores for the attention layer. A low-rank decomposition with an initial LoRA rank of r=8 is performed on the top 20% of key attention heads. A dynamic importance reassessment mechanism is then introduced to recalculate the importance of attention heads every N training steps, eliminating low-contributing heads and activating new candidates. When the importance of attention heads is dynamically adjusted, a flexible LoRA rank assignment strategy is implemented. The discretized rank assignment rule is: r=12 for the top 10% of heads; r=8 for the middle 70% of heads; and r=4 for the bottom 20% of heads.

[0157] Adapter component: To reduce the complexity of large model processing while maintaining model performance, a 128-dimensional bottleneck layer is inserted between the Transformer feedforward network and the GELU activation function is used.

[0158] Through fine-tuning, the structured prompts provided by dynamic knowledge injection are aligned with the model parameters of the large model, so that the large model pays more attention to the attention area guided by the knowledge graph. This step can achieve efficient alignment of the large model capabilities with the knowledge graph with a small amount of training data, avoiding excessive computing power overhead caused by fine-tuning all model parameters. In one embodiment, combined with 8-bit quantization technology, the above method can achieve resource optimization effects such as parameter reduction (reducing fine-tuning parameters by 72%), video memory compression (reducing memory usage from 120GB to 30GB), and training acceleration (reducing fine-tuning time by 70%, based on A100 80GB GPU testing) while retaining the general capabilities of the model.

[0159] Step 2043, feedback optimization of the large model.

[0160] Among them, the problem data is processed by the large model after fine-tuning the model parameters to generate results corresponding to the problem data, and then a reverse optimization link is constructed based on the results corresponding to the problem data to optimize the large model after fine-tuning the model parameters to achieve iterative enhancement of the large model.

[0161] Among them, the results corresponding to the problem data generated by the large model are judged by the cross entropy loss function and the reward function, and the large model is optimized based on the judgment results.

[0162] Among them, the cross entropy loss function is defined as: Where N is the total number of input question data, that is, the number of results corresponding to the question data. It is The true answer corresponding to the question data in the updated knowledge graph. It is the result corresponding to the problem data generated by the large model.

[0163] Optionally, in order to prevent the large model from overfitting, L2 regularization technology is used, that is, a regularization term is added to the above cross entropy loss function to limit the model parameters of the large model. As shown below: .in, is the cross entropy loss function mentioned above, is the regularization coefficient, is the total number of model parameters of the large model; For the model parameters.

[0164] In the embodiment of the present application, feedback optimization of large models is achieved by calibrating the results through reinforcement learning feedback constrained by knowledge graphs. Specifically, a dual-channel reward function is designed to suppress model "hallucinations" and force the output to be consistent with the knowledge graph logic, including: (1) Structural incentives: SPARQL queries with timing constraints are executed to verify logical consistency. A cache mechanism is established for SPARQL query results, and query depth restrictions are implemented. The initial query hop count = 3. If the number of returned entities is less than the threshold, the hop count is gradually increased, and the upper limit is set to 8. (2) Anti-hallucination constraints: Dynamic penalty terms are constructed through adversarial negative sampling to suppress incorrect reasoning. The positive and negative sample ratio adopts a curriculum learning strategy, and the complexity of negative samples is gradually increased with each training round, starting from 1:3 and ending at 1:6.

[0165] Step 205 , fine-tuning the model parameters and optimizing the hyperparameters of the large model.

[0166] Cross-validation is used to evaluate the performance of large models, using metrics such as accuracy, recall, F1 score, and response time. Grid search or Bayesian optimization are then used to fine-tune the model's hyperparameters.

[0167] In one implementation, step 205 may specifically include the following steps 2051 to 2054 .

[0168] Step 2051: Obtain candidate hyperparameter combinations from a predefined hyperparameter space.

[0169] Step 2052 , by dividing the training data set into K non-overlapping subsets, and selecting one of the subsets as the validation set and the remaining subsets as the training set.

[0170] Step 2053: In each cross-validation, the large model is trained using the training set, and the performance of the large model is evaluated using the validation set. The evaluation indicators of each validation are recorded, including accuracy, recall, F1 value, and response time.

[0171] Step 2054: Determine the target hyperparameter combination from the candidate hyperparameter combinations based on the average performance index. The calculation of the average performance index is as follows: Metric , where Metric is the average value of the performance index; is the number of cross-validation folds; Metric For the Folding performance indicators.

[0172] Step 206: Customize the functions of the large model according to the specific needs of the enterprise.

[0173] In one implementation, step 206 may specifically include the following steps 2061 to 2064 .

[0174] Step 2061: Analyze the specific needs of the enterprise and collect feedback from various departments and users to identify the fields and knowledge units that need to be added. Step 2062: Based on the results of the needs analysis, define the knowledge units of the newly added fields. Each knowledge unit contains a question text, an answer text, and its corresponding multimodal feature representation. Step 2063: Annotate the newly added knowledge units. The annotation process includes proofreading the question text and answer text, as well as extracting and annotating multimodal features. Step 2064: Add the newly added knowledge units to the existing knowledge base, use graph database technology to insert new nodes and edges, and update the knowledge graph based on the relationship and similarity between nodes.

[0175] Step 207: deploy the optimized large model to the enterprise's actual application environment.

[0176] Deployment includes configuring servers, developing interfaces, and ensuring seamless integration with the company's existing systems.

[0177] In one implementation, step 207 may specifically include the following steps 2071 to 2074 .

[0178] Step 2071: Configure the target deployment environment based on the enterprise's specific needs, including server hardware, operating system, network environment, and required software libraries and frameworks. Step 2072: Export the large model after fine-tuning and optimizing model parameters, convert and optimize it into a format suitable for the actual application environment, such as ONNX or TensorFlow Lite. Step 2073: Develop an application programming interface (API) for enterprise systems to call the large model. Interface development includes defining request and response formats, implementing interface logic, and performing security authentication and permission control. Step 2074: Deploy the converted large model to the target server.

[0179] In the examples of this application, we first comprehensively collect and meticulously preprocess multimodal data, including unstructured documents, images, audio, and video from both within and outside the enterprise. This covers multiple data sources, breaking the limitations of traditional reliance on text data and providing a rich information foundation for the model. Furthermore, through unified ETL operations and standardized processing, we standardize data formats, eliminate noise, and ensure data consistency, laying a solid foundation for subsequent feature extraction and fusion.

[0180] Secondly, deep feature extraction from multimodal data is performed using techniques such as convolutional neural networks, long short-term memory networks, and Mel-frequency cepstral coefficients. By setting appropriate parameters (for example, setting 8 or 16 attention heads in a multi-head attention mechanism to project the input multimodal feature vector into a low-dimensional subspace), the multi-head attention mechanism adaptively weights and integrates the features to generate a unified multimodal feature representation. This mechanism simultaneously focuses on different aspects of features from different modalities, capturing cross-modal correlations through parallel subspaces and fully exploring potential connections between data from different modalities. A knowledge graph is constructed based on the multimodal feature representation. Each knowledge unit, as a node, contains a unique identifier, question text, multimodal feature representations, and attribute information specific to image, audio, and video nodes. Logical connections between knowledge units, such as similarity and causal relationships, are represented by edges. A graph database is used to store and establish full-text and attribute information, supporting efficient knowledge retrieval and management. The knowledge graph supports the real-time insertion of new knowledge units. Upon insertion, a consistency check process is initiated to verify that the relationships between the new node and existing nodes conform to logical rules and avoid logical conflicts. At the same time, relevant indexes are updated to ensure that the knowledge graph can promptly reflect the latest enterprise knowledge information, maintain logical integrity and efficient retrieval performance. The consistency check process and index update mechanism of the dynamic update of the knowledge graph ensure the real-time and accuracy of the knowledge graph.

[0181] Furthermore, large models are fine-tuned and optimized. A closed-loop design of "dynamic knowledge injection → efficient parameter fine-tuning → reinforcement learning feedback" is proposed, achieving breakthroughs in knowledge real-time, model lightweightness, and result controllability. This addresses the core issues of traditional methods, such as high resource consumption, update lag, and poor logical consistency. By encoding knowledge graph relationships as dynamic templates, combined with TransE path vectorization and dynamic threshold pruning, invalid search paths are reduced, enabling real-time knowledge injection and optimized inference efficiency, effectively resolving the contradiction between knowledge dynamics and template statics. Leveraging elastic LoRA+Adapter technology, parameter efficiency is balanced with knowledge alignment capabilities, enabling collaborative optimization of knowledge prompt templates and models, breaking through the computing bottleneck of fine-tuning and effectively addressing the imbalance between resources and results in fine-tuning large speech models. A reinforcement learning reverse optimization chain is established. Through a dual-channel reward mechanism combining "structural + adversarial constraints," this improves the consistency of generated results with knowledge logic and significantly mitigates the "hallucination" problem of generated results.

[0182] Finally, based on the company's specific needs, by collecting feedback from various departments and users, we analyze and identify new domains and knowledge units, define and annotate them, and then add them to the existing knowledge graph. This knowledge graph is then updated to improve the model's fit and applicability with the company's business. We rationally configure server hardware, operating systems, network environments, and software dependency libraries and frameworks. After fine-tuning and optimizing model parameters, we export and convert the large model into a new format. We develop application programming interfaces (APIs), perform security authentication, and control permissions to ensure seamless integration of the large model with the company's existing systems, providing efficient knowledge services.

[0183] It should be noted that the execution entity of the knowledge graph-based large-model intelligent question-answering system optimization method provided in the embodiment of the present application can be a knowledge graph-based large-model intelligent question-answering system optimization device, or a control module in the knowledge graph-based large-model intelligent question-answering system optimization device for executing the knowledge graph-based large-model intelligent question-answering system optimization method. In the embodiment of the present application, the execution method of the knowledge graph-based large-model intelligent question-answering system optimization device is taken as an example to illustrate the knowledge graph-based large-model intelligent question-answering system optimization device provided in the embodiment of the present application.

[0184] Figure 3 The schematic diagram of the structure of a large model intelligent question answering system optimization device based on knowledge graph provided by the embodiment of the present application is shown. Figure 3 As shown, the device 300 includes: an acquisition module 31, a knowledge graph update module 32, a fine-tuning module 33, and an optimization module 34.

[0185] Among them, the acquisition module 31 is used to acquire the newly added multimodal data in the target field; the knowledge graph update module 32 is used to perform consistency check on the newly added multimodal data and the historical knowledge graph, and update the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph; the fine-tuning module 33 is used to fine-tune the model parameters of the large model based on the updated knowledge graph; the optimization module 34 is used to generate results corresponding to the problem data through the large model after fine-tuning the model parameters, construct a reverse optimization link based on the results corresponding to the problem data, and optimize the large model after fine-tuning the model parameters.

[0186] In one implementation, the fine-tuning module 33 may be configured to encode the updated knowledge graph into a structured prompt template, and align the model parameters of the large model according to the structured prompt template.

[0187] In one implementation, the fine-tuning module 33 can be specifically used to: generate various path vectors between the query entity and the target entity based on the knowledge entities in the updated knowledge graph and the relationship between the knowledge entities; wherein the knowledge entities include the query entity, the target entity and the intermediate entities from the query entity to the target entity; prune the large model based on the similarity between the query vector corresponding to the problem data and the various path vectors and the dynamic similarity threshold; perform importance evaluation on the attention layer of the large model after pruning to obtain the importance value of each attention head; perform low-rank decomposition processing according to the importance value of each attention head according to the low-rank adaptation strategy corresponding to the importance value to obtain a large model in which the structure of the model parameters matches the structured prompt template.

[0188] In one implementation, the optimization module 34 can be used to: input the problem data into the large model after fine-tuning the model parameters for processing, and generate the results corresponding to the problem data according to the structured prompt template; determine the cross-entropy loss function and reward function of the large model after fine-tuning the model parameters based on the results corresponding to the problem data and the corresponding multimodal feature representation in the updated knowledge graph; optimize the model parameters of the large model after fine-tuning the model parameters according to the cross-entropy loss function and the reward function until the preset convergence conditions are met to obtain the target large model.

[0189] In one implementation, the above-mentioned device 300 may also include an adjustment module for obtaining candidate hyperparameter combinations of the target large model from a predefined hyperparameter space; performing performance evaluation on the target large model configured with the candidate hyperparameter combination through cross-validation to determine the evaluation index of the target large model; and determining the target hyperparameter combination from the candidate hyperparameter combinations based on the evaluation index.

[0190] In one implementation, the knowledge graph update module 32 can be used to extract the knowledge entities, knowledge attributes, and the relationship between the newly added multimodal data and the historical knowledge graph; when the knowledge entities corresponding to the newly added multimodal data are the same as the knowledge entities in the historical knowledge graph, or the knowledge attributes corresponding to the newly added multimodal data are the same as the knowledge attributes in the historical knowledge graph, the knowledge entities corresponding to the newly added multimodal data or the knowledge attributes corresponding to the newly added multimodal data are deduplicated; when the relationship between the newly added multimodal data and the historical knowledge graph conflicts with the basic logic, the newly added multimodal data is intercepted; when the relationship between the newly added multimodal data and the historical knowledge graph is consistent with the basic logic, the newly added multimodal data is adopted or discarded or manually reviewed according to the confidence score threshold to which the confidence score of the newly added multimodal data belongs.

[0191] In one implementation, the above-mentioned device 300 may also include a construction module for obtaining multimodal data of the target field; extracting the corresponding multiple features through the corresponding feature extraction network; splicing the multiple features according to the dimensions corresponding to the features through a multi-head attention mechanism to obtain at least one multimodal feature representation; and constructing the historical knowledge graph based on at least one multimodal feature representation and the relationship between the multimodal features.

[0192] The knowledge graph-based large-model intelligent question-answering system optimization device in the embodiments of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. Non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PC), televisions (TV), ATMs, or self-service machines, etc., which are not specifically limited in the embodiments of the present application. The knowledge graph-based large-model intelligent question-answering system optimization device in the embodiments of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application. The knowledge graph-based large-model intelligent question-answering system optimization device provided in the embodiments of the present application can implement the various processes implemented in the method embodiments described in the figures. To avoid repetition, they are not further described here.

[0193] Figure 4 A schematic diagram of the structure of a large-scale intelligent question-answering system based on a knowledge graph provided in an embodiment of the present application is shown. Figure 4 , the system 400 includes the following modules.

[0194] The data acquisition and processing module 41 is used to collect multimodal text data from inside and outside the enterprise, and the text data includes text, pictures, audio, and video formats, and to denoise and standardize the collected text data. The multimodal feature extraction module 42 is used to extract image features using a convolutional neural network, text features using a long short-term memory network, audio features using Mel-frequency cepstral coefficients, and video features using a three-dimensional convolutional network based on the preprocessed multimodal data provided by the data acquisition and preprocessing module, and all extracted features are fused through a multi-head attention mechanism to generate a unified multimodal feature representation. The knowledge graph construction module 43 is used to build an enterprise-specific knowledge graph based on the multimodal feature representation generated by the multimodal feature extraction module. And the knowledge graph is updated according to the newly added multimodal data. The hybrid fine-tuning module 44 is used to fine-tune and optimize the model parameters of the large model. The model deployment integration module 45 is used to deploy the optimized large model into the actual application environment of the enterprise, including configuring the deployment environment, exporting and converting the model format, developing interfaces for system calls, configuring load balancing and fault tolerance mechanisms, and performing system integration.

[0195] The large-model intelligent question-answering system based on the knowledge graph provided by the embodiment of the present application realizes the collection and preprocessing of multimodal data, the construction and dynamic update of the knowledge graph, which not only ensures that the knowledge graph in the system can be updated in real time and holds the latest knowledge in the target field, but also provides a more accurate basis for the subsequent generation of results corresponding to the question data. And based on the knowledge graph, the model parameters of the large model are fine-tuned and optimized to achieve the lightweight of the large model, improve the efficiency of result generation and reduce resource consumption, ensure the adaptability and logical consistency of the generated results with the target field, suppress the hallucination of the generated results, and realize the comprehensive optimization of the large-model intelligent question-answering system. In addition, the large model can be deployed in different application environments to provide users or enterprises with large models that are suitable for different fields, have high accuracy, good field adaptability and logical consistency, so as to efficiently realize intelligent question and answer.

[0196] Based on the same technical concept, an embodiment of the present application further provides an electronic device for executing the above-mentioned large-model intelligent question-answering system optimization method based on knowledge graph. Figure 5A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. Electronic devices may have relatively large differences due to different configurations or performances, and may include a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504. The processor 501 can call a computer program stored in the memory 503 and can run on the processor 501 to execute the various steps of the above-mentioned embodiment of the large-model intelligent question-answering system optimization method based on the knowledge graph, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0197] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices other than terminals, such as cars, robots, and handheld devices.

[0198] The above electronic device structure does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components or arrange them differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured as a display panel in the form of a liquid crystal display, organic light-emitting diode, or the like. The user input unit includes at least one of a touch panel and other input devices. A touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons and on / off buttons), a trackball, a mouse, and a joystick, which are not described in detail here. Memory can be used to store software programs and various data. The memory may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store an operating system, applications or instructions required for at least one function (such as sound playback or image playback), etc. Furthermore, the memory may include volatile memory or non-volatile memory, or it may include both volatile and non-volatile memory. Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DRRAM). The processor may include one or more processing units; optionally, the processor may integrate an application processor and a modem processor, wherein the application processor primarily handles operations related to the operating system, user interface, and application programs, and the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into the processor.

[0199] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by the processor, each process of the above-mentioned embodiment of the large-model intelligent question-answering system optimization method based on the knowledge graph is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0200] The present application also provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned large-model intelligent question-answering system optimization method based on the knowledge graph, and can achieve the same technical effect. To avoid repetition, it will not be repeated here. It should be understood that the chip mentioned in the embodiment of the present application can also be called a system-level chip, a system chip, a chip system, or a system-on-chip chip, etc.

[0201] An embodiment of the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes a program or instruction. When the program or instruction is executed, the various processes of the above-mentioned embodiment of the large-model intelligent question-answering system optimization method based on the knowledge graph are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0202] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0203] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0204] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A large-model intelligent question-answering system optimization method based on knowledge graph, characterized in that: include: Acquire new multimodal data in the target domain; Performing a consistency check on the newly added multimodal data and the historical knowledge graph, and updating the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph; Fine-tuning model parameters of the large model based on the updated knowledge graph; The large model after fine-tuning the model parameters is used to generate results corresponding to the problem data, and a reverse optimization link is constructed based on the results corresponding to the problem data to optimize the large model after fine-tuning the model parameters.

2. The method according to claim 1, characterized in that Fine-tuning the model parameters of the large model based on the updated knowledge graph includes: Encoding the updated knowledge graph into a structured prompt template; The model parameters of the large model are aligned according to the structured prompt template.

3. The method according to claim 2, characterized in that The aligning of the model parameters of the large model according to the structured prompt template includes: Generate path vectors between a query entity and a target entity based on the knowledge entities in the updated knowledge graph and the relationships between the knowledge entities; wherein the knowledge entities include the query entity, the target entity, and intermediate entities between the query entity and the target entity; Pruning the large model according to the similarity between the query vector corresponding to the question data and each path vector and a dynamic similarity threshold; Performing importance evaluation on the attention layer of the large model after pruning to obtain the importance value of each attention head; According to the importance values of the respective attention heads, low-rank decomposition processing is performed according to a low-rank adaptation strategy corresponding to the importance values, so as to obtain a large model whose structure of the model parameters matches the structured prompt template.

4. The method according to claim 2, characterized in that The large model after fine-tuning the model parameters generates a result corresponding to the problem data, and a reverse optimization link is constructed based on the result corresponding to the problem data to optimize the large model after fine-tuning the model parameters, including: Inputting the question data into the large model after fine-tuning the model parameters for processing, and generating a result corresponding to the question data according to the structured prompt template; Determine the cross entropy loss function and reward function of the large model after fine-tuning the model parameters according to the result corresponding to the question data and the corresponding multimodal feature representation in the updated knowledge graph; The large model after fine-tuning the model parameters is optimized according to the cross entropy loss function and the reward function until the preset convergence conditions are met to obtain the target large model.

5. The method according to claim 4, characterized in that After obtaining the target large model, the method further includes: Obtaining a candidate hyperparameter combination of the target large model from a predefined hyperparameter space; Performing a performance evaluation on the target large model configured with the candidate hyperparameter combination through cross-validation to determine an evaluation index of the target large model; A target hyperparameter combination is determined from the candidate hyperparameter combinations according to the evaluation index.

6. The method according to claim 1, characterized in that The consistency check of the newly added multimodal data and the historical knowledge graph is performed, and the historical knowledge graph is updated using a hybrid resolution strategy according to the result of the consistency check to obtain an updated knowledge graph, including: Extracting the knowledge entities and knowledge attributes corresponding to the newly added multimodal data, and the relationship between the knowledge entities and the historical knowledge graph; If the knowledge entity corresponding to the newly added multimodal data is the same as the knowledge entity in the historical knowledge graph, or the knowledge attribute corresponding to the newly added multimodal data is the same as the knowledge attribute in the historical knowledge graph, deduplication processing is performed on the knowledge entity corresponding to the newly added multimodal data or the knowledge attribute corresponding to the newly added multimodal data; If the relationship between the newly added multimodal data and the historical knowledge graph conflicts with the basic logic, intercepting the newly added multimodal data; When the relationship between the newly added multimodal data and the historical knowledge graph is consistent with the basic logic, the newly added multimodal data is adopted, discarded, or manually reviewed according to the confidence score threshold to which the confidence score of the newly added multimodal data belongs.

7. The method according to claim 1, characterized in that The method further comprises: Acquiring multimodal data of the target domain; Extracting a plurality of features respectively corresponding to the multimodal data through a feature extraction network corresponding to the multimodal data; splicing the multiple features according to the dimensions corresponding to the features through a multi-head attention mechanism to obtain at least one multimodal feature representation; The historical knowledge graph is constructed based on at least one of the multimodal feature representations and the relationship between the multimodal features.

8. A large-model intelligent question-answering system optimization device based on knowledge graph, characterized in that: include: The acquisition module is used to obtain new multimodal data in the target field; A knowledge graph update module is used to perform a consistency check on the newly added multimodal data and the historical knowledge graph, and to update the historical knowledge graph using a hybrid resolution strategy based on the result of the consistency check to obtain an updated knowledge graph; A fine-tuning module, configured to fine-tune model parameters of the large model based on the updated knowledge graph; The optimization module is used to generate results corresponding to the problem data through the large model after fine-tuning the model parameters, build a reverse optimization link based on the results corresponding to the problem data, and optimize the large model after fine-tuning the model parameters.

9. An electronic device, characterized in that: It includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the method for optimizing a large-model intelligent question-answering system based on a knowledge graph are implemented as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the large-model intelligent question-answering system optimization method based on the knowledge graph are implemented as described in any one of claims 1 to 7.

Images