Multi-layer semantic understanding large model agent construction and application method

Abstract

The invention discloses a multi-layer semantic understanding large model agent construction and application method. The method comprises the following steps: S1, preprocessing input data; s2, analyzing the preprocessed data by using a large language model and dialogue history, identifying user intentions and extracting deep semantic features; s3, key information is extracted from data intending to be recognized, structured representation is carried out, and model configuration is adjusted through dynamic hyper-parameter optimization; s4, querying the knowledge graph, associating and organizing the extracted key information with the existing data, and generating a structured knowledge representation; s5, vectorizing the structured knowledge representation, and storing the vectorized structured knowledge representation in a vector database to support rapid similarity search; and S6, in combination with the generated unified semantic representation and the large model, performing deep analysis on user input, generating coherent and logical personalized answers, and returning the personalized answers to the user. The method can support various application scenes such as intelligent question and answer, content generation, sentiment analysis and multi-modal search.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-layer semantic understanding large model intelligent agent construction and application method. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the continuous progress in natural language processing, computer vision and audio processing, how to achieve high-precision semantic understanding and fusion in multimodal data has become a hot topic and difficulty in current research. Traditional single-modality processing methods can no longer meet the increasingly complex application needs, and multimodal fusion technology has emerged to achieve a comprehensive understanding of complex scenes by integrating information from multiple modalities such as text, images, and audio. However, existing technologies still have many shortcomings in the fusion and semantic understanding of multimodal data.

[0003] Although the existing technologies have made certain progress in the fields of multimodal data processing and semantic understanding, there are still many defects. First, when processing multimodal data, the existing technologies usually adopt simple feature splicing or early fusion strategies to directly splice features from different modalities together. This method ignores the deep interactive information between different modalities, resulting in insufficient information fusion depth and failure to fully explore the potential correlations of multimodal data. Many existing technologies rely on fixed model architectures and hyperparameter settings, lack flexibility and adaptability, and are difficult to adapt to changes in different tasks and data distributions. This makes the model show poor generalization ability when facing a variety of practical application scenarios. The fixed model architecture cannot dynamically adjust the hyperparameters to adapt to the new data distribution, resulting in inconsistent performance of the model on different tasks and failure to effectively process complex data in practical applications.

[0004] In intelligent question-answering and dialogue systems, existing technologies often ignore the full use of contextual information and historical dialogues, and the generated answers lack coherence and logic. This makes it difficult for the system to maintain consistent semantic fluency and logical continuity in multiple rounds of dialogue. The answers generated by the dialogue system are often isolated and fail to make full use of contextual information, resulting in answers that are not natural and intelligent enough and cannot meet the personalized needs of users. In the process of information extraction and semantic representation, existing technologies often have problems of information loss and insufficient expression. This is because traditional methods mostly rely on rule matching and shallow feature extraction, and cannot capture deep semantic relationships and complex entity associations. When faced with changeable and complex practical applications, rule matching methods seem to be powerless and difficult to process large amounts of unstructured data, resulting in the accuracy and comprehensiveness of information extraction being affected.

[0005] In terms of storage and retrieval of high-dimensional vectors, existing technologies often show inefficiency. Traditional databases and storage technologies are difficult to efficiently process large-scale high-dimensional vector data, resulting in slow similarity search and retrieval speeds, which cannot meet the needs of real-time applications. The storage and retrieval of high-dimensional data require more efficient indexing and optimization technologies. The research and application of existing methods in this regard are relatively lagging, affecting the overall performance of the system. In content generation and question-answering systems, the answers generated by existing technologies often lack depth and relevance, and cannot provide valuable and meaningful content. This is mainly due to the lack of understanding of deep semantics and the effective fusion of multimodal information, and the generated content is difficult to meet the actual needs of users. The answers generated by existing methods are mostly templated and superficial, lacking a deep understanding of user questions and personalized responses, resulting in a poor user experience.

[0006] Multimodal data often comes from different sources and is heterogeneous. Existing technologies often lack effective fusion and parsing technologies when processing heterogeneous data, which makes it difficult to eliminate the semantic differences between data from different sources. The semantic differences of heterogeneous data affect the unified representation and comprehensive utilization of multimodal data. Existing technologies have limited processing capabilities in this regard and it is difficult to achieve comprehensive and consistent semantic understanding.

[0007] Therefore, how to provide a method for constructing and applying a multi-layer semantic understanding large model intelligent agent is an urgent problem that technicians in this field need to solve. Summary of the invention

[0008] One purpose of the present invention is to propose a multi-layer semantic understanding large model intelligent agent construction and application method. The present invention makes full use of multimodal data fusion, deep learning and semantic understanding technology, and describes in detail the algorithm for realizing high-precision semantic understanding and fusion in multimodal data, and has the following advantages: First, through multi-level semantic feature extraction and deep semantic understanding, the comprehensive processing capability of text, image and audio data is effectively improved; secondly, through dynamic hyperparameter optimization and hybrid neural symbolic reasoning network, the flexibility and adaptability of the model are enhanced; in addition, the present invention also demonstrates wide applicability and high efficiency in various application scenarios such as intelligent question and answer, content generation, sentiment analysis and multimodal search.

[0009] A multi-layer semantic understanding large model intelligent agent construction and application method according to an embodiment of the present invention includes the following steps:

[0010] S1. Preprocess the input data to adapt to the subsequent semantic understanding process. The preprocessing includes data cleaning and annotation, multimodal data alignment and fusion, data enhancement and expansion;

[0011] S2, use semantic understanding agents to analyze preprocessed data, identify user intentions and needs through large language models and conversation history, and use hybrid neural symbolic reasoning networks to extract deep semantic features;

[0012] S3, extract key information from the identified intent data and extracted deep semantic features, use information extraction technology to identify entities, relationships and events in the data, form structured data, and adjust the model hyperparameter configuration to adapt to different data distributions and task requirements through dynamic hyperparameter optimization based on evolutionary strategies, and generate optimized model parameters;

[0013] S4. Based on the extracted key information and optimized model parameters, the knowledge graph is queried to associate and organize the information and generate a structured knowledge representation;

[0014] S5. Vectorize the generated structured knowledge representation and store it in a vector database. Use a unified attention mechanism to weight and fuse different modal features to generate a unified semantic representation.

[0015] S6. Combine the generated unified semantic representation and large model to conduct in-depth analysis of user input, generate relevant answers based on context information and conversation history, apply adaptive context generation algorithms to provide coherent and logical output, and return the results to the user.

[0016] Optionally, the S1 specifically includes:

[0017] S11, perform preliminary processing on the input data to remove noise and redundant information;

[0018] S12, label the data after preliminary processing to provide a data set for model training;

[0019] S13, aligning data from different modalities in time and space, so that text, image and audio data can be processed in a unified time axis and space range;

[0020] S14, using a fusion algorithm to fuse the aligned multimodal data to generate a comprehensive data representation of the multimodal features;

[0021] S15. Generate more training samples based on the generated comprehensive data representation through data expansion technology to enrich the data set;

[0022] S16. On the basis of data expansion, further expand the size of the data set and optimize the model's adaptability under different data distributions by adding diversified data samples.

[0023] Optionally, the S2 specifically includes:

[0024] S21, receiving and processing the generated comprehensive data representation;

[0025] S22. Use a large language model to perform in-depth analysis on the comprehensive data representation and extract deep semantic features;

[0026] S23, combining the conversation history and context information, performing context association analysis on the comprehensive data representation to identify the user's intentions and needs;

[0027] S24. Using a hybrid neural symbolic reasoning network, deep semantic features are reasoned and analyzed to extract high-level semantic information. The hybrid neural symbolic reasoning network combines the representation ability of neural networks and the reasoning ability of symbolic logic:

[0028]

[0029] Among them, P(Q|F) represents the probability of querying Q given feature F, λ i is the weight parameter, f i is the characteristic function, Z is the normalization factor;

[0030] S25, matching the extracted high-level semantic information with the identified user intentions and needs to generate a structured semantic representation;

[0031] S26. Through adaptive knowledge embedding and dynamic logical reasoning mechanism, the structured semantic representation is optimized to generate the final deep semantic features and data representation of user intention.

[0032] Optionally, the S3 specifically includes:

[0033] S31, extracting key information from the identified intent data and the extracted deep semantic features;

[0034] S32. Use information extraction technology to identify entities, relationships and events in the data:

[0035]

[0036] Among them, Score(e i ,r,e j ) represents entity e i and e j The score under relation r, h r Represents the embedding vector of relation r, W r represents the weight matrix of relation r, b r Represents the bias vector of relation r, g(e i ) and g(e j ) represents entity e i and e j The embedding vector ofi ); g(e j )] represents the concatenation operation of entity vectors;

[0037] S33, structurally representing the identified entities, relationships and events to form structured data;

[0038] S34. Use dynamic hyperparameter optimization based on evolution strategy to adjust the model's hyperparameter configuration to adapt to different data distributions and task requirements:

[0039]

[0040] Among them, θ t represents the model parameters of the tth generation, α is the learning rate, β is the regularization parameter, J(θ; z i ) is the model in sample z i The loss function on , N is the total number of samples, and M is the number of samples of the regularization term;

[0041] S35. Generate final optimized model parameters according to the structured data and the optimized model parameters.

[0042] Optionally, the S4 specifically includes:

[0043] S41. According to the extracted key information and the optimized model parameters, query the knowledge graph to obtain data related to the key information;

[0044] S42, associate and organize the newly extracted key information with the information obtained from the knowledge graph to generate a preliminary structured knowledge representation;

[0045] S43. Use semantic similarity calculation and graph matching algorithm to find the connection points between the preliminary structured knowledge representation and the nodes and relationships in the existing knowledge graph:

[0046]

[0047] Among them, Sim(x,y) represents the semantic similarity between two entities x and y, α i represents the weight of the i-th feature, represents the cosine similarity of the i-th feature vector;

[0048] S44. Based on the found association points, semantic fusion of information is performed to unify the information representation from different sources. In the process of semantic fusion, a multi-layer perceptron MLP is used for nonlinear transformation:

[0049]

[0050] Among them, v fused represents the fused semantic vector, βi represents the weight of the i-th information source, v i Represents the semantic vector of the i-th information source;

[0051] S45. Use ontology matching and semantic parsing technology to solve the semantic differences of heterogeneous data:

[0052]

[0053] Among them, Match(O1,O2) represents the matching degree between ontology O1 and ontology O2, C represents the concept set in the ontology, and sim ontology (c1,c1) represents the similarity between concepts c1 and c2, depth(c) represents the depth of concept c in the ontology tree, which is used for weight adjustment;

[0054] S46. Associate the newly extracted information with the nodes and relationships in the existing knowledge graph after parsing and matching to form a unified structured knowledge representation, and use the generated unified structured knowledge representation as the final structured knowledge representation.

[0055] Optionally, the S5 specifically includes:

[0056] S51. Vectorize the generated structured knowledge representation. The vectorization process includes embedding generation, feature extraction, and dimension reduction. Embedding generation uses a pre-trained language model to represent knowledge as a high-dimensional vector:

[0057] v embed =BCE_Embedding(K);

[0058] Among them, v embed A high-dimensional vector representation representing the knowledge K;

[0059] S52, perform feature extraction, and use a unified attention mechanism to weight and fuse features of different modalities to generate a unified semantic representation:

[0060]

[0061] Among them, h unified represents the feature vector after weighted fusion, α l is the weight coefficient in the attention mechanism, W l is the weight matrix of the lth layer, h l is the feature vector of the lth layer, σ is the activation function;

[0062] S53, perform dimensionality reduction, and use multi-level principal component analysis technology to reduce the high-dimensional vector to a low-dimensional vector suitable for storage and retrieval:

[0063]

[0064] Among them, v reduced represents the vector after dimensionality reduction, W i represents the principal component analysis matrix at level i;

[0065] S54, using a vector database to store vectorized knowledge data and optimize the storage and retrieval of high-dimensional vectors to facilitate similarity search and processing;

[0066] S55. Through the unified attention mechanism, weighted fusion of different modal features is performed to generate the final unified semantic representation:

[0067]

[0068] Among them, v final represents the final unified semantic representation, β i Represents the weight coefficient in the attention mechanism, v i represents the vector of different modal features, and σ is the activation function.

[0069] Optionally, the S6 specifically includes:

[0070] S61. Combined with the generated unified semantic representation v final The large model performs in-depth analysis of user input. The analysis process includes comprehensive processing of user input, context information, and conversation history to generate context semantic vectors:

[0071] h context =f Qwen (input,context,history);

[0072] Among them, h context represents the semantic representation that combines user input, context information, and conversation history, f Qwen The semantic vector generation function representing the large model;

[0073] S62, based on h context Generate relevant answers and use the adaptive context generation algorithm for answer generation. The adaptive context generation algorithm uses a combination of dynamic attention mechanism and recurrent neural network to generate high-quality answers:

[0074]

[0075] Among them, r gen represents the generated answer, g context represents the adaptive context generation algorithm, W is the weight matrix, b is the bias vector, α t is the dynamic attention weight, is the hidden state of RNN;

[0076] S63, through the multi-level processing of adaptive context generation algorithm, generate coherent and logical output:

[0077]

[0078] Among them, r final Indicates the final coherent and logical output, α i Represents the weight coefficient in the adaptive context generation algorithm, W i and b i They represent the weight matrix and bias vector of the i-th layer respectively, σ is the activation function, and ELU is the exponential linear unit activation function;

[0079] S64, the final coherent and logical output r final Return to the user, completing the entire process of semantic understanding and response generation.

[0080] The beneficial effects of the present invention are:

[0081] Through multi-level semantic feature extraction and deep semantic understanding, the present invention can deeply understand the semantics of text, image and audio data layer by layer, greatly improving the depth and accuracy of semantic understanding. By using a hybrid neural symbolic reasoning network, the representation ability of neural networks and the reasoning ability of symbolic logic are combined to extract high-level semantic information, effectively solving the problems of information loss and insufficient feature expression. In addition, the dynamic hyperparameter optimization technology based on evolutionary strategies can dynamically adjust the hyperparameter configuration of the model according to different data distributions and task requirements, significantly enhancing the flexibility and adaptability of the model, and improving the generalization ability on different tasks and data sets, solving the problem of poor generalization ability of traditional methods when dealing with changing application scenarios.

[0082] The present invention combines a large model with an adaptive context generation algorithm to perform in-depth analysis of user input, making full use of context information and conversation history to generate personalized and context-related answers. Through the adaptive context generation algorithm, a coherent and logical output is generated, which improves the relevance and naturalness of the generated content and solves the problem of lack of coherence and logic in the answers in the prior art. In addition, through an innovative attention mechanism and a multi-level interactive network, the features of different modalities are weighted and fused to generate a unified semantic representation, ensuring the deep fusion and complementarity of different modal information, and solving the problem of insufficient correlation between modalities in existing methods.

[0083] The present invention uses a vector database to store vectorized knowledge data, optimizes the storage and retrieval efficiency of high-dimensional vectors, and allows for rapid similarity search and efficient processing. This not only improves the storage and retrieval efficiency of the system, but also enhances the performance of real-time applications. Through these innovative technologies, it demonstrates wide applicability and efficiency in a variety of application scenarios such as intelligent question and answer, content generation, sentiment analysis, and multimodal search, significantly improving the semantic understanding and practical application effects of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0085] Figure 1 A flowchart of a multi-layer semantic understanding large model intelligent agent construction and application method proposed by the present invention;

[0086] Figure 2 A flowchart of data preprocessing and semantic understanding in the present invention;

[0087] Figure 3 Flowchart for structured knowledge representation vectorization and response generation in the present invention. DETAILED DESCRIPTION

[0088] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0089] refer to Figure 1-3 , a multi-layer semantic understanding large model intelligent agent construction and application method, comprising the following steps:

[0090] S1. Preprocess the input data to adapt to the subsequent semantic understanding process. The preprocessing includes data cleaning and annotation, multimodal data alignment and fusion, data enhancement and expansion;

[0091] S2, use semantic understanding agents to analyze preprocessed data, identify user intentions and needs through large language models and conversation history, and use hybrid neural symbolic reasoning networks to extract deep semantic features;

[0092] S3, extract key information from the identified intent data and extracted deep semantic features, use information extraction technology to identify entities, relationships and events in the data, form structured data, and adjust the model hyperparameter configuration to adapt to different data distributions and task requirements through dynamic hyperparameter optimization based on evolutionary strategies, and generate optimized model parameters;

[0093] S4. Based on the extracted key information and optimized model parameters, the knowledge graph is queried to associate and organize the information and generate a structured knowledge representation;

[0094] S5. Vectorize the generated structured knowledge representation and store it in a vector database. Use a unified attention mechanism to weight and fuse different modal features to generate a unified semantic representation.

[0095] S6. Combine the generated unified semantic representation and large model to conduct in-depth analysis of user input, generate relevant answers based on context information and conversation history, apply adaptive context generation algorithms to provide coherent and logical output, and return the results to the user.

[0096] In this implementation, S1 specifically includes:

[0097] S11, perform preliminary processing on the input data to remove noise and redundant information;

[0098] S12, label the data after preliminary processing to provide a data set for model training;

[0099] S13, aligning data from different modalities in time and space, so that text, image and audio data can be processed in a unified time axis and space range;

[0100] S14, using a fusion algorithm to fuse the aligned multimodal data to generate a comprehensive data representation of the multimodal features;

[0101] S15. Generate more training samples based on the generated comprehensive data representation through data expansion technology to enrich the data set;

[0102] S16. On the basis of data expansion, further expand the size of the data set and optimize the model's adaptability under different data distributions by adding diversified data samples.

[0103] In this implementation, S2 specifically includes:

[0104] S21, receiving and processing the generated comprehensive data representation;

[0105] S22. Use a large language model to perform in-depth analysis on the comprehensive data representation and extract deep semantic features;

[0106] S23, combining the conversation history and context information, performing context association analysis on the comprehensive data representation to identify the user's intentions and needs;

[0107] S24. Using a hybrid neural symbolic reasoning network, deep semantic features are reasoned and analyzed to extract high-level semantic information. The hybrid neural symbolic reasoning network combines the representation ability of neural networks and the reasoning ability of symbolic logic:

[0108]

[0109] Among them, P(Q|F) represents the probability of querying Q given feature F, λ i is the weight parameter, f i is the characteristic function, Z is the normalization factor;

[0110] S25, matching the extracted high-level semantic information with the identified user intentions and needs to generate a structured semantic representation;

[0111] S26. Through adaptive knowledge embedding and dynamic logical reasoning mechanism, the structured semantic representation is optimized to generate the final deep semantic features and data representation of user intention.

[0112] In this implementation, S3 specifically includes:

[0113] S31, extracting key information from the identified intent data and the extracted deep semantic features;

[0114] S32. Use information extraction technology to identify entities, relationships and events in the data:

[0115]

[0116] Among them, Score(e i ,r,e j ) represents entity e i and e j The score under relation r, h r Represents the embedding vector of relation r, W r represents the weight matrix of relation r, b r Represents the bias vector of relation r, g(e i ) and g(e j ) represents entity e i and e j The embedding vector of i ); g(e j )] represents the concatenation operation of entity vectors;

[0117] S33, structurally representing the identified entities, relationships and events to form structured data;

[0118] S34. Use dynamic hyperparameter optimization based on evolution strategy to adjust the model's hyperparameter configuration to adapt to different data distributions and task requirements:

[0119]

[0120] Among them, θ t represents the model parameters of the tth generation, α is the learning rate, β is the regularization parameter, J(θ; z i ) is the model in sample z i The loss function on , N is the total number of samples, and M is the number of samples of the regularization term;

[0121] S35. Generate final optimized model parameters according to the structured data and the optimized model parameters.

[0122] In this implementation manner, the S4 specifically includes:

[0123] S41. According to the extracted key information and the optimized model parameters, query the knowledge graph to obtain data related to the key information;

[0124] S42, associate and organize the newly extracted key information with the information obtained from the knowledge graph to generate a preliminary structured knowledge representation;

[0125] S43. Use semantic similarity calculation and graph matching algorithm to find the connection points between the preliminary structured knowledge representation and the nodes and relationships in the existing knowledge graph:

[0126]

[0127] Among them, Sim(x,y) represents the semantic similarity between two entities x and y, α i represents the weight of the i-th feature, represents the cosine similarity of the i-th feature vector;

[0128] S44. Based on the found association points, semantic fusion of information is performed to unify the information representation from different sources. In the process of semantic fusion, a multi-layer perceptron MLP is used for nonlinear transformation:

[0129]

[0130] Among them, v fused represents the fused semantic vector, β i represents the weight of the i-th information source, v i Represents the semantic vector of the i-th information source;

[0131] S45. Use ontology matching and semantic parsing technology to solve the semantic differences of heterogeneous data:

[0132]

[0133] Among them, Match(O1,O2) represents the matching degree between ontology O1 and ontology O2, C represents the concept set in the ontology, and sim ontology (c1,c2) ​​represents the similarity between concepts c1 and c2, depth(c) represents the depth of concept c in the ontology tree, which is used for weight adjustment;

[0134] S46. Associate the newly extracted information with the nodes and relationships in the existing knowledge graph after parsing and matching to form a unified structured knowledge representation, and use the generated unified structured knowledge representation as the final structured knowledge representation.

[0135] In this implementation manner, S5 specifically includes:

[0136] S51. Vectorize the generated structured knowledge representation. The vectorization process includes embedding generation, feature extraction, and dimension reduction. Embedding generation uses a pre-trained language model to represent knowledge as a high-dimensional vector:

[0137] v embed =BCE_Embedding(K);

[0138] Among them, v embed A high-dimensional vector representation representing the knowledge K;

[0139] S52, perform feature extraction, and use a unified attention mechanism to weight and fuse features of different modalities to generate a unified semantic representation:

[0140]

[0141] Among them, h unified represents the feature vector after weighted fusion, α l is the weight coefficient in the attention mechanism, W l is the weight matrix of the lth layer, h l is the feature vector of the lth layer, σ is the activation function;

[0142] S53, perform dimensionality reduction, and use multi-level principal component analysis technology to reduce the high-dimensional vector to a low-dimensional vector suitable for storage and retrieval:

[0143]

[0144] Among them, v reduced represents the vector after dimensionality reduction, W i represents the principal component analysis matrix at level i;

[0145] S54, using a vector database to store vectorized knowledge data and optimize the storage and retrieval of high-dimensional vectors to facilitate similarity search and processing;

[0146] S55. Through the unified attention mechanism, weighted fusion of different modal features is performed to generate the final unified semantic representation:

[0147]

[0148] Among them, v final represents the final unified semantic representation, β i Represents the weight coefficient in the attention mechanism, v i represents the vector of different modal features, and σ is the activation function.

[0149] In this implementation manner, S6 specifically includes:

[0150] S61. Combined with the generated unified semantic representation v final The large model performs in-depth analysis of user input. The analysis process includes comprehensive processing of user input, context information, and conversation history to generate context semantic vectors:

[0151] h context =f Qwen (input,context,history);

[0152] Among them, h context represents the semantic representation that combines user input, context information, and conversation history, f Qwen The semantic vector generation function representing the large model;

[0153] S62, based on h context Generate relevant answers and use the adaptive context generation algorithm for answer generation. The adaptive context generation algorithm uses a combination of dynamic attention mechanism and recurrent neural network to generate high-quality answers:

[0154]

[0155] Among them, r gen represents the generated answer, g context represents the adaptive context generation algorithm, W is the weight matrix, b is the bias vector, α t is the dynamic attention weight, is the hidden state of RNN;

[0156] S63, through the multi-level processing of adaptive context generation algorithm, generate coherent and logical output:

[0157]

[0158] Among them, r final Indicates the final coherent and logical output, α i Represents the weight coefficient in the adaptive context generation algorithm, Wi and b i They represent the weight matrix and bias vector of the i-th layer respectively, σ is the activation function, and ELU is the exponential linear unit activation function;

[0159] S64, the final coherent and logical output r final Return to the user, completing the entire process of semantic understanding and response generation.

[0160] Embodiment 1:

[0161] In order to verify the feasibility of the present invention in implementation, the present invention is applied to a specific application process in a medical consultation scenario, aiming to solve the problem of multimodal data fusion and high-precision semantic understanding. The data input by the user is a natural language text, such as "How to get reimbursement for seeing a doctor at the People's Hospital?" First, the system standardizes the input text, including synonym normalization and word form restoration. For example, "see a doctor" can be converted into "see a doctor". This preprocessing helps to eliminate the ambiguity caused by language diversity and improve the accuracy of subsequent semantic understanding.

[0162] The preprocessed text is input into the semantic understanding agent. The agent uses a large language model for analysis, combined with the conversation history and context information, to identify that the user's intention is to consult the medical insurance policy. From the results of semantic understanding, the system extracts key information. In this embodiment, the entity "People's Hospital" is identified using named entity recognition technology. Based on the extracted entities, the system queries the knowledge graph. Since there are many records associated with "People's Hospital" in the existing knowledge graph of medical institutions, the agent will consider the information incomplete and trigger multiple rounds of questions and answers. The agent will combine the relevant records in the knowledge graph and ask the user: "Which hospital do you specifically refer to when you mention the 'People's Hospital', such as xx People's Hospital, xx People's Hospital..." in order to further clarify the user's specific intentions.

[0163] When the user determines the name of the specific medical institution, the system obtains detailed information about the institution from the knowledge graph. For example, after the user enters "xx People's Hospital", the system confirms that the institution is a "tertiary medical institution." Subsequently, the agent updates the previous query processing result to "How to reimburse medical treatment at xx People's Hospital (tertiary medical institution)?" After each step, the agent will evaluate whether the current information is complete enough. If the agent detects that information is missing, it will continue to guide the user to supplement the necessary information. For example, when the "medical insurance type" is missing, the agent will ask: "Do you want to consult about urban employee medical insurance or urban and rural residents' medical insurance?" After the user answered "urban employee medical insurance", the agent found that the information was still incomplete and continued to ask: "The reimbursement benefits of urban employee medical insurance are different due to different personnel categories. The personnel categories of urban employee medical insurance are divided into employed personnel (including flexible employment personnel), retirees and old workers before the founding of the People's Republic of China. Which type of person do you want to consult?"

[0164] If the user answers "employed workers", the agent will think that the information is basically complete. The system processes the current query sentence as "How can employed workers' urban employee medical insurance be reimbursed for medical treatment at xx People's Hospital (a third-level medical institution)?" Next, the system vectorizes the updated query sentence, generates the corresponding vector representation, and stores it in the vector database for fast retrieval. Finally, combined with the medical insurance policy knowledge stored in the vector database, the agent generates the final summary text information and returns it to the user.

[0165] In a pilot application of a large hospital information system, the system can complete the entire process from user input to final response in an average of less than 2 seconds, which is significantly improved compared to the average response time of traditional methods (about 5 seconds). At the same time, the user satisfaction survey shows that more than 90% of users are satisfied with the answers generated by the intelligent agent, and believe that the answers are coherent and logically clear. Specific data show that during the pilot period in a tertiary hospital in Beijing, the system handles about 500 user consultations per day, and the system response accuracy rate reaches more than 95%, greatly improving the hospital's information service efficiency. Through this method of multimodal fusion and high-precision semantic understanding, not only the workload of manual customer service is reduced, but also the user experience is improved. In the future, the system can be extended to more medical institutions and application scenarios to provide more users with efficient and accurate intelligent services. In summary, this embodiment demonstrates the superior performance and wide applicability of the present invention in practical applications, and proves the significant advantages of the present invention in multimodal data processing and semantic understanding.

[0166] Table 1 System overall performance and user satisfaction data table

[0167]

[0168]

[0169] Table 2 Statistics of multi-round question-answering processing

[0170] Type of missing information Number of inquiries User answer accuracy Name of medical institution 150 times 98％ Type of medical insurance 200 times 95％ Personnel Category 150 times 97％

[0171] Table 3 Performance data of each module of the system

[0172] Modules Average processing time Processing accuracy Data preprocessing 0.5 seconds 99％ Semantic Understanding 1 second 95％ Key information extraction 0.3 seconds 97％ Knowledge graph query 0.5 seconds 96％ Structured Knowledge Representation Vectorization 0.2 seconds 98％ Final response generation 0.5 seconds 94％

[0173] According to the above table data summary, the present invention performs well in terms of overall performance and user satisfaction in practical applications. The average response time of the system is less than 2 seconds, the number of consultations processed per day reaches more than 500 times, the accuracy of the system response exceeds 95%, and the user satisfaction exceeds 90%. In the process of multiple rounds of question and answer processing, the number of inquiries for medical institution name, medical insurance type and personnel category were 150 times, 200 times and 150 times respectively, and the accuracy of user answers was 98%, 95% and 97% respectively. In terms of the performance of each module of the system, the average processing time for data preprocessing, semantic understanding, key information extraction, knowledge graph query, structured knowledge representation vectorization and final response generation was 0.5 seconds, 1 second, 0.3 seconds, 0.5 seconds, 0.2 seconds and 0.5 seconds respectively, and the processing accuracy was 99%, 95%, 97%, 96%, 98% and 94% respectively. These data show that the method of the present invention has significant advantages and innovations in multimodal data processing and semantic understanding, can efficiently process information and generate responses, and improve user experience and system performance.

[0174] Through the above-mentioned embodiments, the present invention realizes the whole process from user input to final response generation, and verifies the effectiveness and practicality of the method of the present invention. In practical applications, the processing speed and accuracy of the present invention have been significantly improved. In an embodiment, the natural language text input by the user, such as "How to reimburse for medical treatment in the People's Hospital?" is first standardized, including synonym normalization and word form restoration, to improve the accuracy of subsequent semantic understanding. The preprocessed text is input into the semantic understanding agent, analyzed using a large language model, and combined with the conversation history and context information, it is identified that the user's intention is to consult the medical insurance policy. The system uses named entity recognition technology to extract key information, such as "People's Hospital", and queries the knowledge graph for multiple rounds of questions and answers, and finally confirms the specific medical institution and medical insurance type to generate accurate consultation answers.

[0175] In summary, the present invention achieves deep fusion and high-precision semantic understanding of multimodal data through innovative technologies, and effectively solves the problems existing in the prior art, such as insufficient depth of modal fusion, lack of flexibility and adaptability, insufficient use of context and historical information, information loss and insufficient expression, low storage and retrieval efficiency, and lack of depth and relevance in generated answers. The actual application and data in the embodiments verify the effectiveness and practicality of the method of the present invention, showing its wide applicability and efficiency in various application scenarios such as intelligent question and answer, content generation, sentiment analysis and multimodal search, and providing a more efficient and accurate solution for future intelligent systems.

[0176] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-layer semantic understanding large model intelligent agent construction and application method, characterized in that: The steps include: S1. Preprocess the input data to adapt to the subsequent semantic understanding process. The preprocessing includes data cleaning and annotation, multimodal data alignment and fusion, data enhancement and expansion; S2, use semantic understanding agents to analyze preprocessed data, identify user intentions and needs through large language models and conversation history, and use hybrid neural symbolic reasoning networks to extract deep semantic features; S3, extract key information from the identified intent data and extracted deep semantic features, use information extraction technology to identify entities, relationships and events in the data, form structured data, and adjust the model hyperparameter configuration to adapt to different data distributions and task requirements through dynamic hyperparameter optimization based on evolutionary strategies, and generate optimized model parameters; S4. Based on the extracted key information and optimized model parameters, the knowledge graph is queried to associate and organize the information and generate a structured knowledge representation; S5. Vectorize the generated structured knowledge representation and store it in a vector database. Use a unified attention mechanism to weight and fuse different modal features to generate a unified semantic representation. S6. Combine the generated unified semantic representation and large model to conduct in-depth analysis of user input, generate relevant answers based on context information and conversation history, apply adaptive context generation algorithms to provide coherent and logical output, and return the results to the user.

2. According to claim 1, a multi-layer semantic understanding large model intelligent agent construction and application method is characterized in that: The S1 specifically includes: S11, perform preliminary processing on the input data to remove noise and redundant information; S12, label the data after preliminary processing to provide a data set for model training; S13, aligning data from different modalities in time and space, so that text, image and audio data can be processed in a unified time axis and space range; S14, using a fusion algorithm to fuse the aligned multimodal data to generate a comprehensive data representation of the multimodal features; S15. Generate more training samples based on the generated comprehensive data representation through data expansion technology to enrich the data set; S16. On the basis of data expansion, further expand the size of the data set and optimize the model's adaptability under different data distributions by adding diversified data samples.

3. According to the method for constructing and applying a multi-layer semantic understanding large model agent according to claim 1, it is characterized in that: The S2 specifically includes: S21, receiving and processing the generated comprehensive data representation; S22. Use a large language model to perform in-depth analysis on the comprehensive data representation and extract deep semantic features; S23, combining the conversation history and context information, performing context association analysis on the comprehensive data representation to identify the user's intentions and needs; S24. Using a hybrid neural symbolic reasoning network, deep semantic features are reasoned and analyzed to extract high-level semantic information. The hybrid neural symbolic reasoning network combines the representation ability of neural networks and the reasoning ability of symbolic logic: Among them, P(Q|F) represents the probability of querying Q given feature F, λ i is the weight parameter, f i is the characteristic function, Z is the normalization factor; S25, matching the extracted high-level semantic information with the identified user intentions and needs to generate a structured semantic representation; S26. Through adaptive knowledge embedding and dynamic logical reasoning mechanism, the structured semantic representation is optimized to generate the final deep semantic features and data representation of user intention.

4. According to claim 1, a multi-layer semantic understanding large model intelligent agent construction and application method is characterized in that: The S3 specifically includes: S31, extracting key information from the identified intent data and the extracted deep semantic features; S32. Use information extraction technology to identify entities, relationships and events in the data: Among them, Score(e i ,r,e j ) represents entity e i and e j The score under relation r, h r Represents the embedding vector of relation r, W r represents the weight matrix of relation r, b r Represents the bias vector of relation r, g(e i ) and g(e j ) represents entity e i and e j The embedding vector of i ); g(e j )] represents the concatenation operation of entity vectors; S33, structurally representing the identified entities, relationships and events to form structured data; S34. Use dynamic hyperparameter optimization based on evolution strategy to adjust the model's hyperparameter configuration to adapt to different data distributions and task requirements: Among them, θ t represents the model parameters of the tth generation, α is the learning rate, β is the regularization parameter, J(θ; z i ) is the model in sample z i The loss function on , N is the total number of samples, and M is the number of samples of the regularization term; S35. Generate final optimized model parameters according to the structured data and the optimized model parameters.

5. According to claim 1, a multi-layer semantic understanding large model intelligent agent construction and application method is characterized in that: The S4 specifically includes: S41. According to the extracted key information and the optimized model parameters, query the knowledge graph to obtain data related to the key information; S42, associate and organize the newly extracted key information with the information obtained from the knowledge graph to generate a preliminary structured knowledge representation; S43. Use semantic similarity calculation and graph matching algorithm to find the connection points between the preliminary structured knowledge representation and the nodes and relationships in the existing knowledge graph: Among them, Sim(x,y) represents the semantic similarity between two entities x and y, α i represents the weight of the i-th feature, represents the cosine similarity of the i-th feature vector; S44. Based on the found association points, semantic fusion of information is performed to unify the information representation from different sources. In the process of semantic fusion, a multi-layer perceptron MLP is used for nonlinear transformation: Among them, v fused represents the fused semantic vector, β i represents the weight of the i-th information source, v i Represents the semantic vector of the i-th information source; S45. Use ontology matching and semantic parsing technology to solve the semantic differences of heterogeneous data: Among them, Match(O1,O2) represents the matching degree between ontology O1 and ontology O2, C represents the concept set in the ontology, and sim ontology (c1,c2) ​​represents the similarity between concepts c1 and c2, depth(c) represents the depth of concept c in the ontology tree, which is used for weight adjustment; S46. Associate the newly extracted information with the nodes and relationships in the existing knowledge graph after parsing and matching to form a unified structured knowledge representation, and use the generated unified structured knowledge representation as the final structured knowledge representation.

6. The method for constructing and applying a multi-layer semantic understanding large model agent according to claim 1, characterized in that: The S5 specifically includes: S51. Vectorize the generated structured knowledge representation. The vectorization process includes embedding generation, feature extraction, and dimension reduction. Embedding generation uses a pre-trained language model to represent knowledge as a high-dimensional vector: v embed =BCE_Embedding(K); Among them, v embed A high-dimensional vector representation representing the knowledge K; S52, perform feature extraction, and use a unified attention mechanism to weight and fuse features of different modalities to generate a unified semantic representation: Among them, h unified represents the feature vector after weighted fusion, α1 is the weight coefficient in the attention mechanism, W1 is the weight matrix of the lth layer, h1 is the feature vector of the lth layer, and σ is the activation function; S53, perform dimensionality reduction, and use multi-level principal component analysis technology to reduce the high-dimensional vector to a low-dimensional vector suitable for storage and retrieval: Among them, v reduced represents the vector after dimensionality reduction, W i represents the principal component analysis matrix at level i; S54, using a vector database to store vectorized knowledge data and optimize the storage and retrieval of high-dimensional vectors to facilitate similarity search and processing; S55. Through the unified attention mechanism, weighted fusion of different modal features is performed to generate the final unified semantic representation: Among them, v final represents the final unified semantic representation, β i Represents the weight coefficient in the attention mechanism, v i represents the vector of different modal features, and σ is the activation function.

7. The method for constructing and applying a multi-layer semantic understanding large model agent according to claim 1, characterized in that: The S6 specifically includes: S61. Combined with the generated unified semantic representation v final The large model performs in-depth analysis of user input. The analysis process includes comprehensive processing of user input, context information, and conversation history to generate context semantic vectors: h context =f Qwen (input,context,history); Among them, h context represents the semantic representation that combines user input, context information, and conversation history, f Qwen The semantic vector generation function representing the large model; S62, based on h context Generate relevant answers and use the adaptive context generation algorithm for answer generation. The adaptive context generation algorithm uses a combination of dynamic attention mechanism and recurrent neural network to generate high-quality answers: Among them, r gen represents the generated answer, g context represents the adaptive context generation algorithm, W is the weight matrix, b is the bias vector, α t is the dynamic attention weight, is the hidden state of RNN; S63, through the multi-level processing of adaptive context generation algorithm, generate coherent and logical output: Among them, r final Indicates the final coherent and logical output, α i Represents the weight coefficient in the adaptive context generation algorithm, W i and b i They represent the weight matrix and bias vector of the i-th layer respectively, σ is the activation function, and ELU is the exponential linear unit activation function; S64, the final coherent and logical output r final Return to the user, completing the entire process of semantic understanding and response generation.

Images