Method and apparatus for generating reply information, storage medium, and electronic device

Abstract

The application discloses a reply information generation method and device, a storage medium and an electronic device, relates to the technical field of computers, and comprises the following steps: receiving a target query request input by a target account when the target account requests to query information in a target business field; performing feature extraction on a multi-modal input file to obtain semantic features of the input file of each mode; converting an information query intention of the target query request according to the plurality of semantic features; generating target reply information of the target query request according to the information query intention and preset business information possessed by the target business field; and sending the target reply information to the target account, so that the technical problem of low generation efficiency of reply information in the related art is solved, and the technical effect of improving the generation efficiency of reply information is achieved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for generating response information, a storage medium, and an electronic device. Background Technology

[0002] With the popularization and development of artificial intelligence (AI) technology, it has been gradually applied to question-answering (QA) systems. Users input questions into these systems, and the systems provide corresponding answers. Current QA systems use a prompt-based response method. Users need to trigger the system to perform corresponding actions using specific prompts. Users embed their query requirements into the corresponding prompt structure according to the prompt format, thus constructing the query question. The QA system identifies the prompts in the query question; only when a match is found can it understand the user's query intent. This response method is highly dependent on prompts; if the set prompts are not matched, the system cannot perceive the user's true query intent. Especially when the query content is complex, it requires input files with different modalities to convey the user's query requirements more nuancedly. For example, there might be text input describing the query request, as well as event screenshots describing the event state to which the queried information belongs. In such cases, it is difficult for the user to construct accurate prompts for the QA system to perceive their query intent, thus affecting the accuracy of the output response. Summary of the Invention

[0003] This application provides a method and apparatus for generating response information, a storage medium and an electronic device, to at least solve the problem of low efficiency in generating response information in related technologies.

[0004] This application provides a method for generating response information, comprising: receiving a target query request input by a target account when requesting to query information within a target business domain, wherein the target query request includes a multimodal input file, the multimodal input file being used to characterize the information query conditions of the target query request from multiple dimensions; performing feature extraction on the multimodal input file to obtain semantic features of the input file for each modality, wherein the semantic features are used to indicate the information query needs described by the corresponding input file; converting the current information query intent of the target query request based on the multiple semantic features; generating target response information for the target query request based on the information query intent and preset business information of the target business domain; and sending the target response information to the target account.

[0005] This application also provides a response information generation apparatus, comprising: a receiving module, configured to receive a target query request input by a target account when requesting to query information within a target business domain, wherein the target query request includes a multimodal input file, the multimodal input file being used to characterize the information query conditions of the target query request from multiple dimensions; an extraction module, configured to perform feature extraction on the multimodal input file to obtain semantic features of the input file for each modality, wherein the semantic features are used to indicate the information query requirements described by the corresponding input file; a conversion module, configured to convert the current information query intent of the target query request based on the multiple semantic features; a generation module, configured to generate target response information for the target query request based on the information query intent and preset business information of the target business domain; and a sending module, configured to send the target response information to the target account.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described methods for generating response information.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described methods for generating response information.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described methods for generating response information.

[0009] This application incorporates multimodal input files into the target query request, which carry the user's actual query needs. By extracting features from these multimodal input files, semantic features of each modality are obtained, enabling the perception of the information query needs described in the input files. Furthermore, by converting multiple semantic features, the information query intent indicated by the target query request is derived. This achieves the accuracy of the information query intent by semantically understanding the multimodal input files. Based on the refined query intent and pre-defined business information of the target business domain, target response information is generated, improving the matching degree between the target response information and the target query request. Therefore, this addresses the technical problem of low response information generation efficiency in related technologies, achieving the technical effect of improving response information generation efficiency. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a hardware structure block diagram of the response information generation method according to an embodiment of this application;

[0012] Figure 2 This is a flowchart of a method for generating response information according to an embodiment of this application;

[0013] Figure 3 This is a question-and-answer system architecture diagram according to an embodiment of this application;

[0014] Figure 4 This is a structural block diagram of a response information generation method according to an embodiment of this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0016] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0017] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] The specific application environment architecture or specific hardware architecture on which the execution of the response information generation method depends is described here.

[0019] The methods and embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1This is a hardware structure block diagram of the response information generation method according to an embodiment of this application. For example... Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0020] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the operating system startup method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0021] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0022] The embodiments of this application provide a method for generating response information. The method is described in detail below in conjunction with the execution flow of the response information generation method.

[0023] This embodiment provides a method for generating response information. Figure 2 This is a flowchart of a method for generating response information according to an embodiment of this application, such as... Figure 2As shown, the method includes the following steps:

[0024] Step S202: Receive a target query request input by the target account when requesting to query information in the target business domain. The target query request includes a multimodal input file, which is used to characterize the information query conditions of the target query request from multiple dimensions.

[0025] Step S204: Extract features from the multimodal input files to obtain semantic features of the input files for each modality, wherein the semantic features are used to indicate the information query requirements described by the corresponding input file;

[0026] Step S206: Based on the multiple semantic features, the current information query intent of the target query request is converted;

[0027] Step S208: Generate target response information for the target query request based on the information query intent and the preset business information of the target business domain;

[0028] Step S210: Send the target response information to the target account.

[0029] Through the above steps, the target account includes multimodal input files in the target query request. These multimodal input files carry the user's actual query needs. By extracting features from the multimodal input files, the semantic features of each modality are obtained, enabling the perception of the information query needs described in the input files. Furthermore, by converting multiple semantic features, the information query intent indicated by the target query request is derived. This achieves the conversion of the true information query intent of the target query request through semantic understanding of the multimodal input files, ensuring the accuracy of the information query intent. Finally, target response information is generated based on the information query intent and preset business information in the target business domain, improving the matching degree between the target response information and the target query request. Therefore, this solves the technical problem of low response information generation efficiency in related technologies, achieving the technical effect of improving the efficiency of response information generation.

[0030] The above-mentioned method for generating response information can be applied to, but is not limited to, information question-and-answer systems. When a target account inputs a target query request, including a multimodal input file, into the question-and-answer system, the system generates target response information for the target query request by running the above-mentioned method for generating response information.

[0031] In the embodiment provided in step S202, the modality of the input file is used to indicate the carrier of the input file. The file carriers of input files with different modalities are different. Multimodal input files can be, but are not limited to, files in the form of text, audio, video, images, sensor data, etc.

[0032] In this application embodiment, users' demands for answers from the question-and-answer system are becoming increasingly sophisticated, no longer satisfied with the traditional single-answer retrieval method. The questions entered by users are more professional, the question description information required to describe the question to be answered is more extensive, the information involved in the description information has more dimensions, and the information used for description may be carried in multiple different modal files. For example, when requesting the question-and-answer system to diagnose faults based on the server's operating status, the multimodal input files required may include, but are not limited to, screenshots of system logs, data collected by sensors, multimedia images of the system's operating status, user-input voice commands, text commands, etc. This solution does not limit this.

[0033] In the embodiment provided in step S204, the purpose of feature extraction of the input file is to perceive the query requirements described in the corresponding input file. Since the file carriers of input files of different modalities are different, feature extraction is performed using a feature extraction method corresponding to the modality of the input file. For example, when the input file is a text file, text processing is performed to remove special characters or stop words, and then semantic recognition and feature extraction are performed on the processed text to obtain the semantic features of the text. When the input file is a multimedia image file, image feature extraction is performed on the image to obtain the semantic features corresponding to the image. When the input file is an audio file, the audio file is converted to text to obtain the corresponding text file, and semantic recognition and feature extraction are performed on the text file to obtain the semantic features of the text.

[0034] In the embodiment provided in step S206, the information query intent can be converted by aligning and fusing multiple semantic features, but is not limited to this method. The purpose of aligning and fusing multiple semantic features is to map semantic features of different modalities to a unified feature space, and to represent the target account’s total query intent in the target query request through the fused information query intent.

[0035] Optionally, in this embodiment, the information query intent can be obtained by recognizing and converting multiple semantic features through an intent conversion model. The intent conversion model records the conversion relationship between semantic features and query intent. By inputting multiple semantic features into the intent conversion model, the content output by the intent conversion model is used as the information query intent. This solution does not limit this.

[0036] In the embodiment provided in step S208, the preset business information refers to the business information configured within the target business domain. This preset business information serves as the basis for responding to query requests within the target business domain; that is, the target response information is generated based on the information content included in the preset business information. The method for generating target response information based on the information query intent and the preset business information may include, but is not limited to, querying candidate business information related to the information query intent from the preset business information, and using the candidate business information as the information basis to generate corresponding target response information for the target query request according to the response method indicated by the information query intent.

[0037] Optionally, in this embodiment, to improve the efficiency of querying business information, preset business information of the target business domain can be stored in a knowledge graph database. The knowledge graph database includes multiple entity points and connecting edges between these entity points. Each entity point corresponds one-to-one with a sub-business included in the target business domain, and each entity point represents the sub-business information of its corresponding sub-business. Connecting edges indicate the transfer relationships between the sub-businesses corresponding to the entity points. The information query intent may include, but is not limited to, reference entity points and corresponding reference connecting edges involved in the target query request. The database searches for entity points corresponding to the reference entity points and reference connecting edges to obtain the answer. When no existing answer exists in the knowledge graph database, a generative model is used to generate a natural language answer. The generative model is based on the existing knowledge graph database structure (e.g., knowledge-based generation). Incremental updates are supported; when new data arrives, newly added entity points or connecting edges are automatically identified, and the knowledge graph database is updated. User feedback-driven updates are also supported: after a user marks an incorrect answer, the system automatically repairs the knowledge graph database. Supports multi-version control: Retains multiple versions of the knowledge graph library, supporting backtracking and auditing. Component-driven updates: Monitors code repository changes via Git Webhook, triggering incremental knowledge extraction. Streaming processing: Uses Apache Flink to analyze log streams in real time, detecting new error patterns and updating the graph. Example: Associates the error code "Nova 500" in the log with the "resource contention" node in the knowledge graph and maps it to the voice description "virtual machine startup failed." In this embodiment, the knowledge graph library can be constructed by extracting entity points from information within the target business domain, obtaining the relationship between the entity point and other entity points in the knowledge graph library, storing the entity point in the knowledge graph library, and constructing connection edges between the entity point and existing entity points in the graph library (eliminating duplicate entities, merging identical information from different sources, and ensuring the consistency and accuracy of the knowledge graph) for easy retrieval. Domain-adaptive models (such as fine-tuned BERT-NER) are used to extract entity relationships from unstructured text (such as logs and community discussions).

[0038] As an optional implementation, the step of converting the current information query intent of the target query request based on multiple semantic features includes:

[0039] The semantic features are fused to obtain the fused multimodal semantic features;

[0040] The multimodal semantic features are semantically parsed to obtain the information query intent of the target query request.

[0041] Optionally, in this embodiment, multimodal semantic features can be generated by aligning and fusing multiple semantic features. The purpose of aligning and fusing multiple semantic features is to map semantic features from different modalities to a unified feature space. The method of fusing multimodal semantic features can be, but is not limited to, using a dynamic gating network to introduce a learnable gating mechanism. Based on the multiple semantic features input into the dynamic gating network, the contribution weights of the semantic features corresponding to different modalities to the query intent representing the target query request are dynamically adjusted. Then, the contribution weights are used to perform a weighted summation of the multiple semantic features to determine the fused semantic features as the information query intent.

[0042] Optionally, in this embodiment, the method of converting multiple semantic features into multimodal semantic features can also be based on Generative Adversarial Networks (GANs) to achieve effective multimodal fusion. The generator maps multimodal inputs (such as text, images, and speech) into a unified vector, and the discriminator determines whether this vector can reconstruct the original modal data. Through adversarial training, the vector is forced to retain the core information of all modalities, so that even if a certain modality is missing (such as only text input), a complete multimodal vector can still be generated. The specific implementation steps are as follows:

[0043] 1. Define the generator;

[0044] Input: Semantic vectors of different modalities (e.g., text vector v) t Image vector v i Speech vector v s ).

[0045] Objective: To fuse these vectors into a single, unified multimodal vector v. mm .

[0046] operate:

[0047] Feature extraction: Extract semantic vectors from the input of each modality.

[0048] Fusion Module: Design a fusion module to merge vectors from different modalities. This can be achieved using methods such as weighted summation or concatenation followed by a fully connected layer.

[0049] For example, weighted summation:

[0050] v min =α t v t +α i v i +α s v s

[0051] Wherein, weight α t ,αi ,α s It can be learned through training.

[0052] Alternatively, the pieces can be combined and then passed through a fully connected layer: v min =ReLU(W[v t ;v i ;v s ]+b)

[0053] Output: The generator outputs a uniform multimodal vector v. mm .

[0054] 2. Define the discriminator;

[0055] Input: A multimodal vector v generated by the generator mm .

[0056] Objective: To determine whether the vector can reconstruct the original modal data.

[0057] operate:

[0058] Refactoring Module: Design a refactoring module that attempts to refactor from v mm The original modal data (such as text, image, and speech) is reconstructed from it.

[0059] For example, a decoder can be used for text reconstruction:

[0060]

[0061] For image reconstruction, a generative network can be used:

[0062]

[0063] For speech reconstruction, an acoustic model can be used:

[0064]

[0065] Loss calculation: Calculate the difference between the reconstructed data and the original data, which serves as the loss function for the discriminator.

[0066] For example, using mean squared error (MSE):

[0067]

[0068] Output: The discriminator outputs a probability value indicating whether the vector can reconstruct the original modal data.

[0069] 3. Combat training;

[0070] Objective: Through adversarial training between the generator and the discriminator, the multimodal vectors generated by the generator can retain the core information of all modalities.

[0071] operate:

[0072] Generator Loss: The goal of the generator is to cause the discriminator to misclassify the generated vectors as real data. The generator's loss function can be defined as:

[0073]

[0074] Where D(vmm) is the output probability of the discriminator.

[0075] Discriminator Loss: The goal of the discriminator is to distinguish between generated vectors and real data. The discriminator's loss function can be defined as:

[0076]

[0077] Where Pdata is the distribution of the real data and PG is the distribution of the data generated by the generator.

[0078] Training process:

[0079] Alternately train the generator and discriminator:

[0080] Fix the generator and train the discriminator to better distinguish between real and generated data.

[0081] Fix the discriminator and train the generator so that the vectors it generates can better deceive the discriminator.

[0082] Through multiple iterations, the vectors generated by the generator can pass the discriminator's test while retaining all the core modal information.

[0083] 4. Handling modal missing cases;

[0084] Objective: The generator should still be able to generate robust multimodal vectors even if one modality is missing (e.g., only text input is available).

[0085] operate:

[0086] Modal completion: During training, some modal inputs are randomly discarded, forcing the generator to learn how to generate complete multimodal vectors by relying only on other modalities.

[0087] Regularization: Add a regularization term to the generator's loss function to ensure that the generated vectors remain stable and consistent even when modes are missing.

[0088] For example, adding reconstruction loss:

[0089]

[0090] Where λ is the regularization coefficient. It is the reconstruction loss.

[0091] Through the steps described above, the generator learns how to map multimodal inputs to a unified vector while preserving the core information of all modalities. The discriminator supervises the generator by determining whether this vector can reconstruct the original modal data. Through adversarial training, the model can handle cases of missing modalities and generate complete multimodal vectors.

[0092] Optionally, in the embodiments of this application, the information query intent may be, but is not limited to, obtained by using a deep learning model to semantically understand the multimodal semantic features based on the fused multimodal semantic features, using a pre-trained language model to encode the multimodal semantic features, learning the semantic information therein, understanding the user's intent and semantics, and identifying the information query intent corresponding to the multimodal semantic features through the analysis of the language information. In this embodiment, to improve the reliability of information intent, dynamic context modeling can be used to determine the target account's current information query intent. This can be achieved by using a pre-trained language model to encode multimodal semantic features, learning the semantic information within them, understanding the user's intent and semantics, analyzing the semantic information to identify the reference information query intent corresponding to the multimodal semantic features, and obtaining multiple initial information query intents corresponding to multiple information query requests from the target account before the current moment. These initial information query intents are arranged chronologically. The intent transformation information of the target account is then transformed using these initial information query intents. This intent transformation information characterizes the relationship between the target account's information query intent and time. Finally, the reference information query intent is corrected based on the intent transformation amount indicated by the intent transformation information to obtain the final information query intent. This method, by combining the target account's query intents in adjacent historical time periods to model the current information query intent, ensures the accuracy and reliability of the information query intent.

[0093] Through the above, by fusing multiple semantic features, multimodal semantic features are obtained, thereby perceiving the semantic features included in the multimodal input file. Then, semantic parsing is performed based on the multimodal semantic features to perceive the information query intent corresponding to the target query request. This realizes the transformation of information query intent based on the semantic features of the multimodal input file, thus improving the accuracy of information query intent.

[0094] As an optional implementation, the feature fusion of multiple semantic features to obtain fused multimodal semantic features includes one of the following:

[0095] Text recognition is performed on the file content carried by the multimodal input files to obtain text information carried by each input file; based on the information content of the text information, association information of the input files for each modality is extracted, wherein the association information is used to indicate the dependency relationship between the information content of the current modality's input file and the information content of other modal input files in the multimodal input files; target weight parameters are assigned to the multimodal input files according to the dependency relationship, wherein the target weight parameters are used to indicate the influence of the corresponding input file on characterizing the current query intent of the target account; the target weight parameters are used to perform a weighted summation calculation on the corresponding semantic features to obtain the multimodal semantic features;

[0096] By inputting multiple semantic features into a target generation model, target vector features corresponding to each semantic feature are obtained from the output of the target generation model. These multiple target vector features are vector features in a target vector space. The target generation model records the transformation relationship between vector features in the vector space corresponding to the multimodal input file and vector features in the target vector space. The multiple target vector features are then input into a target fusion model to obtain the multimodal semantic features output by the target fusion model. The target fusion model records the fusion relationship between the various vector features.

[0097] Optionally, in this embodiment, the association information of the input files can be transformed, but is not limited to, in the following ways: extracting target keywords from the text information, wherein the target keywords are used to characterize the core content described by the corresponding input file; calculating the association degree between the target keywords in each target input file included in multiple input files and the target keywords in a reference input file to obtain an association degree parameter; wherein the association information includes the association degree parameter, and the reference input file is other files in the multiple input files besides the reference input file. Furthermore, the method for assigning target weight parameters to multiple modal input files according to the dependency relationship can be to calculate the sum of the association degree parameters of all target keywords included in each input file to obtain the target association degree parameter of each input file, and assign corresponding target weight parameters according to the level of the target association degree parameter of the input files.

[0098] As described above, by assigning corresponding target weight parameters to each semantic feature based on the dependencies between input files, multiple semantic features are weighted and merged according to these dependencies. This allows for feature merging based on the influence of input files on the query semantics, resulting in more realistic and reliable semantic content expressed by the merged multimodal semantic features. By using a target generation model to process multiple semantic features, features with different dimensions are transformed to the same semantic feature dimension. Finally, a fusion model is used to fuse these transformed target vector features to obtain multimodal semantic features, ensuring their accuracy.

[0099] As an optional implementation, the step of semantically parsing the multimodal semantic features to obtain the information query intent of the target query request includes:

[0100] The multimodal semantic features are input into the target semantic generation model, wherein the target semantic generation model records the conversion relationship between semantic features and query intent;

[0101] Obtain the information query intent output by the target semantic generation model.

[0102] As an optional implementation, generating target response information for the target query request based on the information query intent and preset business information of the target business domain includes:

[0103] Searching for candidate entities with target association relationships with target entities and target relationship information in the target knowledge graph, wherein the information query intent includes the target entity and the target relationship information, the target entity is used to indicate the sub-business to which the query information requested by the target query request belongs, the target relationship information is used to indicate the transfer relationship between the target entities, the target knowledge graph records multiple entities and relationship information used to indicate the transfer relationship between the multiple entities, and the entity is used to indicate a sub-business included in the target business domain;

[0104] Obtain candidate business information of the sub-business corresponding to the candidate entity in the preset business information;

[0105] The target response information is generated based on the candidate service information.

[0106] Optionally, in this embodiment, candidate business information provides information support for the generated target response information. When the correlation between the target entity and the candidate entity is greater than a certain threshold, the candidate business information can be considered to be the content requested by the target query request, and the candidate business information can be directly used as the target response information. When the correlation between the target entity and the candidate entity is less than or equal to a certain threshold, the candidate business information can be considered to be related to the target query request, but not yet a standard response to the target query question. Therefore, a target response information can be generated based on the content of the candidate business information and the information query intent. When there is no target response information to answer the target query question within the target business domain, the target indication map can be updated according to the currently generated target response information, and the target response information can be updated to the preset business information included in the target business domain, thereby enriching the information content in the information database of the target business domain.

[0107] Through the above, by constructing a knowledge graph, each sub-business is stored as an entity in the knowledge graph, and the transfer relationships between sub-businesses are used to construct relationship information between entities, thereby improving the efficiency of maintaining business information and information retrieval within the target business domain.

[0108] As an optional implementation, generating the target response information based on the candidate service information includes:

[0109] When the correlation between the target entity and the candidate entity is less than or equal to the target correlation, the candidate business information and the information query intent are input into the target language model to obtain the target response information output by the target language model. The target language model records the business information and the transfer relationship between the query intent and the response information.

[0110] Optionally, in this embodiment of the application, when the correlation between the target entity and the candidate entity is less than or equal to the target correlation, it can be considered that the candidate business information is not the standard answer to the target query request. However, there is a certain correlation between the candidate business information and the target query request, so the content of the candidate business information can be used as a reference to generate the response content for the target query request.

[0111] Based on the above, when the correlation between the target entity and the candidate entity is less than or equal to the target correlation, the target semantic model is used to identify and process the candidate business information and information query intent, thereby using the candidate business information as the information basis for answering the target query request, thus generating more accurate target response information.

[0112] As an optional implementation, generating the target response information based on the candidate service information includes:

[0113] If the correlation between the target entity and the candidate entity is greater than the target correlation, the candidate business information is determined as the target response information.

[0114] Figure 3 This is a question-answering system architecture diagram according to an embodiment of this application, such as... Figure 3 As shown, the question-answering system adopts a layered architecture, and the core modules are as follows:

[0115] Multimodal sensing layer:

[0116] It supports input of text, voice, image, video, and sensor data (such as IoT devices), which are uniformly encoded into semantic vectors through a cross-modal alignment network;

[0117] An integrated emotion recognition module analyzes users' emotional states (such as urgency or confusion) in real time to optimize response strategies.

[0118] Intelligent Understanding Layer:

[0119] Generate context-aware intent vectors based on a large model (integrating BERT and GPT-4 architecture);

[0120] A bidirectional attention mechanism is used to link historical dialogues with the current question, and multi-granularity intent parsing is used: user questions are decomposed into atomic intents ("comparing performance") and an intent dependency graph is constructed.

[0121] Dynamic knowledge engine:

[0122] Knowledge graph construction: Based on the Neo4j graph database, it includes a three-layer structure of entities, relations, and attributes;

[0123] Real-time data stream processing: Real-time access to external data sources (such as news APIs and academic databases) via Apache Flink to trigger incremental updates of the knowledge graph (delay <30 seconds);

[0124] Cross-domain reasoning module: Utilizes graph neural networks (GNNs) to achieve associative reasoning of knowledge nodes.

[0125] Decision generation layer:

[0126] Multi-strategy answer generation: Combining rule templates, generative models (such as T5), and retrieval enhancement generation, the optimal output format is selected according to the scenario.

[0127] Closed-loop optimization layer:

[0128] Feedback perception module: collects explicit feedback (user ratings) and implicit signals (dialogue turns, answer acceptance rate, etc.);

[0129] Incremental learning engine: Based on the Elastic Weight Fixation (EWC) algorithm, it only updates key parameters to avoid catastrophic forgetting.

[0130] As shown in the figure, the intelligent question-answering method process is as follows:

[0131] Step 1: Multimodal input parsing and alignment;

[0132] • Voice input: Converts speech to text via end-to-end speech recognition (ASR) and extracts pitch and speech rate features;

[0133] • Image input: Descriptive text is generated through a visual-semantic joint embedding model;

[0134] Text, speech, and image features are fused into a unified semantic vector via the CMAN network.

[0135] Step 2: Deep semantic understanding and intent decomposition;

[0136] • Context encoding: Encode the current question and historical dialogue into a temporal vector, and capture long-term dependencies through an LSTM network;

[0137] • Use large models to generate context-aware semantic representations to identify user intent (such as inquiries, complaints, suggestions) and improve classification accuracy;

[0138] Step 3: Dynamic knowledge retrieval and causal reasoning;

[0139] • Parallel retrieval of local knowledge graphs, external databases (encyclopedias, etc.), and real-time data streams;

[0140] • Integrate multi-source results using a confidence-weighted fusion algorithm, excluding low-quality answers with a confidence level <90%, and retaining only results with a confidence level >=90%.

[0141] • Knowledge Reasoning: Generates reasoning paths based on GNN and provides the correct path;

[0142] Step 4: Personalized answer generation and multimodal output;

[0143] • User profile matching: Generate customized answers based on user history and search type;

[0144] • Multimodal output: Generates answer formats adapted to different terminals through TTS (text-to-speech) and data visualization engines;

[0145] Step 5: Feedback-driven system self-optimization;

[0146] • Collect user behavior data (such as answer acceptance rate, follow-up question depth, and user ratings);

[0147] • Incremental learning is triggered daily. Every day at midnight, the model parameters are updated using a gradient pruning + elastic weight solidification strategy to ensure that historical knowledge is not overwritten.

[0148] • The knowledge graph dynamically adjusts node priority based on popularity, and frequently accessed nodes are cached at the edge.

[0149] The embodiments of this application have the following beneficial effects: a) Real-time knowledge: The delay from data collection to knowledge graph update is less than 1 minute, supporting immediate response to emergencies; b) Multimodal collaboration: Improved cross-modal question answering accuracy, supporting 10+ input / output formats, and providing more accurate answers to users' needs; c) Adaptability: Through feedback loop, the system automatically optimizes the model monthly, reducing the error rate; d) Improved accuracy: Question answering accuracy is improved compared to traditional systems, and the ability to handle complex problems is increased by 3 times, greatly reducing the cost of manual maintenance.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0151] Embodiments of this application also provide a response information generation apparatus. Figure 4 This is a structural block diagram of a response information generation method according to an embodiment of this application, such as... Figure 4 As shown, the device includes:

[0152] The receiving module is used to receive a target query request input by a target account when requesting to query information in the target business domain. The target query request includes a multimodal input file, which is used to characterize the information query conditions of the target query request from multiple dimensions.

[0153] The extraction module is used to extract features from the multimodal input files to obtain semantic features of the input files for each modality, wherein the semantic features are used to indicate the information query requirements described by the corresponding input files;

[0154] The conversion module is used to convert the current information query intent of the target query request based on multiple semantic features;

[0155] The generation module is used to generate target response information for the target query request based on the information query intent and the preset business information of the target business domain;

[0156] The sending module is used to send the target reply information to the target account.

[0157] Through the above device, the target account includes multimodal input files in the input target query request. These multimodal input files carry the user's real query needs. By extracting features from the multimodal input files, the semantic features of each modality are obtained, enabling the perception of the information query needs described in the input files. Furthermore, by converting multiple semantic features, the information query intent indicated by the target query request is derived. This achieves the conversion of the true information query intent of the target query request through semantic understanding of the multimodal input files, ensuring the accuracy of the information query intent. Then, based on the more refined query intent and the preset business information of the target business domain, target response information is generated, improving the matching degree between the target response information and the target query request. Therefore, it can solve the technical problem of low response information generation efficiency in related technologies, achieving the technical effect of improving the efficiency of response information generation.

[0158] Optionally, the conversion module includes:

[0159] The fusion unit is used to fuse multiple semantic features to obtain fused multimodal semantic features;

[0160] The parsing unit is used to perform semantic parsing on the multimodal semantic features to obtain the information query intent of the target query request.

[0161] Optionally, the fusion unit is configured to perform one of the following operations:

[0162] Text recognition is performed on the file content carried by the multimodal input files to obtain text information carried by each input file; based on the information content of the text information, association information of the input files for each modality is extracted, wherein the association information is used to indicate the dependency relationship between the information content of the current modality's input file and the information content of other modal input files in the multimodal input files; target weight parameters are assigned to the multimodal input files according to the dependency relationship, wherein the target weight parameters are used to indicate the influence of the corresponding input file on characterizing the current query intent of the target account; the target weight parameters are used to perform a weighted summation calculation on the corresponding semantic features to obtain the multimodal semantic features;

[0163] By inputting multiple semantic features into a target generation model, target vector features corresponding to each semantic feature are obtained from the output of the target generation model. These multiple target vector features are vector features in a target vector space. The target generation model records the transformation relationship between vector features in the vector space corresponding to the multimodal input file and vector features in the target vector space. The multiple target vector features are then input into a target fusion model to obtain the multimodal semantic features output by the target fusion model. The target fusion model records the fusion relationship between the various vector features.

[0164] Optionally, the parsing unit is used for:

[0165] The multimodal semantic features are input into the target semantic generation model, wherein the target semantic generation model records the conversion relationship between semantic features and query intent;

[0166] Obtain the information query intent output by the target semantic generation model.

[0167] Optionally, the generation module includes:

[0168] The search unit is used to search for candidate entities in the target knowledge graph that have a target association relationship with the target entity and the target relationship information. The information query intent includes the target entity and the target relationship information. The target entity is used to indicate the sub-business to which the query information requested by the target query request belongs. The target relationship information is used to indicate the transfer relationship between the target entities. The target knowledge graph records multiple entities and relationship information used to indicate the transfer relationship between the multiple entities. The entity is used to indicate a sub-business included in the target business domain.

[0169] The acquisition unit is used to acquire candidate business information of the sub-business corresponding to the candidate entity in the preset business information;

[0170] The generation unit is used to generate the target response information based on the candidate service information.

[0171] Optionally, the generation unit is used for:

[0172] When the correlation between the target entity and the candidate entity is less than or equal to the target correlation, the candidate business information and the information query intent are input into the target language model to obtain the target response information output by the target language model. The target language model records the business information and the transfer relationship between the query intent and the response information.

[0173] Optionally, the generation unit is used for:

[0174] If the correlation between the target entity and the candidate entity is greater than the target correlation, the candidate business information is determined as the target response information.

[0175] For a description of the features in the embodiment corresponding to the response information generation device, please refer to the relevant description in the embodiment corresponding to the response information generation method, which will not be repeated here.

[0176] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described methods for generating response information.

[0177] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described methods for generating response information when it is run.

[0178] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0179] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described methods for generating response information.

[0180] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described methods for generating response information.

[0181] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] The foregoing has provided a detailed description of a method and apparatus for generating response information, a storage medium, and an electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for generating response information, characterized in that, include: The system receives a target query request input by a target account when requesting to query information within a target business domain. The target query request includes a multimodal input file, which is used to characterize the information query conditions of the target query request from multiple dimensions. Feature extraction is performed on the input files of the multimodal approaches to obtain semantic features of the input files for each modality, wherein the semantic features are used to indicate the information query requirements described by the corresponding input files; The current information query intent of the target query request is derived from multiple semantic features; Based on the information query intent and the preset business information of the target business domain, generate target response information for the target query request; Send the target response information to the target account; The step of converting the current information query intent of the target query request based on multiple semantic features includes: performing feature fusion on multiple semantic features to obtain fused multimodal semantic features; and performing semantic parsing on the multimodal semantic features to obtain the information query intent of the target query request. The step of semantically parsing the multimodal semantic features to obtain the information query intent of the target query request includes: inputting the multimodal semantic features into a target semantic generation model, wherein the target semantic generation model records the conversion relationship between semantic features and query intent; obtaining the reference information query intent output by the target semantic generation model; obtaining multiple initial information query intents corresponding to multiple information query requests entered by the target account before the current time; arranging the multiple initial information query intents in chronological order, determining the intent transformation information of the target account through the multiple initial information query intents, wherein the intent transformation information is used to characterize the relationship between the information query intent of the target account and time; and correcting the reference information query intent according to the intent transformation amount indicated by the intent transformation information to obtain the final information query intent. The step of generating target response information for the target query request based on the information query intent and preset business information of the target business domain includes: searching for candidate entities with target association relationships with target entities and target relationship information in the target knowledge graph, wherein the information query intent includes the target entity and the target relationship information, the target entity is used to indicate the sub-business to which the query information requested by the target query request belongs, the target relationship information is used to indicate the transfer relationship between the target entities, the target knowledge graph records multiple entities and relationship information used to indicate the transfer relationship between the multiple entities, and the entity is used to indicate a sub-business included in the target business domain; obtaining candidate business information of the sub-business corresponding to the candidate entity in the preset business information; and generating the target response information based on the candidate business information. The step of generating the target response information based on the candidate business information includes: when the correlation between the target entity and the candidate entity is less than or equal to the target correlation, inputting the candidate business information and the information query intent into the target language model to obtain the target response information output by the target language model, wherein the target language model records the business information and the transfer relationship between the query intent and the response information.

2. The method according to claim 1, characterized in that, The feature fusion of multiple semantic features to obtain fused multimodal semantic features includes one of the following: Text recognition is performed on the file content carried by the multimodal input files to obtain text information carried by each input file; based on the information content of the text information, association information of the input files for each modality is extracted, wherein the association information is used to indicate the dependency relationship between the information content of the current modality's input file and the information content of other modal input files in the multimodal input files; target weight parameters are assigned to the multimodal input files according to the dependency relationship, wherein the target weight parameters are used to indicate the influence of the corresponding input file on characterizing the current query intent of the target account; the target weight parameters are used to perform a weighted summation calculation on the corresponding semantic features to obtain the multimodal semantic features; By inputting multiple semantic features into a target generation model, target vector features corresponding to each semantic feature are obtained from the output of the target generation model. These multiple target vector features are vector features in a target vector space. The target generation model records the transformation relationship between vector features in the vector space corresponding to the multimodal input file and vector features in the target vector space. The multiple target vector features are then input into a target fusion model to obtain the multimodal semantic features output by the target fusion model. The target fusion model records the fusion relationship between the various vector features.

3. The method according to claim 1, characterized in that, The step of generating the target response information based on the candidate service information includes: If the correlation between the target entity and the candidate entity is greater than the target correlation, the candidate business information is determined as the target response information.

4. A device for generating response information, characterized in that, include: The receiving module is used to receive a target query request input by a target account when requesting to query information in the target business domain. The target query request includes a multimodal input file, which is used to characterize the information query conditions of the target query request from multiple dimensions. The extraction module is used to extract features from the multimodal input files to obtain semantic features of the input files for each modality, wherein the semantic features are used to indicate the information query requirements described by the corresponding input files; The conversion module is used to convert the current information query intent of the target query request based on multiple semantic features; The generation module is used to generate target response information for the target query request based on the information query intent and the preset business information of the target business domain; The sending module is used to send the target reply information to the target account; The conversion module includes: a fusion unit for performing feature fusion on multiple semantic features to obtain fused multimodal semantic features; and a parsing unit for performing semantic parsing on the multimodal semantic features to obtain the information query intent of the target query request. The parsing unit is further configured to: input the multimodal semantic features into a target semantic generation model, wherein the target semantic generation model records the conversion relationship between semantic features and query intent; obtain the reference information query intent output by the target semantic generation model; obtain multiple initial information query intents corresponding to multiple information query requests entered by the target account before the current time; arrange the multiple initial information query intents in chronological order, determine the intent transformation information of the target account through the multiple initial information query intents, wherein the intent transformation information is used to characterize the relationship between the information query intent of the target account and time; and correct the reference information query intent according to the intent transformation amount indicated by the intent transformation information to obtain the final information query intent. The generation module includes: a search unit, configured to search in the target knowledge graph for candidate entities that have a target association relationship with the target entity and target relationship information, wherein the information query intent includes the target entity and the target relationship information, the target entity is used to indicate the sub-business to which the query information requested by the target query request belongs, the target relationship information is used to indicate the transfer relationship between the target entities, the target knowledge graph records multiple entities and relationship information used to indicate the transfer relationship between the multiple entities, and the entity is used to indicate a sub-business included in the target business domain; an acquisition unit, configured to acquire candidate business information of the sub-business corresponding to the candidate entity in the preset business information; and a generation unit, configured to generate the target response information based on the candidate business information. The generation unit is further configured to: when the correlation between the target entity and the candidate entity is less than or equal to the target correlation, input the candidate business information and the information query intent into the target language model to obtain the target response information output by the target language model, wherein the target language model records the business information and the transfer relationship between the query intent and the response information.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the method for generating response information as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method for generating response information as described in any one of claims 1 to 3.

Images