Digital employee automated question answering method and system based on cross-modal retrieval enhancement

Abstract

The present application relates to the technical field of data processing, in particular to a digital employee automatic question and answer method and system based on cross-modal retrieval enhancement. The method comprises the following steps: 1) based on the multi-modal data of the digital employee, constructing candidate scheduling materials; 2) based on the user question, determining the task type included in the user question, and determining the evidence value of each modality under each task type; based on the evidence value of each modality under each task type, calculating the task modality adaptation degree of each candidate scheduling material for the user question; based on the task modality adaptation degree, calculating the evidence adaptation degree of each candidate scheduling material for the user question; 3) based on the evidence adaptation degree, selecting evidence materials from each candidate scheduling material, and generating a reply text for the user question based on the evidence materials. The evidence value of each modality under each task type is considered, thereby improving the effectiveness of the reply to the user question.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an automated question-answering method and system for digital employees based on cross-modal retrieval enhancement. Background Technology

[0002] As enterprise office systems, customer management systems, work order systems, knowledge bases, meeting systems, and business log platforms become increasingly digitized, digital employees are no longer just chatbots used to answer policy requests or retrieve document content. Instead, they are intelligent digital employees capable of handling business inquiries, anomaly diagnosis, task follow-up, process reminders, information aggregation, and automated execution. When users actually use digital employees, the questions they ask are often not simple knowledge queries, but rather continuous tasks revolving around a specific customer, project, device, contract, work order, or process status. To complete these tasks, digital employees typically need to simultaneously access information from different modalities, including text policy documents, image evidence, spreadsheet data, voice conference minutes, business system logs, historical conversation memories, and automated interface information.

[0003] Existing retrieval-enhanced question-answering methods typically convert all information from different modalities into text or vector fragments, then select relevant fragments based on the similarity between the user's question and the text or vector fragments, and input these fragments into a large language model to obtain the answer. Because information from different modalities has varying values ​​for answering different questions, and because the above approach ignores the intrinsic value of modal information by converting it all into text or vector fragments for subsequent processing, the generated response text has weak validity when dealing with cross-modal user question responses. For example, if a user's question requires an audio recording, but the response provides the text form of the audio recording, although the semantic similarity may be high, it lacks evidentiary value, resulting in a weakly valid response. Summary of the Invention

[0004] To address the technical problem of existing technologies neglecting the value of modal information itself, resulting in weak effectiveness of response results, the present invention aims to provide a digital employee automated question-answering method and system based on cross-modal retrieval enhancement. The specific technical solution adopted is as follows: In a first aspect, the present invention provides an automated question-answering method for digital employees based on cross-modal retrieval enhancement, comprising the following steps: 1) Construct candidate scheduling data based on multimodal data of digital employees; 2) Based on the user question, determine the task types included in the user question, and determine the evidentiary value of each modality under each task type; Based on the evidentiary value of each modality under each task type, calculate the task modality fit of each candidate scheduling data for the user problem. Based on the task modality fit, calculate the evidence fit of each candidate scheduling data for the user question; 3) Based on the evidence fit, select evidence data from each candidate scheduling data, and generate a response text for the user's question based on the evidence data.

[0005] In conjunction with the first aspect mentioned above, among some possible implementations, methods for calculating the task modality fit of each candidate scheduling data to the user problem include: Calculate the weights of each task type included in the user's question; For any candidate scheduling data, determine its mode; Based on the weights of each task type included in the user problem, the evidence value of that modality under each task type is weighted and summed to obtain the task modality fit of the candidate scheduling data for the user problem.

[0006] In conjunction with the first aspect above, among some possible implementations, methods for calculating the weights of the various task types included in the user problem include: Based on user questions, determine the degree of matching between user questions and each task type, as well as the confidence level of each task type; For any task type, the matching degree and confidence of the task type are combined, and the weight of the task type is obtained based on the fusion result.

[0007] In conjunction with the first aspect mentioned above, among some possible implementation methods, the methods for determining the degree of matching between user questions and various task types include: Based on each task type, keywords for each task type are determined. User questions are then matched with the keywords for each task type, and the degree of matching between the user questions and each task type is obtained based on the matching results.

[0008] In conjunction with the first aspect mentioned above, among some possible implementation methods, the methods for determining the evidentiary value of each modality under each task type include: Based on prior human experience data, the initial evidence value of each modality under each task type is calculated. Then, based on the historical question-and-answer data of the current dialogue, the initial evidence value of each modality under each task type is updated to obtain the evidence value of each modality under each task type.

[0009] In conjunction with the first aspect mentioned above, among some possible implementations, methods for updating the initial evidence value of each modality under each task type include: Based on the historical question-and-answer data of the current dialogue, calculate the effective contribution of each modality under each task type; By integrating the effective contribution of each modality under each task type and the preset update step size, the evidence value update amount of the modality under the task type is obtained, and the initial evidence value of the modality under the task type is updated based on the evidence value update amount.

[0010] In conjunction with the first aspect mentioned above, among some possible implementation methods, the methods for calculating the effective contribution of each modality under each task type include: For any modality under any task type, the effective contribution of that modality under that task type is obtained by dividing the number of times the candidate scheduling data of that modality is cited in the answer text of user questions of that task type in the historical question-answering data by the total number of times the candidate scheduling data of that modality is processed as evidence data and input into the large language model when answering user questions of that task type in the historical question-answering data.

[0011] In conjunction with the first aspect mentioned above, among some possible implementations, methods for calculating the evidence fit of each candidate scheduling data for the user problem include: Based on the user's question, the task object is determined, and the matching degree between each candidate scheduling data and the business entity of the user's question is calculated by combining the business entities of each candidate scheduling data. Based on the user question, the time range is determined, and the time matching degree between each candidate scheduling data and the user question is calculated by combining the generation time or update time of each candidate scheduling data. Based on the content representation of each candidate scheduling data and the user question, calculate the semantic similarity between each candidate scheduling data and the user question; By integrating the task modality fit, the business entity matching degree, the time matching degree, and the semantic similarity, the evidence fit degree of each candidate scheduling data for the user question is obtained.

[0012] In conjunction with the first aspect mentioned above, among some possible implementations, methods for calculating the time matching degree between each candidate scheduling data and the user problem include: For any candidate scheduling data, if the generation time or update time of the candidate scheduling data is not within the time range, then the time deviation between the generation time or update time of the candidate scheduling data and the time range is calculated, and the time deviation is negatively correlated to obtain the time matching degree between the candidate scheduling data and the user question.

[0013] Secondly, the present invention also provides an automated question-and-answer system for digital employees based on cross-modal retrieval enhancement, including a question-and-answer module and a multimodal data acquisition module. The multimodal data acquisition module is used to acquire multimodal data and send it to the question-and-answer module. The question-and-answer module is used to acquire user questions and, based on the received data and user questions, uses the automated question-and-answer method for digital employees based on cross-modal retrieval enhancement of the present invention to generate answer text for user questions.

[0014] The present invention has the following beneficial effects: The digital employee automated question answering method and system based on cross-modal retrieval enhancement of the present invention determines the evidentiary value of each modality under each task type, and calculates the task modality fit of each candidate scheduling data to the user question based on the evidentiary value of each modality under each task type. The modality fit indicates what type of evidence is needed to answer the user question, and based on the task modality fit, evidence data is selected to ensure that the selected evidence data is the evidence data needed to answer the user question. Finally, the answer text is generated based on the evidence data, thereby improving the effectiveness of the answer text. Attached Figure Description

[0015] Figure 1 This is a structural diagram of the automated question-answering system for digital employees based on cross-modal retrieval enhancement, according to an embodiment of the present invention. Figure 2 This is a flowchart of an embodiment of the digital employee automated question answering method based on cross-modal retrieval enhancement according to the present invention. Detailed Implementation

[0016] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0017] The specific structure of the digital employee automated question-answering system based on cross-modal retrieval enhancement of the present invention is as follows: Figure 1As shown, the system includes a question-and-answer module and a multimodal data acquisition module. The multimodal data acquisition module, based on the digital employee's data access permissions, retrieves multimodal data from other systems (such as Customer Relationship Management (CRM), historical databases, enterprise communication management systems, and enterprise meeting management systems) within the permitted scope of permissions. It records the data source of each piece of multimodal data and sends the multimodal data and its source to the question-and-answer module. In this embodiment, the multimodal data includes: textual materials, such as policy documents, process descriptions, contract texts, knowledge base articles, etc.; image materials, such as on-site photos, equipment screenshots, invoice images, and anomaly screenshots, etc.; tabular materials, such as customer payment tables, sales data tables, project progress tables, and equipment ledgers, etc.; audio materials, such as meeting recordings and telephone communication records, etc.; log materials, such as CRM operation records, work order flow records, and equipment operation logs, etc.; and memory information already saved by the digital employee, such as user preferences, historical task conclusions, previous confirmation results, and long-term business rules, etc.

[0018] The question-and-answer module is used to obtain user questions and, based on the received data, generates response content using the digital employee automated question-and-answer method based on cross-modal retrieval enhancement of the present invention.

[0019] The specific process of the digital employee automated question answering method based on cross-modal retrieval enhancement of the present invention is as follows: Figure 2 As shown, it includes the following steps: Step 1: Based on the multimodal data of digital employees, construct candidate scheduling data and determine the task objects and time range of user problems.

[0020] 1.1) Preprocess each data point in the multimodal data of digital employees to determine the content representation of each data point, and then construct each candidate scheduling data.

[0021] In this embodiment, the multimodal data is denoted as... ,in, This refers to text-based documents, such as policy documents, process descriptions, contract texts, knowledge base articles, etc. This refers to image-based data, such as on-site photos, equipment screenshots, invoice images, and screenshots of abnormal situations. This represents tabular data, such as customer payment forms, sales data tables, project schedules, equipment ledgers, etc. This refers to audio data, such as meeting recordings and telephone communication logs. This represents business system logs, such as CRM (Customer Relationship Management) operation records, work order flow records, equipment operation logs, etc. This refers to the memory information that digital employees have saved, such as user preferences, historical task conclusions, previous confirmation results, and long-term business rules.

[0022] In this embodiment, the preprocessing methods differ for different modalities of data. For image data, preprocessing includes extracting text descriptions, object labels, and key area descriptions using OCR or image description models. For audio data, transcribed text and speaker information are extracted using speech recognition algorithms. For table and log data, preprocessing includes retaining field names, field values, timestamps, and data sources. The specific preprocessing methods are all existing technologies and will not be elaborated upon in this embodiment.

[0023] The main content of each data point is extracted based on the preprocessed data to obtain the content representation of each data point. Based on the data format of each data point, determine the modality of each data point. The data source of each data point is determined based on the data source retained during preprocessing. The generation or update time of each data item is obtained based on the timestamp of each preprocessed data item. From the content expression of each data point The filter identifies the relevant business entities by keywords, thus obtaining the business entities for each data entry. .

[0024] In this embodiment, the candidate scheduling data is denoted as follows: , Indicates the first One candidate scheduling data, For the first The content of each candidate scheduling data can be represented as a text fragment, image description, table field summary, speech-to-text segment, or log event summary. For the first Modal type of each candidate scheduling data, For the first Data sources for candidate scheduling information For the first The generation time or update time of each candidate scheduling data. For the first The candidate scheduling data involves business entities, such as customer ID, equipment, project, contract, or work order. In this embodiment, the types of business entities differ in different scenarios, and the business entities are divided into different levels. The hierarchical architecture of business entities differs in different scenarios. For example, in a project scenario, the first-level business entity is the project ID, and the second-level business entity is the contract ID. If business entities have the same project ID but different contract IDs, it means that they belong to different contracts of the same project.

[0025] 1.2) Based on current user issues And the historical question-and-answer data of the current dialogue, to determine the task target. and time range .

[0026] From the current user issues By filtering for keywords that indicate the relevant business entities, the task object for the current user problem can be obtained. .

[0027] From the current user issues Explicit time phrases (such as "May 2023", "last week", "last three months") are extracted from the historical question-and-answer data of the current conversation. Then, using the current time as the anchor point, the relative time phrases are converted into absolute start and end timestamps, thus forming a time range. If no specific time range is found, the default setting is the system's preset full time period or the most recent year. The start time of the time range. This is the end time of the time range.

[0028] In this embodiment, the task object This includes the type of the business entity object, the number of the business entity object, and the number of the parent business entity object to which the business entity object belongs.

[0029] In this implementation, the hierarchical architecture of business entity objects differs depending on the scenario. For example, in a sales scenario, the hierarchical architecture of business entity objects is "Group (Company) - Customer"; in a project scenario, it is "Project - Contract"; and in an equipment operation and maintenance scenario, it is "System (Plant) - Equipment". Therefore, the types of business entity objects in this implementation include Group (Company), Customer, Project, Contract, System (Plant), and Equipment.

[0030] Step 2: Calculate the task modality fit, business entity matching, time matching, and semantic similarity of each candidate scheduling data for the current user problem, and then calculate the evidence fit of each candidate scheduling data for the current user problem.

[0031] 2.1) Based on the current user problem, determine the task types included in the current user problem and calculate the weight of each task type.

[0032] In this embodiment, the method for determining the task type included in the current user problem includes: First, address the current user issues. The input is fed into the task type recognizer, which outputs the current user question. The matching degree and confidence level of each task type are calculated. Then, the matching degree and confidence level of each task type are merged and normalized to obtain the weight of each task type. All task types with weights greater than a preset weight threshold are included as the task types covered by the current user question. In this embodiment, the choice of normalization method and value normalization process ensures that the sum of the weights of all task types is 1.

[0033] The specific formulas for calculating the weights of each task type are as follows: In the formula, Indicates the first Task types, Indicates the first Task types, For the first Weights for each task type For current user issues and the The degree of matching between task types For the set of all task types, For the first Confidence level for each task type As the matching degree weight, In this embodiment, to balance the degree of matching and the confidence level, the confidence level is set as a weight. .

[0034] In this embodiment, the task type identifier includes keyword triggering rules and a text classification model. The keyword triggering rules are used to output the degree of matching between the current user question and each task type, and the text classification model is used to output the confidence level of each task type.

[0035] Methods for outputting the degree of match between the current user's question and various task types include: Based on the various task types, keywords for each task type are determined. The current user question Q is then matched against the keywords for each task type. If a match is found with any keyword of a particular task type, the degree of match for that task type is calculated. Setting it to 1 indicates the match level for a given task type if none of the keywords match. Set to 0. In this implementation, task types include: information query, anomaly diagnosis, process explanation, data analysis, and task follow-up. In this implementation, the method for determining keywords includes: extracting the action words and business objects from the task type name as keywords for that task type. For example, keywords for information query include the action word "query" and "business object information"; keywords for anomaly diagnosis include the action word "judgment" and "business object anomaly"; keywords for process explanation include the action word "explanation" and "business object process"; and keywords for data analysis include the action word "analysis" and "business object data".

[0036] The text classification model consists of a feature extraction module and a fully connected layer. First, the feature extraction module encodes the current user question using a pre-trained language model (such as BERT), extracting and outputting a sentence vector representing the semantic features of the question. Then, this sentence vector is input into the fully connected layer, which performs a linear transformation using weight matrices and bias terms to calculate the initial score for each task type. Finally, the Softmax activation function is used to non-linearly map the initial scores, transforming them into continuous values ​​between 0 and 1, thus obtaining the confidence score for each task type.

[0037] In this embodiment, all task types with a weight greater than a preset weight threshold are considered as the task types included in the current user question. In this embodiment, the preset weight threshold is 0.

[0038] 2.2) Determine the evidentiary value of different modalities under different task types.

[0039] Because the evidentiary value of different modalities varies across different task types—for example, tabular data, business logs, and historical communication records have higher evidentiary value in risk assessment tasks, while image data, operation logs, and maintenance records have higher evidentiary value in anomaly diagnosis tasks—this implementation constructs a task modality evidentiary value matrix to quantify the evidentiary value of different modalities across different task types. The first of the matrix Line 1 Column data Indicates task type Lower mode The evidentiary value. In this embodiment, the task modality evidentiary value matrix. The construction methods include: Based on prior human experience data, the initial evidence value of each modality under different task types was calculated using the Analytic Hierarchy Process (AHP). Then, the initial evidence value of each modality under different task types was updated based on the historical question-and-answer data of the current dialogue, resulting in a task modality evidence value matrix. The analytic hierarchy process (AHP) is an existing technology and will not be described in detail in this embodiment.

[0040] In this embodiment, the generated question answer text is required to have an evidence number label that can identify the evidence, so as to count the number of times the candidate scheduling data of each modality is cited in the answer text of user questions of each task type in the historical question answer data of the current dialogue, and the total number of times the candidate scheduling data of each modality is processed as evidence data and input into the large language model when answering user questions of each task type in the historical question answer data.

[0041] For any modality under any task type, the effective contribution of the modality under that task type is calculated by dividing the number of times the candidate scheduling data of that modality is cited in the answer text of user questions of that task type in the historical question-answering data by the total number of times the candidate scheduling data of that modality is processed as evidence data and input into the large language model when answering user questions of that task type in the historical question-answering data. The initial evidence value of the modality under that task type is then updated.

[0042] The specific calculation methods for effective contribution include: In the formula, For task type Lower mode Effective contribution Task type in historical question-and-answer data Modal in the response text of user questions The number of times the candidate scheduling data is cited. For the types of response tasks in historical question-and-answer data User problem modality The total number of times candidate scheduling data is input into the large language model.

[0043] It should be noted that, in the same response, whether the candidate scheduling data is cited once or repeatedly, it is counted as one citation.

[0044] Update methods include: In the formula, For the updated task type Lower mode Evidence value For the task type before the update Lower mode Evidence value, i.e., task type Lower mode The initial evidentiary value, To update the step size, This represents the update amount of the evidentiary value. This represents the level of trust in and the speed of absorption of new rounds of real question-and-answer feedback data. If the value is too large (e.g., close to 1), the accumulated prior human experience within the matrix will be quickly erased by a single or small amount of abnormal reference data, causing drastic fluctuations in the parameters; if the value is too small (e.g., less than 0.001), the system's closed-loop update will be extremely slow, unable to adapt to changes in business preferences in a timely manner. In summary, in this implementation, Using 0.01 to 0.1 as a smoothing buffer can both stably preserve long-term experience and ensure that it steadily converges to actual business preferences in multiple rounds of the current conversation.

[0045] It should be noted that, if If the value is 0, then directly set 2.3) Combining the weights of each task type and the evidentiary value of different modalities under different task types, calculate the task modality fit of each candidate scheduling data for the current user problem.

[0046] Taking any candidate scheduling data as an example, the method for calculating its suitability for the current user problem's task modality includes: The modality of the candidate scheduling data is determined. Based on the task modality evidence value matrix, the evidence value of this modality under each task type included in the current user problem is determined. Then, according to the weight of each task type, the evidence values ​​of this modality under each task type are weighted and summed to obtain the task modality fit between the candidate scheduling data and the current user problem. The formula is expressed as follows: In the formula, For the first The fit between the candidate scheduling data and the task modality of the current user problem. This represents the set of task types included in the current user question. For the first The modalities of candidate scheduling data, For task type Next The modality of candidate scheduling data is considered as evidence. A higher task modality fit value indicates that the modality of the candidate scheduling data is more valuable in answering the current user question.

[0047] Based on candidate scheduling data, it is possible to determine whether the modality of each candidate scheduling data is suitable for the current task. However, relying solely on modality fit cannot directly determine whether the data is suitable as evidence for the current problem, because there may be a large amount of data with different content, different objects, and different times within the same modality. Therefore, this invention also needs to calculate business entity matching degree, time matching degree, and semantic similarity, as detailed below: 2.4) Calculate the matching degree between each candidate scheduling data and the business entity of the current user problem.

[0048] Based on task object For each candidate scheduling data item and its associated business entity, calculate the matching degree between each candidate scheduling data item and the business entity of the current user problem. The specific calculation method includes: For any candidate scheduling data, the business entity of that candidate scheduling data is determined, including the type of the business entity object, the number of the business entity object, and the number of the parent business entity object to which the business entity object belongs. If the business entity of the candidate scheduling data matches the task object... If the business entity objects are completely identical (including the type, number, and number of the parent business entity object), then the business entity matching degree between the candidate scheduling data and the current user problem is set as the first business entity matching degree; if the business entity object number of the candidate scheduling data is identical to the task object... The differences lie in the type of the business entity object and the number of the parent business entity object to which the business entity object belongs, as well as the task object. If they are the same, it means that they belong to different business entity objects under the same superior level. This candidate scheduling data has certain value. The matching degree between this candidate scheduling data and the current user problem's business entity is set as the second business entity matching degree. If the business entity object number of the candidate scheduling data, and the number of the superior business entity object to which the business entity object belongs, match the task object... They are different, but the type of the business entity object is the same as the task object. If they are the same, it means that they are only business entities of the same type. This candidate scheduling data has reference value, but the reference value is small. Therefore, the matching degree of this candidate scheduling data with the business entity of the current user problem is set as the third business entity matching degree. If the business entity of the candidate scheduling data is the same as the task object... If the business entity objects are completely different (including the type, number, and parent business entity number), then the business entity matching degree between the candidate scheduling data and the current user problem is set as the fourth business entity matching degree. The first business entity matching degree... Second business entity matching degree Third business entity matching degree The fourth business entity matching degree is as follows: in this embodiment, the first business entity matching degree is 1, the second business entity matching degree is 0.7, the third business entity matching degree is 0.4, and the fourth business entity matching degree is 0.

[0049] The formula for calculating the matching degree between each candidate scheduling data and the business entity of the current user problem is as follows: In the formula, For the first The degree of matching between candidate scheduling data and the business entity of the current user problem. Indicates the first Business entities and task objects of each candidate scheduling data Completely identical For task object The ID of the parent business entity object to which the business entity object belongs. For the first The number of the parent business entity object to which the business entity object of the candidate scheduling data belongs. For task object Business entity object type, For the first The business entity object type of each candidate scheduling data Indicates other cases, that is, the first The business entity with candidate scheduling data, and the task object Completely different.

[0050] It should be noted that if the task object or the Candidate scheduling data The business entity object does not have a parent business entity, that is... or If it does not exist, then the default is... Not equal to If the task object If the business entities in the candidate scheduling data contain multiple objects, then calculate the matching degree of all business entities in the set, and take the maximum value as the final business entity matching degree of the candidate scheduling data.

[0051] 2.5) Calculate the time matching degree between each candidate scheduling data and the current user problem.

[0052] Based on the generation time or update time of each candidate scheduling data and the time frame of current user issues. Calculate time matching degree The specific calculation methods include: In the formula, For the first The time matching degree between candidate scheduling data and the current user problem For the first The time deviation between the generation or update time of each candidate scheduling data and the time range of the current user question. The attenuation coefficient is... This indicates the operation of finding the minimum value. In this embodiment, when the first... When the generation or update time of candidate scheduling data is outside the time frame of the current user question, through Time deviation A negative correlation mapping is performed, mapping the data to the (0,1] interval. This ensures that the greater the time deviation, the smaller the time matching degree. In other implementations, other negative correlation mapping methods can also be used, such as... Methods such as taking the reciprocal.

[0053] In this embodiment, , and All are measured in days as the smallest unit. The dimensions are In this implementation, day -1 . This controls the rate at which the value of evidence decays after the time frame exceeds the specified limit. If the value is too large, the time matching accuracy will plummet to near zero as soon as the data time frame deviates from the specified range, even by just one or two days, which is overly stringent. If the value is too small, the matching accuracy will remain high even if the data time frame deviates from the range by several years, thus negating the purpose of the time range constraint. A value of 0.1 means that the matching accuracy decays to around 0.5 approximately 7 days after the boundary is crossed, and to an extremely low value after 30 days. This smooth exponential decay curve aligns with the time-sensitive expectations of typical business operations. Therefore, in this implementation... The value is 0.1.

[0054] 2.6) Calculate the semantic similarity between each candidate scheduling data and the current user problem.

[0055] Content representation based on each candidate scheduling data With current user issues Calculate the semantic similarity between each candidate scheduling data and the current user problem.

[0056] Specific calculation methods include: For any candidate scheduling data, express its content. With current user issues The data is input into the semantic matching model to obtain the semantic similarity between the candidate scheduling data and the current user question.

[0057] In this implementation, the semantic matching model uses the BERT model, which first combines the two input texts (content expression) With current user issues The input text is encoded into vectors, the cosine similarity between two vectors is calculated, and then mapped to the range of 0-1 to obtain the semantic similarity between the two input texts. This model is existing technology, and its specific structure and principle will not be described in detail in this embodiment.

[0058] 2.7) By integrating the task modality fit, business entity matching, time matching and semantic similarity of each candidate scheduling data with the current user problem, we can obtain the evidence fit between each candidate scheduling data and the current user problem.

[0059] For any candidate scheduling data, the method for calculating its evidence fit with the current user problem includes: In the formula, For the first The fit between candidate scheduling data and the evidence for the current user problem , , and These are the preset semantic similarity weight coefficient, task modality adaptation weight coefficient, business entity matching weight coefficient, and time matching weight coefficient. In this embodiment, to fully reflect the impact of different modalities of evidence having different values, a task modality fit weighting coefficient is set. The maximum value is 0.4, which is the semantic similarity weight coefficient. The value is 0.3, which is the weighting coefficient for business entity matching. The value is 0.2, representing the time matching degree weighting coefficient. The minimum is 0.1. It should be noted that... , , and The value of is not limited to the above values, but can be adjusted according to the actual situation. In one implementation, the specific values ​​of each weight coefficient can be adjusted based on the historical question and answer data of the current dialogue. For example, after a set number of question and answer sessions, if it is found that the time matching degree between the candidate scheduling data cited in the reply and the user's question is small, the time matching degree weight coefficient is further reduced, and other weight coefficients are increased accordingly to keep the total weight coefficient of 1.

[0060] Step 3: Based on the evidence fit between each candidate scheduling data and the current user question, select evidence data from each candidate scheduling data and parse it to generate evidence text; based on the current user question, the task types included in the current user question and the evidence text, generate the response text for the current user question.

[0061] 3.1) Based on the evidence fit between each candidate scheduling data and the current user problem, select evidence data from each candidate scheduling data.

[0062] In this embodiment, the candidate scheduling data are sorted according to the evidence fit from high to low to obtain the evidence sequence of the current user question. A predetermined number of candidate scheduling data or candidate scheduling data whose total text length is less than or equal to a preset length threshold are extracted from the evidence sequence (either condition is met first, and extraction stops there). In this embodiment, the predetermined number is 10, and the preset length threshold is 4000 characters.

[0063] 3.2) Analyze each of the selected pieces of evidence to generate the text of each piece of evidence.

[0064] Because the selected evidence includes multimodal data, the data analysis methods differ for different modalities, as detailed below: For text-based evidence, using user question Q as the query condition, keyword matching or semantic similarity matching is performed on the text, and the matched paragraphs and their pre-set context are extracted. For tabular modal evidence, depending on the task recipient and time range Locate the corresponding row, column, or field, and extract the field name, field value, statistical results, or outliers, then convert them into a table summary; For evidence data in the image modality, read the preprocessing results from step one and retain the descriptions related to the task object, abnormal state, or identification object. For evidence of speech modalities, transcribe it into text data and locate the speech segments containing task objects or related action words in the transcribed text; For log-based evidence, extract log time, operation object, operation type, state change, and operation result; The memory information already saved by digital employees is also stored in text mode, and its parsing method is the same as that used for parsing text-modal evidence.

[0065] Through the above processing, standardized evidence texts that can be processed by the large language model of each piece of evidence are obtained. ,in Indicates the first One piece of evidence, Indicates the first Evidence text of each piece of evidence.

[0066] 3.3) Based on each piece of evidence text, construct each piece of input evidence to obtain the input evidence sequence. Input the input evidence sequence, the current user question, the task types included in the current user question, and the format requirements for answering the question into the large language model to obtain the response text for the current user question.

[0067] In this embodiment, based on the first The modalities and data sources of candidate scheduling data for each piece of evidence are used to determine the first... The evidence modality and data source, together with the evidence text, constitute the first [evidence / evidence]. One piece of input evidence. Represented as: In the formula, Indicates the first Input evidence, Indicates the first Modalities of input evidence, For the first The data source for each piece of input evidence.

[0068] Each piece of evidence forms an input evidence, resulting in an input evidence sequence.

[0069] In this implementation, the large language model is required to prioritize citing the evidence text of the top-ranked candidate scheduling data when answering the current user question. When the top-ranked evidence text can already support the conclusion, the later evidence is only used as supplementary information. When there are inconsistencies among different pieces of evidence, the top-ranked evidence with the best data source is preferred. More concrete evidence, and explain the differences in your response.

[0070] In this implementation, the generated question-and-answer text is required to include an evidence number tag that identifies the evidence. After the answer is completed, the dialogue record is saved. The evidence number tags contained in the text are extracted from the response text Y of the large language model. Then, the number of times candidate scheduling data of a certain modality type was actually cited and the number of times it was input into the large language model in this generation are counted, and... , and Update the task modality evidence value matrix before answering the next user question. Update.

[0071] Through the above steps, this invention can first identify "what type of evidence the current task requires," thereby reducing the likelihood of semantically similar but insufficiently relevant data being prioritized for recall, and improving the effectiveness of the response text. Furthermore, this invention comprehensively determines evidence data based on task modality fit, business entity matching, time matching, and semantic similarity, rather than solely using semantic similarity for ranking. Even if a candidate scheduling data has low similarity to the user's question text, as long as its modality type, business entity, and time range are highly matched, it can still obtain a higher ranking position, thereby increasing the chances of key business evidence entering the response context.

Claims

1. A digital employee automated question-answering method based on cross-modal retrieval enhancement, characterized in that: Includes the following steps: 1) Construct candidate scheduling data based on multimodal data of digital employees; 2) Based on the user's question, determine the types of tasks included in the user's question; Based on prior human experience data, the initial evidence value of each modality under each task type is calculated. Then, based on the historical question-and-answer data of the current dialogue, the initial evidence value of each modality under each task type is updated to obtain the evidence value of each modality under each task type. Based on the evidentiary value of each modality under each task type, calculate the task modality fit of each candidate scheduling data for the user problem. Based on the content representation of each candidate scheduling data and the user question, calculate the semantic similarity between each candidate scheduling data and the user question; Based on the task modality fit and the semantic similarity, calculate the evidence fit of each candidate scheduling data for the user question; 3) Based on the evidence fit, select evidence data from each candidate scheduling data, and generate a response text for the user's question based on the evidence data.

2. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 1, characterized in that, Methods for calculating the task modality fit of each candidate scheduling data to the user problem include: Calculate the weights of each task type included in the user's question; For any candidate scheduling data, determine its mode; Based on the weights of each task type included in the user problem, the evidence value of that modality under each task type is weighted and summed to obtain the task modality fit of the candidate scheduling data for the user problem.

3. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 2, characterized in that, Methods for calculating the weights of the various task types included in a user question include: Based on user questions, determine the degree of matching between user questions and each task type, as well as the confidence level of each task type; For any task type, the matching degree and confidence of the task type are combined, and the weight of the task type is obtained based on the fusion result.

4. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 3, characterized in that, Methods for determining the degree of match between user questions and various task types include: Based on each task type, keywords for each task type are determined. User questions are then matched with the keywords for each task type, and the degree of matching between the user questions and each task type is obtained based on the matching results.

5. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 1, characterized in that, Methods for updating the initial evidence value for each modality under each task type include: Based on the historical question-and-answer data of the current dialogue, calculate the effective contribution of each modality under each task type; By integrating the effective contribution of each modality under each task type and the preset update step size, the evidence value update amount of the modality under the task type is obtained, and the initial evidence value of the modality under the task type is updated based on the evidence value update amount.

6. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 5, characterized in that, Methods for calculating the effective contribution of each modality under each task type include: For any modality under any task type, the effective contribution of that modality under that task type is obtained by dividing the number of times the candidate scheduling data of that modality is cited in the answer text of user questions of that task type in the historical question-answering data by the total number of times the candidate scheduling data of that modality is processed as evidence data and input into the large language model when answering user questions of that task type in the historical question-answering data.

7. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 1, characterized in that, Methods for calculating the evidence fit of each candidate scheduling data for the user problem include: Based on the user's question, the task object is determined, and the matching degree between each candidate scheduling data and the business entity of the user's question is calculated by combining the business entities of each candidate scheduling data. Based on the user question, the time range is determined, and the time matching degree between each candidate scheduling data and the user question is calculated by combining the generation time or update time of each candidate scheduling data. By integrating the task modality fit, the business entity matching degree, the time matching degree, and the semantic similarity, the evidence fit degree of each candidate scheduling data for the user question is obtained.

8. The automated question-answering method for digital employees based on cross-modal retrieval enhancement according to claim 7, characterized in that, Methods for calculating the time matching degree between each candidate scheduling data and the user problem include: For any candidate scheduling data, if the generation time or update time of the candidate scheduling data is not within the time range, then the time deviation between the generation time or update time of the candidate scheduling data and the time range is calculated, and the time deviation is negatively correlated to obtain the time matching degree between the candidate scheduling data and the user question.

9. A digital employee automated question-answering system based on cross-modal retrieval enhancement, characterized in that: It includes a question-and-answer module and a multimodal data acquisition module. The multimodal data acquisition module is used to acquire multimodal data and send it to the question-and-answer module. The question-and-answer module is used to acquire user questions and, based on the received data and user questions, to generate the answer text for the user questions using the digital employee automated question-and-answer method based on cross-modal retrieval enhancement as described in any one of claims 1-8.

Images