Information determination method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an information determination method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: in response to a received first text, retrieving from a first knowledge base to obtain first information; determining a first prompt word of the first text from a second knowledge base; fusing the first text with the first prompt word to obtain a second text; based on the second text, retrieving from the first knowledge base to obtain second information; and determining third information corresponding to the first text based on the first information and the second information. According to the invention, the accuracy of information determination can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an information determination method, device, electronic device, storage medium, and program product. Background Art

[0002] In information retrieval and knowledge management systems, efficiently and accurately retrieving relevant information is crucial for improving user experience. Traditional information determination methods often suffer from significant semantic discrepancies between the user input question and the correct information to be retrieved. As a result, the information retrieved from the knowledge base is incomplete or inaccurate. Summary of the Invention

[0003] The embodiments of the present application provide an information determination method, apparatus, electronic device, storage medium, and program product, which can improve the accuracy of information determination.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] This embodiment of the present application provides a method for determining information, the method comprising:

[0006] Retrieving first information from a first knowledge base in response to the received first text;

[0007] Determining a first prompt word of the first text from a second knowledge base;

[0008] fusing the first text with the first prompt word to obtain a second text;

[0009] Retrieving second information from the first knowledge base based on the second text;

[0010] Based on the first information and the second information, third information corresponding to the first text is determined.

[0011] An embodiment of the present application provides an information determination device, including:

[0012] A first retrieval module is configured to retrieve first information from a first knowledge base in response to a received first text;

[0013] a prompt word determination module, configured to determine a first prompt word of the first text from a second knowledge base;

[0014] a text fusion module, configured to fuse the first text with the first prompt word to obtain a second text;

[0015] A second retrieval module is configured to retrieve second information from the first knowledge base based on the second text;

[0016] An information determination module is used to determine third information corresponding to the first text based on the first information and the second information.

[0017] An embodiment of the present application provides an electronic device, comprising:

[0018] a memory for storing computer-executable instructions or computer programs;

[0019] The processor is used to implement the information determination method provided in the embodiment of the present application when executing the computer-executable instructions or computer program stored in the memory.

[0020] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the information determination method provided in the embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the information determination method provided in the embodiment of the present application is implemented.

[0022] The embodiments of the present application have the following beneficial effects:

[0023] First, based on the received first text, the first information is retrieved directly from the first knowledge base. Then, the first text is optimized using prompt words from the second knowledge base to generate a more accurate or semantically rich second text. Based on the second text, the second information is retrieved from the first knowledge base. Finally, the first and second information retrieved from these two searches are combined to comprehensively determine the final third information, improving the quality and accuracy of the information determination. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the architecture of the information determination system provided in an embodiment of the present application;

[0025] Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0026] Figure 3 This is a flow chart of the information determination method provided in the embodiment of the present application. Figure 1 ;

[0027] Figure 4 This is a flow chart of the information determination method provided in the embodiment of the present application. Figure 2 ;

[0028] Figure 5 This is a flow chart of the information determination method provided in the embodiment of the present application. Figure 3 ;

[0029] Figure 6 This is a flow chart of the information determination method provided in the embodiment of the present application. Figure 4 ;

[0030] Figure 7 This is a schematic diagram of the framework of the information recall system provided in the embodiment of the present application;

[0031] Figure 8 It is a schematic diagram of the principles of each memory module provided in the embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0034] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0035] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0036] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0037] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0038] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0039] 1) Information Recall: refers to retrieving correct information related to a user's query or need from a large amount of data. In the embodiment of the present application, information recall refers to retrieving information related to the first text input by the user from the first knowledge base.

[0040] 2) Knowledge base: A database or information system used to store, organize, and retrieve information. It aims to provide users with structured and unstructured data to support query, decision-making, and problem-solving. A knowledge base typically contains facts, rules, concepts, relationships, and other types of knowledge, and can be in various formats, such as text, images, and audio.

[0041] 3) Token: This refers to the basic unit of text segmentation, used for further analysis, processing, and understanding in Natural Language Processing (NLP), such as word frequency statistics, grammatical analysis, and sentiment analysis. A token can be a word, punctuation mark, phrase, or any character sequence divided according to a rule.

[0042] 4) Raw Output (Logits): This refers to the output value of the last layer of the model, which has not yet been converted to a probability distribution through an activation function (such as the Softmax function). In generation or classification tasks, Logits is a set of numerical values ​​calculated for each possible output token. For example, for a task with N possible output tokens, the model will output a Logit value for each token. Each Logit value reflects the model's unnormalized score for the likelihood of each token appearing at that position. Through the Softmax function, these Logit values ​​can be converted to a probability distribution.

[0043] 5) Fine-tuning: In natural language processing, fine-tuning refers to the process of further training a model pre-trained on a large-scale corpus to adapt it to a specific downstream task. This process leverages the general language understanding capabilities captured by the pre-trained model and, by training it on a smaller dataset in a specific domain, enables the model to learn domain knowledge.

[0044] 6) Retrieval-Augmented Generation (RAG): This is a natural language processing method that combines information retrieval and generative models to improve the quality and accuracy of generated text. First, information retrieval: Retrieving information relevant to the generation task from a large knowledge base or document collection. Then, generation: Combining the retrieved information with the internal generation process of the generative model to generate the answer or text. RAG enables the generated text to not only be based on the inherent understanding of the generative model but also reference external, specific, relevant information, thereby improving the accuracy and richness of the generated text.

[0045] 7) Prompt: A piece of text or instruction entered by the user when the large model generates or predicts a task. The prompt guides the large model to generate output of the type or content specified by the prompt. The prompt can be a question, command, rule, etc. The large model will generate a corresponding answer or continuation based on the context and semantics of the prompt.

[0046] 8) Agent Big Model: This refers to an agent built on a large-scale pre-trained model. It has access to application programming interfaces (APIs) and functions, and is capable of performing complex tasks such as understanding and generating text or other forms of data, as well as autonomously learning and adapting to new environments or objectives. Agent Big Models combine deep learning and reinforcement learning techniques, offering strong generalization and flexibility.

[0047] 9) Planning Big Model: A large, pre-trained model for planning tasks that enables strategy formulation and optimization in complex decision-making environments. It is suitable for applications such as automated plan generation, resource scheduling, and policy formulation. Planning Big Models typically combine deep learning techniques and reinforcement learning principles to improve decision accuracy and efficiency.

[0048] 10) Large Recall Model: This refers to a model used in information retrieval and recommendation systems that uses large-scale pre-training to efficiently identify and filter large amounts of recalled information relevant to a query or user's interests. The large recall model aims to improve recall accuracy and coverage, and is suitable for use in fields such as search engines and recommendation systems.

[0049] 11) Keywords: These are significant and representative words in a text that concisely and accurately reflect the core content and theme of the text. Keywords play a key role in tasks such as information retrieval, text summarization, and classification.

[0050] 12) Syntactic Structure: This refers to the organization and arrangement of the various components of a sentence, describing the rules for how words are combined into phrases, and how phrases are combined into sentences. Syntactic structure is typically represented as a tree, showing the hierarchical relationships between words and phrases. Syntactic structure includes syntax trees and dependency trees.

[0051] 13) Syntax Tree: Also known as a phrase structure tree or syntactic tree, this is a hierarchical structure that shows the relationships between the various components of a sentence, particularly how phrases are composed. Syntax trees are based on Phrase Structure Grammar, which defines a series of rules describing how phrases are constructed from words, and then sentences from phrases. For example, in a syntax tree for a text, the root node is the text, each node represents a phrase category (such as a noun phrase, a verb phrase, etc.), and leaf nodes represent specific words in the text.

[0052] 14) Dependency Tree: This is a representation method based on dependency grammar that emphasizes direct dependencies between words. A dependency tree describes sentence structure by annotating the relationships between each word and other words (such as subject-verb, verb-object, etc.). For example, in a text dependency tree, the root node is the central word (verb) of the text, and each edge connects nodes to other words in the text. Each edge represents a dependency relationship between the central word and other words, such as subject, object, and prepositional object.

[0053] In retrieval tasks, we often face the problem of significant semantic differences between the query and the correct item to be matched. When the query is the name of an entity, additional specific descriptions are usually required to enhance the matching degree, and the specific descriptions and additional definitions of the entity may change dynamically, making the matching more difficult. For example, various projects in an enterprise have different project names. When using a large model RAG to answer the recent dynamic progress of the project, the retrieval process usually searches the knowledge base for candidates with a high degree of match with the query (such as meeting content, document content, and project member progress that discussed the project). Matching is usually based on keyword regularization, vector similarity (embedding similarity), large model reasoning, etc.

[0054] For example, the query input by the user is: What are the recent progress and risks of the "intelligent knowledge" project?

[0055] Retrieval process: Break down the problem into "intelligent knowledge". "Intelligent knowledge" refers to projects that automatically help employees generate weekly reports, manage projects, and dynamic Kanban meeting minutes.

[0056] Items to be matched (item): item1 "original text of meeting 1", item2 "original text of meeting 2", item3 "document 1", etc.

[0057] Recall results: original text of Meeting 1 (the keyword "meeting minutes" appeared in the original text of Meeting 1, which is likely to be discussing the project); Document 4 (the content "automation helps employees generate weekly reports" appeared in Document 4, which is likely to be discussing the project).

[0058] Answer generation: Integrate the content answers from meeting 1 and document 4.

[0059] It is often difficult to directly match the project (query) name with the project description, project dynamics, meetings, and documents to be matched semantically. In addition, the project definition, specific work direction, and content may change constantly. For example, if the company's focus on the project changes or a new project emerges, it will become difficult to search and recall the database to answer questions about the project's recent progress risks.

[0060] There are three types of recalls:

[0061] 1) Rule-based approach: Keywords are mapped to each query, and manually defined regular expressions are used to determine whether the text of the item to be matched matches the keywords. This method is fast, but has low recall accuracy and poor results.

[0062] 2) Text matching. This method uses traditional machine learning methods such as convolutional neural networks (CNN), recurrent neural networks (RNN), pre-trained models (Bidirectional Encoder Representations from Transformers, BERT), or large models to extract semantic vector representations of the query and item to be matched. A classification model is then trained to determine the degree of match between the query and item. This method requires a large amount of training data and requires retraining when new item entities appear or when the definition of an item changes. This method is not scalable and, if a large model is used, the training cost is extremely high.

[0063] 3) Large-scale model judgment. Using the project definition and any available additional information, a prompt word is constructed, allowing the large-scale model to infer the degree of match between the query and the item to be matched. This method requires customized prompt words and requires retraining when new project entities appear or when the project definition changes, resulting in limited scalability.

[0064] Based on the problems existing in the related art, the embodiments of the present application provide an information determination method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of information determination. In the information determination method provided in the embodiments of the present application, first, in response to a received first text, first information is retrieved from a first knowledge base; then, a first prompt word of the first text is determined from a second knowledge base; the first text and the first prompt word are fused to obtain a second text; then, based on the second text, second information is retrieved from the first knowledge base; finally, based on the first information and the second information, third information corresponding to the first text is determined.

[0065] The following describes an exemplary application of the information determination device provided in the embodiment of the present application, which is an electronic device for implementing the information determination method. The electronic device provided in the embodiment of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as servers. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. Below, the exemplary application of the information determination device when it is implemented as a terminal or a server will be described.

[0066] See also Figure 1 , Figure 1 : It is a schematic diagram of the architecture of the information determination system provided in the embodiment of the present application. In order to perform the information determination operation, an information determination application can be provided. For example, the information determination application can be an application dedicated to information determination, or it can be a functional module in other applications (such as an information determination module in an intelligent dialogue application, etc.). The information determination system 100 in the embodiment of the present application includes at least a terminal 400, a network 300 and a server 200, wherein the server 200 is a server for the information determination application. The server 200 can constitute the information determination device of the embodiment of the present application, that is, the information determination method of the embodiment of the present application is implemented through the server 200. The terminal 400 is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two.

[0067] See also Figure 1, the user can perform interactive operations on the client side of the information determination application through the terminal 400. The interactive operations may be, for example, inputting the first text to be queried, searching for information, etc. After receiving the interactive operation of the user, the client side sends an information determination request to the server 200 through the network 300. After receiving the information determination request, the server 200 responds to the information determination request sent by the terminal and retrieves the first information from the first knowledge base in response to the received first text; the server 200 determines the first prompt word of the first text from the second knowledge base; the server 200 fuses the first text with the first prompt word to obtain the second text; the server 200 retrieves the second information from the first knowledge base based on the second text; the server 200 determines the third information corresponding to the first text based on the first information and the second information. The server 200 may send the third information to the terminal 400. The terminal 400 displays the third information on the current interface.

[0068] In some embodiments, the terminal 400 may also perform the information determination method of the embodiment of the present application. That is, after the user performs an interactive operation on the client of the information determination application through the terminal 400, the terminal 400 retrieves the first information from the first knowledge base in response to the interactive operation and the received first text; the terminal 400 determines the first prompt word of the first text from the second knowledge base; the terminal 400 fuses the first text with the first prompt word to obtain the second text; the terminal 400 retrieves the second information from the first knowledge base based on the second text; the terminal 400 determines the third information corresponding to the first text based on the first information and the second information. The terminal 400 displays the third information on the current interface.

[0069] In the scenario of an intelligent customer service system on an e-commerce platform, users can ask questions in natural language to obtain product information or solve common problems. The question entered by the user is determined as a first text. In response to the received first text, first information is retrieved from a first knowledge base. For example, if the first text is "What is the battery life of this phone?", basic information about the phone (such as brand, model, screen size, etc.) is retrieved from the first knowledge base as the first information. A first prompt word for the first text is determined from a second knowledge base, such as a first prompt word related to "battery life" (such as charging time, standby time, and frequency of use). The first text and the first prompt word are fused to obtain a second text. Based on the second text, second information is retrieved from the first knowledge base. Based on the first and second information, third information corresponding to the first text is determined, such as "The battery capacity of this phone is 4000 mAh, the battery life under normal use is one and a half days, the charging time is 1.5 hours, and the standby time is up to 10 days." The third information is then presented to the user.

[0070] In the enterprise knowledge management system scenario, users can ask questions in natural language to obtain relevant information about the company's internal policies, processes, and projects. The question entered by the user is determined as the first text. In response to the received first text, the first information is retrieved from the first knowledge base. For example, if the first text is "How to apply for travel reimbursement", the basic process of travel reimbursement is retrieved from the first knowledge base as the first information; the first prompt word of the first text is determined from the second knowledge base (such as invoice requirements, approval process, reimbursement time limit, special expenses, etc.); the first text and the first prompt word are merged to obtain the second text; based on the second text, the second information is retrieved from the first knowledge base; based on the first information and the second information, the third information corresponding to the first text is determined. For example, the third information is "Fill out the travel application form and submit it for approval within 7 working days after the end of the business trip." The third information is presented to the user.

[0071] See also Figure 2 , Figure 2 is a structural diagram of an electronic device provided in an embodiment of the present application, Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the electronic device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2 Various buses are labeled as bus system 440 .

[0072] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0073] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0074] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0075] The memory 450 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0076] In some embodiments, the memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0077] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0078] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0079] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0080] The input processing module 454 is configured to detect one or more user inputs or interactions from one of the one or more input devices 432 and to translate the detected inputs or interactions.

[0081] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The information determination device 455 stored in the memory 450 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a first search module 4551, a prompt word determination module 4552, a text fusion module 4553, a second search module 4554, and an information determination module 4555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0082] In other embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the information determination method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0083] The following describes the information determination method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the information determination method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0084] It should be noted that, based on their understanding of the following text, those skilled in the art can apply the information determination method provided in the embodiments of the present application to any scenario that requires information determination or intelligent dialogue, such as: enterprise knowledge management system scenarios, intelligent document retrieval scenarios, human resources management system scenarios, medical auxiliary consultation scenarios, education and learning platform scenarios, tourism service platform scenarios, intelligent dialogue robot scenarios, etc.

[0085] See also Figure 3 , Figure 3 This is a flow chart of the information determination method provided in the embodiment of the present application. Figure 1 , will combine Figure 3 The steps shown are explained as Figure 3 As shown, the information determination method is described as an example in which the execution subject is a server. The method includes the following steps 101 to 105:

[0086] In step 101, in response to a received first text, first information is retrieved from a first knowledge base.

[0087] Here, the first text refers to the initial query content input by the user or received by the information determination system. The first text can be a sentence, phrase, or keyword in natural language that expresses the user's information needs. The user can enter the query content via voice, text, or video. If the user enters the query content via text, the text input by the user is directly determined as the first text. If the user enters the query content via voice or video, the voice or video input by the user is subjected to voice recognition processing to obtain the first text. The first knowledge base is a database or data set that stores at least multiple pieces of information to be recalled. Exemplarily, the first knowledge base can be an internal enterprise knowledge base, a public database, a literature database, etc. The first information refers to the information retrieved from the first knowledge base by the information determination system based on the first text. The first information can be in various forms, such as text, tables, charts, images, voice, links, and videos. Based on the first text, one or more pieces of first information can be retrieved from the first knowledge base.

[0088] For example, the user enters the first text "What is the recent progress of the intelligent knowledge project?" The first knowledge base is an internal enterprise knowledge management system or project management platform, containing documents, meeting minutes, progress reports, and other information from all projects. The first knowledge base contains 1,000 pieces of information to be retrieved. Based on the first text, 50 pieces of first information are retrieved from the first knowledge base. These 50 pieces of first information include original meeting texts and documents related to the intelligent knowledge project.

[0089] In some embodiments, after a user inputs a first text, the first text can be rewritten to obtain at least one rewritten text, and the first text can be decomposed to obtain at least one subtext, and at least one hypothetical answer text of the first text can be determined. The first text, at least one rewritten text, at least one subtext, and at least one hypothetical answer text are constructed as a first text set. The first text in subsequent embodiments can be any text in the first text set. The embodiments of the present application can improve the comprehensiveness of information determination and thus improve accuracy by expanding the original first text in different ways.

[0090] In some embodiments, retrieving the first information from the first knowledge base in response to the received first text in step 101 can be achieved in the following manner: first, performing keyword extraction on the first text to obtain the first keyword; then, performing grammatical analysis on the first text to obtain the first syntactic structure; finally, based on the first keyword in the first text and the first syntactic structure of the first text, retrieving the first information from the first knowledge base; and / or, based on the first feature vector of the first text, retrieving the first information from the first knowledge base.

[0091] Here, the first keyword is the core word or phrase in the first text, which can reflect the core content of the first text and represent the user's query intention. The method for extracting keywords from the first text in the embodiments of this application is not limited. For example, the first keyword can be identified based on a rule-based keyword extraction method. First, the first text can be tokenized to obtain multiple words. After removing meaningless words (such as "of", "is", "excuse me") from the multiple words, each remaining word is tagged with a词性 (such as noun, verb, adjective, etc.). Based on the词性, representative words are selected as the first keyword. The number of first keywords can be one or more. Generally, nouns, verbs, and adjectives are selected as the first keyword. Or, the first keyword can be identified based on a statistical keyword extraction method. First, the word frequency and inverse document frequency of each word in the first text are calculated, and then several words with higher word frequency and inverse document frequency are selected as the first keyword. Exemplary, for the first text "Excuse me, what is the recent progress of the intelligent knowledge project?", the first keywords can include "intelligent knowledge" and "progress".

[0092] The first syntactic structure can be the structure obtained by performing syntactic analysis on the first text, which is used to represent the subject-predicate-object relationship and modifying words and other components of the first text. The first syntactic structure can include a syntax tree and / or a dependency tree. A syntax tree is a tree-like structure based on constituent syntactic analysis that decomposes the first text into phrases and shows the hierarchical relationship between these phrases. The syntax tree is constructed as follows: First, the first text is decomposed into multiple words, and the词性 of each word is tagged. Then, noun words, verb words, prepositional words, etc. in the first text are identified. The syntax tree is constructed based on the relationship between the multiple words. Exemplary, for the first text "Excuse me, what is the recent progress of the intelligent knowledge project?", the root node of the syntax tree is the first text, and the root node has three child nodes, namely the subject, the predicate, and the modifier. There is no subject in the first text. The child node of the predicate node is the word "is", and the object is "progress". The child nodes of the modifier node are the adjective word "recent" and the noun phrase "intelligent knowledge project". A dependency tree is a tree-like structure based on dependency syntactic analysis that shows the dependency relationship between the words in the first text. Each word is used as a node, and the edge represents the dependency relationship between the words (such as subject-predicate relationship, verb-object relationship, etc.). A dependency syntactic analysis tool can be used to parse the first text to identify the dependency relationship between the words, and the dependency tree is constructed based on the dependency relationship.

[0093] It should be noted that there may be some inaccuracies in the translation of "词性" as it's not clear what the specific Chinese term refers to in this context. It might be better to have more context clarification for a more accurate translation.In some embodiments, retrieving first information from a first knowledge base based on a first keyword in a first text and a first syntactic structure of the first text can be achieved in the following manner: first, for each piece of information to be recalled in the first knowledge base, determining a second keyword in the information and a second syntactic structure of the information from the first knowledge base; then, determining a ratio of a first quantity to a second quantity, wherein the first quantity is the number of first keywords that are identical to the second keyword, and the second quantity is the total number of first keywords; finally, when the ratio is greater than a first threshold and the first similarity is greater than a second threshold, determining the information as first information, wherein the first similarity is the similarity between the first syntactic structure and the second syntactic structure.

[0094] Here, the first knowledge base may also store the second keyword and second syntactic structure of each piece of information. The first knowledge base may store a first mapping table that stores the associations between the second keyword and the second syntactic structure, respectively, and the information. That is, for each piece of information to be recalled in the first knowledge base, the second keyword and second syntactic structure associated with the information can be directly obtained from the first mapping table of the first knowledge base, and the second keyword and second syntactic structure associated with the information can be used as the second keyword and second syntactic structure in the information. Alternatively, if the first knowledge base does not store the second keyword and second syntactic structure, the second keyword and second syntactic structure in each piece of information can be determined using a method consistent with determining the first keyword and first syntactic structure of the first text. For each piece of information, the presence of the first keyword is checked against at least one second keyword in the information, and the number of first keywords that are identical to the second keyword is determined as the first number. The total number of the multiple first keywords in the first text is determined as the second number. The first similarity between the first and second syntactic structures is determined. It should be noted that the specific values ​​of the first and second thresholds are not limited in this embodiment of the present application and can be set based on actual needs.

[0095] For example, the first text is "What is the recent progress of the intelligent knowledge project?", and the multiple first keywords in the first text include "intelligent knowledge" and "progress", and the second quantity is 2. The first information to be recalled in the first knowledge base is the original text of Meeting 1. The multiple second keywords of the original text of Meeting 1 include "intelligent knowledge", "meeting minutes", "discussion" and "progress", then the second keywords that are the same as the first keywords are "intelligent knowledge" and "progress", the first quantity is 2, and the ratio of the first quantity to the second quantity is 1. The first similarity between the first syntactic structure and the second syntactic structure is calculated to be 0.67, the first threshold is assumed to be 0.8, and the second threshold is assumed to be 0.6, then the ratio of the first quantity to the second quantity "1" is greater than the first threshold of 0.8, the first similarity is greater than the second threshold, and the original text of Meeting 1 is determined to be the first information.

[0096] In some embodiments, for each piece of information to be recalled, the first syntactic structure of the information includes a first grammatical tree and a first dependency tree, and the second syntactic structure includes a second grammatical tree and a second dependency tree. The fourth similarity between the first grammatical tree and the second grammatical tree, and the fifth similarity between the first dependency tree and the second dependency tree can be determined respectively. A preset fifth threshold set for the grammatical tree and a sixth threshold set for the dependency tree are obtained. The embodiment of the present application does not limit the specific values ​​of the fifth threshold and the sixth threshold, and can be set based on actual needs. If the ratio of the first quantity to the second quantity is greater than the first threshold and the fourth similarity is greater than the fifth threshold, the information is determined to be the first information; or if the ratio of the first quantity to the second quantity is greater than the first threshold and the fifth similarity is greater than the sixth threshold, the information is determined to be the first information. If the ratio of the first quantity to the second quantity is less than or equal to the first threshold, or if the ratio of the first quantity to the second quantity is greater than the first threshold and the fourth similarity is less than or equal to the fifth threshold and the fifth similarity is less than or equal to the sixth threshold, the information is skipped and it is determined whether the next information in the first knowledge base is the first information.

[0097] The fourth similarity between the first syntax tree and the second syntax tree can be determined in the following way: first, extract multiple first subtrees from the first syntax tree, and extract multiple second subtrees from the second syntax tree. Then, determine the number of subtrees that match between the multiple first subtrees and the multiple second subtrees, and determine the ratio of the number of matched subtrees to the total number of subtrees as the fourth similarity. For example, the first text is "What is the recent progress of the intelligent knowledge project?" and the information to be recalled is "The progress of the intelligent knowledge project was discussed at the most recent meeting." Then, the first syntax tree of the first text has three first subtrees, and the second syntax tree of the information also has three second subtrees. The matched subtrees are the subject "intelligent knowledge project" and the predicate + object "progress." The number of matched subtrees is 2, and the total number of subtrees is 6. Then the fourth similarity is 2 / 6=0.33.

[0098] The fifth similarity between the first dependency tree and the second dependency tree can be calculated by comparing the dependency paths of the first dependency tree and the second dependency tree, and calculating the ratio of the number of matched paths to the total number of paths as the fifth similarity. For example, if the first text is "What is the recent progress of the intelligent knowledge project?" and the information to be recalled is "The progress of the intelligent knowledge project was discussed at the most recent meeting," then the matched dependency path is "intelligent knowledge project → discussion," the number of matched paths is 1, and the total number of paths is 6, then the fifth similarity is 1 / 6.

[0099] The embodiment of the present application ensures that the core vocabulary of the query intent of the first text is fully considered by extracting the first keyword from the first text and accurately matching it with the second keyword in the first knowledge base. Calculating the ratio can quantify the degree of keyword matching, thereby screening out highly relevant information to be recalled. The similarity calculation of the first syntactic structure and the second syntactic structure is introduced to achieve not only focusing on the matching at the lexical level, but also going deep into the grammatical and semantic structure of the text, which can more comprehensively understand the logical relationship and context of the user query, thereby improving the relevance and accuracy of the information corresponding to the first text.

[0100] The first feature vector is a numerical vector obtained by encoding the first text, representing the semantic features of the first text. The first feature vector can be generated using natural language processing (NLP) techniques, such as bag-of-words, word embedding, or a pre-trained language model (such as BERT).

[0101] In an embodiment of the present application, the first information can be retrieved from the first knowledge base only through the first keyword in the first text and the first syntactic structure of the first text, or the first information can be retrieved from the first knowledge base only based on the first feature vector of the first text, or the information retrieved through the first keyword in the first text and the first syntactic structure of the first text, and the information retrieved based on the first feature vector of the first text are jointly determined as the first information.

[0102] In some embodiments, retrieving first information from a first knowledge base based on a first feature vector of a first text can be achieved in the following manner: first, for each piece of information to be recalled in the first knowledge base, determining a second feature vector of the information from the first knowledge base; then, when the second similarity between the first feature vector and the second feature vector is greater than a third threshold, determining the information as the first information.

[0103] Here, the first knowledge base may also store a second feature vector for each piece of information, and the first knowledge base may store a second mapping table that stores the association between the second feature vector and the information. The first mapping table and the second mapping table may be the same mapping table or different mapping tables. In this case, for each piece of information to be recalled in the first knowledge base, the second feature vector associated with the information is directly obtained from the second mapping table of the first knowledge base. Alternatively, if the first knowledge base does not store the second feature vector, if the information is in text form, the content of the information can be first summarized using a pre-trained large model to obtain a summary text; if the information is in audio or video form, the information can be subjected to text-to-speech recognition processing to obtain text, which is then summarized using the pre-trained large model to obtain a summary text. The summary text is encoded using a method consistent with the method used to determine the first feature vector of the first text to obtain the second feature vector of the information. This embodiment of the present application does not limit the method for calculating the second similarity. For example, the second similarity can be obtained by calculating the cosine similarity, Euclidean distance, or Manhattan distance between the first and second feature vectors. A third threshold (e.g., 0.8) is set based on the application scenario and requirements to filter out information that meets the conditions. If the second similarity between the first eigenvector and the second eigenvector is greater than the third threshold, the information is determined to be the first information. If the second similarity between the first eigenvector and the second eigenvector is less than or equal to the third threshold, the information is skipped and the next information in the first knowledge base is determined to be the first information.

[0104] The embodiment of the present application retrieves the first information matching the first text by combining multiple methods such as keyword matching, syntactic structure similarity, and text vector similarity, which can improve the accuracy of information determination.

[0105] In step 102, a first prompt word of a first text is determined from a second knowledge base.

[0106] Here, the second knowledge base is a database for storing multiple first prompt words. The second knowledge base and the first knowledge base can be two independent knowledge bases or two subsets of a single knowledge base. A first text corresponds to one or more first prompt words, and a first prompt word corresponds to a type. The present embodiment does not limit the specific method for classifying types, and can be set based on actual needs. For example, in an enterprise project query scenario, the type can be the project type involved in the first text. The second knowledge base pre-stores the first prompt words for multiple project types. The first prompt word corresponding to the project type involved in the first text is selected from the first prompt words of the multiple project types and used as the first prompt word for the first text. The first prompt word can be not only a word, but also a short sentence or a long text. After the user enters the first text, the type involved in the first text can be determined, and based on the type, the first prompt word for the first text can be retrieved from the second knowledge base. For example, in an enterprise knowledge management scenario, the first text entered by the user may be a query question for different projects. Therefore, each project in the enterprise can be considered a type, and a first prompt word can be determined for each project and stored in the second knowledge base. For the first text "What is the recent progress of the intelligent knowledge project?", the project name is "intelligent knowledge", the type is intelligent knowledge, and the first prompt words obtained from the second knowledge base are "automatically generated weekly report", "management project", "dynamic Kanban meeting minutes", "documents involving artificial intelligence", etc.

[0107] The first prompt word can be set in advance or generated in advance based on the big model. The generation of the first prompt word in advance based on the big model can be achieved in the following way: obtain a training set, which includes a query text and multiple information matching the query text. The query text and the multiple information matching the query text are input into the planning big model (Planning big model). The planning big model generates an initialization definition description, a note text, etc. based on the rules and paradigms of the multiple information matching the query text, and determines the initialization definition description, the note text, etc. as the first prompt word for the type of the query text. For example, if the query text also involves an intelligent knowledge project, a first prompt word for the type of intelligent knowledge is obtained.

[0108] In step 103, the first text and the first prompt word are fused to obtain a second text.

[0109] Here, the first text and the first prompt word are fused, that is, directly combined to obtain the second text. For example, if the first text is "What is the recent progress of the intelligent knowledge project?" and the first prompt word is "Automated weekly report generation," "Project management," "Dynamic Kanban meeting minutes," and "Documents involving artificial intelligence," the fused second text will be "What is the recent progress of the intelligent knowledge project? Automated weekly report generation, Project management, Dynamic Kanban meeting minutes, Documents involving artificial intelligence."

[0110] In step 104, second information is retrieved from the first knowledge base based on the second text.

[0111] Here, the second text can be input into the pre-trained recall model, and the pre-trained recall model performs prediction processing on multiple information in the first knowledge base based on the second text to obtain second information corresponding to the second text.

[0112] In some embodiments, see Figure 4 , Figure 4 It is shown that in step 104, the second information is retrieved from the first knowledge base based on the second text, which can be achieved by the following steps 1041 to 1043:

[0113] In step 1041 , for each piece of information to be recalled in the first knowledge base, the second text and the information are concatenated to obtain a third text.

[0114] Here, for each piece of information to be recalled in the first knowledge base, the second text may be combined with the information to generate a new third text. Combining the second text and the information may be directly concatenating the second text and the information.

[0115] In step 1042 , a matching degree prediction process is performed on the third text to obtain a first matching degree between the information and the second text.

[0116] Here, for each third text in the plurality of third texts, the pre-trained recall model can be used to perform a matching prediction process on the third text to obtain a first matching degree between the information corresponding to the third text and the second text. It should be noted that the embodiment of the present application does not limit the model structure of the recall model, for example, it can be a deep learning model, etc. The pre-trained recall model can be implemented in the following way to perform a matching prediction process on the third text: first, the word segmenter in the recall model performs a word segmentation process on the third text to obtain multiple word element tokens, where the word element token can be any word in the third text. Each word element is embedded (Embedding) by a word embedding method to obtain a feature vector of each word element. The feature vector of each word element is positionally encoded to obtain an encoded input vector sequence. The input vector sequence is encoded by the multi-layer encoder (multi-head self-attention mechanism and feedforward neural network) of the recall model to obtain a context representation vector. The context representation vector is decoded by the decoder of the recall model to obtain a first matching degree between the information and the second text. The first matching degree can be the original output logit of the recall large model without activation function processing. Alternatively, the first matching degree can be the probability obtained by activating the original output logit based on the activation function (such as the softmax function). Exemplarily, when the first matching degree is a probability, the first matching degree can be a value between 0 and 1, or the first matching degree can be a value between 0 and 1.

[0117] In step 1043 , when the first matching degree is greater than a fourth threshold, the information is determined as second information.

[0118] Here, the embodiment of the present application does not specifically limit the value of the fourth threshold, and the fourth threshold can be a value between 0 and 1. For example, if the first matching degree between the information "Meeting 1 Original Text" in the first knowledge base and the second text is 1 and the fourth threshold is 0.9, the information "Meeting 1 Original Text" is determined to be the second information.

[0119] This embodiment of the application combines the second text with each piece of information to be recalled in the first knowledge base to form a third text, and uses a pre-trained recall model to predict the matching degree of the third text, thereby calculating a first matching degree between the information and the second text. When the first matching degree exceeds a set fourth threshold, the corresponding information is determined to be the second information. This effectively filters out items highly relevant to the query text from a large amount of knowledge base data, improving the accuracy and efficiency of information retrieval.

[0120] In step 105 , third information corresponding to the first text is determined based on the first information and the second information.

[0121] Here, the first information and the second information can be directly determined as the third information corresponding to the first text. Alternatively, at least one piece of information that overlaps between multiple first information and multiple second information can be determined, and the at least one piece of information that overlaps can be determined as the third information. Alternatively, multiple first information and multiple second information can be used as candidate information, and for each candidate information, the first similarity, second similarity, and first matching degree corresponding to the candidate information can be determined. The first similarity corresponding to the candidate information is normalized to obtain a first normalized value, the second similarity corresponding to the candidate information is normalized to obtain a second normalized value, and the first matching degree corresponding to the candidate information is normalized to obtain a third normalized value. The first normalized value, the second normalized value, and the third normalized value are weighted to obtain a comprehensive normalized value. The multiple candidate information are sorted from large to small according to the comprehensive normalized value, and the candidate information corresponding to the largest comprehensive normalized value is selected as the third information, or a preset number of candidate information ranked first is selected as the third information.

[0122] In some embodiments, after obtaining the third information, text generation processing may be performed on the third information based on a pre-trained generative model to obtain a reply text for the user.

[0123] In this embodiment, first information is retrieved directly from a first knowledge base based on a received first text. The first text is optimized using prompt words from a second knowledge base to generate a more accurate or semantically rich second text, which is then used to retrieve second information from the first knowledge base based on the second text. The final third information is determined by combining the first and second information retrieved from these two searches, thereby improving the quality and accuracy of the information determined.

[0124] In some embodiments, the first knowledge base includes a third keyword of the first information. Figure 5 The information determination method provided in the embodiment of the present application further includes the following steps 201 to 202:

[0125] In step 201 , when a feedback result indicating that the first information does not match the first text is received, a fourth keyword is determined.

[0126] The fourth keyword is the third keyword that is the same as the first keyword of the first text.

[0127] Here, the first information does not match the first text, which means that the first information does not match the actual intention of the user to enter the first text. The feedback result refers to the feedback provided by the user or the information determination system after evaluating the first information, which is used to indicate whether the first information meets expectations. Exemplarily, when the user finds that one or more first information among multiple first information do not match the first text, one or more feedback results that the first information does not match the first text can be actively returned. For each unmatched first information, the first knowledge base stores multiple third keywords in the first information. A third keyword that is the same as any first keyword in the first text is obtained from the multiple third keywords and determined as the fourth keyword.

[0128] In step 202, the fourth keyword in the first information is deleted from the first knowledge base to obtain an updated first knowledge base.

[0129] Here, for each unmatched first information, the fourth keyword corresponding to the first information is deleted from the plurality of third keywords in the first information stored in the first knowledge base, thereby generating an updated first knowledge base. The updated first knowledge base is then used to retrieve the first information from the updated first knowledge base based on the new query text when the user enters a new query text the next time.

[0130] After receiving mismatch feedback, the present embodiment identifies and deletes the key factor causing the mismatch (i.e., the fourth keyword), thereby continuously optimizing the first knowledge base and ensuring that subsequent information retrieval is more accurate and relevant. This not only improves the accuracy of the information determination system, but also enhances its adaptive capabilities, enabling it to better meet user needs.

[0131] In some embodiments, the user may also proactively return feedback indicating that the fifth information matches the first text, where the fifth information is different from the first and second information. When the fifth information already exists in the first knowledge base, the keywords of the fifth information are updated, and the keywords of the updated fifth information are added to the first knowledge base to obtain an updated first knowledge base. When the fifth information does not exist in the first knowledge base, the keywords of the fifth information are determined, and the fifth information and the keywords of the fifth information are added to the first knowledge base to obtain an updated first knowledge base.

[0132] It should be noted that, when a feedback result is received indicating that the first information does not match the first text, or when a feedback result is received indicating that the fifth information matches the first text, the feature vector of the first information in the first knowledge base can also be updated, that is, the content of the first information is summarized through a large model to obtain an updated summary text, and the updated summary text is encoded to obtain a feature vector.

[0133] In some embodiments, see Figure 6 The information determination method provided in the embodiment of the present application further includes the following steps 301 to 303:

[0134] In step 301, when a feedback result indicating that the first information does not match the first text is received, the first prompt word is corrected based on the first information to obtain a second prompt word.

[0135] Here, when a feedback result indicating that the first information does not match the first text is received, the first information and the first prompt word can be input into a pre-trained planning model (Planning model). The planning model corrects the first prompt word based on the first information to obtain a second prompt word.

[0136] In step 302 , a second matching degree between the first information and the second text is determined.

[0137] Here, the specific process of determining the second matching degree between the first information and the second text can refer to the embodiment of performing matching degree prediction processing on the third text in the above step 1042 to obtain the first matching degree between the information and the second text, which will not be explained here.

[0138] In step 303, based on the second prompt word and the second matching degree, the first prompt word in the second knowledge base is updated to obtain an updated second knowledge base.

[0139] The embodiment of the present application can dynamically correct the first prompt word and re-evaluate the matching degree after receiving the mismatch feedback result, thereby continuously optimizing the content of the second knowledge base, which not only improves the accuracy of information determination, but also enhances the adaptive ability of the information determination system, enabling it to better respond to changes in user needs.

[0140] In some embodiments, based on the second prompt word and the second matching degree, the first prompt word in the second knowledge base is updated to obtain an updated second knowledge base, which can be achieved in the following way: first, the first text and the second prompt word are merged to obtain a fifth text, and the third matching degree between the first information and the fifth text is determined; then, when the third matching degree is less than the second matching degree, the first prompt word in the second knowledge base is replaced with the second prompt word to obtain an updated second knowledge base.

[0141] Here, the fusion of the first text and the second prompt word can be performed by directly concatenating the first text and the second prompt word to obtain a fifth text. The fifth text is concatenated with the first information to generate a new sixth text. The sixth text is subjected to a matching degree prediction process to obtain a third matching degree between the first information and the fifth text. The specific process of performing a matching degree prediction process on the sixth text to obtain the third matching degree between the first information and the fifth text can refer to the embodiment of performing a matching degree prediction process on the third text to obtain the first matching degree between the information and the second text in step 1042 above, which will not be described here. If the third matching degree is less than the second matching degree, it indicates that the updated second prompt word has a positive effect on information determination. The first prompt word in the second knowledge base can be replaced with the second prompt word to obtain an updated second knowledge base. Alternatively, if the third matching degree is greater than or equal to the second matching degree, it indicates that the updated second prompt word has a negative effect on information determination. Step 301 is repeated to generate a new second prompt word until the third matching degree is less than the second matching degree.

[0142] In some embodiments, upon receiving feedback indicating a match between the fifth information and the first text, the fifth information and the first prompt word are input into a pre-trained large planning model. The large planning model then modifies the first prompt word based on the fifth information to produce a second prompt word. At this point, a fourth degree of match between the fifth information and the second text can be determined. If the fourth degree of match is greater than or equal to the second degree of match, the updated second prompt word has a positive effect on information determination, and the first prompt word in the second knowledge base can be replaced with the second prompt word to produce an updated second knowledge base.

[0143] The embodiment of the present application dynamically corrects the prompt word and re-evaluates the matching degree after receiving the mismatch feedback result, thereby ensuring that the prompt word in the second knowledge base can better reflect the user's query intention and improving the accuracy of information determination.

[0144] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0145] An embodiment of the present application provides an information determination method, which is a large-scale intelligent recall and sorting method based on iterative knowledge construction, which can realize automatic updating and iteration, and improve the recall and sorting performance of large-scale pre-trained models. The method is described in detail below.

[0146] Figure 7 This is a schematic diagram of the framework of the information recall system provided by the embodiment of the present application. Figure 7The information recall system includes a decision module 700, a knowledge base 704, and an analysis and learning module 705. The decision module 700 includes a deep memory module 701, a long-term memory module 702, and a shallow fast memory module 703. First, the user performs a query, that is, inputs the query question (query, corresponding to the first text in the above embodiment) into the decision module 700. After the question enters the decision module 700, the pre-trained large model ( Figure 7 The system uses various enhancement methods (not shown) to generate enhanced questions related to the query: rewritten questions, hypothetical answers, sub-questions, etc., by performing rewriting (modifying or expanding the user's question while keeping the original query intent unchanged), decomposition (splitting the original input question into multiple parts, searching or analyzing each part separately, and then synthesizing the results to give the final answer), and hypothetical answer generation (Hypothetical Answer Generation, used to predict or construct possible answers based on the question, generated by the large model. In the absence of a clear answer, the large model attempts to generate a reasonable hypothesis as an answer by analyzing relevant information and background knowledge, which helps exploratory search or provides a way of thinking when it is difficult to find an exact answer directly. It can also be used as an auxiliary tool to guide the direction of the conversation or stimulate further discussion in the interactive question-answering process). For each enhancement method, a set of enhanced questions with a maximum length of no more than 5 is maintained by matching the similarity with the initial question. For example, for the rewritten question, an enhanced question set containing 5 rewritten questions is obtained. After obtaining the recalled information, the decision module 700 may receive result feedback, and process the result feedback based on the analysis and learning module 705 to update the knowledge base 704 of the deep memory module 701 , the long-term memory module 702 and the shallow fast memory module 703 .

[0147] The knowledge base 704 (corresponding to the first and second knowledge bases in the above embodiment) may include various sub-knowledge bases corresponding to the shallow fast memory module 703 (corresponding to the first knowledge base in the above embodiment), where each sub-knowledge base refers to a pre-defined type, such as preparing a sub-knowledge base for each project in an enterprise, or preparing a sub-knowledge base for each product in a shopping platform. It should be noted that it is also possible to not classify and only use a sub-knowledge base corresponding to the shallow fast memory module 703. Figure 8 This is a schematic diagram of the principles of each memory module provided in the embodiment of this application. Figure 8The shallow fast memory module 703 is used for rule matching, that is, matching rule-based keywords, syntax trees, and dependency trees. After obtaining the enhanced question set, the shallow fast memory module 703 is used to perform rule matching on each to-be-matched item in the sub-knowledge base (corresponding to the information to be recalled in the above embodiment) and each enhanced question in the enhanced question set (corresponding to the first text in the above embodiment). The rule matching methods include the following:

[0148] Rule-based keyword matching: directly calculate the hit situation, for each enhanced question (query), perform word segmentation on the enhanced question to obtain multiple keywords (corresponding to the first keyword in the above embodiment), and calculate the hit situation of the multiple keywords in the keywords of the to-be-matched item (item) in the sub-knowledge base (corresponding to the second keyword in the above embodiment) (corresponding to the ratio of the first number to the second number in the above embodiment). For example, if the enhanced question has 5 keywords, and these 5 keywords exist among the 10 keywords in item1, then the hit rate is 100%, and item1 will be recalled.

[0149] Syntax tree and dependency tree matching: For each enhanced question, the Natural Language Toolkit (NLTK) library is directly used to generate syntax trees, dependency trees, and other features. Similarity is calculated between these trees and the items in the sub-knowledge base. If the similarity exceeds a second threshold, the item is recalled.

[0150] The shallow fast memory module 703 performs direct hit matching on the three types of features (keywords, syntax trees, and dependency trees) mentioned above, and a match is considered successful only if the keyword is hit and one of the other features is hit at the same time.

[0151] The long-term memory module 702 is used as a large model, combining the prompt words (a type of text information, including detailed descriptions, operating instructions, precautions and experience summaries, etc., such as the prompt word "the appearance of artificial intelligence-related items is Project 1", or the appearance of xx person's name is Project 1) stored in the corresponding various sub-knowledge bases with enhanced questions, and inputting the combined prompt words and enhanced questions into the large model to return the judgment result. If there are n items in the sub-knowledge base, the large model will be called n times, and the large model will directly output a 0 / 1 judgment for each item. If the output is 0, it will not be recalled, and if the output is 1, it will be recalled, and the logits of the output token (corresponding to the first matching degree in the above embodiment) will be retained as the result judgment after recall, and the reflection indicator for measuring the iterative effect will be used.

[0152] The deep memory module 701 uses vector similarity matching: the vector of each item in the corresponding sub-knowledge base of each category (corresponding to the second eigenvector in the above embodiment) is calculated with the encoded vector of the query (corresponding to the first eigenvector in the above embodiment) to achieve similarity (corresponding to the second similarity in the above embodiment). If it passes the threshold (corresponding to the third threshold in the above embodiment), it is considered a recall. The vector of each item can include the vector of the historically correct item to be matched, the vector of the question rewrite content, and the vector of the detailed description of the question.

[0153] In actual use, the results calculated by the three memory modules can be combined to adjust their weights as needed. For example, the shallow fast memory module 703 focuses on word and syntactic matching, while the deep memory module 701 emphasizes semantic similarity, even if the degree of word and syntactic matching is not high. Optionally, the score of each item output by each memory module is normalized to between 0 and 1, and the normalized scores are weighted, and multiple items with higher weighted scores are selected (corresponding to the third information in the above embodiment).

[0154] In the embodiment of the present application, the framework can be dynamically trained and updated during the online inference phase. The purpose of initializing training to form the knowledge base 704 is to enhance the model's reflection and judgment capabilities, thereby updating samples during iteration. Only samples incorrectly judged by the model enter the iteration, thereby reducing the introduction of simple samples and noise. The initialization training process is as follows: A large number of matching sets of positive and negative examples [query, item1, item2, ...] are collected in a ratio of 2:1. These serve as the initialization framework and the data set for verifying the effectiveness of the framework's self-learning iterations.

[0155] The process of initializing the shallow fast memory module 703 is as follows: randomly sampling 30% of the positive examples in the matching set, using a statistical learning library, extracting keywords from multiple positive example items, extracting features such as the syntactic-parsing tree and the dependency tree, and maintaining a first-in-first-out stack with a maximum element size of 50 to ensure certain performance and accuracy. The process of initializing the long-term memory module 702 is as follows: based on the rules and paradigms of the sampled positive examples, the planning model plans to generate initialization definition descriptions, precautions text, etc. for each type of recall task to generate prompt words (prompt) as long-term memory, which serve as prompt words for the recall model. That is, the query and item in the positive example are input into the planning model, which summarizes and outputs the initialization prompt words corresponding to the query and stores them in the knowledge base (corresponding to the second knowledge base in the above embodiment). The process of initializing the deep memory module 701 is as follows: using a fine-tuned BERT-based encoder model, the sampled positive examples are summarized by the large model and encoded into a deep memory vector (embedding) library. The maximum length of the embedding is 512 and the dimension is 1024. The feature vector of each item is stored in the vector library.

[0156] The recall model generates a 0 / 1 judgment for each item based on the initialization prompt word and the query, and then outputs the recalled item. If the recalled item does not match the actual item in the positive example, for example, if the wrong item is recalled, the agent model is called. The negative example and the current prompt are input into the planning model. The agent model modifies the prompt word for the query in the knowledge base until the recall model outputs no negative examples / hit positive examples. The agent model (reflexion framework) is used to maintain a maximum of 10 prompt words (when a positive example cannot be recalled and there are already 10 prompt words, the agent model modifies the integrated prompt words to 8).

[0157] See also Figure 8The analysis and learning module includes a planning large model and an intelligent agent large model. After initialization, you can continue to use the various memory modules of the current framework to serially generate a result based on the remaining examples (positive and negative examples) in each data set. That is, if a memory module hits, it is considered a recall. The analysis and learning module 705 uses the planning large model (planning large model) to self-learn and generate specific content, judges through feedback, learns by reasoning, and calls the intelligent agent large model (Agent large model) when the result is wrong, and iterates each memory module. The intelligent agent large model writes to the rule library and vector library through component calls, and automatically recalls prompt words. For example, the intelligent agent large model can perform the following operations: shallow memory and deep memory: when a negative example is hit / a positive example is not hit, delete / add matching items in the shallow fast memory and deep memory. Long-term memory: input the example and the current prompt to the Agent large model, and let it modify the prompt in the long-term memory until the negative example is not hit / the positive example is hit.

[0158] By comparing the performance of the model on the validation set before and after the memory module is updated, that is, the confidence of the logits output of the long-term memory recall large model on the correct answer, the score of the vector embedding similarity, and the success rate of rule matching, it is evaluated whether the iteration within the knowledge base has a positive impact on the model performance. Otherwise, the result of the planning output of the analysis learning module 705 is invalid, and the planning large model re-infers the result and repeats the above iterative process. The maximum number of retries is set to 3. If it exceeds, the example is skipped to avoid falling into a local infinite loop. This evaluation mechanism ensures that the iterative process of the model is beneficial and effective, thereby continuously pushing the model towards a more efficient and accurate direction. The training of the framework can support large-scale training with existing data offline (equivalent to initialization), and because it does not require updating model parameters, it can dynamically complete an answer task online and self-learn when receiving user feedback.

[0159] It can be based on parallel reasoning of each module online, and when at least two modules match, it is considered as a recall, which improves the reasoning speed and accuracy. The recall model in the long-term memory module 702 sets the large model sampling random parameter (temperature) to 1.2 and the maximum output token to 8, so that it generates a judgment token for each example, sampling 5 times, making the judgment more robust, and more hits determine whether the recall match is successful. The deep memory module 701 sets the similarity threshold to 0.65. Experiments show that after iterating using this method, the recall rate is increased by 50% when compared with the iterated pre-trained model in the initial stage. After the specific meaning and definition of the actual online recall task change, there is no need to re-train and deploy, and the recall rate still has a certain stability.

[0160] The embodiment of the present application proposes a model architecture that cleverly integrates an adaptive memory experience module based on historical data and feedback mechanism, which can realize automatic update and iteration. Through this design, the embodiment of the present application effectively avoids the problem of consuming too many resources in parameter adjustment, thereby significantly improving the recall and sorting performance of large-scale pre-training models in specific field application scenarios, and providing a more efficient and accurate solution for solving related problems in the field. The embodiment of the present application emphasizes the adaptive learning ability of the model, and improves the generalization ability of the model by continuously absorbing and integrating historical information and user feedback, thereby achieving a deeper understanding of data in a specific field and more accurate sorting and recall without significantly increasing the computing cost. The architecture is based on the method of imitating the human learning process, integrating statistical machine learning, deep learning small models and large models, improving the reasoning speed, and making the black box model decision process more interpretable and visual.

[0161] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent must be obtained, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0162] The following continues to describe the exemplary structure of the information determination device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the information determination device 455 of the memory 450 may include: a first retrieval module 4551, used to retrieve the first information from the first knowledge base in response to the received first text; a prompt word determination module 4552, used to determine the first prompt word of the first text from the second knowledge base; a text fusion module 4553, used to fuse the first text with the first prompt word to obtain the second text; a second retrieval module 4554, used to retrieve the second information from the first knowledge base based on the second text; an information determination module 4555, used to determine the third information corresponding to the first text based on the first information and the second information.

[0163] In some embodiments, the first retrieval module 4551 is also used to extract keywords from the first text to obtain first keywords; perform grammatical analysis on the first text to obtain a first syntactic structure; retrieve first information from the first knowledge base based on the first keyword in the first text and the first syntactic structure of the first text; and / or retrieve first information from the first knowledge base based on the first feature vector of the first text.

[0164] In some embodiments, the first retrieval module 4551 is also used to determine, for each information to be recalled in the first knowledge base, the second keyword and the second syntactic structure of the information in the information from the first knowledge base; determine the ratio of the first quantity to the second quantity, wherein the first quantity is the number of first keywords that are the same as the second keyword, and the second quantity is the total number of first keywords; when the ratio is greater than the first threshold and the first similarity is greater than the second threshold, determine the information as the first information, wherein the first similarity is the similarity between the first syntactic structure and the second syntactic structure.

[0165] In some embodiments, the first retrieval module 4551 is also used to determine the second feature vector of the information from the first knowledge base for each information to be recalled in the first knowledge base; when the second similarity between the first feature vector and the second feature vector is greater than a third threshold, the information is determined as the first information.

[0166] In some embodiments, the second retrieval module 4554 is also used to combine the second text and the information for each information to be recalled in the first knowledge base to obtain a third text; perform matching degree prediction processing on the third text to obtain a first matching degree between the information and the second text; and determine the information as the second information when the first matching degree is greater than a fourth threshold.

[0167] In some embodiments, the first knowledge base includes a third keyword of the first information. The information determining device 455 further includes a knowledge base updating module configured to, upon receiving feedback indicating that the first information does not match the first text, determine a fourth keyword, wherein the fourth keyword is the same third keyword as the first keyword of the first text; and delete the fourth keyword from the first information from the first knowledge base to obtain an updated first knowledge base.

[0168] In some embodiments, the knowledge base update module is also used to, upon receiving a feedback result indicating that the first information does not match the first text, correct the first prompt word based on the first information to obtain a second prompt word; determine a second matching degree between the first information and the second text; and update the first prompt word in the second knowledge base based on the second prompt word and the second matching degree to obtain an updated second knowledge base.

[0169] In some embodiments, the knowledge base update module is also used to fuse the first text and the second prompt word to obtain a fifth text, and determine a third matching degree between the first information and the fifth text; when the third matching degree is less than the second matching degree, the first prompt word in the second knowledge base is replaced with the second prompt word to obtain an updated second knowledge base.

[0170] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the information determination method described in the embodiment of the present application.

[0171] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the information determination method provided by the embodiment of the present application, for example, Figure 3 The information determination method shown.

[0172] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0173] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0174] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0175] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0176] In summary, the first knowledge base and the second knowledge base are updated in real time through the embodiments of the present application, thereby improving the accuracy of information determination.

[0177] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A method for determining information, characterized in that: include: Retrieving first information from a first knowledge base in response to the received first text; Determining a first prompt word of the first text from a second knowledge base; fusing the first text with the first prompt word to obtain a second text; Retrieving second information from the first knowledge base based on the second text; Based on the first information and the second information, third information corresponding to the first text is determined.

2. The method according to claim 1, characterized in that The step of retrieving first information from a first knowledge base in response to the received first text includes: Performing keyword extraction on the first text to obtain a first keyword; Performing grammatical analysis on the first text to obtain a first syntactic structure; Retrieving the first information from the first knowledge base based on a first keyword in the first text and a first syntactic structure of the first text; and / or, The first information is retrieved from the first knowledge base based on the first feature vector of the first text.

3. The method according to claim 2, characterized in that The retrieving the first information from the first knowledge base based on the first keyword in the first text and the first syntactic structure of the first text includes: For each piece of information to be recalled in the first knowledge base, determining a second keyword in the information and a second syntactic structure of the information from the first knowledge base; determining a ratio of a first number to a second number, wherein the first number is the number of the first keywords identical to the second keyword, and the second number is the total number of the first keywords; When the ratio is greater than a first threshold and the first similarity is greater than a second threshold, the information is determined as the first information, wherein the first similarity is the similarity between the first syntactic structure and the second syntactic structure.

4. The method according to claim 2, characterized in that The retrieving the first information from the first knowledge base based on the first feature vector of the first text includes: For each piece of information to be recalled in the first knowledge base, determining a second feature vector of the information from the first knowledge base; When the second similarity between the first feature vector and the second feature vector is greater than a third threshold, the information is determined as the first information.

5. The method according to claim 1, wherein The retrieving second information from the first knowledge base based on the second text includes: For each piece of information to be recalled in the first knowledge base, concatenate the second text and the information to obtain a third text; Performing a matching degree prediction on the third text to obtain a first matching degree between the information and the second text; When the first matching degree is greater than a fourth threshold, the information is determined as the second information.

6. The method according to claim 1, characterized in that The first knowledge base includes a third keyword of the first information; The method further comprises: In a case where a feedback result indicating that the first information does not match the first text is received, determining a fourth keyword from the third keywords, wherein the fourth keyword is the third keyword that is the same as the first keyword of the first text; The fourth keyword in the first information is deleted from the first knowledge base to obtain an updated first knowledge base.

7. The method according to claim 1, characterized in that The method further comprises: When receiving a feedback result indicating that the first information does not match the first text, modifying the first prompt word based on the first information to obtain a second prompt word; determining a second degree of matching between the first information and the second text; Based on the second prompt word and the second matching degree, the first prompt word in the second knowledge base is updated to obtain an updated second knowledge base.

8. The method according to claim 7, characterized in that The updating of the first prompt word in the second knowledge base based on the second prompt word and the second matching degree to obtain an updated second knowledge base includes: fusing the first text and the second prompt word to obtain a fifth text, and determining a third matching degree between the first information and the fifth text; When the third matching degree is less than the second matching degree, the first prompt word in the second knowledge base is replaced with the second prompt word to obtain an updated second knowledge base.

9. An information determination device, characterized in that: The device comprises: A first retrieval module is configured to retrieve first information from a first knowledge base in response to a received first text; a prompt word determination module, configured to determine a first prompt word of the first text from a second knowledge base; a text fusion module, configured to fuse the first text with the first prompt word to obtain a second text; A second retrieval module is configured to retrieve second information from the first knowledge base based on the second text; An information determination module is used to determine third information corresponding to the first text based on the first information and the second information.

10. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions or computer programs; A processor, configured to implement the information determination method according to any one of claims 1 to 8 when executing the computer-executable instructions or computer program stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the information determination method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the information determination method according to any one of claims 1 to 8 is implemented.

Images