Data processing method, model training method, electronic equipment and storage medium

Abstract

The invention discloses a data processing method, a model training method, electronic equipment and a storage medium, and relates to the field of large model technologies and data processing. The method comprises the following steps: acquiring current query data; on the basis of a matching result of the current query data, a prompt text is constructed, and the matching result is used for determining whether the current query data contains a target type word matched from a preset word bank or not; a target retrieval judgment model is adopted to conduct query analysis on the prompt text, a judgment result is obtained, and the judgment result is used for determining whether retrieval needs to be conducted on the current open domain query of the current query data or not; and generating a target answer according to the current query data and the judgment result. The technical problems of high resource consumption and low query processing efficiency of a query processing mode provided in related technologies are solved.

Description

Technical Field

[0001] This application relates to large model technology and data processing, and more specifically, to a data processing method, a model training method, an electronic device, and a storage medium. Background Technology

[0002] With the rapid development of Natural Language Processing (NLP) technology, large-scale language models have demonstrated excellent capabilities in multiple fields. However, facing the increasing demand for real-time information from users, these models, while providing rapid responses, also present key challenges in terms of information timeliness, accuracy, and computational resource optimization. To address these challenges, integrating external knowledge bases and utilizing Retrieve-Augmented Generation (RAG) technology can effectively supplement the model's information deficiencies, thereby improving the accuracy and comprehensiveness of the model's responses. However, query processing methods relying on RAG technology in these technologies are time-consuming, making it difficult to efficiently and intelligently identify and process diverse real-time information queries, easily leading to resource waste and poor timeliness and accuracy of query results.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a data processing method, a model training method, an electronic device, and a storage medium to at least solve the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0005] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring current query data; constructing prompt text based on the matching results of the current query data, wherein the matching results are used to determine whether the current query data contains words of the target type matched from a preset thesaurus; performing query analysis on the prompt text using a target retrieval judgment model to obtain a judgment result, wherein the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data; and generating a target answer based on the current query data and the judgment result.

[0006] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring a news report query request; constructing a news report query prompt text based on the matching result of the news report query request, wherein the matching result is used to determine whether the news report query request contains news report type words; performing query analysis on the news report query prompt text using a target retrieval judgment model to obtain a judgment result, wherein the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the news report query request; and generating a target news report based on the news report query request and the judgment result.

[0007] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: current query data; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target answer, wherein the target answer is generated based on the current query data and a determination result, the determination result is obtained by performing query analysis on the prompt text using a target retrieval determination model, the determination result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data, and the prompt text is constructed based on the matching result of the current query data, the matching result is used to determine whether the current query data contains target type words.

[0008] According to one aspect of the embodiments of this application, a data processing method is provided, comprising: acquiring a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: current query data; responding to the data processing dialogue request and returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: a target answer, wherein the target answer is generated based on the current query data and a judgment result, the judgment result is obtained by performing query analysis on the prompt text using a target retrieval judgment model, the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data, the prompt text is constructed based on the matching result of the current query data, the matching result is used to determine whether the current query data contains target type words; and displaying the target answer in a graphical user interface.

[0009] According to one aspect of the embodiments of this application, a model training method is provided, comprising: acquiring training data, wherein the training data includes: query samples and corresponding tags for the query samples, the query samples containing target type words, and the tags being used to determine whether an open-domain search is needed for the query samples; training an initial search determination model based on the training data to generate a target search determination model, wherein the target search determination model is used to perform query analysis on prompt text to obtain a determination result, the determination result being used to determine whether a search is needed for the current open domain query of the current query data, the prompt text being constructed based on the matching result of the current query data, and the matching result being used to determine whether the current query data contains target type words.

[0010] According to one aspect of the embodiments of this application, a computer terminal is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0011] According to one aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0012] According to one aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0013] According to one aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods in various embodiments of this application.

[0014] According to one aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0015] In this embodiment, by acquiring the current query data and then constructing prompt text based on the matching results of the current query data, it is determined whether the current query data contains words of the target type matched from a preset thesaurus. Subsequently, a target retrieval judgment model is used to analyze the prompt text and obtain a judgment result to determine whether a retrieval is needed for the current open domain query of the current query data. Finally, a target answer is generated based on the current query data and the judgment result. Thus, by dynamically analyzing the current query data and matching it with a preset thesaurus, it is possible to intelligently determine whether the current query data involves information that needs to be updated in real time, thereby selectively triggering open domain query retrieval and effectively avoiding resource waste. Using a target retrieval judgment model to analyze the prompt text makes the retrieval decision more accurate, ensuring both the timeliness and accuracy of the target answer and avoiding unnecessary retrieval operations. The target answer generated based on the current query data and the judgment result not only quickly responds to the user's query needs but also ensures the accuracy and real-time relevance of the information, thereby significantly optimizing the user's query experience. Therefore, this embodiment achieves the technical effect of saving resource consumption during query processing and improving query processing efficiency, thereby solving the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0016] It is worth noting that the general description above and the detailed description that follow are merely for illustrative purposes and do not constitute a limitation on this application. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a data processing method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a data processing method according to an embodiment of this application;

[0021] Figure 4 This is a flowchart of another data processing method according to an embodiment of this application;

[0022] Figure 5 This is a flowchart of another data processing method according to an embodiment of this application;

[0023] Figure 6This is a flowchart of another data processing method according to an embodiment of this application;

[0024] Figure 7 This is a flowchart of a model training method according to an embodiment of this application;

[0025] Figure 8 This is a structural block diagram of a data processing apparatus according to an embodiment of this application;

[0026] Figure 9 This is a structural block diagram of another data processing apparatus according to an embodiment of this application;

[0027] Figure 10 This is a structural block diagram of another data processing apparatus according to an embodiment of this application;

[0028] Figure 11 This is a structural block diagram of another data processing apparatus according to an embodiment of this application;

[0029] Figure 12 This is a structural block diagram of a model training device according to an embodiment of this application;

[0030] Figure 13 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] The technical solution provided in this application is mainly implemented using large-scale model technology. Here, "large-scale model" refers to a deep learning model with a massive number of parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of parameters. Large-scale models can also be called foundation models. They are pre-trained using large-scale unlabeled corpora to produce pre-trained models with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0034] It should be noted that, in practical applications, large models can be fine-tuned using a small number of samples to adapt them to different tasks. For example, large models can be widely applied in Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios for large models include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this embodiment, the data processing using a target retrieval and determination model in a query scenario is used as an example for explanation.

[0035] In recent years, with the rapid development of Natural Language Processing (NLP) technology, LLM has demonstrated excellent capabilities in various fields such as text generation, automatic summarization, machine translation, dialogue systems, and sentiment analysis. These models have significant advantages in capturing semantics, context, and syntactic structure, and can generate natural and fluent text. However, with the increasing demand for real-time information from users, ensuring information accuracy and reducing computational resource consumption while providing rapid responses has become an urgent challenge.

[0036] Content illusion in LLM (Local Modeling) significantly impacts model reliability. Even when the generated text is grammatically correct, in real-time information scenarios, the model may rely on outdated or inaccurate data, leading to misleading conclusions. This is particularly pronounced in rapidly changing environments such as news reports, social media updates, and e-commerce product descriptions, where users expect the most timely and accurate information. Furthermore, existing models are often based on static training datasets, failing to effectively reflect dynamic social, economic, or technological changes, limiting the understanding of emerging phenomena and the timeliness of information feedback. In addition, in high-concurrency request environments, real-time search and information processing place high demands on system computing power, leading to increased system load and operational costs. Especially in large-scale user interaction scenarios, efficiently managing and processing requests while ensuring service stability and responsiveness becomes a significant technical challenge. In multi-turn dialogues, information consistency and coherence are particularly important; models need to effectively integrate current user input with previous dialogue context to improve user experience.

[0037] To address these challenges, RAG technology, by integrating external knowledge bases, can effectively supplement the information deficiencies of models and improve the accuracy and comprehensiveness of user answers. When a user asks a question, the system can intelligently retrieve relevant information and combine it with the model's inherent knowledge to generate a targeted answer. However, in complex open-domain environments, how to efficiently perform search and decision-making remains an unresolved issue. Query processing methods relying on RAG technology in related technologies consume significant time, making it difficult to efficiently and intelligently identify and process diverse real-time information queries, easily leading to resource waste and poor timeliness and accuracy of query results.

[0038] According to an embodiment of this application, a data processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] Considering the large number of model parameters in large models and the limited computing resources of mobile terminals, the data processing method provided in this application embodiment can be applied to, for example, Figure 1 The application scenarios shown are not limited to these. In, for example... Figure 1In the application scenario shown, the large model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. These client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to access the large model, thereby implementing the method provided in this embodiment.

[0040] In this embodiment, the system consisting of a client device and a server can perform the following steps: The server obtains a data processing dialogue request from the client device. The request data carried in the data processing dialogue request includes the current query data. In response to the data processing dialogue request, the server returns a data processing dialogue response to the client device. The response carries information including the target answer, which is generated based on the current query data and a judgment result. The judgment result is obtained by analyzing the prompt text using a target retrieval judgment model. The judgment result is used to determine whether a retrieval is needed for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data, which is used to determine whether the current query data contains the target type words. After receiving the data processing dialogue response, the client device can display the target answer within a graphical user interface.

[0041] It should be noted that, provided that the client device's operating resources can meet the deployment and operation conditions of the large model, the embodiments of this application can be performed on the client device.

[0042] Under the aforementioned operating environment, this application provides the following: Figure 2 The data processing method shown. Figure 2 This is a flowchart of a data processing method according to an embodiment of this application. Figure 2 As shown, the method may include the following steps:

[0043] Step S21: Obtain the currently queried data;

[0044] Step S22: Based on the matching results of the current query data, construct prompt text, wherein the matching results are used to determine whether the current query data contains words of the target type matched from the preset thesaurus;

[0045] Step S23: Use the target retrieval judgment model to perform query analysis on the prompt text and obtain the judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data.

[0046] Step S24: Generate the target answer based on the current query data and the judgment result.

[0047] The aforementioned current query data can be user-input queries, specifically text, voice, or image-based inquiries or requests submitted by users to interactive applications such as question-and-answer systems, search engines, and customer service chatbots to obtain information, solve problems, or seek assistance. In text-based question-and-answer systems, the current query data is typically natural language text. In voice assistants or intelligent customer service scenarios, the current query data can be voice commands or questions converted to text through speech recognition. In image-based queries or vision-based interactive systems, the current query data can be user-submitted images used to identify content, search for similar images, or obtain image-related information. The current query data not only includes the user's direct questions or instructions but can also include contextual information such as the user's historical interaction records, geographical location, and timestamps. This contextual information helps the system more accurately understand user needs and provide personalized answers or services.

[0048] After acquiring the current query data, matching is performed on it within a preset thesaurus to obtain the matching results. The preset thesaurus is a high-frequency thesaurus, containing current trending topics and commonly used user query terms, and is continuously updated based on real-time data or user behavior. Target type words are high-frequency words that appear more frequently than a preset threshold in the text corpus, reflecting the main topics, background information, or trending events involved in the query. The definition and scope of high-frequency words may differ in different corpora. For example, in news texts, "welfare," "economy," and "market" can be considered high-frequency words; while on social media, words like "like," "share," and "emoji" are more common.

[0049] Based on the matching results of the current query data, construct a prompt text. The prompt text can be a piece of input text designed to guide the model to generate a specific type or content output. The prompt text can contain specific information about the user's current query, as well as context history, guiding statements, or special tags to help the model understand the query intent and make a corresponding answer.

[0050] Furthermore, a target retrieval decision model is employed to perform query analysis on the prompt text, obtaining a decision result to determine whether a search is needed for the current open-domain search of the current query data. The target retrieval decision model performs query analysis based on multiple dimensions such as the context, keywords, and semantics of the input content, and determines whether further open-domain search is necessary. The output of the target retrieval decision model is two options: search or not search. Open-domain query refers to the process of information retrieval from a broad, unrestricted data source. Compared to closed-domain search, which typically searches for answers in a predefined, specific domain, or limited dataset, open-domain query faces the infinite information resources on the internet, including but not limited to web pages, social media, news, academic papers, and specialized databases. The goal of open-domain query is to obtain the most relevant, comprehensive, and timely information to answer user questions that may cover any topic. Utilizing open-domain query can obtain the latest information not covered during model training, compensating for the limitations of model knowledge, especially in rapidly updating domains such as news and social media, providing more accurate and user-relevant answers.

[0051] The target retrieval determination model described above can be obtained by training the initial retrieval determination model based on training data. The training data includes: query samples and the tags corresponding to the query samples. The query samples contain target type words, and the tags are used to determine whether an open search is needed for the query samples.

[0052] Specifically, during the training of the target retrieval judgment model, prompt_1 is the prompt text constructed when the current query data does not contain target type words, and prompt_2 is the prompt text constructed when the current query data contains target type words. Based on prompt_1 and prompt_2, data creation and instruction fine-tuning (Supervised Fine-Tuning, SFT) are performed to obtain training data. This training data is then used to train the initial retrieval judgment model to improve its ability to recognize user query intent. The training data includes user queries with high-frequency words and their corresponding labels. These labels indicate whether a retrieval should be performed for the current open-domain query of the user query. For example: {"text": Given a question, determine whether to invoke the search tool: History: "User: Hello. Reply: Hello, how can I help you?" Current question: What is the weather like in City A today? Reply: xxx, "Label": "Search"}. The above training data can be collected from open datasets and manually labeled to obtain a high-quality dataset. Before fine-tuning, all datasets from all sources were converted into a uniform structured format to ensure that the model could consistently interpret each task.

[0053] By employing a target retrieval judgment model to query and analyze the prompt text and obtain judgment results, it can intelligently determine whether to perform an open domain search and generate appropriate target answers based on the judgment results. This ensures the timeliness and accuracy of information while avoiding unnecessary consumption of computing resources, thereby improving overall response efficiency and user satisfaction.

[0054] Based on steps S21 to S24 above, by acquiring the current query data and then constructing prompt text based on the matching results of the current query data, it is determined whether the current query data contains words of the target type matched from the preset thesaurus. Subsequently, a target retrieval judgment model is used to analyze the prompt text and obtain a judgment result to determine whether a retrieval is needed for the current open domain query of the current query data. Finally, the target answer is generated based on the current query data and the judgment result. Thus, by dynamically analyzing the current query data and matching it with the preset thesaurus, it is possible to intelligently determine whether the current query data involves information that needs to be updated in real time, thereby selectively triggering open domain query retrieval and effectively avoiding resource waste. Using a target retrieval judgment model to analyze the prompt text makes the retrieval decision more accurate, ensuring both the timeliness and accuracy of the target answer and avoiding unnecessary retrieval operations. The target answer generated based on the current query data and the judgment result not only quickly responds to the user's query needs but also ensures the accuracy and real-time relevance of the information, thereby significantly optimizing the user's query experience. Therefore, the embodiments of this application achieve the technical effects of saving resource consumption during query processing and improving query processing efficiency, thereby solving the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0055] It should be noted that the data processing method in this embodiment can be applied to real-time information query scenarios, including but not limited to rapidly changing scenarios such as news reports, social media platforms, and e-commerce product recommendation systems. Specifically, in the news field, the timeliness and accuracy of information are crucial. When processing news report query requests, the data processing method in this embodiment can quickly supplement the latest news events, data, and background information to generate corresponding target news reports, further improving the timeliness and comprehensiveness of news reports. Information on social media platforms changes rapidly, and users have a strong demand for real-time interaction and access to the latest information. The data processing method in this embodiment can quickly respond to user questions, provide accurate information related to the current topic, and enhance the consistency and coherence of multi-turn dialogues, improving the user experience on social media platforms. On e-commerce platforms, product information and user needs are constantly changing. The data processing method in this embodiment can obtain information such as product inventory, price changes, and user reviews in real time, optimize product descriptions, provide more personalized product recommendations, and improve the e-commerce user experience and sales conversion rate.

[0056] The data processing methods in the embodiments of this application will be further described below.

[0057] In an optional embodiment, the data processing method in this application further includes:

[0058] The current query data is segmented into words to obtain the first segmentation result; the first segmentation result is matched with a preset thesaurus to obtain the matching result; the matching result records the target query markers corresponding to the current query data. The target query markers include a first query marker and a second query marker. The first query marker is used to indicate that the target type word corresponding to the first segmentation result is matched in the preset thesaurus, and the second query marker is used to indicate that the target type word corresponding to the first segmentation result is not matched in the preset thesaurus. The query level of the first query marker is higher than that of the second query marker.

[0059] Specifically, word segmentation is a crucial step in understanding query intent, constructing suggestion text, and making search decisions. A reasonable word segmentation strategy can significantly improve the efficiency and accuracy of subsequent processing. Furthermore, since the construction and updating of the high-frequency thesaurus also depend on the initial word segmentation results, the quality of word segmentation directly affects the effectiveness of the high-frequency thesaurus and the overall performance of the system. When performing word segmentation on the current query data, word segmentation toolkits can be used, offering a wealth of segmentation functions and algorithms. Depending on specific needs and the characteristics of the current query data, custom word segmentation algorithms can also be implemented. For example, word segmentation can be based on rule-based, statistical, or machine learning methods.

[0060] After obtaining the first word segmentation result, each segmented word is traversed to check if it exists in the preset thesaurus. During the traversal, the surface form of the words can be compared, such as literal matching, and the latent semantics can also be compared, such as part-of-speech and semantic similarity, to ensure accurate matching. For each word, if it matches any target type word in the preset thesaurus, a first query tag is generated for the current query data, marking it as an "important query"; if the word does not match any target type word in the preset thesaurus, a second query tag is generated for the current query data, marking it as a "normal query". The query level of the first query tag is higher than that of the second query tag, meaning that query data containing target type words will be processed first or trigger a deeper open-domain search.

[0061] Based on the above optional embodiments, by performing word segmentation on the current query data to obtain the first word segmentation result, and then matching the first word segmentation result with the preset thesaurus to obtain the matching result, the query tag corresponding to the current query data can be quickly determined to improve the processing efficiency and information accuracy of the question answering system.

[0062] In an optional embodiment, in step S22, constructing the prompt text based on the matching result of the current query data includes: in response to the first query tag being recorded in the matching result, constructing the prompt text using the current query data, target type words, and historical interaction records associated with the current query data.

[0063] Specifically, if the first word segmentation result of the current query data matches a target type word in the preset thesaurus, a prompt text is constructed using the current query data, the target type word, and the historical interaction records associated with the current query data. Based on the user's identity or session context, historical interaction records related to the current query data are retrieved. These historical interaction records include previous questions, system answers, and any user feedback, aiming to build a comprehensive contextual environment to help the model better understand the user's intent.

[0064] For example, in constructing prompt text using current query data, target type words, and historical interaction records associated with the current query data, the current query data and matched target type words are first integrated into the prompt text, helping the model focus on the key points of the query. Subsequently, historical interaction records are fused, incorporating summaries or key information from these records into the prompt text to provide context for a continuous dialogue, ensuring the model's responses are coherent and consistent with previous conversations. Furthermore, the prompt text can be organized according to a predefined format or template to ensure the model can correctly recognize and parse the input. When constructing the prompt text, the model's input length limitations can also be considered, and historical interaction records can be simplified or summarized to keep the prompt text length within the model's processing range.

[0065] Based on the above optional embodiments, by responding to the first query tag recorded in the matching result, prompt text is constructed using the current query data, target type words, and historical interaction records associated with the current query data. This can improve the accuracy and timeliness of the response, and further enhance the consistency of the dialogue and the user experience.

[0066] In an optional embodiment, in step S22, constructing the prompt text based on the matching result of the current query data includes: in response to the second query tag being recorded in the matching result, constructing the prompt text using the current query data and the historical interaction records associated with the current query data.

[0067] Specifically, if the first word segmentation result of the current query data does not match the target type word in the preset thesaurus, a prompt text is constructed using the current query data and the historical interaction records associated with the current query data. For example, a summary or key information of the historical interaction records and the current query data are added to the prompt text to provide context for a continuous dialogue, ensuring that the model's response is coherent and consistent with previous dialogues.

[0068] Based on the above optional embodiments, by responding to the second query tag recorded in the matching result, the prompt text is constructed using the current query data and the historical interaction records associated with the current query data, thereby improving the accuracy and timeliness of the response and further enhancing the consistency of the dialogue and the user experience.

[0069] In an optional embodiment, step S24, generating the target answer based on the current query data and the determination result includes: in response to the determination result indicating that it is not necessary to retrieve the current open domain query, using the target answer generation model to perform answer generation analysis on the current query data and the historical interaction records associated with the current query data to obtain the target answer.

[0070] Specifically, if the determination result indicates that it is not necessary to retrieve the current open domain query, the query suggestion text is constructed using the current query data and the historical interaction records associated with the current query data, and then the query suggestion question is input into the target answer generation model for answer generation analysis.

[0071] For example, the target answer generation module first understands the specific content of the current query data and the context of historical dialogues, including previous user questions and system responses, as well as possible changes in user preferences or state. This process relies on the model's contextual understanding and memory capabilities to ensure that historical information is taken into account when generating the answer. After understanding the context, the target answer generation model generates an answer that matches the current query data based on its internal knowledge and logic. The generated answer should be as accurate and complete as possible, and consistent with historical dialogues in tone and style to improve the user experience. The generated answer may contain elements that require post-processing, such as formatting adjustments, grammatical corrections, or information verification. Further post-processing of the generated result ensures that the output answer meets the expected format and quality standards. After post-processing, the final target answer is output. In this case, the target answer is directly presented to the user without the need for additional search or information retrieval processes, relying on the target answer generation model's internal knowledge base and language generation capabilities.

[0072] Based on the above optional embodiments, in response to the determination result indicating that it is not necessary to retrieve the current open domain query, the target answer generation model is used to perform answer generation analysis on the current query data and the historical interaction records associated with the current query data. This can quickly generate accurate and coherent targets, thereby significantly improving the response speed and resource utilization efficiency of the question-and-answer system, while maintaining a high level of user experience and information accuracy.

[0073] In an optional embodiment, step S24, generating the target answer based on the current query data and the judgment result includes: in response to the judgment result indicating that the current open domain query needs to be retrieved, calling the retrieval engine to perform information retrieval on the current query data and obtaining the target retrieval result; and using the target answer generation model to perform answer generation analysis on the current query data, the historical interaction records associated with the current query data, and the target retrieval result to obtain the target answer.

[0074] Specifically, if the determination result indicates that a retrieval of the current open domain query is required, a retrieval request is issued based on the current query data, and the retrieval engine is invoked to perform information search within the open domain according to the retrieval request, crawling relevant web page content and data to obtain the target retrieval results.

[0075] Furthermore, a target answer generation model is employed to analyze the current query data, the historical interaction records associated with the current query data, and the target search results to generate the target answer. This allows for efficient use of open-domain search to enhance answer generation, leveraging the richness of external information while maintaining consistency with the user's historical interactions. This provides users with highly accurate and detailed responses, especially when dealing with complex queries that require the latest data or cross-domain knowledge, significantly improving the performance and user experience of the question-answering system.

[0076] In one optional embodiment, calling the search engine to perform information retrieval on the current query data and obtaining the target search result includes: calling the search engine to perform information retrieval on the current query data and obtaining initial search results; sorting the initial search results to obtain the target search result.

[0077] Specifically, the initial search results include multiple data chunks. The retrieved data chunks undergo preliminary processing to remove irrelevant or duplicate information. This preliminary processing includes, but is not limited to, text cleaning, removal of webpage tags, and standardization of text formatting, in order to improve the efficiency and accuracy of subsequent processing.

[0078] Furthermore, the recall ranking algorithm is used to further filter and rank the data blocks after the initial processing. The recall ranking algorithm can evaluate the relevance of each data block to the query, the reliability and timeliness of the information, and thus select the more relevant information fragments as the target retrieval results.

[0079] Based on the above optional embodiments, by calling the search engine to retrieve information from the current query data, initial search results are obtained, and then the initial search results are sorted to obtain the target search results. This can effectively obtain real-time data related to the user query, and the quality of the search results can be significantly improved through sorting, thereby further improving the accuracy of the question-and-answer system and user satisfaction.

[0080] In an optional embodiment, the data processing method in this application further includes: obtaining evaluation data corresponding to the target answer, wherein the evaluation data is determined based on user feedback on the target answer, and the evaluation data is used to update at least one of the following: model parameters of the target retrieval determination model, words contained in the preset thesaurus, and model parameters of the target answer generation model.

[0081] Specifically, evaluation data can be collected through user surveys, user feedback, and user ratings. The collected evaluation data can be used to update the model parameters of the target retrieval decision model to improve retrieval accuracy and efficiency. Evaluation data can also be used to update the words in the pre-set thesaurus to ensure coverage of more questions users might ask. Furthermore, evaluation data can be used to update the model parameters of the target answer generation model to generate more accurate and useful target answers. By continuously collecting and analyzing user feedback, relevant models can be continuously optimized and improved, enhancing user experience and satisfaction.

[0082] In an optional embodiment, the data processing method in this application further includes:

[0083] Obtain historical query data; preprocess the historical query data to obtain the second word segmentation result; perform word frequency statistics on the second word segmentation result to obtain the statistical result; in response to the statistical result meeting the preset conditions, establish a preset word library based on the statistical result, wherein the preset conditions are determined according to the popularity and frequency threshold of the target topic.

[0084] The aforementioned historical query data can be obtained from the user's client, including but not limited to the user's search records, feedback information and related interaction data in the system. That is, the historical query data is the query data of the user in the client within a historical time period, rather than the historical interaction data in each round of dialogue.

[0085] Specifically, a scheduled task can be used to automatically collect historical query data from the user's end every day. During this process, it is necessary to ensure the integrity and validity of the data, and the data coverage should be as broad as possible to capture diverse user needs. Further, the historical query data is preprocessed to obtain the second word segmentation result; word frequency statistics are then performed on the second word segmentation result to obtain statistical results. These results are iterated through, and if the statistical results meet preset conditions, a preset thesaurus is built based on the statistical results, and the preset thesaurus is updated in real time in subsequent processes.

[0086] The frequency thresholds mentioned above can be set based on historical query data or query requirements. The frequency threshold can be a specific numerical value, such as a word appearing more than 50 times, or a relative value, such as a word appearing in more than 0.1% of all queries. The statistical results for new queries are dynamically updated. If a word reaches the threshold again in future queries, it is also included in the hot word library. Users can also manually update the library, storing selected hot words in a high-frequency word library to form a structured database, including information such as word, frequency, last update time, and word type. In addition, a stop word library is maintained. The general word library can include words such as "I," "you," "he," "she," "it," "recently," and "whether." The stop word library is updated regularly to increase the proportion of meaningful words in the second word segmentation results.

[0087] Based on the above optional embodiments, by acquiring historical query data and then preprocessing the historical query data to obtain the second word segmentation result, and then performing word frequency statistics on the second word segmentation result to obtain the statistical result, and finally responding to the statistical result meeting the preset conditions, a preset thesaurus is established based on the statistical result, thereby significantly improving the efficiency, accuracy and coherence of the question-answering system in processing user queries, while optimizing resource allocation, and providing users with a more intelligent and demand-oriented query experience.

[0088] In one optional embodiment, preprocessing historical query data to obtain the second word segmentation result includes: performing data cleaning on the historical query data to obtain a cleaning result; performing data format conversion on the cleaning result to obtain a conversion result; and performing word segmentation on the conversion result to obtain the second word segmentation result.

[0089] Specifically, during data cleaning, irrelevant symbols, special characters, and duplicate information in historical query data can be removed, and all data can be converted to a consistent format, such as unifying capitalization and removing spaces. Finally, the conversion results are processed by word segmentation. Commonly used Chinese word segmentation techniques can be selected to ensure accuracy. A dictionary-based method combined with an implicit Markov model can be used for word segmentation to improve accuracy.

[0090] Figure 3 This is a schematic diagram of a data processing method according to an embodiment of this application, such as... Figure 3As shown, the current query data is retrieved, and word segmentation is performed on it to obtain the first word segmentation result. This first word segmentation result is then matched against a preset thesaurus to obtain the matching result. The matching result records the target query markers corresponding to the current query data. These markers include a first query marker and a second query marker. The first query marker indicates that a target type word corresponding to the first word segmentation result is matched in the preset thesaurus, while the second query marker indicates that a target type word corresponding to the first word segmentation result is not matched in the preset thesaurus. The query level of the first query marker is higher than that of the second query marker. In response to the first query marker being recorded in the matching result, prompt text 1 is constructed using the current query data, the target type word, and the historical interaction records associated with the current query data. In response to the second query marker being recorded in the matching result, prompt text 2 is constructed using the current query data and the historical interaction records associated with the current query data.

[0091] Furthermore, a target retrieval judgment model is used to perform query analysis on the prompt text to obtain a judgment result. This judgment result determines whether a retrieval is needed for the current open domain query based on the current query data. If the judgment result indicates that a retrieval is not needed for the current open domain query, a target answer generation model is used to analyze the current query data and its associated historical interaction records to obtain the target answer. If the judgment result indicates that a retrieval is needed for the current open domain query, a retrieval engine is invoked to retrieve information from the current query data, obtaining the target retrieval result; the target answer generation model is then used to analyze the current query data, its associated historical interaction records, and the target retrieval result to obtain the target answer.

[0092] In summary, the data processing method in this embodiment combines high-frequency word rule matching, intelligent judgment, and open-domain search, effectively improving the accuracy and flexibility of the overall response. It possesses a powerful ability to handle complex queries, enhancing user experience and providing data support for subsequent system optimization. By introducing high-frequency word matching, popular topics can be quickly identified in the early stages of user queries, thus guiding subsequent processing more effectively, reducing reliance on complex rules, and improving the system's response quality. Since the target retrieval judgment model has undergone instruction fine-tuning, it can dynamically judge based on user input and context. Therefore, when handling new types of queries, it exhibits higher adaptability compared to static rule systems, enabling the system to quickly respond to diverse practical needs. Utilizing open-domain search, the system can acquire extensive information, better addressing open-ended questions without clear answers, thereby improving the system's completeness and flexibility. Due to the high degree of automation, high-frequency word matching and intelligent search judgment reduce the need for manual intervention, effectively reducing labor costs and improving overall system efficiency.

[0093] Figure 4 This is a flowchart of another data processing method according to an embodiment of this application. Figure 4 As shown, the method may include the following steps:

[0094] Step S41: Obtain a news report query request;

[0095] Step S42: Based on the matching results of the news report query request, construct the news report query prompt text, wherein the matching results are used to determine whether the news report query request contains news report type words;

[0096] Step S43: Use the target retrieval judgment model to perform query analysis on the news report query prompt text and obtain the judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the news report query request.

[0097] Step S44: Generate the target news report based on the news report query request and the judgment result.

[0098] Based on steps S41 to S44 above, a news report query request is obtained, and then a news report query prompt text is constructed based on the matching results of the news report query request to determine whether the news report query request contains news report type words. Subsequently, a target retrieval judgment model is used to analyze the news report query prompt text to obtain a judgment result, determining whether a retrieval is needed for the current open domain query of the news report query request. Finally, the target news report is generated based on the news report query request and the judgment result. Thus, by dynamically analyzing the news report query request, it is possible to intelligently determine whether the news report query request contains news report type words, thereby selectively triggering open domain query retrieval and effectively avoiding resource waste. Using a target retrieval judgment model to analyze the news report query prompt text makes the retrieval decision more accurate, ensuring both the timeliness and accuracy of the target news report and avoiding unnecessary retrieval operations. The target news report generated based on the news report query request and the judgment result not only quickly responds to user query needs but also ensures the accuracy and real-time relevance of information, thereby significantly optimizing the user query experience. Therefore, the embodiments of this application achieve the technical effects of saving resource consumption during query processing and improving query processing efficiency, thereby solving the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0099] Figure 5 This is a flowchart of another data processing method according to an embodiment of this application. Figure 5 As shown, the method may include the following steps:

[0100] Step S51: Obtain a data processing request through the first application programming interface, wherein the request data carried in the data processing request includes: the currently queried data;

[0101] Step S52: Return a data processing response through the second application programming interface. The response data carried in the data processing response includes: the target answer, which is generated based on the current query data and the judgment result. The judgment result is obtained by performing query analysis on the prompt text using a target retrieval judgment model. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

[0102] Based on steps S51 to S52 above, a data processing request is obtained through the first application programming interface (API). The request data includes the current query data. A data processing response is then returned through the second API. This response includes the target answer, generated based on the current query data and a judgment result. The judgment result is obtained by analyzing the prompt text using a target retrieval judgment model. This judgment result determines whether a retrieval of the current open domain query for the current query data is necessary. The prompt text is constructed based on the matching results of the current query data. The matching results determine whether the current query data contains target type words. By dynamically analyzing the current query data and matching it with a preset thesaurus, it can intelligently determine whether the current query data involves information requiring real-time updates, thus selectively triggering open domain query retrieval and effectively avoiding resource waste. Using a target retrieval judgment model to analyze the prompt text makes retrieval decisions more accurate, ensuring both the timeliness and accuracy of the target answer and avoiding unnecessary retrieval operations. The target answer generated based on the current query data and the judgment result not only quickly responds to user query needs but also ensures the accuracy and real-time relevance of information, thereby significantly optimizing the user query experience. Therefore, the embodiments of this application achieve the technical effects of saving resource consumption during query processing and improving query processing efficiency, thereby solving the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0103] Figure 6 This is a flowchart of another data processing method according to an embodiment of this application. Figure 6 As shown, the method may include the following steps:

[0104] Step S61: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the currently queried data;

[0105] Step S62: In response to the data processing dialogue request, a data processing dialogue response is returned. The data processing dialogue response carries the following information: the target answer, which is generated based on the current query data and the judgment result. The judgment result is obtained by performing query analysis on the prompt text using a target retrieval judgment model. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

[0106] Step S63: Display the target answer within the graphical user interface.

[0107] Based on steps S61 to S63 above, by acquiring the currently input data processing dialogue request, which carries the request data including the current query data, and then responding to the data processing dialogue request by returning a data processing dialogue reply, the information carried in the data processing dialogue reply includes the target answer. The target answer is generated based on the current query data and the judgment result. The judgment result is obtained by analyzing the prompt text using a target retrieval judgment model. The judgment result is used to determine whether a retrieval is needed for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words. Finally, the target answer is displayed in the graphical user interface. Thus, by dynamically analyzing the current query data and matching it with a preset thesaurus, it is possible to intelligently determine whether the current query data involves information that needs to be updated in real time, thereby selectively triggering open domain query retrieval and effectively avoiding resource waste. Using a target retrieval judgment model to analyze the prompt text makes the retrieval decision more accurate, ensuring both the timeliness and accuracy of the target answer and avoiding unnecessary retrieval operations. The target answer generated based on the current query data and judgment results not only quickly responds to user query needs but also ensures the accuracy and real-time relevance of information, thereby significantly optimizing the user query experience. Therefore, this embodiment of the application achieves the technical effects of saving resource consumption during query processing and improving query processing efficiency, thus solving the technical problems of high resource consumption and low query processing efficiency in related technologies.

[0108] Figure 7 This is a flowchart of a model training method according to an embodiment of this application. Figure 7 As shown, the method may include the following steps:

[0109] Step S71: Obtain training data, which includes: query samples and corresponding labels for the query samples. The query samples contain target type words, and the labels are used to determine whether an open search is needed for the query samples.

[0110] Step S72: Train the initial retrieval judgment model based on the training data to generate a target retrieval judgment model. The target retrieval judgment model is used to perform query analysis on the prompt text to obtain a judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval on the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

[0111] Based on steps S71 to S72 above, training data is acquired, and the initial retrieval judgment model is trained using this data to generate a target retrieval judgment model. This model analyzes the prompt text to obtain a judgment result, which determines whether a retrieval is needed for the current open domain query of the current query data. The prompt text is constructed based on the matching results of the current query data, which determine whether the current query data contains target type words. By dynamically analyzing the current query data and matching it with a preset thesaurus, the model can intelligently determine whether the current query data involves information requiring real-time updates, thus selectively triggering open domain query retrieval and effectively avoiding resource waste. Using the target retrieval judgment model to analyze the prompt text makes retrieval decisions more accurate, ensuring both the timeliness and accuracy of the target answer and avoiding unnecessary retrieval operations. The target answer generated based on the current query data and the judgment result not only quickly responds to user query needs but also ensures the accuracy and real-time relevance of information, thereby significantly optimizing the user query experience. Therefore, the embodiments of this application achieve the technical effects of saving resource consumption during query processing and improving query processing efficiency, thereby solving the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0112] In an optional embodiment, step S72, training the initial retrieval decision model based on training data to generate the target retrieval decision model includes: generating prompt samples based on training data; using the initial retrieval decision model to perform query analysis on the prompt samples to obtain prediction results; determining the target loss based on the prediction results and labels; and using the target loss to fine-tune the model parameters of the initial retrieval decision model to obtain the target retrieval decision model.

[0113] The target retrieval determination model described above can be obtained by training the initial retrieval determination model based on training data. The training data includes: query samples and the tags corresponding to the query samples. The query samples contain target type words, and the tags are used to determine whether an open search is needed for the query samples.

[0114] Specifically, prompt_1 is the suggestion text constructed when the current query data does not contain the target type words, and prompt_2 is the suggestion text constructed when the current query data contains the target type words. Data processing and SFT are performed based on prompt_1 and prompt_2 to obtain training data. This training data is then used to train the initial retrieval decision model to improve its ability to recognize user query intent. The training data includes user queries with high-frequency words and their corresponding labels. These labels indicate whether a retrieval should be performed for the current open-domain query of the user query. For example: {"text": Given a question, determine whether to invoke the search tool: History: "User: Hello. Reply: Hello, how can I help you?" Current question: What is the weather like in City A today? Reply: xxx, "Label": "Search"}. The above training data can be collected from open datasets and manually labeled to obtain a high-quality dataset. Before fine-tuning, all datasets from various sources are converted into a unified structured format to ensure that the model can consistently interpret each task.

[0115] The SFT process involves exposing the model to a large number of training instances with explicit instruction labels, each instance containing a task instruction, input data, and expected output. The model learns how to extract key information from the input data and produce appropriate output based on two given prompt instructions.

[0116] Next, the model loads training data and trains an initial retrieval decision model via the DeepSpeed ​​interface, receiving samples containing instructions and expected outputs in each training batch. The initial retrieval decision model performs query analysis on the suggested samples to obtain prediction results. Based on the prediction results and labels, the target loss is determined, and the target loss is backpropagated to update the model's weights, making the model closer to the correct answer in subsequent predictions. As training progresses, the model gradually learns how to generate corresponding high-quality outputs based on different instructions, whether it's improving query wording, evaluating document relevance, or enhancing response accuracy. Accelerated by the DeepSpeed ​​interface, this process can be completed in a shorter time while maintaining high accuracy. The final target retrieval decision model possesses the ability to perform diverse tasks, greatly expanding its application scenarios in the real world.

[0117] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0118] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0120] According to an embodiment of this application, a data processing apparatus for implementing the above-described data processing method is also provided. Figure 8 This is a structural block diagram of a data processing apparatus according to an embodiment of this application, such as... Figure 8 As shown, the device includes:

[0121] Module 801 is used to retrieve the currently queried data;

[0122] The construction module 802 is used to construct prompt text based on the matching results of the current query data, wherein the matching results are used to determine whether the current query data contains words of the target type matched from the preset thesaurus;

[0123] The analysis module 803 is used to perform query analysis on the prompt text using a target retrieval judgment model to obtain a judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data.

[0124] The generation module 804 is used to generate the target answer based on the current query data and the judgment result.

[0125] Optionally, the data processing device further includes: a word segmentation module 805, used to perform word segmentation processing on the current query data to obtain a first word segmentation result; and a matching module 806, used to perform matching processing on the first word segmentation result and a preset thesaurus to obtain a matching result; wherein, the matching result records the target query marker corresponding to the current query data, and the target query marker includes: a first query marker and a second query marker, the first query marker is used to indicate that the first word segmentation result matches the target type word corresponding to the first word segmentation result in the preset thesaurus, and the second query marker is used to indicate that the target type word corresponding to the first word segmentation result is not matched in the preset thesaurus, and the query level of the first query marker is higher than the query level of the second query marker.

[0126] Optionally, the construction module 802 is also configured to: in response to the first query tag being recorded in the matching result, construct a prompt text using the current query data, the target type words, and the historical interaction records associated with the current query data.

[0127] Optionally, the construction module 802 is also configured to: in response to the fact that a second query tag is recorded in the matching result, construct a prompt text using the current query data and the historical interaction records associated with the current query data.

[0128] Optionally, the generation module 804 is further configured to: in response to the determination result indicating that it is not necessary to retrieve the current open domain query, use the target answer generation model to perform answer generation analysis on the current query data and the historical interaction records associated with the current query data to obtain the target answer.

[0129] Optionally, the generation module 804 is further configured to: respond to the determination result indicating that a retrieval of the current open domain query is required, call the retrieval engine to perform information retrieval on the current query data and obtain the target retrieval result; and use the target answer generation model to perform answer generation analysis on the current query data, the historical interaction records associated with the current query data, and the target retrieval result to obtain the target answer.

[0130] Optionally, the generation module 804 is also used to: call the retrieval engine to perform information retrieval on the current query data and obtain initial retrieval results; sort the initial retrieval results to obtain the target retrieval results.

[0131] Optionally, the acquisition module 801 is further configured to acquire evaluation data corresponding to the target answer, wherein the evaluation data is determined based on user feedback on the target answer, and the evaluation data is used to update at least one of the following: model parameters of the target retrieval determination model, words contained in the preset thesaurus, and model parameters of the target answer generation model.

[0132] Optionally, the acquisition module 801 is also used to acquire historical query data; the data processing device further includes: a preprocessing module 807, used to preprocess the historical query data to obtain a second word segmentation result; a statistics module 808, used to perform word frequency statistics on the second word segmentation result to obtain statistical results; and an update module 809, used to establish a preset word library based on the statistical results in response to the statistical results meeting preset conditions, wherein the preset conditions are determined based on the popularity and frequency threshold of the target topic.

[0133] Optionally, the preprocessing module 807 is also used to: perform data cleaning on the historical query data to obtain the cleaning result; perform data format conversion on the cleaning result to obtain the conversion result; and perform word segmentation on the conversion result to obtain the second word segmentation result.

[0134] It should be noted that the acquisition module 801, construction module 802, analysis module 803, and generation module 804 correspond to steps S21 to S24 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of a device in the server provided in the above embodiments.

[0135] Figure 9 This is a structural block diagram of another data processing apparatus according to an embodiment of this application, such as... Figure 9 As shown, the device includes:

[0136] Module 901 is used to obtain news report query requests;

[0137] Module 902 is used to construct news report query prompt text based on the matching results of the news report query request. The matching results are used to determine whether the news report query request contains news report type words.

[0138] Analysis module 903 is used to perform query analysis on the news report query prompt text using a target retrieval judgment model to obtain a judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the news report query request.

[0139] The generation module 904 is used to generate target news reports based on news report query requests and judgment results.

[0140] It should be noted that the acquisition module 901, construction module 902, analysis module 903, and generation module 904 correspond to steps S41 to S44 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. These modules can also run as part of a device in the server provided in the above embodiments.

[0141] Figure 10 This is a structural block diagram of another data processing apparatus according to an embodiment of this application, such as... Figure 10 As shown, the device includes:

[0142] The acquisition module 1001 is used to acquire a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: the currently queried data;

[0143] The return module 1002 is used to return a data processing response through the second application programming interface. The response data carried in the data processing response includes: the target answer, which is generated based on the current query data and the judgment result. The judgment result is obtained by performing query analysis on the prompt text using a target retrieval judgment model. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

[0144] It should be noted that the acquisition module 1001 and return module 1002 correspond to steps S51 to S52 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0145] Figure 11 This is a structural block diagram of another data processing apparatus according to an embodiment of this application, such as... Figure 11 As shown, the device includes:

[0146] The acquisition module 1101 is used to acquire the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the currently queried data;

[0147] Return module 1102 is used to respond to the data processing dialogue request and return a data processing dialogue response. The data processing dialogue response carries the following information: the target answer, which is generated based on the current query data and the judgment result. The judgment result is obtained by performing query analysis on the prompt text using a target retrieval judgment model. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

[0148] Display module 1103 is used to display the target answer within the graphical user interface.

[0149] It should be noted that the acquisition module 1101, return module 1102, and display module 1103 correspond to steps S61 to S63 in the above embodiments. The three modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0150] Figure 12 This is a structural block diagram of a model training device according to an embodiment of this application, such as... Figure 12 As shown, the device includes:

[0151] The acquisition module 1201 is used to acquire training data, which includes: query samples and corresponding labels for the query samples. The query samples contain target type words, and the labels are used to determine whether an open search is needed for the query samples.

[0152] Training module 1202 is used to train the initial retrieval judgment model based on training data and generate a target retrieval judgment model. The target retrieval judgment model is used to perform query analysis on the prompt text to obtain a judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval on the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

[0153] Optionally, the training module 1202 is also used to: generate prompt samples based on training data; perform query analysis on the prompt samples using the initial retrieval decision model to obtain prediction results; determine the target loss based on the prediction results and labels; and fine-tune the model parameters of the initial retrieval decision model using the target loss to obtain the target retrieval decision model.

[0154] It should be noted that the acquisition module 1201 and training module 1202 correspond to steps S71 to S72 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules or units can be hardware or software components stored in memory and processed by one or more processors. The above modules can also be part of a device and run in the server provided in the above embodiments.

[0155] It should be noted that the preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, but are not limited to the solutions provided in the above embodiments.

[0156] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic devices. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0157] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0158] In this embodiment, the computer terminal described above can execute the program code in the method.

[0159] Optionally, Figure 13 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 13 As shown, the electronic device may include: one or more (only one is shown in the figure) processors 132, memory 134, memory controller, and peripheral interfaces, wherein the peripheral interfaces are connected to a radio frequency module, an audio module, and a display.

[0160] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0161] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtain the current query data; construct prompt text based on the matching results of the current query data, wherein the matching results are used to determine whether the current query data contains words of the target type matched from a preset thesaurus; perform query analysis on the prompt text using a target retrieval determination model to obtain a determination result, wherein the determination result is used to determine whether a retrieval is needed for the current open domain query of the current query data; and generate the target answer based on the current query data and the determination result.

[0162] Optionally, the processor may also execute program code that performs the following steps: performing word segmentation on the current query data to obtain a first word segmentation result; performing matching processing on the first word segmentation result and a preset thesaurus to obtain a matching result; wherein the matching result records the target query marker corresponding to the current query data, the target query marker including: a first query marker and a second query marker, the first query marker being used to indicate that the target type word corresponding to the first word segmentation result is matched in the preset thesaurus, the second query marker being used to indicate that the target type word corresponding to the first word segmentation result is not matched in the preset thesaurus, and the query level of the first query marker is higher than the query level of the second query marker.

[0163] Optionally, the processor may also execute program code that performs the following steps: in response to the first query tag being recorded in the matching result, constructing a prompt text using the current query data, target type words, and historical interaction records associated with the current query data.

[0164] Optionally, the processor may also execute program code that performs the following steps: in response to the second query tag being recorded in the matching result, constructing a prompt text using the current query data and the historical interaction records associated with the current query data.

[0165] Optionally, the processor may also execute program code that performs the following steps: in response to the determination result indicating that it is not necessary to retrieve the current open domain query, the target answer generation model is used to perform answer generation analysis on the current query data and the historical interaction records associated with the current query data to obtain the target answer.

[0166] Optionally, the processor may also execute program code that performs the following steps: in response to the determination result indicating that a retrieval of the current open domain query is required, the processor calls the retrieval engine to perform information retrieval on the current query data and obtains the target retrieval result; the processor uses a target answer generation model to perform answer generation analysis on the current query data, the historical interaction records associated with the current query data, and the target retrieval result to obtain the target answer.

[0167] Optionally, the processor may also execute program code that performs the following steps: calls the retrieval engine to perform information retrieval on the current query data and obtains initial retrieval results; sorts the initial retrieval results to obtain the target retrieval results.

[0168] Optionally, the processor may also execute program code that performs the following steps: obtaining evaluation data corresponding to the target answer, wherein the evaluation data is determined based on user feedback on the target answer, and the evaluation data is used to update at least one of the following: model parameters of the target retrieval determination model, words contained in the preset thesaurus, and model parameters of the target answer generation model.

[0169] Optionally, the processor may also execute program code that performs the following steps: acquiring historical query data; preprocessing the historical query data to obtain a second word segmentation result; performing word frequency statistics on the second word segmentation result to obtain statistical results; and, in response to the statistical results meeting preset conditions, establishing a preset lexicon based on the statistical results, wherein the preset conditions are determined based on the popularity and frequency thresholds of the target topic.

[0170] Optionally, the processor may also execute program code that performs the following steps: cleansing the historical query data to obtain a cleansing result; converting the cleansing result into a data format to obtain a conversion result; and segmenting the conversion result into words to obtain a second word segmentation result.

[0171] The processor can access information and applications stored in memory via a transmission device to perform the following steps: obtaining a news report query request; constructing a news report query prompt text based on the matching results of the news report query request, wherein the matching results are used to determine whether the news report query request contains news report type words; performing query analysis on the news report query prompt text using a target retrieval judgment model to obtain a judgment result, wherein the judgment result is used to determine whether a retrieval is needed for the current open domain query of the news report query request; and generating a target news report based on the news report query request and the judgment result.

[0172] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: the current query data; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: the target answer, which is generated based on the current query data and the judgment result, the judgment result being obtained by performing query analysis on the prompt text using a target retrieval judgment model, the judgment result being used to determine whether a retrieval is needed for the current open domain query of the current query data, and the prompt text being constructed based on the matching result of the current query data, the matching result being used to determine whether the current query data contains target type words.

[0173] The processor can invoke information and applications stored in memory via a transmission device to perform the following steps: acquiring the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the current query data; responding to the data processing dialogue request, returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: the target answer, which is generated based on the current query data and the judgment result, the judgment result being obtained by performing query analysis on the prompt text using a target retrieval judgment model, the judgment result being used to determine whether a retrieval is needed for the current open domain query of the current query data, the prompt text being constructed based on the matching results of the current query data, the matching results being used to determine whether the current query data contains target type words; and displaying the target answer within the graphical user interface.

[0174] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: acquiring training data, wherein the training data includes: query samples and corresponding tags for the query samples, the query samples contain target type words, and the tags are used to determine whether an open-domain search is needed for the query samples; training an initial retrieval decision model based on the training data to generate a target retrieval decision model, wherein the target retrieval decision model is used to perform query analysis on the prompt text to obtain a decision result, the decision result is used to determine whether a search is needed for the current open-domain query of the current query data, the prompt text is constructed based on the matching results of the current query data, and the matching results are used to determine whether the current query data contains target type words.

[0175] Optionally, the processor may also execute program code that performs the following steps: generating prompt samples based on training data; performing query analysis on the prompt samples using an initial retrieval decision model to obtain prediction results; determining the target loss based on the prediction results and labels; and fine-tuning the model parameters of the initial retrieval decision model using the target loss to obtain the target retrieval decision model.

[0176] By employing the embodiments of this application, current query data is acquired, and then, based on the matching results of the current query data, prompt text is constructed to determine whether the current query data contains words of the target type matched from a preset thesaurus. Subsequently, a target retrieval judgment model is used to analyze the prompt text and obtain a judgment result to determine whether a retrieval is needed for the current open domain query of the current query data. Finally, a target answer is generated based on the current query data and the judgment result. Thus, by dynamically analyzing the current query data and matching it with a preset thesaurus, it is possible to intelligently determine whether the current query data involves information that needs to be updated in real time, thereby selectively triggering open domain query retrieval and effectively avoiding resource waste. The use of a target retrieval judgment model to analyze the prompt text makes the retrieval decision more accurate, ensuring both the timeliness and accuracy of the target answer and avoiding unnecessary retrieval operations. The target answer generated based on the current query data and the judgment result not only quickly responds to the user's query needs but also ensures the accuracy and real-time relevance of the information, thereby significantly optimizing the user's query experience. Therefore, the embodiments of this application achieve the technical effects of saving resource consumption during query processing and improving query processing efficiency, thereby solving the technical problems of high resource consumption and low query processing efficiency in the query processing methods provided in related technologies.

[0177] Those skilled in the art will understand that, Figure 13 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. This diagram does not limit the structure of the aforementioned electronic devices. For example, electronic devices may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the diagram, or have a different configuration than shown in the diagram.

[0178] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0179] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0180] Optionally, in this embodiment, the storage medium may be located in any one of the electronic devices in the group of electronic devices in the computer network, or in any one of the mobile terminals in the group of mobile terminals.

[0181] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining current query data; constructing prompt text based on the matching results of the current query data, wherein the matching results are used to determine whether the current query data contains words of the target type matched from a preset thesaurus; performing query analysis on the prompt text using a target retrieval determination model to obtain a determination result, wherein the determination result is used to determine whether a retrieval is needed for the current open domain query of the current query data; and generating a target answer based on the current query data and the determination result.

[0182] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: performing word segmentation on the current query data to obtain a first word segmentation result; performing matching processing on the first word segmentation result with a preset thesaurus to obtain a matching result; wherein the matching result records the target query marker corresponding to the current query data, the target query marker including: a first query marker and a second query marker, the first query marker being used to indicate that the target type word corresponding to the first word segmentation result is matched in the preset thesaurus, the second query marker being used to indicate that the target type word corresponding to the first word segmentation result is not matched in the preset thesaurus, and the query level of the first query marker is higher than the query level of the second query marker.

[0183] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: in response to a first query tag being recorded in the matching results, constructing a prompt text using the current query data, target type words, and historical interaction records associated with the current query data.

[0184] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: in response to a second query tag being recorded in the matching result, constructing a prompt text using the current query data and historical interaction records associated with the current query data.

[0185] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: in response to a determination result indicating that a retrieval of the current open domain query is not required, an answer generation analysis is performed on the current query data and the historical interaction records associated with the current query data using a target answer generation model to obtain the target answer.

[0186] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: in response to a determination result indicating that a retrieval of the current open domain query is required, calling a retrieval engine to perform information retrieval on the current query data and obtaining the target retrieval result; and using a target answer generation model to perform answer generation analysis on the current query data, the historical interaction records associated with the current query data, and the target retrieval result to obtain the target answer.

[0187] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: invoking a retrieval engine to retrieve information from the current query data and obtaining initial retrieval results; sorting the initial retrieval results to obtain the target retrieval results.

[0188] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: obtaining evaluation data corresponding to the target answer, wherein the evaluation data is determined based on user feedback on the target answer, and the evaluation data is used to update at least one of the following: model parameters of the target retrieval determination model, words contained in a preset thesaurus, and model parameters of the target answer generation model.

[0189] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: obtaining historical query data; preprocessing the historical query data to obtain a second word segmentation result; performing word frequency statistics on the second word segmentation result to obtain statistical results; and, in response to the statistical results meeting preset conditions, establishing a preset lexicon based on the statistical results, wherein the preset conditions are determined based on the popularity and frequency thresholds of the target topic.

[0190] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: cleaning historical query data to obtain cleaning results; converting the cleaning results into a data format to obtain conversion results; and segmenting the conversion results into words to obtain second word segmentation results.

[0191] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a news report query request; constructing a news report query prompt text based on the matching result of the news report query request, wherein the matching result is used to determine whether the news report query request contains news report type words; performing query analysis on the news report query prompt text using a target retrieval judgment model to obtain a judgment result, wherein the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the news report query request; and generating a target news report based on the news report query request and the judgment result.

[0192] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a data processing request through a first application programming interface, wherein the request data carried in the data processing request includes: current query data; returning a data processing response through a second application programming interface, wherein the response data carried in the data processing response includes: a target answer, wherein the target answer is generated based on the current query data and a judgment result, the judgment result is obtained by performing query analysis on the prompt text using a target retrieval judgment model, the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data, and the prompt text is constructed based on the matching result of the current query data, the matching result is used to determine whether the current query data contains target type words.

[0193] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: obtaining a currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: current query data; responding to the data processing dialogue request, returning a data processing dialogue response, wherein the information carried in the data processing dialogue response includes: a target answer, the target answer being generated based on the current query data and a judgment result, the judgment result being obtained by performing query analysis on the prompt text using a target retrieval judgment model, the judgment result being used to determine whether a retrieval is needed for the current open domain query of the current query data, the prompt text being constructed based on the matching result of the current query data, the matching result being used to determine whether the current query data contains target type words; and displaying the target answer within a graphical user interface.

[0194] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: acquiring training data, wherein the training data includes: query samples and corresponding tags for the query samples, the query samples contain target type words, and the tags are used to determine whether an open domain search is needed for the query samples; training an initial search judgment model based on the training data to generate a target search judgment model, wherein the target search judgment model is used to perform query analysis on the prompt text to obtain a judgment result, the judgment result is used to determine whether a search is needed for the current open domain query of the current query data, the prompt text is constructed based on the matching result of the current query data, and the matching result is used to determine whether the current query data contains target type words.

[0195] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: generating prompt samples based on training data; performing query analysis on the prompt samples using an initial retrieval decision model to obtain prediction results; determining the target loss based on the prediction results and labels; and fine-tuning the model parameters of the initial retrieval decision model using the target loss to obtain the target retrieval decision model.

[0196] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the methods provided in the embodiments described above.

[0197] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which can be used to store a computer program that, when executed by a processor, implements the method provided in the above embodiments.

[0198] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0199] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0201] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0203] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0204] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve the currently queried data; Based on the matching results of the current query data, a prompt text is constructed, wherein the matching results are used to determine whether the current query data contains words of the target type matched from a preset thesaurus; The target retrieval judgment model is used to perform query analysis on the prompt text to obtain a judgment result, wherein the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data; The target answer is generated based on the current query data and the judgment result.

2. The data processing method according to claim 1, characterized in that, The data processing method further includes: The current query data is segmented into words to obtain the first segmentation result; The first word segmentation result is matched with the preset dictionary to obtain the matching result; The matching result records the target query marker corresponding to the current query data. The target query marker includes a first query marker and a second query marker. The first query marker is used to indicate that the target type word corresponding to the first word segmentation result is matched in the preset word library. The second query marker is used to indicate that the target type word corresponding to the first word segmentation result is not matched in the preset word library. The query level of the first query marker is higher than that of the second query marker.

3. The data processing method according to claim 2, characterized in that, Based on the matching results of the current query data, the prompt text is constructed as follows: In response to the first query tag being recorded in the matching result, the prompt text is constructed using the current query data, the target type words, and the historical interaction records associated with the current query data.

4. The data processing method according to claim 2, characterized in that, Based on the matching results of the current query data, the prompt text is constructed as follows: In response to the fact that the second query tag is recorded in the matching result, the prompt text is constructed using the current query data and the historical interaction records associated with the current query data.

5. The data processing method according to claim 1, characterized in that, The target answer is generated based on the current query data and the determination result, including: In response to the determination result indicating that it is not necessary to retrieve the current open domain query, the target answer generation model is used to perform answer generation analysis on the current query data and the historical interaction records associated with the current query data to obtain the target answer.

6. The data processing method according to claim 1, characterized in that, The target answer is generated based on the current query data and the determination result, including: In response to the determination result indicating that a retrieval of the current open domain query is required, the retrieval engine is invoked to perform information retrieval on the current query data and obtain the target retrieval result; The target answer is obtained by using a target answer generation model to analyze the current query data, the historical interaction records associated with the current query data, and the target search results.

7. The data processing method according to claim 6, characterized in that, The search engine is invoked to retrieve information from the current query data, and the target search results are obtained by: The search engine is invoked to retrieve information from the current query data and obtain initial search results. The initial search results are sorted to obtain the target search results.

8. The data processing method according to claim 1, characterized in that, The data processing method further includes: Obtain the evaluation data corresponding to the target answer, wherein the evaluation data is determined based on user feedback on the target answer, and the evaluation data is used to update at least one of the following: the model parameters of the target retrieval judgment model, the words contained in the preset thesaurus, and the model parameters of the target answer generation model.

9. The data processing method according to claim 1, characterized in that, The data processing method further includes: Retrieve historical query data; The historical query data is preprocessed to obtain the second word segmentation result; The second word segmentation result was analyzed by word frequency statistics to obtain the statistical results; In response to the statistical results meeting preset conditions, the preset word library is established based on the statistical results, wherein the preset conditions are determined according to the popularity and frequency thresholds of the target topic.

10. The data processing method according to claim 9, characterized in that, The historical query data is preprocessed to obtain the second word segmentation result, which includes: The historical query data is cleaned to obtain the cleaned result; The cleaning results are converted into a data format to obtain the conversion result; The conversion result is then segmented into words to obtain the second segmentation result.

11. A data processing method, characterized in that, include: Retrieve news report query request; Based on the matching results of the news report query request, a news report query prompt text is constructed, wherein the matching results are used to determine whether the news report query request contains news report type words; The target retrieval judgment model is used to perform query analysis on the news report query prompt text to obtain a judgment result, wherein the judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the news report query request; The target news report is generated based on the news report query request and the judgment result.

12. A data processing method, characterized in that, include: A data processing request is obtained through a first application programming interface, wherein the request data carried in the data processing request includes: the currently queried data; The data processing response is returned through the second application programming interface. The response data carried in the data processing response includes: the target answer, which is generated based on the current query data and the judgment result. The judgment result is obtained by querying and analyzing the prompt text using a target retrieval judgment model. The judgment result is used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

13. A data processing method, characterized in that, include: Obtain the currently input data processing dialogue request, wherein the request data carried in the data processing dialogue request includes: the currently queried data; In response to the data processing dialogue request, a data processing dialogue response is returned, wherein the information carried in the data processing dialogue response includes: the target answer, which is generated based on the current query data and the judgment result, the judgment result being obtained by performing query analysis on the prompt text using a target retrieval judgment model, the judgment result being used to determine whether it is necessary to perform a retrieval for the current open domain query of the current query data, and the prompt text being constructed based on the matching result of the current query data, the matching result being used to determine whether the current query data contains target type words; The target answer is displayed within the graphical user interface.

14. A model training method, characterized in that, include: Acquire training data, wherein the training data includes: query samples and labels corresponding to the query samples, the query samples contain target type words, and the labels are used to determine whether an open search is needed for the query samples; The initial retrieval judgment model is trained based on the training data to generate a target retrieval judgment model. The target retrieval judgment model is used to perform query analysis on the prompt text to obtain a judgment result. The judgment result is used to determine whether it is necessary to perform a retrieval on the current open domain query of the current query data. The prompt text is constructed based on the matching result of the current query data. The matching result is used to determine whether the current query data contains target type words.

15. The model training method according to claim 14, characterized in that, The initial retrieval and determination model is trained based on the training data to generate the target retrieval and determination model, including: Based on the training data, generate prompt samples; The initial retrieval and judgment model is used to perform query analysis on the suggested samples to obtain prediction results; The target loss is determined based on the prediction results and the labels; The target loss is used to fine-tune the model parameters of the initial retrieval decision model to obtain the target retrieval decision model.

16. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, performs the data processing method according to any one of claims 1 to 13 or the model training method according to any one of claims 14 to 15.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method of any one of claims 1 to 13 or the model training method of any one of claims 14 to 15.

18. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the data processing method of any one of claims 1 to 13 or the model training method of any one of claims 14 to 15.

Images