Multi-source data search method and device based on artificial intelligence model

Abstract

The invention provides a multi-information-source data searching method and device based on an artificial intelligence model, and the method comprises the steps: responding to a received to-be-processed request, screening out at least one target information source from a plurality of candidate information sources based on the semantic features of the to-be-processed request, and determining a corresponding information source searching rule; searching according to the to-be-processed request and the information source searching rule to obtain target data of each target information source; and based on the to-be-processed request and target data, the artificial intelligence model generates reply content. Therefore, the semantic analysis capability of the artificial intelligence model for the complex request can be enhanced, the pertinence of data search is improved, the reply content generated by the artificial intelligence model based on the search data is ensured to meet the requirement, and the quality of the reply content is improved. Moreover, by screening the information sources, the data processing quantity of the model can be reduced while the data is ensured to be comprehensive and correct, and the model performance is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for searching multi-source data based on an artificial intelligence model. Background Technology

[0002] When users interact with artificial intelligence models (such as large language models), they typically ask the model a question, and the model then searches through a large amount of data before generating a response based on the search results.

[0003] In existing related technologies, vector knowledge bases are typically constructed to compensate for the slow updates and high error rates of web crawlers. However, due to the complex sources of knowledge in vector knowledge bases, a large amount of data with low reference value may exist, leading to inaccurate search results. Furthermore, existing methods still suffer from scheduling defects when performing data searches. When faced with different complex questions, the responses cannot take into account both the reference value of the data and all dimensions of complex requirements, resulting in the AI ​​model's response quality failing to meet expectations. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multi-source data search method and apparatus based on an artificial intelligence model, so as to solve the problem that existing artificial intelligence models cannot find high-quality data when faced with complex needs, resulting in poor quality of generated response content.

[0005] This application provides a multi-source data search method based on an artificial intelligence model, the method comprising: In response to receiving a request to be processed, at least one target information source is selected from multiple candidate information sources based on the semantic features of the request to be processed, and the corresponding information source search rules are determined. The search is performed based on the pending request and the source search rules to obtain the target data for each target source. Based on the request to be processed and the target data, the artificial intelligence model generates a response.

[0006] This application embodiment also provides a multi-source data search device based on an artificial intelligence model, the device comprising: The determination module is used to, in response to receiving a request to be processed, filter out at least one target information source from multiple candidate information sources based on the semantic features of the request to be processed, and determine the corresponding information source search rules; The search module is used to search according to the request to be processed and the source search rules to obtain the target data of each target source; The generation module is used to generate response content based on the request to be processed and the target data, using the artificial intelligence model.

[0007] This application provides a multi-source data search method and apparatus based on an artificial intelligence model. By extracting semantic features of the request to be processed, filtering target sources based on these features, and determining source search rules, the method enhances the semantic parsing ability of the artificial intelligence model for complex requests and improves the targeting of data searches. Furthermore, it ensures that the response content generated by the artificial intelligence model based on the search data meets the requirements, thus improving the quality of the response content. Moreover, by filtering the sources, the method ensures the comprehensiveness and accuracy of the data while reducing the amount of data processing required by the model, balancing the accuracy, comprehensiveness, and real-time nature of the artificial intelligence model's output, thereby improving model performance. To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 The flowchart illustrates a multi-source data search method based on an artificial intelligence model provided in an embodiment of this application. Figure 2 This illustration shows a schematic diagram of the structure of a multi-source data search device based on an artificial intelligence model provided in an embodiment of this application; Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0011] Research has found that when users engage in dialogue with artificial intelligence models (such as large language models), the user typically poses a question to the AI ​​model, which then needs to search through a large amount of data before generating a response based on the search results.

[0012] In existing technologies, vector knowledge bases are typically used to compensate for the slow updates and high error rates of web crawlers. However, due to the complex sources of knowledge in vector knowledge bases, a large amount of data with low reference value may exist. Furthermore, existing methods still suffer from scheduling defects when performing data searches. When faced with different complex queries, they cannot provide both valuable data and cover all dimensions of complex needs, resulting in deficiencies in the answers provided by artificial intelligence models.

[0013] Based on this, this application provides a multi-source data search method based on an artificial intelligence model to improve the data search quality and efficiency of the artificial intelligence model in the face of complex needs, and to improve the quality of the generated response content.

[0014] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-source data search method based on an artificial intelligence model, provided as an embodiment of this application. The method provided in this application embodiment can be implemented using software and / or hardware. Figure 1 As shown in the embodiments of this application, the method includes: S101. In response to receiving a request to be processed, based on the semantic features of the request to be processed, at least one target source is selected from multiple candidate sources, and the corresponding source search rules are determined.

[0015] Here, pending requests can be initiated by users or automatically initiated by specific programs when certain conditions are triggered. Each pending request includes at least one interactive intent, and each interactive intent corresponds to a processing task of the artificial intelligence model. Each processing task may include a search subtask in the corresponding data search stage and a generation subtask in the corresponding response generation stage. Considering the case where a pending request includes multiple interactive intents simultaneously, or the case where a large language model receives multiple pending requests simultaneously, the search subtasks corresponding to each interactive intent can be processed in parallel, and the corresponding generation subtasks can also be processed in parallel. However, since some generation subtasks require data from the search subtasks to be implemented, the search subtasks and generation subtasks corresponding to the same intent must be processed sequentially. For example, a pending request from a user may contain two interactive intents: writing travel suggestions based on today's traffic conditions in a city and writing clothing suggestions based on today's weather forecast for a city. In this case, the traffic search task and the weather search task, as well as the travel suggestion generation task and the clothing suggestion generation task, are independent of each other and can be processed in parallel. However, the traffic search task and the travel suggestion generation task must be completed sequentially, and the weather search task and the clothing suggestion generation task must also be completed sequentially.

[0016] In this step, semantic features at different levels can be extracted through semantic analysis of the request to be processed; for example, semantic features include keywords and interaction intents. Keywords include the original keywords included in the request to be processed. In some cases, the keywords included in the request to be processed are insufficiently comprehensive; therefore, synonym expansion and / or near-synonym expansion can be performed on the original keywords in various dimensions, and the original keywords and expanded keywords are combined into a keyword dataset to prevent keyword deficiencies from affecting search result quality. That is, the keywords in the semantic features can also include synonyms and near-synonyms expanded from the original keywords, together forming a keyword dataset. Keyword dimensions refer to the expanded dimensions under a single keyword: for example, keywords A1, A2, A3… expanded from keyword A are independent dimensions; if B1, B2, B3… expanded from keyword B is another independent dimension, the keywords of each independent dimension are stored in different independent parts of the keyword dataset.

[0017] Furthermore, this application embodiment also considers that if subsequent search subtasks are performed directly based on the keyword dataset, a large amount of irrelevant and redundant search noise may be introduced. Therefore, this application embodiment also assigns different weights to the keywords in each dimension of the keyword dataset, and determines the logical relationship (e.g., AND, OR, NOT relationship) between the keywords in each dimension, as well as at least one keyword with the highest weight in each dimension.

[0018] In practical implementation, the TF-IDF algorithm can be used to evaluate the weight of keywords in each dimension. TF (Term Frequency) represents the frequency of a word appearing in a specific topic database; IDF (Inverse Document Frequency) represents the frequency of a word appearing in the overall database. A higher IDF value indicates a more unique word and its representation of a specific topic. The overall database is composed of several different topic databases, including data from all sources and topics, including but not limited to data from search engines, social media, and self-built source databases. It is a collection of numerous topic databases, storing the most comprehensive data. Here, the overall database, each topic database, and each source database are vector databases. The data in these databases are vectors, requiring conversion into spatial vector form using a vector transformation model. This vector transformation model can be BERT or similar models, and this application does not impose any restrictions on its implementation.

[0019] The specific evaluation method for the weight of keywords in each dimension may include: Step 1, calculating the TF value for all extended keywords in the corresponding keyword database. Step 2, calculating the IDF value for the entire database. Step 3, calculating W = TF * IDF value, which serves as the weight W of the keyword in its dimension. The higher the W value, the more frequently the keyword appears in the current topic and the more unique it is within the entire database, meaning the higher the quality of the keyword, and therefore the higher its weight.

[0020] It is worth noting that this application's embodiments also reveal through research that searching all sources in a single search would result in an enormous volume of data, leading to substantial search overhead, storage overhead, and subsequent writing overhead. Furthermore, an excessive amount of data without prioritization is generally detrimental to generating effective responses. Not only does the marginal benefit of increasing the number of search sources diminish, but referencing inappropriate sources significantly increases the probability of providing incorrect data, thereby affecting the quality of the generated content and reducing user experience. For example, when a user's requirement is to create a travel plan for a city, the depth and breadth of reference data obtained from searching the city's official website are significantly less than that from social media platforms. Without sorting and distinguishing sources, a comprehensive keyword search across all sources may result in a massive amount of data with insufficient relevance, causing the AI ​​model to miss valuable information during search subtasks or waste excessive computing power on processing information with insufficient real-time relevance and effectiveness, ultimately hindering the provision of reasonable travel arrangements in subsequent writing.

[0021] To address this issue, embodiments of this application sort and filter information sources based on semantic features and define information source search rules. This method of filtering information sources based on semantic features narrows the data search scope and ensures that the selected target information sources align with semantic features and are highly relevant to interaction needs, thus balancing search efficiency and search quality.

[0022] In one possible implementation, step S101, based on the semantic features of the request to be processed, filters at least one target source from multiple candidate sources and determines the corresponding source search rules, which may include: S1011. Determine the index value of each candidate source with respect to at least one source index, and determine the weight value of each source index according to the semantic features of the request to be processed.

[0023] On the one hand, information source indicators may include at least one of the following: timeliness indicators, authority indicators, fine-grained indicators, adoption rate indicators, cost indicators, and comprehensive indicators.

[0024] For timeliness indicators The timeliness index value can be obtained through the formula. Calculate; where, This represents the time difference between the publication time of relevant data in the candidate information source and the generation time of the request to be processed, and the unit can be hours; It is an adjustable attenuation coefficient, related to the timeliness type of the candidate information source. For example, for real-time information sources such as news and social media, A higher value causes the timeliness score to decay more quickly; for information sources such as historical documents, official websites, and academic journals, The value is relatively small.

[0025] For authoritative (important) indicators The authority index value can be calculated based on the existing PageRank algorithm. The PageRank value is the authority index value. A higher PageRank value means that the information source is more important and has higher reference value.

[0026] For fine-grained indicators Fine-grained metric values ​​can be evaluated through keyword matching to assess whether the content related to keywords in candidate information sources is close to user experience. If it is close to user experience, the fine-grained score will be higher. It should be noted that the data used to calculate the fine-grained metrics in this application embodiment is obtained only after the user has given explicit authorization. If the user has not given authorization, the fine-grained metrics will use the default data configuration.

[0027] For the adoption rate metric The adoption rate metric includes short-term adoption rate and / or long-term adoption rate. Short-term adoption rate refers to the number of times the candidate source was adopted by the AI ​​model or user in recent tasks (e.g., the most recent 100 tasks, or a recent period) / total number of tasks. Long-term adoption rate is the total number of times the candidate source was adopted by the AI ​​model or user throughout history / total number of tasks. Preferably, the adoption rate metric can combine short-term and long-term adoption rates, for example, by a weighted summation, with the weights of the short-term and long-term adoption rates determined based on each interaction intent. This allows for focusing on more valuable adoption data while effectively avoiding getting trapped in short-term local optima.

[0028] For cost indicators The cost metric refers to the expense of calling the candidate information source, including API price and / or time cost of obtaining its data.

[0029] For comprehensive indicators It is used to measure whether the data in a candidate source is comprehensive enough and whether it can cover different news, opinions, academics, etc.

[0030] On the other hand, when determining the weight value of each source indicator based on the semantic features of the request to be processed, the coefficients of each weight can be adjusted by analyzing the interaction intent in the semantic features. For example, the weight value of a certain indicator that the user or the artificial intelligence model is interested in can be appropriately increased. In one example, if the user explicitly requests to obtain "the latest developments of a certain event," then the timeliness indicator and the cost indicator can be increased. Corresponding weights ,make Greater than 0.5; when system resources are scarce and overhead is sensitive, increase the cost indicator. Corresponding weights When users need a comprehensive understanding, improve the comprehensiveness metrics. Corresponding weights And so on.

[0031] S1012. Determine the priority score of each candidate source based on the index value of each candidate source with respect to at least one source index and the weight value of each source index.

[0032] In this step, the priority score of each candidate source can be determined by weighted summation based on the index value of each candidate source with respect to at least one source indicator and the weight value of each source indicator. The formula can be expressed as: ,and .in, Indicates the first The priority score of each candidate information source These represent the weights of each information source indicator.

[0033] S1013. Based on the priority score of each candidate information source, select the target information source from the plurality of candidate information sources, and define the information source search rules according to the identifier of the target information source.

[0034] In this step, candidate sources with higher priority scores indicate higher data quality and a better match to the interaction intent in the request to be processed. Therefore, a certain number or proportion of candidate sources with higher priority scores can be selected as target sources. The proportion or number can be determined based on the number of candidate sources.

[0035] Furthermore, the higher-ranked candidate sources have the highest scores, but the marginal value of their scores decays rapidly as their ranking decreases. Therefore, embodiments of this application can also dynamically select target sources based on the data decay characteristics of priority scores.

[0036] In another possible implementation, step S1013, which involves selecting the target information source from the plurality of candidate information sources based on the priority score of each candidate information source, may further include: Step a1: Sort the multiple candidate information sources in descending order of priority score, so that the candidate information sources ranked higher have higher priority scores.

[0037] Step a2: Starting from the (i+1)th candidate source in the sorted order, compare the priority score of the selected (i)th candidate source with the priority score of the (i+1)th candidate source to determine whether the (i+1)th candidate source can meet the target source condition, until any candidate source fails to meet the target source condition, where i is a positive integer.

[0038] The first candidate source in the sorted order has the highest priority score and is directly designated as a target source. Starting from the second candidate source in the sorted order, the priority score of the currently selected (i+1)th candidate source is compared with the priority score of the previously selected (i)th candidate source to determine the decrease in priority score. This determines whether the (i+1)th candidate source meets the target source condition. If it does, it is designated as a selected candidate source (i.e., the target source), and the process continues to evaluate the next candidate source until any candidate source fails to meet the target source condition. At this point, the selected candidate source that meets the target source condition is designated as the target source.

[0039] In one example, the priority score difference can be determined based on the priority score of the i-th candidate source and the priority score of the (i+1)-th candidate source. The attenuation rate of the i-th priority score is determined based on the ratio between the priority score difference and the priority score of the i-th candidate source; if the attenuation rate of the i-th priority score is greater than or equal to the first attenuation rate threshold, then the (i+1)-th candidate source is determined to meet the target source condition.

[0040] Here, we can determine when to stop adding new sources by observing the score differences between adjacent candidate sources. When the score difference reaches a certain threshold, it means that the value of adding new sources is significantly lower than the data value brought by the new sources, and we can cut off the addition of candidate sources.

[0041] Furthermore, the impact of absolute score counts can be reduced by calculating the priority score decay rate, allowing for more precise screening of target information sources. The formula is expressed as: ,in, This represents the priority score attenuation rate of the (i+1)th source. A threshold T is set for the attenuation rate, and only candidate sources with a priority score attenuation rate greater than or equal to T are selected as target sources. The value of T is a dynamic threshold, and the initial value of T can be the median or average attenuation rate of each source. Furthermore, to ensure that the number of target sources is not too small due to a high threshold, a minimum number of target sources can also be set.

[0042] After identifying the target information sources, source search rules can be defined based on their identifiers. Furthermore, these source search rules also specify rule parameters for searching within the corresponding identified target information sources, such as the priority between target information sources and the depth of each search within each source.

[0043] S102. Perform a search based on the pending request and the source search rules to obtain the target data for each target source.

[0044] In this step, data matching the semantic features of the request to be processed can be searched within the target information source according to the information source search rules to obtain the target data under each target information source. As an example, the information source search principle is as follows: calculate the spatial distance between at least one keyword vector in the keyword database corresponding to the request to be processed and each data vector in the information source database of the target information source, and take the data corresponding to the data vector whose spatial distance with the keyword vector meets the search distance threshold as the data for feature matching. The information source search principle can also be other forms in the prior art. The spatial distance can be calculated by cosine similarity, Euclidean distance, etc., or it can be calculated based on other spatial distance calculation methods. This application does not limit this. In addition, in order to ensure that the spatial distance calculated during retrieval is effective, the calculation related to spatial distance in this application uses the same calculation method. For example, when calculating the spatial distance, the Euclidean distance between each vector is uniformly used as the spatial distance, or the cosine similarity between each vector is uniformly used as the spatial distance. Moreover, before calculating the spatial distance, each vector needs to be normalized. This application does not limit the vector normalization method.

[0045] S103. Based on the request to be processed and the target data, the artificial intelligence model generates a response.

[0046] In this step, the AI ​​model combines the request to be processed with the searched target data to generate a response. This response can be a dialogue output by a conversational AI model, or it can be delivery content generated by an intelligent agent system (AI Agent), such as code, text, files, or websites (or website links). Files can include documents, images, etc.

[0047] In one possible implementation, before writing the public data of each information source into the information source database in vector form, the following steps are included: The source data is structured to obtain at least one reference data block.

[0048] Here, through structured processing, the searched target data can be segmented into text or image-text blocks with reference value, resulting in at least one reference data block. Each reference data block carries a unique identifier corresponding to the information source. Furthermore, data denoising can be achieved through structured processing, i.e., removing navigation, advertising, and script code elements irrelevant to the core content, followed by block segmentation to remove irrelevant data, improve data quality and effective data density, and finally convert each reference data block into an embedded vector.

[0049] The request to be processed includes at least one target keyword, and step S103 further includes: S1031. Obtain at least one reference data block.

[0050] S1032. Determine the relevance score between each reference data block and the request to be processed, based at least on the density of the target keyword in each reference data block.

[0051] S1033. Based on each of the aforementioned correlation scores, select the target reference data block from multiple reference data blocks.

[0052] Regarding step S1031, the reference data block refers to a data block obtained from each information source according to the information source search rules, whose spatial distance to the at least one target keyword meets the search distance threshold. Here, the target keywords include original keywords and extended keywords.

[0053] For steps S1032 and S1033, after obtaining the target data from the target information source through keywords, the amount of information may be large. On the one hand, it may exceed the context that the artificial intelligence model can process. On the other hand, information with poor relevance may also affect the information quality of the artificial intelligence model when writing, while increasing the time cost and processing cost of reasoning. Therefore, it is necessary to determine the relevance score of each reference data block based at least on the density of keywords, and to sort each reference data block based on the relevance score.

[0054] Therefore, in specific implementation, regarding step S1032, the relevance score between each reference data block and the request to be processed can be determined based on the density of the target keyword in each reference data block and the distribution position of the target keyword in each reference data block.

[0055] Density refers to the frequency of the target keyword in the reference data block; distribution location refers to the location of the target keyword in the reference data block. When the target keyword appears in the title, abstract, or document header or footer of the reference data block, its distribution dimension will have a higher score.

[0056] In specific implementation, regarding step S1033, the method of selecting the target reference data block from multiple reference data blocks based on the correlation score can refer to the aforementioned method of selecting the target information source from multiple candidate information sources, and can achieve the same technical effect. For example, the multiple reference data blocks can be sorted in descending order of correlation score, and the correlation score difference can be determined based on the correlation score of the j-th reference data block and the correlation score of the (j+1)-th reference data block. The correlation score decay rate of the j-th reference data block is determined based on the ratio between the correlation score difference and the correlation score of the j-th reference data block; if the correlation score decay rate of the j-th reference data block is greater than or equal to a second decay rate threshold, then the (j+1)-th reference data block meets the target reference data block condition, where j is a positive integer.

[0057] In this way, the artificial intelligence model performs a second screening on the searched target data. This is because although the target information source has a high priority score, the data in the target information source is not necessarily of reference value. By processing the data in the target information source again, we can further ensure that the reference data is low in noise, highly referential, and structured, and reduce the data size, making it easier for the model to understand and process, thus improving the model's processing efficiency.

[0058] S1035. Generate prompt words based on the pending request and each target reference data block, send them to the artificial intelligence model, and call the artificial intelligence model to generate the response content corresponding to the pending request according to the prompt words.

[0059] In this step, prompts are constructed based on the requests to be processed and the interactive intents obtained from them, which are used to limit the style requirements, format requirements, etc. of the writing report. The mapping relationship between the unique identifier of the information source and the target reference data block, the address of each target reference data block or data block, prompts, and other information are sent to the artificial intelligence model through interface calls, so that the artificial intelligence model can perform the task of replying.

[0060] After the AI ​​model generates the response content, it can also be asked to review whether the response content matches the prompt words and user intent, and whether the facts and occurrence time in the writing report match the data output by the source. If the review passes, the response content is output; if the review fails, the reason for the failure is sent to the AI ​​model so that it can regenerate the response.

[0061] This application provides a multi-source data search method based on an artificial intelligence model. By extracting semantic features of the request to be processed, filtering target sources based on these features, and determining source search rules, this method enhances the semantic parsing ability of the artificial intelligence model for complex requests and improves the targeting of data searches. Furthermore, it ensures that the response content generated by the artificial intelligence model based on the search data meets the requirements, thus improving the quality of the response content. Moreover, by filtering the sources, it ensures that the data is comprehensive and accurate while reducing the amount of data processing required by the model, thereby improving model performance.

[0062] In another embodiment of this application, in some scenarios, the request to be processed needs to be executed multiple times at a certain period or according to certain triggering conditions. Therefore, step S101, determining the corresponding source search rule, may further include: Step b1: Determine the number of times the pending request will be executed based on the pending request.

[0063] Here, the pending requests can be used to indicate the triggering conditions. For example, a pending request might indicate that execution should be triggered periodically. This can be directly specified by the user, such as requesting that a user write travel and clothing suggestions for the day based on the weather at 8 AM every morning. In this case, the search and writing tasks will be triggered once a day, meaning the pending requests will be executed more than once. Alternatively, semantic analysis can be used to determine if the user needs periodic feedback.

[0064] In addition, pending requests can also indicate the conditions for stopping the interaction. For example, if a specific action is required to stop the interaction, then if this specific action is not detected, the pending request needs to be executed multiple times, meaning the pending request is executed more than once.

[0065] Step b2: If the number of times the pending request is executed exceeds one, obtain the execution condition parameters in the pending request, define the source search rule based on the execution condition parameters and the target source, and store the pending request, execution condition parameters and source search rule in the task database.

[0066] In this way, it is possible to monitor in real time whether the pending request needs to be executed again based on the execution condition parameters.

[0067] Accordingly, in response to the object indicated by each pending request satisfying the execution condition parameters, the pending request and the source search rules are retrieved from the task database, and the search is performed again according to the pending request and the source search rules, generating new response content. Preferably, for the same pending request, incremental updates can be implemented based on the new search content and historical search data content. Corresponding incremental updates are also performed during content generation; that is, the additions and deletions in the latest search results and historical search data are used to partially add or delete from the historical search data. Simultaneously, the output content is updated based on the incremental updates, thereby reducing the overhead in the search, storage, and generation chain.

[0068] In this way, responses can be retrieved multiple times from a single pending request, reducing the need for multiple interactions with the AI ​​model and improving human-computer interaction efficiency and ease of use. Furthermore, the source search rules are already stored in the database, avoiding the need for repeated generation of rules and saving computational resources.

[0069] On the other hand, if the execution is limited to one time and no triggering condition is specified, the search can be executed directly and an answer generated. Alternatively, if the execution is limited to one time but a triggering condition is specified, the constraint object of the triggering condition can be monitored, and the search can be executed and an answer generated when the constraint object meets the triggering condition.

[0070] Furthermore, the source search rules stored in the database also include the index value and / or weight value and / or priority score of each candidate source with respect to at least one source indicator.

[0071] In practical implementation, the database in this application embodiment can be divided into a source content database and a source management database. The source content database has the ability to perform retrieval through indexing within massive amounts of data. It stores relatively stable attributes such as: search results obtained by using keywords to search for target sources, the type and link of target sources, original keywords for each task, extended keywords, and the importance of sources. The source management database has high flexibility and is suitable for storing frequently updated information, including priority scores of various target sources under different tasks, the adoption status of artificial intelligence models for each source, and user feedback tags. The source content database is preferably Elasticsearch, a vector database, Apache Solr, Meilisearch, TypeSense, etc., while the source management database can use NoSQL databases such as MongoDB and CouchDB. It should be noted that the above are only examples of optional database types; in practice, this solution does not restrict the choice of database.

[0072] If the execution count exceeds one, the AI ​​model's historical processing history for the request to be processed, including its adoption of reference data from each target information source and / or user feedback, is collected. Based on the AI ​​model's adoption and / or user feedback, the information source search rules stored in the database are dynamically iterated.

[0073] In this way, with the use of artificial intelligence models, this dynamic iterative optimization can make the source data search more in line with the needs of artificial intelligence models and / or users, thereby ensuring that the response content generated by the artificial intelligence model based on the search data meets the requirements and helps to improve the quality of the response content.

[0074] In another embodiment of this application, the method further includes: Step c1: In response to detecting that the similarity between the request to be processed and at least one historical request in the database is greater than or equal to a third similarity threshold, the historical request with the similarity greater than or equal to the third similarity threshold is taken as the target historical request, and the historical association data of the target historical request is obtained from the database. The historical association data includes at least the target keywords, target information sources, and target data associated with the target historical request.

[0075] Step c2: If the request to be processed has new keywords compared to the target historical request, then perform an incremental search based on the new keywords to obtain the incremental reference data corresponding to the new keywords.

[0076] Step c3: Based on the request to be processed, the historical associated data, and the incremental reference data, generate response content using the artificial intelligence model.

[0077] In this embodiment, to conserve computing resources and improve search speed, the source search rules and the target data obtained can be reused after removing sensitive and personal information for different requests from the same user or similar requests from different users. Therefore, when a new request to be processed is detected to have a high similarity to a stored historical request—for example, when the keyword overlap rate between the request to be processed and at least one historical request is greater than or equal to the third similarity threshold—existing data can be retrieved from the source content database and the source management database to avoid repeating the search process. Furthermore, incremental search and updates are performed using newly added keywords that do not overlap in the new request to be processed to prevent information omission. Here, reusing only the source search rules and the target data obtained from the search, without reusing the generated content, not only reduces the overhead of the search steps but also reduces the information independence between different requests to be processed and between different users.

[0078] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a multi-source data search device based on an artificial intelligence model, provided in an embodiment of this application. Figure 2 As shown, the multi-source data search device 200 includes: The determining module 210 is configured to, in response to receiving a request to be processed, filter out at least one target information source from multiple candidate information sources based on the semantic features of the request to be processed, and determine the corresponding information source search rules; The search module 220 is used to search according to the request to be processed and the source search rules to obtain the target data of each target source; The generation module 230 is used to generate response content based on the request to be processed and the target data through the artificial intelligence model.

[0079] Furthermore, when the determining module 210 is used to, in response to receiving a request to be processed, filter out at least one target source from multiple candidate sources based on the semantic features of the request to be processed, and determine the corresponding source search rule, the determining module 210 is used to: Determine the index value of each candidate source with respect to at least one source index, and determine the weight value of each source index based on the semantic features of the request to be processed; The priority score of each candidate source is determined based on the index value of each candidate source with respect to at least one source indicator and the weight value of each source indicator. Based on the priority score of each candidate information source, the target information source is selected from the plurality of candidate information sources, and the information source search rules are defined according to the identifier of the target information source.

[0080] Furthermore, when the determining module 210 selects the target information source from the plurality of candidate information sources based on the priority score of each candidate information source, the determining module 210 is used to: The candidate information sources are sorted in descending order of priority score; Starting from the (i+1)th candidate source in the sorted order, compare the priority score of the selected (i)th candidate source with the priority score of the (i+1)th candidate source to determine whether the (i+1)th candidate source can meet the target source condition, until any candidate source fails to meet the target source condition, where i is a positive integer.

[0081] Furthermore, when the determining module 210 compares the priority score of the selected ith candidate source with the priority score of the (i+1)th candidate source, starting from the (i+1)th candidate source in the sorted order, to determine whether the (i+1)th candidate source satisfies the target source condition, the determining module 210 is used to: The priority score difference is determined based on the priority score of the i-th candidate source and the priority score of the (i+1)-th candidate source. The attenuation rate of the i-th priority score is determined based on the ratio between the priority score difference and the priority score of the i-th candidate source. If the attenuation rate of the i-th priority score is greater than or equal to the first attenuation rate threshold, then the (i+1)-th candidate source is determined to meet the target source condition.

[0082] Furthermore, if the request to be processed includes at least one target keyword, then when the generation module 230 generates response content based on the request to be processed and the target data using the artificial intelligence model, the generation module 230 is used to: Obtain at least one reference data block; Based at least on the density of the target keyword in each reference data block, determine the relevance score between each reference data block and the request to be processed; Based on each of the aforementioned relevance scores, a target reference data block is selected from multiple reference data blocks; Based on the pending request and each target reference data block, a prompt word is generated and sent to the artificial intelligence model. The artificial intelligence model is then invoked to generate the corresponding response content for the pending request based on the prompt word.

[0083] Furthermore, when determining the relevance score between each reference data block and the request to be processed, based at least on the density of the target keyword in each reference data block, the generation module 230 is used to: Based on the density of the target keyword in each reference data block and the distribution position of the target keyword in each reference data block, the relevance score between each reference data block and the request to be processed is determined; Accordingly, when the generation module 230 is used to filter out a target reference data block from multiple reference data blocks based on each of the relevance scores, the generation module 230 is used to: The multiple reference data blocks are sorted in descending order of their relevance scores, and the relevance score difference is determined based on the relevance score of the j-th reference data block and the relevance score of the (j+1)-th reference data block. The correlation score decay rate of the j-th reference data block is determined based on the ratio between the correlation score difference and the correlation score of the j-th reference data block; if the correlation score decay rate of the j-th reference data block is greater than or equal to the second decay rate threshold, then the (j+1)-th reference data block meets the target reference data block condition, where j is a positive integer.

[0084] Furthermore, when determining the corresponding source search rule, the determining module 210 is used to: Based on the pending requests, determine the number of times the pending requests will be executed; If the number of executions exceeds one, the execution condition parameters in the pending request are obtained, and the source search rule is defined based on the execution condition parameters and the target source. The pending request, the execution condition parameters, and the source search rule are then stored in the database. In response to each pending request indicating that the object satisfies the execution condition parameters, the pending request and the source search rule are retrieved from the database, and the search is performed again according to the pending request and the source search rule, and new response content is generated.

[0085] Furthermore, when determining the corresponding source search rule, the determining module 210 is used to: Based on the pending requests, determine the number of times the pending requests will be executed; If the number of executions exceeds one, the execution condition parameters in the pending request are obtained, and the source search rule is defined based on the execution condition parameters and the target source. The pending request, the execution condition parameters, and the source search rule are then stored in the database. In response to each pending request indicating that the object satisfies the execution condition parameters, the pending request and the source search rule are retrieved from the database, and the search is performed again according to the pending request and the source search rule, and new response content is generated.

[0086] Furthermore, the source search rules stored in the database also include the indicator value and / or weight value and / or priority score of each candidate source with respect to at least one source indicator; the multi-source data search device 200 further includes an update module; the update module is used for: If the number of executions exceeds one, the adoption status of reference data for each target information source and / or user feedback status of the artificial intelligence model in the history of processing the request to be processed are collected. The source search rules stored in the database are dynamically iterated based on the adoption of the artificial intelligence model and / or user feedback.

[0087] Furthermore, the search module is also configured to, in response to detecting that the similarity between the request to be processed and at least one historical request in the database is greater than or equal to a third similarity threshold, take the historical request with the similarity greater than or equal to the third similarity threshold as the target historical request, and obtain historical association data of the target historical request from the database, wherein the historical association data includes at least the target keywords, target information sources, and target data associated with the target historical request; if the request to be processed has new keywords compared to the target historical request, then perform incremental search based on the new keywords to obtain incremental reference data corresponding to the new keywords; The generation module is further configured to generate response content using the artificial intelligence model based on the request to be processed, the historical associated data, and the incremental reference data.

[0088] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0089] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the multi-source data search method based on the artificial intelligence model in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0090] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the multi-source data search method based on the artificial intelligence model in the method embodiment shown are specifically implemented in the method embodiment and will not be repeated here.

[0091] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0093] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-source data search method based on an artificial intelligence model, characterized in that, The method comprises: in response to receiving a to-be-processed request, screening at least one target source from a plurality of candidate sources based on semantic features of the to-be-processed request, and determining a corresponding source search rule; searching according to the to-be-processed request and the source search rule to obtain target data of each target source; based on the to-be-processed request and the target data, the artificial intelligence model generates reply content.

2. The method of claim 1, wherein, Based on the semantic features of the to-be-processed request, at least one target source is screened from a plurality of candidate sources, and a corresponding source search rule is determined, comprising: determine the index value of each candidate source on at least one source index, and determine the weight value of each source index according to the semantic features of the to-be-processed request; determine the priority score of each candidate source according to the index value of each candidate source on at least one source index and the weight value of each source index; According to the priority score of each candidate source, the target source is screened from the plurality of candidate sources, and the source search rule is defined according to the identification of the target source.

3. The method of claim 2, wherein, According to the priority score of each candidate source, the target source is screened from the plurality of candidate sources, comprising: sort the plurality of candidate sources in order of priority score from high to low; starting from the i+1th candidate source according to the sorting, compare the priority score of the i th candidate source that has been selected with the priority score of the i+1th candidate source to determine whether the i+1th candidate source meets the target source condition, until any candidate source cannot meet the target source condition, i is a positive integer.

4. The method of claim 3, wherein, Starting from the i+1th candidate source according to the sorting, compare the priority score of the i th candidate source that has been selected with the priority score of the i+1th candidate source to determine whether the i+1th candidate source meets the target source condition, comprising: determine the priority score difference value according to the priority score of the i th candidate source and the priority score of the i+1th candidate source; determine the i priority score decay rate according to the ratio between the priority score difference value and the priority score of the i th candidate source; if the i priority score decay rate is greater than or equal to the first decay rate threshold, it is determined that the i+1th candidate source meets the target source condition.

5. The method of claim 1, wherein, The to-be-processed request comprises at least one target keyword; then based on the to-be-processed request and the target data, the artificial intelligence model generates reply content, comprising: obtain at least one reference data block; determine the relevance score of each reference data block to the to-be-processed request based on at least the density of the target keyword in each reference data block; select target reference data blocks from a plurality of reference data blocks according to each of the relevance scores; generate prompt words based on the to-be-processed request and each target reference data block, and send them to the artificial intelligence model, and call the artificial intelligence model to generate reply content corresponding to the to-be-processed request according to the prompt words.

6. The method of claim 5, wherein, determining a relevance score of each reference data block to the to-be-processed request based on at least a density of the target keyword in each reference data block, comprising: determining a relevance score of each reference data block to the to-be-processed request based on at least a density of the target keyword in each reference data block and a distribution position of the target keyword in each reference data block; the step of screening a target reference data block from the plurality of reference data blocks according to each of the relevance scores comprises: sorting the plurality of reference data blocks in an order from high to low of the relevance scores, determining a relevance score difference value according to a relevance score of a jth reference data block and a relevance score of a j+1th reference data block, j being a positive integer; determining a jth relevance score decay rate according to a ratio between the relevance score difference value and the relevance score of the jth reference data block, and if the jth relevance score decay rate is greater than or equal to a second decay rate threshold, the j+1th reference data block meets a target reference data block condition.

7. The method of claim 2, wherein, determining a corresponding source search rule comprises: determining an execution number of the to-be-processed request according to the to-be-processed request; if the execution number exceeds once, obtaining an execution condition parameter in the to-be-processed request, and defining the source search rule based on the execution condition parameter and the target source, and storing the to-be-processed request, the execution condition parameter and the source search rule in a database; in response to an object indicated by each of the to-be-processed request meeting the execution condition parameter, obtaining the to-be-processed request and the source search rule from the database, and performing search again according to the to-be-processed request and the source search rule, and generating new reply content.

8. The method of claim 7, wherein, the source search rule stored in the database further comprises an index value and / or a weight value and / or a priority score of each candidate source with respect to at least one source index; if the execution number exceeds once, collecting a reference data adoption situation and / or a user feedback situation of each target source in a historical record of the artificial intelligence model in processing the to-be-processed request; dynamically iterating the source search rule stored in the database according to the adoption situation and / or the user feedback situation of the artificial intelligence model.

9. The method of claim 1, wherein, the method further comprises: in response to monitoring that a similarity between the to-be-processed request and at least one historical request in the database is greater than or equal to a third similarity threshold, taking the historical request with the similarity greater than or equal to the third similarity threshold as a target historical request, and obtaining historical association data of the target historical request from the database, the historical association data at least comprising a target keyword, a target source and target data associated with the target historical request; if there is a new keyword in the to-be-processed request compared with the target historical request, performing incremental search based on the new keyword to obtain incremental reference data corresponding to the new keyword; generating reply content using the artificial intelligence model based on the to-be-processed request, the historical association data and the incremental reference data. 10.A multi-source data search device based on an artificial intelligence model, characterized by, the device comprises: A determining module is configured to, in response to receiving a to-be-processed request, filter at least one target source from a plurality of candidate sources based on semantic features of the to-be-processed request, and determine a corresponding source search rule; A searching module is configured to search according to the to-be-processed request and the source search rule, and obtain target data of each target source; A generating module is configured to generate reply content by using the artificial intelligence model based on the to-be-processed request and the target data.

Images