Methods, devices, computer equipment, storage media, and computer program products for obtaining network information

Abstract

This application provides a method, apparatus, computer device, storage medium, and computer program product for obtaining online information. The method includes: in response to an online information query request, obtaining query text and corresponding keywords and text vectors; performing parallel retrieval based on the keywords and text vectors to obtain an initial search result set; the initial search result set includes multiple search results; based on document identifiers in each search result, deduplicating the initial search result set to obtain a candidate search result set; determining the relevance score between the online information document and the query text for each candidate search result in the candidate search result set, and based on each relevance score, selecting and returning the target search result from the candidate search result set to respond to the online information query request. This method can improve the accuracy of the response results returned by the information retrieval system, thereby improving the efficiency of users obtaining online information.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, storage medium and computer program product for obtaining network information. Background Technology

[0002] With the explosive growth of internet information, users expect to quickly and accurately obtain the information they need from massive amounts of online information, which places higher demands on the accuracy and efficiency of information retrieval systems.

[0003] Currently, traditional information retrieval systems primarily rely on keyword matching. However, this keyword-based approach fails to accurately retrieve results when dealing with complex and diverse user queries. This forces users to expend significant time and effort on secondary filtering and identification, impacting their efficiency in obtaining relevant online information.

[0004] Therefore, traditional technologies suffer from the problem that the inaccurate response results returned by information retrieval systems lead to low efficiency in users obtaining online information. Summary of the Invention

[0005] Based on this, the purpose of this application is to at least solve one of the above-mentioned technical defects, in particular the technical defect in the prior art that the response results returned by the information retrieval system are not accurate enough, resulting in low efficiency for users to obtain online information. This application provides a method, apparatus, computer equipment, computer-readable storage medium and computer program product for obtaining online information that can improve the accuracy of the response results returned by the information retrieval system and thus improve the efficiency for users to obtain online information.

[0006] Firstly, this application provides a method for obtaining online information, the method comprising:

[0007] In response to a network information query request, the query text is obtained, along with the keywords and text vectors corresponding to the query text.

[0008] Parallel retrieval is performed based on the keywords and text vectors to obtain an initial set of search results; the initial set of search results includes multiple search results; each search result includes a web information document and the document identifier of the web information document;

[0009] Based on the document identifiers in each of the search results, the initial search result set is deduplicated to obtain a candidate search result set;

[0010] The relevance score between the online information document and the query text of each candidate search result in the candidate search result set is determined, and the target search result is selected from the candidate search result set and returned based on the relevance score to respond to the online information query request.

[0011] In an exemplary embodiment, each search result further includes a similarity score corresponding to the online information document; the similarity score characterizes the degree of similarity between the online information document and the query text; based on the document identifiers in each search result, the initial search result set is deduplicated to obtain a candidate search result set, including:

[0012] Determine whether there are multiple search results with the same document identifier in the initial search result set;

[0013] If they exist, the search result with the highest similarity among multiple search results with the same document identifier is retained to obtain a set of candidate search results;

[0014] If it does not exist, the initial search result set will be used as the candidate search result set.

[0015] In an exemplary embodiment, determining the relevance score between the online information document and the query text for each candidate search result in the candidate search result set includes:

[0016] The input text for each candidate search result is constructed based on the title and summary of the online information document in each candidate search result.

[0017] The input texts of each candidate retrieval result are combined into an input text list, and the query text and the input text list are fed into a pre-trained relevance determination model to obtain the relevance score between the online information document and the query text of each candidate retrieval result.

[0018] In an exemplary embodiment, the target search result is selected from the candidate search result set based on each relevance score, including:

[0019] Based on each relevance score, the candidate search results are sorted in descending order to obtain a list of candidate search results;

[0020] Filter the candidate search results to select the target search results.

[0021] In one exemplary embodiment, filtering the target search result from the candidate search result list includes:

[0022] Select the top-ranked candidate search results from the candidate search results list;

[0023] For each selected candidate search result, the query text and the titles and contents of the online information documents in the candidate search results are input into the large language model to obtain the relevance judgment results of the candidate search results;

[0024] Based on the relevance judgment result of each selected candidate search result, the candidate search results that do not meet the preset relevance conditions are filtered out to obtain the filtered search result set.

[0025] Select the target search results from the filtered search results set.

[0026] In one exemplary embodiment, obtaining the keywords and text vectors corresponding to the query text includes:

[0027] The query text is rewritten using a large language model to obtain the rewritten query text;

[0028] Based on the rewritten query text, a text vector is generated, and the rewritten query text is segmented to obtain multiple keywords.

[0029] In one exemplary embodiment, parallel retrieval based on keywords and text vectors is performed to obtain an initial set of search results, including:

[0030] Based on multiple keywords, parallel keyword searches are performed on the field content storage data under multiple preset key fields in a full-text search engine to obtain multiple field content search result sets. Also, based on query vectors, parallel matching is performed on the field vector storage data of multiple preset key fields in a vector database to obtain multiple field vector matching result sets. The preset key fields include at least title fields, abstract fields, and body text fields.

[0031] The initial search result set is obtained by aggregating the search results set of multiple field content and the vector matching result set of multiple fields.

[0032] In one exemplary embodiment, the vector database and full-text search engine are pre-built through the following steps:

[0033] Obtain the page code of each online news page, and determine the news modality type of each online news page based on the page code;

[0034] By utilizing a large language model based on the data extraction method corresponding to the information modality type of each online information page, structured metadata of each online information page is extracted;

[0035] Based on the field content of each preset key field in each structured metadata, generate a field vector for each web information page under each preset key field;

[0036] A vector database is constructed based on the field vectors of each web news page under each preset key field, and a full-text search engine is constructed based on the field content of each web news page under each preset key field.

[0037] Secondly, this application provides a network information acquisition device, the device comprising:

[0038] The acquisition module is used to respond to network information query requests, acquire the query text, and acquire the keywords and text vectors corresponding to the query text;

[0039] The retrieval module is used to perform parallel retrieval based on keywords and text vectors to obtain an initial set of retrieval results. The initial set of retrieval results includes multiple retrieval results. Each retrieval result includes an online information document and its document identifier.

[0040] The deduplication module is used to remove duplicates from the initial search result set based on the document identifiers in each search result, and obtain a candidate search result set.

[0041] The return module is used to determine the relevance score between the online information document and the query text of each candidate search result in the candidate search result set, and based on each relevance score, to filter out the target search result from the candidate search result set and return it to respond to the online information query request.

[0042] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0043] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0044] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0045] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0046] The network information acquisition method, apparatus, computer equipment, storage medium, and computer program product provided in this application, in response to a network information query request, acquire query text and corresponding keywords and text vectors; perform parallel retrieval based on keywords and text vectors to obtain an initial search result set; the initial search result set includes multiple search results; each search result includes a network information document and its document identifier; based on the document identifiers in each search result, the initial search result set is deduplicated to obtain a candidate search result set; the relevance score between the network information document and the query text in each candidate search result set is determined, and target information documents are selected from the candidate search result set based on the relevance scores. The system retrieves and returns search results in response to online information query requests. By constructing a retrieval architecture that combines keyword and text vector matching, it integrates the precision of keyword matching with the generalization ability of semantic vector matching. This solves the problems of incomplete recall and inaccurate results caused by semantic gaps (inconsistencies between query text and online information documents) or keyword limitations (missing key terms) in traditional single-method retrieval. A more comprehensive candidate set is obtained in the initial retrieval stage. Further deduplication and refined filtering based on relevance scores ensure that the final returned target search results possess both high relevance and high information quality, improving the accuracy and recall of the retrieval system. This allows users to obtain the online information they need more quickly and accurately, significantly improving information retrieval efficiency. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is an application environment diagram of a network information acquisition method in one embodiment;

[0049] Figure 2 This is a flowchart illustrating a method for obtaining network information in one embodiment;

[0050] Figure 3 This is a flowchart illustrating a network information retrieval method in one embodiment;

[0051] Figure 4 This is a flowchart illustrating a network information collection method in one embodiment;

[0052] Figure 5 This is a flowchart illustrating a method for obtaining network information in another embodiment;

[0053] Figure 6 This is a structural block diagram of a network information acquisition device in one embodiment;

[0054] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The network information acquisition method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. In response to a network information query request, server 104 obtains the query text and the corresponding keywords and text vectors. Server 104 performs parallel retrieval based on the keywords and text vectors to obtain an initial set of search results. The initial set of search results includes multiple search results. Each search result includes a network information document and its document identifier. Server 104 deduplicates the initial set of search results based on the document identifiers in each search result to obtain a set of candidate search results. Server 104 determines the relevance score between the network information document and the query text in each candidate search result in the candidate search result set, and based on the relevance scores, selects and returns the target search result from the candidate search result set to respond to the network information query request. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0057] In one exemplary embodiment, such as Figure 2 As shown, a method for obtaining network information is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps S202 to S208. Wherein:

[0058] Step S202: In response to the network information query request, obtain the query text, and obtain the keywords and text vectors corresponding to the query text.

[0059] Among them, a network information query request is a request submitted by a user who hopes to obtain relevant network information.

[0060] The query text is a natural language statement contained in the request that describes the user's information needs.

[0061] Keywords are one or more words or phrases extracted from the query text that represent the core concepts of the query.

[0062] Among them, the text vector is a high-dimensional numerical vector that represents the semantics of the query text, which is transformed by the embedding model.

[0063] Parallel retrieval refers to the server simultaneously using two different query representations, keywords and text vectors, to launch queries to different retrieval subsystems in parallel, thereby obtaining results in a parallel manner.

[0064] Optionally, the server first extracts the query text. Then, the server processes the query text in parallel: on the one hand, it performs word segmentation and keyword extraction to obtain keywords; on the other hand, it converts the query text into text vectors using a text embedding model.

[0065] Step S204: Perform parallel retrieval based on the keywords and text vectors to obtain an initial retrieval result set; the initial retrieval result set includes multiple retrieval results; each retrieval result includes a web information document and the document identifier of the web information document.

[0066] The initial search result set is obtained by summarizing all unprocessed search results directly returned by multiple searches.

[0067] Among them, online information documents are pre-processed web pages, articles, or other information content units.

[0068] The document identifier is a unique ID that identifies a web information document. In this application, the document identifier is represented by a GUID.

[0069] Optionally, the server concurrently uses keywords and text vectors to perform similarity searches, merging the returned multi-path results into an initial search result set.

[0070] Step S206: Based on the document identifiers in each of the search results, deduplication is performed on the initial search result set to obtain a candidate search result set.

[0071] The candidate search result set is the result set after removing duplicate online information documents from the initial search result set.

[0072] Optionally, the server deduplicates the results in the set based on the document identifier GUID to ensure that the same document does not appear repeatedly, thus forming a candidate search result set.

[0073] Step S208: Determine the relevance score between the online information document and the query text of each candidate search result in the candidate search result set, and based on each relevance score, filter out the target search result from the candidate search result set and return it to respond to the online information query request.

[0074] The relevance score is a numerical value that quantifies the degree of relevance between a web information document and the query text.

[0075] The target search results set consists of refined and sorted high-quality results that are ultimately returned to the user.

[0076] Optionally, the server calculates a fine relevance score for each online information document in the candidate retrieval set, and then selects the most relevant part from the candidate retrieval set based on these scores as the target retrieval result and returns it to the user.

[0077] The above-described method for obtaining online information involves responding to an online information query request, acquiring query text, and obtaining keywords and text vectors corresponding to the query text; performing parallel retrieval based on the keywords and text vectors to obtain an initial set of search results; the initial set of search results includes multiple search results; each search result includes an online information document and a document identifier for the online information document; based on the document identifiers in each search result, deduplication is performed on the initial set of search results to obtain a set of candidate search results; the relevance score between the online information document and the query text of each candidate search result in the candidate set of search results is determined, and based on each relevance score, a target search result is selected from the candidate set of search results. The system returns the results in response to the online information query request. Thus, by constructing a retrieval architecture that combines keyword and text vector matching, it integrates the precision of keyword matching with the generalization ability of semantic vector matching. This solves the problems of incomplete recall and inaccurate results caused by semantic gaps (inconsistencies between the query text and the online information document) or keyword limitations (missing key terms) in traditional single-search methods. A more comprehensive candidate set is obtained in the initial retrieval stage, and further refinement through deduplication and relevance score-based screening ensures that the final returned target retrieval results possess both high relevance and high information quality. This improves the accuracy and recall rate of the retrieval system, enabling users to obtain the required online information more quickly and accurately, thereby significantly improving information acquisition efficiency.

[0078] In an exemplary embodiment, each search result further includes a similarity score corresponding to the online information document; the similarity score characterizes the degree of similarity between the online information document and the query text; based on the document identifiers in each search result, the initial search result set is deduplicated to obtain a candidate search result set, including: determining whether there are multiple search results with the same document identifier in the initial search result set; if so, retaining the search result with the highest similarity among the multiple search results with the same document identifier to obtain a candidate search result set; if not, using the initial search result set as the candidate search result set.

[0079] In the context of multi-path retrieval, similarity refers to the quantitative value of the degree of matching between online information documents and query alerts, calculated by each retrieval path according to its own path algorithm. The similarity value range and meaning may differ for different retrieval paths.

[0080] Optionally, during concurrent retrieval, the server records which retrieval path each retrieval result comes from and its corresponding original similarity score. When deduplication is performed, all retrieval results are traversed and grouped by document identifier. If the same online information document (with the same document identifier) ​​is found to be hit in multiple vector retrieval paths and multiple full-text retrieval paths, their respective similarity scores are compared, and the retrieval result with the higher similarity score is retained while the rest are discarded. If the online information document appears in only one retrieval path, it is directly retained. Through this optimal retention strategy, the final set of candidate retrieval results is obtained.

[0081] In practical applications, a web news document may rank highly in keyword searches due to high keyword frequency, but its semantic matching degree (vector similarity) with the core query intent may be lower than that of another document. The technical solution in this embodiment ensures that in the final candidate search result set, each unique web news document appears with the aspect that best matches the query (i.e., the highest similarity score), laying a fairer and more accurate foundation for subsequent unified relevance ranking, thereby indirectly improving the accuracy of the final returned results.

[0082] In this embodiment, by adopting a strategy of retaining the search result with the highest similarity when deduplicating multi-path search results, intelligent selection of search results from different sources for the same document is achieved, solving the problem that simple deduplication may discard high-quality matching information from better search paths.

[0083] In an exemplary embodiment, determining the relevance score between the online information document and the query text of each candidate retrieval result in the candidate retrieval result set includes: constructing the input text of each candidate retrieval result based on the title and summary of the online information document in each candidate retrieval result; combining the input text of each candidate retrieval result into an input text list; and inputting the query text and the input text list into a pre-trained relevance determination model to obtain the relevance score between the online information document and the query text of each candidate retrieval result.

[0084] The title is the name of the online information document, and it usually summarizes the content in a very concise way.

[0085] The abstract is a brief summary or extraction of the core content of an online information document.

[0086] The input text is a text fragment constructed for the relevance determination model to represent the candidate result, usually consisting of a title and summary spliced ​​together or combined in a specific format.

[0087] Among them, the pre-trained relevance determination model is a pre-trained model used to determine the relevance score between each candidate retrieval result and the query text, specifically used to judge the deep semantic relevance between a given query text and a document fragment.

[0088] Optionally, for each web information document in the candidate retrieval result set, the server extracts the title and summary fields from its structured metadata, connects the title and summary with "\n" to construct the input text for the model, and then inputs the original query text and the list of input texts for all candidate retrieval results into a pre-trained relevance determination model. This model performs inference based on (query text, list of input texts) and outputs a relevance score, which reflects complex semantic relevance and intent matching.

[0089] For example, based on the candidate search result set and the query text, an input text list `text_list` for re-ranking can be constructed. `text_list` includes multiple `text` values, corresponding to the input text of multiple candidate search results, where `text = document title + "\n" + document summary`. Then, the Rerank model is called to obtain the ranking score (i.e., relevance score) for re-ranking, represented as `scores = rerank_model(query, text_list)`. Finally, according to the ranking score, the candidate search results in the candidate search result set are sorted in descending order, and the re-ranked candidate search result list is output. The re-ranking model in this application can use different Cross-Encoder models.

[0090] In this embodiment, a pre-trained relevance determination model is used to perform deep semantic matching scoring on the query text and the title and summary of the document. This achieves a leap from shallow matching to deep understanding in relevance evaluation, solving the problem that traditional retrieval ranking mainly relies on statistical features or shallow semantics, which cannot accurately grasp the subtle and deep-level relationship between complex query intent and document content. By understanding the context and reasoning about implicit relationships, a more accurate relevance score is calculated, making the final result ranking closer to the user's real needs and improving the accuracy of the returned results.

[0091] In an exemplary embodiment, selecting target search results from the candidate search result set based on each relevance score includes: sorting each candidate search result in descending order based on each relevance score to obtain a candidate search result list; and selecting target search results from the candidate search result list.

[0092] Among them, descending order sorting refers to ranking the candidate search results in descending order of relevance scores.

[0093] The candidate search results list is a list formed by sorting the candidate results from high to low relevance scores.

[0094] Optionally, after calculating the relevance scores of all candidate search results, the server sorts them according to these scores, with the highest score at the top of the list. Then, from this ordered list of candidate search results, the server filters out the final target search results according to preset rules (e.g., selecting the first N results, or selecting results with scores exceeding a certain threshold).

[0095] In this embodiment, by sorting the results in descending order based on relevance scores, an ordered view is constructed, arranged from high to low relevance to the query text. This solves the problem of inefficiency in traditional search results, where the results are coarsely sorted and users need to manually search for truly relevant content from a large number of results. The ordered list allows the most relevant results to naturally appear at the top, enabling users to prioritize the most valuable information and greatly reducing their filtering and judgment time, thus directly improving the efficiency of users in obtaining online information.

[0096] In an exemplary embodiment, selecting the target search result from the candidate search result list includes: selecting multiple top-ranked candidate search results from the candidate search result list; for each selected candidate search result, inputting the query text and the title and content of the online information document in the candidate search result into a large language model to obtain the relevance judgment result of the candidate search result; based on the relevance judgment result of each selected candidate search result, filtering out candidate search results that do not meet the preset relevance conditions to obtain a filtered search result set; and selecting the target search result from the filtered search result set.

[0097] Among them, large language models refer to general-purpose models with powerful language understanding and generation capabilities (such as LLM models).

[0098] Among them, the relevance judgment result is a structured judgment (such as strong relevance, moderate relevance, weak relevance, or no relevance) generated by the large language model based on the given query statement and the title and content of the online information document.

[0099] Among them, the preset correlation condition is the rule for determining whether the result is acceptable, such as the model judging it as strongly correlated or relatively correlated.

[0100] The filtered search results set is the set of high-quality results remaining after being filtered by the large language model.

[0101] Optionally, the server first sorts the results according to their relevance scores, selects the top N (preset number) candidate search results from the candidate search result list, and constructs prompt words from the original query text and the titles and contents of the online information documents in each selected candidate search result. These prompt words are then submitted to the large language model for relevance judgment. The large language model understands the entire query text and the titles and contents of the online information documents, outputs its professional judgment, and the server parses these judgment results. Based on preset conditions (e.g., only retaining results that the model judges to be strongly or relatively relevant), the server filters the results. After filtering, the filtered search results are reordered (or kept in their original order) according to the initial relevance scores, and the top K (target number, K≤N) are selected as the final target search result set.

[0102] For example, the process of inputting each selected candidate search result into the large language model for relevance judgment can be represented as "relevance = LLM judgment (query, title, content)". If the relevance is strong or relatively relevant, it is retained; otherwise, it is filtered.

[0103] In this embodiment, by introducing a large language model to enhance relevance filtering, the problem that even after the aforementioned sorting, some document titles may still be exaggerated, or titles may only be relevant to a part of the document content but not to the overall content is solved. The large language model can read the entire text and comprehensively evaluate the overall fit between the document and the query, making its judgment more accurate. This step can effectively remove noisy results that are ranked high but are actually irrelevant or of low quality, ensuring that each item in the final set of target search results returned to the user has high relevance, thereby improving the accuracy of the returned results and ensuring the efficiency of the user in obtaining information.

[0104] In one exemplary embodiment, obtaining keywords and text vectors corresponding to query text includes: rewriting the query text using a large language model to obtain rewritten query text; generating text vectors based on the rewritten query text; and performing word segmentation on the rewritten query text to obtain multiple keywords.

[0105] The rewritten query text is the text obtained by rewriting and expanding the query using a large language model. It may include synonym expansion, intent clarification or structured expression, aiming to improve retrieval performance.

[0106] The text vector is a fixed-dimensional semantic vector transformed from the rewritten query text through an embedding model, which can be represented as a query vector embedding. Different text embedding models can be used, such as BGE, E5, and OpenAI embedding.

[0107] Among them, word segmentation is the process of dividing the rewritten text into independent lexical units.

[0108] Optionally, after obtaining the original query text, the server first calls a large language model to intelligently rewrite the query text. For example, it rewrites colloquial and vague queries into more standardized queries that contain more relevant terms. Then, it performs two operations on the rewritten query text: first, it uses a pre-built text embedding model to convert it into a text vector; second, it performs word segmentation and possible stop word removal, etc., to extract the core keywords.

[0109] For example, the server uses LLM to rewrite and expand the user's query statement to obtain the rewritten query statement, and generates a query vector embedding based on the rewritten query statement, as well as extracting multiple keywords.

[0110] In this embodiment, the user's original expression is optimized by rewriting the query using a large language model. This solves the problems that user queries may have, such as unclear, unprofessional, incomplete, or typos. A rewritten query text that is more accurate and richer in information can provide better input for subsequent keyword extraction and text vector generation, thereby improving the matching accuracy between query intent and document index from the source of retrieval.

[0111] In one exemplary embodiment, parallel retrieval based on keywords and text vectors is performed to obtain an initial set of search results. This includes: performing parallel keyword searches on the field content storage data of multiple preset key fields in a full-text search engine based on multiple keywords to obtain a set of multiple field content search results; and performing parallel matching on the field vector storage data of multiple preset key fields in a vector database based on query vectors to obtain a set of multiple field vector matching results. The preset key fields include at least a title field, a summary field, and a body text field. The initial set of search results is obtained by summarizing the set of multiple field content search results and the set of multiple field vector matching results.

[0112] Among them, the preset key fields are the core fields that are predefined in the document for indexing and retrieval, such as the title field, the description field, and the content field.

[0113] Vector databases are databases that store data in high-dimensional vector form (such as text semantic vectors) and support fast similarity retrieval. Vector databases can be OceanBase, Milvus, Pinecone, etc.

[0114] Among them, a full-text search engine is a system based on inverted index technology that supports keyword matching and retrieval of text content.

[0115] Among them, the field content storage data is the raw text data stored by field in the full-text search engine, which is used to build the inverted index.

[0116] Among them, the field vector storage data is a vector database in which the content of each online information document in different fields (such as title, abstract, and body) is encoded into vectors.

[0117] Parallel search and parallel matching refer to performing retrieval operations on different fields simultaneously.

[0118] Optionally, the server initiates multiple sets of parallel subqueries. In the full-text search engine, multiple extracted keywords are used to concurrently search multiple fields such as title, abstract, and body text to obtain result sets for multiple fields. In the vector database, text vectors of title, abstract, and body text are used to concurrently perform similarity matching with vector indexes of multiple fields such as title vector, abstract vector, and body text vector to obtain vector result sets for multiple fields. Finally, the result sets of all subqueries are aggregated into the final initial search result set.

[0119] For example, based on the rewritten query and multiple keywords, the server concurrently executes the following search paths: title field vector search, description field vector search, content field vector search, title field full-text search, description field full-text search, and content field full-text search. It then aggregates the results from these six paths to form an initial search result set. In practical applications, restrictions are applied to each search path, with each path returning TopK search results, where TopK = original TopK × 2. The similarity threshold is set to "vector search > 0.7, full-text search > 0.5".

[0120] In this embodiment, by implementing a multi-field concurrent retrieval strategy, comprehensive coverage and simultaneous examination of different information levels of documents (highly summarized titles, core abstracts, and detailed text) are achieved. This solves the problem that searching only a single field may miss relevant documents due to the atypical description of that field, or fail to distinguish the differences in importance between different fields. Parallel retrieval of multiple fields ensures that the matching of key fields receives sufficient weight, and makes full use of all available information, greatly improving the recall and robustness of the retrieval, and providing a more comprehensive and higher-quality initial candidate set for subsequent processes.

[0121] In an exemplary embodiment, the vector database and the full-text search engine are pre-constructed through the following steps: obtaining the page code of each online news page, and determining the information modality type of each online news page based on the page code; extracting structured metadata of each online news page using a large language model based on the data extraction method corresponding to the information modality type of each online news page; generating field vectors for each online news page under each preset key field based on the field content of each preset key field in each structured metadata; constructing a vector database based on the field vectors of each online news page under each preset key field; and constructing a full-text search engine based on the field content of each online news page under each preset key field.

[0122] The page code can be the HTML or other source code of a web news page.

[0123] Among them, information modality type is a type classified according to the characteristics of the page content of online information page, such as text type, audio type, or video type.

[0124] Among them, the data extraction method is a set of rules designed for different information modalities to extract information from online information on a webpage.

[0125] Structured metadata is standardized information extracted by a large language model and organized in key-value pairs. In practical applications, large language models are used to classify the content of online news pages, extract tags such as content attributes, subject areas, and industry applications, and generate structured metadata. Subsequently, during the retrieval stage, full-text categorized searches can be performed according to content attributes, subject areas, industry applications, etc.

[0126] Among them, the field vector is the numerical representation obtained by vectorizing the text content of a specific field (such as the title).

[0127] Optionally, during the data preprocessing stage, the server first downloads or crawls the original page code of the web information page and obtains the original content data. It identifies the content as text, audio, or video modal based on MIME type and content features, and outputs information modal type tags (text / audio / video). If it is a text tag, the main text is extracted; if it is an audio tag, ASR speech recognition is performed; if it is a video tag, the audio is extracted and then ASR speech recognition is performed. Then, the extraction results are sent to the large language model, which is instructed to extract structured metadata. The large language model can accurately extract the title, main text, publication time, etc. Next, a unified text embedding model is used to generate field vectors for the text content of key fields such as title, summary, and main text in the metadata. Finally, these field vectors are stored in the corresponding index of the vector database, and the original text content of the fields is stored in the full-text search engine to build an inverted index.

[0128] In this embodiment, by introducing information modality type recognition and using a large language model for intelligent structured metadata extraction, high-quality and standardized preprocessing of multimodal network information content is achieved. This solves the problems of inaccurate, incomplete, and formatted metadata extraction caused by the diversity of web page content types, and provides high-quality and clearly structured raw data for subsequent retrieval.

[0129] For the convenience of those skilled in the art, Figure 3 A web information retrieval method is provided, specifically including: receiving user query requests, parsing user query requests, and obtaining query text; rewriting the query text using LIM to obtain the rewritten query text, and generating query vectors based on the rewritten query text; concurrently executing multi-path retrieval, including vector retrieval paths 1, 2, and 3 and full-text retrieval paths 1, 2, and 3; for vector retrieval paths, filtering retrieval results with a similarity threshold greater than 0.7, and for full-text retrieval paths, filtering retrieval results with a similarity threshold greater than 0.5, with each path performing similarity filtering independently; summarizing the filtered results from each path, deduplicating results based on GUIDs, and retaining the highest score for the same GUID; re-ranking the results using a Rerank model, and determining whether precise filtering is enabled; if enabled, performing LIM relevance judgment, with judgment results including strongly relevant, moderately relevant, weakly relevant, and irrelevant, retaining only the strongly relevant and moderately relevant results, determining whether there are any remaining results, and if so, continuing relevance judgment; if precise filtering is not enabled, skipping this step; and returning the TopK results to the requesting end.

[0130] Figure 4An exemplary method for collecting online information is provided, specifically including: configuring a data source (including parameters such as type and address); a scheduler triggering a data collection task; selecting the appropriate collection method according to the data source type; for RSS type, obtaining a URL list by parsing the RSS feed; for Sitemap type, obtaining a URL list by parsing the Sitemap. The process begins with retrieving a list of URLs. For API-type URLs, this is done by calling the API interface; for webpage-type URLs, it's done by web crawling. The retrieved URLs are deduplicated. If a URL already exists in the system, it's skipped; otherwise, the process continues. HTTP requests are sent to unique URLs to retrieve the original content (HTML, audio, video, etc.). The content is then processed according to its type: text is extracted, audio is converted to text using ASR speech recognition, and video is extracted and converted to text using ASR speech recognition. LLM is used to intelligently classify the processed text, generating metadata (including title, category, author, publication time, etc.). The text content is then vectorized to generate vector representations. The metadata, vectors, and content are stored in the database. Finally, a decision is made to continue processing the next URL or data source. If yes, the process returns to the URL list retrieval step; otherwise, the process ends, completing all data processing tasks.

[0131] This application provides a network information processing system, including a data acquisition layer, a data processing layer, a storage index layer, a retrieval service layer, and an application interface layer.

[0132] The data acquisition layer is configured with functional modules including a multi-source adapter module (supporting multiple data sources such as RSS, Sitemap, API, and web crawling), an anti-crawler processing module (using strategies such as proxy pools, request rate limiting, and User-Agent rotation), and a content extraction module (extracting the main text based on the Readability algorithm and site rules). Its data flow is "data source URL → adapter parsing → HTTP request → raw content → content extraction → structured data". Its technical implementation includes using Playwright to drive the browser to process dynamic pages, using the crawl4ai framework for intelligent content extraction, and using a Redis caching mechanism to avoid duplicate crawling.

[0133] The execution entity of the data processing layer is the data processing server, which performs the above... Figure 4 The process is shown below.

[0134] The main execution entity of the storage index layer is the database server, and its storage architecture includes a master data table (used to store document metadata and original content), a vector index table (stores vector representations of multiple fields), and a full-text index (based on OceanBase's full-text search capabilities).

[0135] The retrieval service layer is executed by the retrieval server, which performs the above-mentioned tasks. Figure 3 The process is shown below.

[0136] The API gateway server is the main execution entity of the application interface layer. Its primary interfaces include the hybrid search interface and the RAG question-and-answer interface. The hybrid search interface POST / database / hybrid_search request parameters include { "query": "query text", "topk": 20,"filter": {filter conditions},"output_fields": [list of returned fields]}. The RAG question-and-answer interface POST / rag / ask_stream generates streaming answers based on search results. Its processing flow includes: calling the hybrid search to retrieve relevant documents; constructing contextual prompts; calling LLM to generate streaming answers; and returning answers with citations.

[0137] The core innovations of this application lie in the unified multimodal content processing mechanism, the multi-path concurrent retrieval architecture, intelligent re-ranking, and relevance filtering. The unified multimodal content processing mechanism establishes a unified multimodal content processing pipeline, converting audio and video content into text using ASR technology to achieve a unified vectorized representation. It not only supports unified retrieval of text, audio, and video modalities but also preserves the original media information while providing text retrieval capabilities. It distinguishes content types using the `medium_type` field and supports modal filtering. The multi-path concurrent retrieval architecture designs a multi-field, multi-algorithm concurrent retrieval architecture, simultaneously executing vector retrieval and full-text retrieval to maximize recall. The pseudocode for concurrently executing all retrieval tasks can be `results = await asyncio.gather(*tasks)`. Intelligent re-ranking and relevance filtering combine the Rerank model and LLM's two-stage relevance assessment to significantly improve retrieval accuracy. In the first stage, a dedicated Rerank model is used to re-rank the retrieval results. In the second stage, LLM is used to classify the results by relevance (strongly relevant / relatively relevant / weakly relevant / irrelevant). In the third stage, low-quality results are filtered based on the relevance level.

[0138] The technical solution of this application can be applied to enterprise intelligent knowledge base scenarios, enabling the construction of internal knowledge bases within enterprises. This integrates multimodal information from official websites, technical blogs, research reports, audio and video conferences, and other sources. Specific processes include: collecting relevant enterprise information via RSS, APIs, etc.; performing ASR transcription on audio and video conference recordings; using hybrid retrieval to support employees in quickly finding relevant information; and providing intelligent customer service capabilities through RAG question-and-answer. It can also be applied to media content aggregation scenarios, allowing media organizations to aggregate content from multiple platforms and provide editors with material retrieval and content generation support. Specific processes include: collecting multimodal content such as text news, podcast audio, and video reports; uniformly converting it into a searchable text representation; supporting filtering by topic, time, source, and other dimensions; and providing editors with relevant material recommendations.

[0139] The technical effects achievable by this application include: First, significantly improved retrieval accuracy, achieved by 30-50% compared to a single retrieval method through multi-path concurrent retrieval and intelligent reordering; Second, unified processing of multimodal content, enabling unified retrieval of text, audio, and video content, solving the problem that traditional systems can only handle a single modality; Third, optimized retrieval performance, maintaining millisecond-level response times even on large-scale datasets through concurrent execution and database optimization; Fourth, intelligent content filtering, effectively filtering low-quality results based on LLM relevance judgment, improving user experience; Fifth, strong system scalability, supporting rapid integration of new data sources and new retrieval algorithms through modular design.

[0140] In another embodiment, such as Figure 5 As shown, a method for obtaining network information is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0141] Step S502: In response to the network information query request, obtain the query text, and obtain the keywords and text vectors corresponding to the query text.

[0142] Step S504: Perform parallel retrieval based on keywords and text vectors to obtain an initial retrieval result set; the initial retrieval result set includes multiple retrieval results; each retrieval result includes a web information document, the document identifier of the web information document, and the similarity of the web information document; the similarity represents the degree of similarity between the web information document and the query text.

[0143] Step S506: Determine whether there are multiple search results with the same document identifier in the initial search result set.

[0144] Step S508: If it exists, retain the search result with the highest similarity among multiple search results with the same document identifier to obtain a set of candidate search results.

[0145] Step S510: If it does not exist, the initial search result set is used as the candidate search result set.

[0146] Step S512: Determine the relevance score between the online information document and the query text of each candidate search result in the candidate search result set, and based on each relevance score, filter out the target search result from the candidate search result set and return it to respond to the online information query request.

[0147] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a method for obtaining online information described above.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] The network information acquisition device provided in the embodiments of this application is described below. The network information acquisition device has the same inventive concept as the network information acquisition method described above. The solution to the problem provided by the device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more network information acquisition device embodiments provided below can be referred to the limitations of the network information acquisition method above. The network information acquisition device described below and the network information acquisition method described above can be referred to each other, and will not be repeated here.

[0150] In one exemplary embodiment, Figure 6 This is a schematic diagram of the structure of a network information acquisition device provided in an embodiment of this application, such as... Figure 6 As shown, the network information acquisition device includes: an acquisition module 602, a retrieval module 604, a deduplication module 606, and a return module 608, wherein:

[0151] The acquisition module 602 is used to respond to a network information query request, acquire the query text, and acquire the keywords and text vectors corresponding to the query text;

[0152] The retrieval module 604 is used to perform parallel retrieval based on the keywords and text vectors to obtain an initial retrieval result set; the initial retrieval result set includes multiple retrieval results; each retrieval result includes a web information document and the document identifier of the web information document;

[0153] The deduplication module 606 is used to deduplicat the initial search result set based on the document identifiers in each of the search results to obtain a candidate search result set;

[0154] The return module 608 is used to determine the relevance score between the online information document and the query text of each candidate search result in the candidate search result set, and based on each relevance score, to filter out the target search result from the candidate search result set and return it to respond to the online information query request.

[0155] In an exemplary embodiment, each search result further includes the similarity of the online information document; the similarity characterizes the degree of similarity between the online information document and the query text; the deduplication module 606 is specifically used to determine whether there are multiple search results with the same document identifier in the initial search result set; if so, the search result with the highest similarity is retained among the multiple search results with the same document identifier to obtain a candidate search result set; if not, the initial search result set is used as the candidate search result set.

[0156] In an exemplary embodiment, the return module 608 is specifically configured to construct the input text for each candidate search result based on the title and summary of the online information document in each candidate search result; combine the input text of each candidate search result into an input text list; and input the query text and the input text list into a pre-trained relevance determination model to obtain the relevance score between the online information document of each candidate search result and the query text.

[0157] In an exemplary embodiment, the return module 608 is specifically used to sort the candidate search results in descending order based on each relevance score to obtain a list of candidate search results; and to filter out the target search results from the list of candidate search results.

[0158] In an exemplary embodiment, the return module 608 is specifically used to select multiple candidate search results that rank highly from the candidate search result list; for each selected candidate search result, the query text and the title and content of the online information document in the candidate search result are input into the large language model to obtain the relevance judgment result of the candidate search result; based on the relevance judgment result of each selected candidate search result, candidate search results that do not meet the preset relevance conditions are filtered out to obtain a filtered search result set; and the target search result is selected from the filtered search result set.

[0159] In an exemplary embodiment, the acquisition module 602 is specifically used to rewrite the query text using a large language model to obtain the rewritten query text; generate a text vector based on the rewritten query text; and perform word segmentation on the rewritten query text to obtain multiple keywords.

[0160] In an exemplary embodiment, the retrieval module 604 is specifically configured to perform parallel keyword searches on the field content storage data under multiple preset key fields in a full-text search engine based on multiple keywords, to obtain a set of multiple field content search results; and to perform parallel matching on the field vector storage data of multiple preset key fields in a vector database based on query vectors, to obtain a set of multiple field vector matching results; the preset key fields include at least a title field, a summary field, and a body text field; and to summarize the set of multiple field content search results and the set of multiple field vector matching results to obtain an initial retrieval result set.

[0161] In one exemplary embodiment, the construction module is configured to: acquire the page code of each online news page; determine the news modality type of each online news page based on the page code; extract structured metadata of each online news page using a large language model based on the data extraction method corresponding to the news modality type of each online news page; generate a field vector for each online news page under each preset key field based on the field content of each preset key field in each structured metadata; construct a vector database based on the field vectors of each online news page under each preset key field; and construct a full-text search engine based on the field content of each online news page under each preset key field.

[0162] In one exemplary embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the network information acquisition methods described above.

[0163] In one exemplary embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the network information acquisition methods described above.

[0164] In one exemplary embodiment, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the network information acquisition methods described in the above embodiments.

[0165] Indicatively, such as Figure 7 As shown, Figure 7This is a schematic diagram of the internal structure of a computer device 700 provided in an embodiment of this application. The computer device 700 can be provided as a server. (Refer to...) Figure 7 The computer device 700 includes a processing component 702, which further includes one or more processors, and memory resources represented by memory 701 for storing instructions, such as application programs, that can be executed by the processing component 702. The application programs stored in memory 701 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 702 is configured to execute instructions to perform the network information acquisition method of any of the above embodiments.

[0166] The computer device 700 may also include a power supply component 703 configured to perform power management of the computer device 700, a wired or wireless network interface 704 configured to connect the computer device 700 to a network, and an input / output (I / O) interface 705. The computer device 700 can operate on an operating system stored in memory 701, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0167] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0169] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0170] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for obtaining online information, characterized in that, The method includes: In response to a network information query request, the query text is obtained, along with the keywords and text vectors corresponding to the query text. Parallel retrieval is performed based on the keywords and text vectors to obtain an initial set of search results; the initial set of search results includes multiple search results; each search result includes a web information document and the document identifier of the web information document; Based on the document identifiers in each of the search results, the initial search result set is deduplicated to obtain a candidate search result set; The relevance score between the online information document and the query text of each candidate search result in the candidate search result set is determined, and the target search result is selected from the candidate search result set and returned based on the relevance score to respond to the online information query request.

2. The method according to claim 1, characterized in that, Each search result also includes the similarity score corresponding to the online information document; the similarity score represents the degree of similarity between the online information document and the query text; The initial search result set is deduplicated based on the document identifiers in each of the search results to obtain a candidate search result set, including: Determine whether there are multiple search results with the same document identifier in the initial search result set; If they exist, the search result with the highest similarity among the multiple search results with the same document identifier is retained to obtain the candidate search result set. If it does not exist, the initial search result set will be used as the candidate search result set.

3. The method according to claim 1, characterized in that, The determination of the relevance score between the online information document and the query text for each candidate search result in the candidate search result set includes: The input text for each candidate search result is constructed based on the title and summary of the online information document in each candidate search result; The input texts of each candidate retrieval result are combined into an input text list, and the query text and the input text list are input into a pre-trained relevance determination model to obtain the relevance score between the online information document of each candidate retrieval result and the query text.

4. The method according to claim 1, characterized in that, The step of selecting target search results from the candidate search result set based on each of the relevance scores includes: Based on the relevance scores, the candidate search results are sorted in descending order to obtain a list of candidate search results; Select the target search result from the list of candidate search results.

5. The method according to claim 4, characterized in that, The step of filtering the target search results from the candidate search result list includes: Select the top-ranked candidate search results from the candidate search results list; For each selected candidate search result, the query text and the titles and contents of the online information documents in the candidate search results are input into the large language model to obtain the relevance judgment result of the candidate search results; Based on the relevance judgment result of each selected candidate search result, the candidate search results that do not meet the preset relevance conditions are filtered out to obtain the filtered search result set. The target search result is selected from the filtered search result set.

6. The method according to claim 1, characterized in that, The step of obtaining the keywords and text vectors corresponding to the query text includes: The query text is rewritten using a large language model to obtain the rewritten query text; Based on the rewritten query text, the text vector is generated, and the rewritten query text is segmented to obtain multiple keywords.

7. The method according to claim 1, characterized in that, The parallel retrieval based on the keywords and text vectors yields an initial set of search results, including: Based on the multiple keywords, parallel keyword searches are performed on the field content storage data under multiple preset key fields in the full-text search engine to obtain multiple field content search result sets. Furthermore, based on the query vector, parallel matching is performed on the field vector storage data of multiple preset key fields in the vector database to obtain multiple field vector matching result sets. The preset key fields include at least a title field, a summary field, and a body text field. The initial search result set is obtained by summing the search results set of the multiple fields and the vector matching result set of the multiple fields.

8. The method according to claim 7, characterized in that, The vector database and the full-text search engine are pre-built through the following steps: Obtain the page code of each online information page, and determine the information modality type of each online information page based on the page code of each online information page; Using a large language model based on the data extraction method corresponding to the information modality type of each of the aforementioned online information pages, structured metadata of each of the aforementioned online information pages is extracted; Based on the field content of each preset key field in each structured metadata, a field vector is generated for each of the web information pages under each preset key field; Based on the field vectors of each of the aforementioned online information pages under each of the aforementioned preset key fields, the vector database is constructed; and based on the field content of each of the aforementioned online information pages under each of the aforementioned preset key fields, the full-text search engine is constructed.

9. A network information acquisition device, characterized in that, The device includes: The acquisition module is used to respond to a network information query request, acquire the query text, and acquire the keywords and text vectors corresponding to the query text; The retrieval module is used to perform parallel retrieval based on the keywords and text vectors to obtain an initial retrieval result set; the initial retrieval result set includes multiple retrieval results; each retrieval result includes a web information document and the document identifier of the web information document; The deduplication module is used to deduplicat the initial search result set based on the document identifiers in each of the search results to obtain a candidate search result set; The return module is used to determine the relevance score between the online information document and the query text of each candidate search result in the candidate search result set, and based on each relevance score, to filter out the target search result from the candidate search result set and return it to respond to the online information query request.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images