Network information fusion method and device, electronic equipment and computer readable storage medium

Abstract

Embodiments of the present application provide a network information fusion method and device, electronic equipment and computer readable storage medium, relating to the technical field of information processing. The method filters high-value target links based on user requests, realizes semantic-guided path optimization, and improves the accuracy and value density of information acquisition. The target messages are filtered based on the semantic relevance of the key information of each message in the target link and the user request, realizing selective attention. Whether to determine a new target link to continue exploration is judged in combination with the user request and the key information of the accessed message, realizing adaptive adjustment of exploration depth. The content of each exploration webpage is purified, realizing high-precision content separation. After stopping exploration, a standard report is automatically generated based on multi-source information, which can simulate human experts to automatically complete intelligent navigation, dynamic collection, semantic understanding, deep integration and report generation, realizing end-to-end automation from user natural language requests to deep reports.

Description

Technical Field

[0001] This invention relates to the field of information processing technology, and more specifically, to a network information fusion method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Currently, obtaining online information mainly relies on two types of tools: one is general search engines, whose returned results are discrete and isolated, making it difficult to reflect the factual connections and logical connections between different information sources; the other is targeted crawlers, whose access paths are determined by pre-set domain names or link lists, and once started, they execute according to fixed rules, making it impossible to dynamically adjust subsequent access targets based on the actual content value of the pages already obtained.

[0003] Both of these approaches treat information acquisition and information understanding as separate stages. The former stops at the list of links or raw HTML data, while the latter requires separate manual judgment or independent analysis. This results in a lack of a coherent, controllable, and verifiable process support between the initial query intent and the final usable conclusion. The entire research process cannot simulate the selective attention, in-depth exploration, and knowledge integration behavior of human experts in the research process. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a network information fusion method, apparatus, electronic device and computer-readable storage medium that can improve the accuracy and usability of network information fusion.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides a network information fusion method, the method comprising: Filter multiple high-value target links from the initial access links based on user requests; Extract key information from each message in the target link; Target messages are filtered from each message based on the semantic relevance of the key information of each message to the user request; Based on the key information of each sub-message in the target message, the key information of the user request and the accessed message, determine whether to identify a new target link and continue exploring; The webpage content corresponding to the target message obtained in each exploration is purified to obtain the effective search content corresponding to the target message; After the exploration stops, a fusion report is generated based on the preset report template and the access addresses, valid search content, and semantic relevance of all the target messages.

[0006] In an optional implementation, the step of filtering multiple high-value target links from the initial access links based on the user request includes: Based on the user request and each of the initial access links, determine the semantic relevance score, authority score, accessibility score, and timeliness score of each initial access link; The value score of each initial access link is obtained by weighted summation of its semantic relevance score, authority score, accessibility score, and timeliness score. The initial access links corresponding to the value scores that meet the high-value criteria are identified as target links.

[0007] In an optional implementation, the key information includes a title, a summary, and a body; the step of filtering target messages from each message based on the semantic relevance of the key information of each message to the user request includes: The semantic relevance of the title, summary, and body of each message to the user request is evaluated to obtain the semantic relevance of each message; Messages whose semantic relevance exceeds the relevance threshold are identified as the target messages.

[0008] In an optional implementation, the key information includes a title, summary, body, and access address; the step of determining whether to identify a new target link to continue exploring based on the key information of each sub-message in the target message, the user request, and the key information of the already accessed messages includes: The semantic relevance of the title, summary, and body of each sub-message in the target message to the user request is evaluated to obtain the semantic relevance of each sub-message; The similarity between the access address of each sub-message and the access address of the accessed messages is evaluated to obtain the URL similarity of each sub-message; The content quality score of each sub-message is determined based on the number of images contained in the webpage content corresponding to each sub-message; The decision to determine whether to identify new target links and continue exploration is based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message.

[0009] In an optional implementation, determining whether to identify a new target link to continue exploring based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message includes: The exploration score of each sub-message is determined based on its semantic relevance, exploration depth, URL similarity, and content quality score. If the exploration depth is less than the depth threshold and the semantic relevance of the sub-message with the highest exploration score is not lower than the relevance exploration threshold and the content quality score of the sub-message with the highest exploration score is not lower than the quality exploration threshold, then it is determined to continue exploration and the link corresponding to the sub-message with the highest exploration score is determined as a new target link. If the exploration depth is not less than the depth threshold, or the semantic relevance of the sub-message with the highest exploration score exceeds the relevance exploration threshold, or the content quality score of the sub-message with the highest exploration score exceeds the quality exploration threshold, then the exploration is stopped.

[0010] In an optional implementation, the step of purifying the webpage content corresponding to the target message obtained in each exploration to obtain the valid search content corresponding to the target message includes: The webpage content corresponding to the target message is divided into multiple text blocks; The core content identifier, text density, and noise similarity of each text block are determined based on each text block and preset noise keywords; The content confidence of each text block is determined based on its core content identifier, text density, noise similarity, and total character length. Text blocks whose content confidence exceeds the confidence threshold are identified as valid text blocks; The valid text blocks corresponding to the target message are concatenated to obtain the valid search content.

[0011] In an optional implementation, determining the core content identifier, text density, and noise similarity of each text block based on each text block and preset noise keywords includes: If the text block is located in the main text, then the core content identifier indicates that the text block belongs to the core content; If the text block is not located in the main text, then the core content identifier indicates that the text block does not belong to the core content; The ratio of the number of text characters in the text block to the total number of characters is determined as the text density of the text block; The noise similarity of the text block is determined based on the text block and preset noise keywords.

[0012] Secondly, the present invention provides a network information fusion device, the device comprising: The semantic navigation module is used to filter multiple high-value target links from the initial access links based on user requests; extract key information of each message in the target links; filter target messages from each message based on the semantic relevance of the key information of each message to the user request; and determine whether to identify new target links to continue exploring based on the key information of each sub-message in the target messages, the user request, and the key information of the accessed messages. The deep acquisition module is used to clean up the web page content corresponding to the target message obtained in each exploration to obtain the effective search content corresponding to the target message; The intelligent analysis module is used to generate a fusion report based on a preset report template and the access addresses, valid search content, and semantic relevance of all the target messages after the exploration stops.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the network information fusion method described in any of the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the network information fusion method as described in any of the foregoing embodiments.

[0015] Compared to existing technologies, the network information fusion method, apparatus, electronic device, and computer-readable storage medium provided in this invention filter multiple high-value target links from initial access links based on user requests, achieving semantically guided path optimization and improving the accuracy and value density of information acquisition. The method extracts key information from each message in the target links and filters target messages based on the semantic relevance of each message's key information to the user request, achieving selective focus. It determines whether to continue exploring new target links based on the key information of each sub-message in the target messages, the user request, and the key information of already accessed messages, achieving adaptive adjustment of exploration depth. This allows for in-depth exploration of high-value content branches and timely mitigation of low-value branches, balancing research depth and breadth.

[0016] The webpage content corresponding to the target message obtained in each exploration is purified to obtain the effective search content corresponding to the target message, achieving high-precision content separation. After the exploration stops, a fusion report is generated based on the preset report template and the access address, effective search content, and semantic relevance of all target messages. This realizes the automated generation of standardized reports from multi-source information, compressing the research work of several hours to several days to be completed in minutes, significantly reducing research time costs. It can simulate human experts to automatically complete intelligent navigation, dynamic collection, semantic understanding, deep integration, and report generation, achieving end-to-end automation from user natural language requests to in-depth reports.

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a network information fusion method provided in an embodiment of the present invention is shown.

[0020] Figure 2 This diagram illustrates another flowchart of the network information fusion method provided in an embodiment of the present invention.

[0021] Figure 3 A block diagram of a network information fusion device provided in an embodiment of the present invention is shown.

[0022] Figure 4 A block diagram of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0025] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0026] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] Please refer to Figure 1 , Figure 1 A schematic flowchart of a network information fusion method provided by an embodiment of the present invention is shown. The method includes the following steps: Step S10: Based on the user request, filter multiple high-value target links from the initial access links.

[0028] In this embodiment of the invention, multiple target links with high value for completing the information retrieval can be selected from the preset initial access links, thereby ensuring that the subsequent information retrieval and collection work always starts from web page sources with high credibility, reliable content and strong timeliness, avoiding the waste of computing resources and time on low-quality web pages that are inaccessible, have outdated content or lack authority.

[0029] Step S20: Extract key information from each message in the target link.

[0030] Step S30: Filter target messages from each message based on the key information of each message and the semantic relevance of the user request.

[0031] In this embodiment of the invention, information mining is performed on each target link. First, structured elements such as the title, summary, body and URL of each message are extracted from the webpage corresponding to each target link to form key information that can be used for semantic comparison. This allows for more accurate screening of core messages that truly focus on user requests, while eliminating semantically deviated or redundant content to obtain the target messages that need to be explored.

[0032] Step S40: Determine whether to identify a new target link and continue exploring based on the key information of each sub-message in the target message, the user request, and the key information of the accessed messages.

[0033] In this embodiment of the invention, the semantic relevance of each sub-message in the current target message, the address similarity with the visited page, and the completeness of the content structure are used to comprehensively determine whether it is necessary to continue to expand the access path. Thus, when a high-value message is discovered, the exploration is actively deepened (that is, the high-value message is identified as a new target link for further exploration), and when duplicate or low-quality branches are identified, the exploration is terminated in a timely manner.

[0034] Step S50: Clean up the webpage content corresponding to the target message obtained in each exploration to obtain the effective search content corresponding to the target message.

[0035] In this embodiment of the invention, the web page content corresponding to the target message obtained in each exploration is accessed, and the web page content is structured and parsed. Non-substantive content areas such as advertising banners, navigation menus, and copyright information are automatically identified and removed, while the main text that truly carries knowledge is retained, which significantly improves the data purity of subsequent analysis.

[0036] It should be noted that if it is necessary to further expand the access path, the new target link is determined from the sub-message in the target message, and steps S20-S50 are executed to obtain the valid search content corresponding to each depth level. If it is not necessary to further expand the access path, the deep crawling of that access path is terminated, and deep crawling of other initial access links selected as target links continues.

[0037] Step S60: After stopping the exploration, generate a fusion report based on the preset report template and the access addresses, valid search content, and semantic relevance of all target messages.

[0038] In this embodiment of the invention, after the initial access link selected as the target link has completed all in-depth exploration, the user request, the preset report template, the access address corresponding to all target messages, the effective search content and semantic relevance are input into the large language model, and information fusion and structured output are performed under the preset report template to generate a fusion report.

[0039] Among them, the preset report template refers to the pre-defined report generation prompt template, which is used to specify the structure, tone and integration requirements of the report. For example, "compare and analyze viewpoints from different sources" or "organize the core conclusions according to logical hierarchy".

[0040] Assuming N target messages are discovered, the number of valid pages included in the fusion report is N. The access address of the i-th target message refers to the access address of the page corresponding to the i-th target message, used for reference annotation in the report. The valid retrieval content of the i-th target message refers to the concatenated result of high-confidence text blocks obtained after cleaning and processing the content of the page corresponding to the i-th target message. The semantic relevance of the i-th target message refers to the semantic relevance between the key information of the i-th message and the user's request.

[0041] The access addresses, valid search content, and semantic relevance corresponding to N target messages are input into a large language model along with a pre-defined report template and the user request. Under the constraints of the pre-defined report template, the large language model extracts features from these data, performs cross-validation and viewpoint integration, constructs an internal knowledge graph about the user request, and finally outputs a fusion report. This fusion report can serve as an in-depth research report, possessing a standardized structure suitable for both commercial and academic use, including sections such as abstract, core viewpoints, wheel spacing, and references.

[0042] In summary, the network information fusion method provided in this invention filters multiple high-value target links from initial access links based on user requests, achieving semantically guided path optimization and improving the accuracy and value density of information acquisition. It extracts key information from each message in the target links and filters target messages based on the semantic relevance of each message's key information to the user request, achieving selective focus. It determines whether to continue exploring new target links based on the key information of each sub-message in the target messages, the user request, and the key information of already accessed messages, achieving adaptive adjustment of exploration depth. It delves deeper into high-value content branches and promptly stops losses in low-value branches, balancing research depth and breadth.

[0043] The webpage content corresponding to the target message obtained in each exploration is purified to obtain the effective search content corresponding to the target message, achieving high-precision content separation. After the exploration stops, a fusion report is generated based on the preset report template and the access address, effective search content, and semantic relevance of all target messages. This realizes the automated generation of standardized reports from multi-source information, compressing the research work of several hours to several days to be completed in minutes, significantly reducing research time costs. It can simulate human experts to automatically complete intelligent navigation, dynamic collection, semantic understanding, deep integration, and report generation, achieving end-to-end automation from user natural language requests to in-depth reports.

[0044] Alternatively, one possible implementation method is provided below for how to select the target links to be explored. Figure 1 The sub-steps of step S10 may include: Step S100: Determine the semantic relevance score, authority score, accessibility score, and timeliness score of each initial access link based on the user request and each initial access link.

[0045] In this embodiment of the invention, a four-dimensional evaluation prompt word is constructed for processing by a large language model based on four evaluation dimensions: semantic relevance score, authority score, accessibility score, and timeliness score. The semantic relevance score characterizes the semantic relevance between the page content corresponding to the initial access link and the user's request. It is generated by the large language model based on its natural language understanding capabilities and ranges from [0,1]. A higher semantic relevance score indicates that the content carried by the initial access link is more semantically relevant to the user's request.

[0046] The authority score is used to characterize the authority of each initial access link. It is quantified according to preset rules. For example, the official websites of well-known institutions and key universities have a score of [0.8,1], the official websites of general enterprises or media platforms have a score of [0.3,0.7], and websites with unclear content sources, no clear operating entities, or a history of publishing a large amount of false information have a score of [0,0.2].

[0047] Accessibility score is used to characterize the accessibility of the initial access link. It is quantified based on the actual access feasibility. For example, a webpage that can be viewed without any authentication or permission scores 1, that requires an account registration scores 0.6, that requires a paid subscription scores 0.2, and that is completely inaccessible due to technical limitations, permission blocking, broken links, or network blocking scores 0.

[0048] The timeliness score is used to characterize the timeliness of the initial access link. It is quantified based on the time of the first publication of the webpage or the time of the most recent substantive update. For example, the score is [0.8,1] for a webpage published or updated within the last three months, [0.4,0.7] for an update within three to twelve months, and [0,0.3] for no update for more than a year.

[0049] The large language model performs semantic reasoning and evaluation on each initial access link based on four-dimensional evaluation prompts and user requests, and outputs the semantic relevance score, authority score, accessibility score and timeliness score of each initial access link.

[0050] Step S110: The value score of each initial access link is obtained by weighted summation of the semantic relevance score, authority score, accessibility score and timeliness score of each initial access link.

[0051] Step S120: The initial access link corresponding to the value score that meets the high value standard is determined as the target link.

[0052] In this embodiment of the invention, for each initial access link, a weighted sum is calculated based on its semantic relevance score, authority score, accessibility score, and timeliness score, taking into account their respective weight coefficients, to obtain a value score for each initial access link. Then, the initial access links corresponding to the high-value criteria are selected as target links to be explored.

[0053] The high-value criteria include, but are not limited to, values ​​exceeding a value threshold (e.g., 0.8) and the top n values ​​with the highest values. High-value criteria can be set according to actual application scenarios, and this invention does not limit them.

[0054] As one possible implementation, assume that the top n initial access links with the highest value scores (empirically 5 to 10, user-defined) are selected as high-value target links. The filtering expression for target links can be:

[0055] in, It is the set of the top n initial access links with the highest value scores, i.e., the set of n target links; It is the semantic relevance score between the i-th initial access link and the user request; It is the weighting coefficient of the semantic relevance score; It is the authority score of the i-th initially accessed link; It is the weighting coefficient for the authority score; It is the accessibility score of the i-th initially accessed link; These are the weighting coefficients for the accessibility score; It is the timeliness score of the i-th initial access link; It is the weighting coefficient for the timeliness score; , , , and The default values ​​are 0.4, 0.3, 0.15 and 0.15, respectively, and can be set according to the actual application scenario; It means taking the first n; These are all the initial access links.

[0056] As can be seen, the embodiments of the present invention comprehensively consider four aspects to filter target links that need to be explored for information, including the semantic fit between the link and the user request, the credibility of the source, the actual openability, and the freshness of the content. This allows for the dynamic and accurate identification of truly high-value access links, avoiding the omission of important information sources or the introduction of low-quality, invalid, or outdated links due to one-sided evaluation, and significantly improving the accuracy and practicality of subsequent information fusion.

[0057] Optionally, key information includes a title, summary, and body. The following provides a possible implementation method for filtering target messages relevant to user requests from the target link based on key information. Figure 1 The sub-steps of step S30 may include: Step S300: Evaluate the semantic relevance of the title, summary, and body of each message to the user request to obtain the semantic relevance of each message.

[0058] In this embodiment of the invention, after extracting the title, summary and body of each message in the target link from the webpage corresponding to the target link, the title, summary and body of each message, as well as the user request, are input into the large language model. The large language model is then used to evaluate and output the semantic relevance between each message and the user request based on the title, summary and body.

[0059] Step S310: Messages whose semantic relevance exceeds the relevance threshold are identified as target messages.

[0060] Next, a preset relevance threshold (e.g., 0.9) is obtained, and messages with semantic relevance exceeding the relevance threshold are identified as target messages.

[0061] As can be seen, the embodiments of the present invention make a comprehensive judgment by integrating the theme orientation reflected in the title, the core viewpoint summarized in the abstract, and the complete semantic context carried by the body text, avoiding misjudgment of relevance caused by relying on a single field, and significantly improving the accuracy and semantic matching depth of target message filtering.

[0062] Optionally, key information includes the title, summary, body text, and access URL. Regarding how to determine whether to continue with in-depth data mining, one possible implementation method is provided below. Figure 1 The sub-steps of step S40 may include: Step S400: Evaluate the semantic relevance of the title, summary, and body of each sub-message in the target message to the user request, and obtain the semantic relevance of each sub-message.

[0063] In this embodiment of the invention, the title, summary text and access address of each sub-message in the target message are extracted in real time during the crawling process. The semantic matching degree between the title, summary and text of each sub-message in the target message and the user request is evaluated using a large language model to obtain the semantic relevance of each sub-message in each target message. The semantic relevance value ranges from [0,1].

[0064] Step S410: Evaluate the similarity between the access address of each sub-message and the access address of the accessed message to obtain the URL similarity of each sub-message.

[0065] In this embodiment of the invention, for each sub-message, the similarity between the access address of the sub-message and the access addresses of each accessed message is calculated using a string edit distance or domain name similarity algorithm, and the maximum similarity is determined as the URL similarity of the sub-message. The value range of URL similarity is [0,1], which is used for deduplication and to avoid circular access.

[0066] Step S420: Determine the content quality score of each sub-message based on the number of images contained in the webpage content corresponding to each sub-message.

[0067] In this embodiment of the invention, the content quality score of each sub-message is determined based on the number of images contained in the webpage content corresponding to the sub-message. The content quality score characterizes the information richness and presentation completeness of the webpage to which the sub-message points; its value is positively correlated with the number of images and ranges from [0,1].

[0068] Specifically, the content quality score of the sub-message can be obtained by dividing the difference between the preset total number of reference images and the actual number of images parsed from the webpage corresponding to the sub-message by the preset total number of reference images. The closer the number of images is to or reaches the preset total number of images, the higher the content quality score.

[0069] Alternatively, multiple image quantity ranges can be pre-defined, each corresponding to a fixed content quality score. When the number of images parsed from the webpage corresponding to a sub-message falls within a certain range, the content quality score corresponding to that range is directly assigned. The content quality score can be calculated according to the actual application scenario, and this invention does not limit this calculation.

[0070] Step S430: Determine whether to identify new target links and continue exploring based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message.

[0071] In this embodiment of the invention, the exploration depth corresponding to the current sub-message is obtained. Then, based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message, it is determined whether to continue exploring to the next level. If exploration continues, the next level of exploration links are determined from the links corresponding to multiple sub-messages of the same target message, resulting in new target links. In this way, highly relevant, low-repetition, and high-information-density link paths can be dynamically identified during the crawling process. While ensuring that the content matches the user's intent, redundant access and ineffective exploration are effectively avoided, thereby improving the targeting, search efficiency, and result reliability in the network information fusion process.

[0072] It should be understood that the exploration depth is counted starting from the initial accessed link. For example, the exploration depth of the target link to be filtered is set to 1, and the exploration depth is increased by 1 for each layer deeper. This is used to control the scope of exploration and prevent infinite recursion.

[0073] Optionally, regarding how to determine whether to continue in-depth exploration based on the semantic relevance of sub-messages, URL similarity, and content quality, the following is a possible implementation method. Please refer to... Figure 2 The sub-steps of step S430 may include: Step S431: Determine the exploration score of each sub-message based on its semantic relevance, exploration depth, URL similarity, and content quality score.

[0074] In this embodiment of the invention, for each sub-message, a weighted sum of semantic relevance and content quality score is calculated to obtain a first score, and a weighted sum of exploration depth and URL similarity is calculated to obtain a second score. The difference between the first score and the second score is then determined as the exploration score for each sub-message. The formula for calculating the exploration score is as follows:

[0075] in, This is a message The exploration points; These are sub-messages The title, abstract, and main text; This is a message semantic relevance; This is a message Current depth of exploration; This is a message URL similarity; Is a message with sub-message The access addresses of the most similar visited messages; This is a message Content quality score; These are the weighting coefficients for semantic relevance, exploration depth, URL similarity, and content quality score, with default values ​​of 1.0, 0.1, 0.2, and 0.15 respectively. They can be dynamically adjusted according to the required exploration precision. The highest percentage is allocated to ensure relevance.

[0076] Step S432: If the exploration depth is less than the depth threshold and the semantic relevance of the sub-message with the highest exploration score is not lower than the relevance exploration threshold and the content quality score of the sub-message with the highest exploration score is not lower than the quality exploration threshold, then it is determined to continue exploring and the link corresponding to the sub-message with the highest exploration score is determined as the new target link.

[0077] Step S433: If the exploration depth is not less than the depth threshold, or the semantic relevance of the sub-message with the highest exploration score exceeds the relevance exploration threshold, or the content quality score of the sub-message with the highest exploration score exceeds the quality exploration threshold, then the exploration is stopped.

[0078] Next, it is determined whether the exploration depth of the sub-message is less than the depth threshold. In one case, if the exploration depth of the sub-message is not less than the depth threshold, it means that the exploration depth limit has been reached and further exploration is not allowed.

[0079] In another scenario, if the exploration depth of a sub-message is less than a depth threshold, it indicates that exploration can continue to the next level. The next exploration depth is then determined based on the semantic relevance and content quality score of the sub-message with the highest exploration score.

[0080] If the exploration depth of the next step is one more than the exploration depth of the sub-message, it means the sub-message is helpful to the user's request, and exploration to the next level is allowed. The link corresponding to the sub-message with the highest exploration score is then identified as the new target link, forming an adaptive crawling path. If the exploration depth of the next step is the same as the exploration depth of the sub-message (i.e., the exploration depth of the sub-message), it means the sub-message is not very helpful to the user's request, and exploration to the next level is terminated.

[0081] The formula for calculating the exploration depth for the next step is:

[0082] in, This is the depth of exploration for the next step; It refers to the depth of exploration of sub-messages; It explores the semantic relevance of the sub-message with the highest score; It is the content quality score of the sub-message with the highest exploration score; These are the relevance exploration threshold and the quality exploration threshold, respectively. The default value for both is 0.6, and they can be dynamically adjusted based on historical exploration results.

[0083] As can be seen, this embodiment of the invention calculates an exploration score based on the semantic relevance of sub-messages, exploration depth, URL similarity, and content quality score, and dynamically decides whether to continue crawling accordingly, thus achieving adaptive control of the crawling path and exploration depth. Specifically, the semantic relevance of each sub-message reflects its degree of matching with the user's request at the topic level; the exploration depth characterizes the distance between the current access level and the initial access link; URL similarity identifies potential risks of repeated access; and the content quality score reflects the page's information carrying capacity.

[0084] Next, an exploration score is calculated based on the semantic relevance, exploration depth, URL similarity, and content quality score of the sub-message. Only when the exploration depth of the sub-message with the highest exploration score is less than the depth threshold, the semantic relevance is not lower than the relevance exploration threshold, and the content quality score is not lower than the quality exploration threshold, is its corresponding link identified as a new target link and exploration continues. Otherwise, if any condition is not met, exploration is stopped. This achieves adaptive control of the crawling path and exploration depth, allowing for in-depth exploration on high-value branches and timely termination on low-value branches, balancing research depth and breadth, and significantly improving the accuracy and efficiency of information acquisition.

[0085] Alternatively, the following is one possible approach to purifying the discovered webpage content. Figure 1 The sub-steps of step S50 may include: Step S500: Divide the webpage content corresponding to the target message into multiple text blocks.

[0086] In this embodiment of the invention, for each target message obtained from the exploration, the webpage text is extracted according to the webpage content structure to obtain multiple text blocks. The webpage content structure includes, but is not limited to, title, date, and body text.

[0087] Step S520: Determine the core content identifier, text density, and noise similarity of each text block based on each text block and preset noise keywords.

[0088] In this embodiment of the invention, the core content identifier is used to characterize whether the text block is located in the body of the webpage corresponding to the target message; the text density is used to characterize the information purity of the text block; and the noise similarity is used to characterize the degree of matching between the text block and the preset noise keywords.

[0089] Step S530: Determine the content confidence of each text block based on the core content identifier, text density, noise similarity, and total character length of each text block.

[0090] In this embodiment of the invention, the content confidence of each text block is quantitatively calculated by combining the core content identifier, text density, noise similarity, and total character length. The formula for calculating the content confidence is:

[0091] in, It is a text block Content confidence; It is a text block Core content identifiers; It is a text block Text density; It is a text block Noise similarity; It is a text block Total character length; These are the weight coefficients for core content identification, text density, and noise similarity, respectively, with default values ​​of 0.4, 0.3, and 0.3 (which can be adjusted according to actual application scenarios). This is used to balance the influence of tag content features and text features; This is the length penalty coefficient, with a default value of 0.1. It can be adjusted according to the actual application scenario to avoid excessive bias towards extremely long text blocks and to balance the confidence assessment of content of different lengths.

[0092] Step S540: Text blocks whose content confidence exceeds the confidence threshold are identified as valid text blocks.

[0093] In this embodiment of the invention, when the confidence level of the content exceeds a preset confidence level threshold (the default is 0.6, which can be adjusted according to the actual application scenario), the text block content is considered to be valid information, and the corresponding text block is determined to be a valid text block.

[0094] Step S550: Concatenate all valid text blocks corresponding to the target message to obtain valid search content.

[0095] In this embodiment of the invention, since the web page content is managed through a DOM tree structure, the valid text blocks are concatenated according to the order of the DOM tree to obtain the valid search content.

[0096] As can be seen, this embodiment of the invention divides the webpage content corresponding to the target message into multiple text blocks, and quantifies the content confidence of each text block based on the webpage content structure, text density, and noise characteristics, thereby separating the core content from the noisy template. Subsequently, text blocks with content confidence exceeding the confidence threshold are concatenated according to the DOM tree structure to obtain the effective retrieval content. This achieves high-precision text separation, effectively filtering over 90% of page noise, including advertisements, navigation bars, and redundant information, thus improving the data purity and usability upon which subsequent information fusion and report generation depend, i.e., improving the signal-to-noise ratio of information extraction.

[0097] Optionally, regarding how to determine the core content identifier, text density, and noise similarity of each text block, a possible implementation is provided below. Sub-steps of step S520 may include: Step S521: If the text block is located in the body text, the core content identifier indicates that the text block belongs to the core content.

[0098] Step S522: If the text block is not located in the main text, the core content identifier indicates that the text block does not belong to the core content.

[0099] In this embodiment of the invention, for example, the core content identifier of a text block located in the main text is set to 1, and the core content identifier of a text block located outside the main text is set to 0, which is used to quickly locate the area of ​​potential core content. The specific form of the core content identifier is not limited in this invention and can be set according to the actual application scenario.

[0100] Step S523: The ratio of the number of text characters in the text block to the total number of characters is determined as the text density of the text block.

[0101] In this embodiment of the invention, the number of plain text characters in the text block is counted to obtain the total number of text characters. The total number of characters (including text, symbols, etc.) in the text block is counted to obtain the total number of characters. The ratio of the number of text characters in the text block to the total number of characters is calculated to obtain the text density of the text block. The text density ranges from [0,1], and a higher text density indicates less non-text interference.

[0102] Step S524: Determine the noise similarity of the text block based on the text block and the preset noise keywords.

[0103] In this embodiment of the invention, the similarity between preset noise keywords in the noise database and text blocks is calculated, and the maximum similarity is determined as the noise similarity of the text block. Noise keywords include, but are not limited to, advertisements, menus, and copyrights. The noise similarity value ranges from [0,1]. For example, if no noise keyword is matched, the noise similarity is 0.

[0104] Based on the same inventive concept, the basic principle and technical effects of the network information fusion device provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0105] Please refer to Figure 3 , Figure 3 This is a block diagram of a network information fusion device 400 provided in an embodiment of the present invention. The network information fusion device 400 includes a semantic navigation module 410, a deep acquisition module 420, and an intelligent analysis module 430.

[0106] The semantic navigation module 410 is used to filter multiple high-value target links from the initial access links based on user requests; extract key information from each message in the target links; filter target messages from each message based on the semantic relevance of the key information of each message to the user request; and determine whether to identify new target links to continue exploring based on the key information of each sub-message in the target message, the user request, and the key information of the accessed messages.

[0107] The deep acquisition module 420 is used to clean up the web page content corresponding to the target message obtained in each exploration to obtain the effective search content corresponding to the target message.

[0108] The intelligent analysis module 430 is used to generate a fusion report based on a preset report template and the access addresses, valid search content, and semantic relevance of all target messages after the exploration stops.

[0109] In summary, the network information fusion device provided in this embodiment of the invention filters multiple high-value target links from the initial access links based on user requests, achieving semantically guided path optimization and improving the accuracy and value density of information acquisition. It extracts key information from each message in the target links and filters target messages based on the semantic relevance of each message's key information to the user request, achieving selective focus. It determines whether to continue exploring new target links based on the key information of each sub-message in the target message, the user request, and the key information of already accessed messages, achieving adaptive adjustment of exploration depth. It delves deeper into high-value content branches and promptly stops losses in low-value branches, balancing research depth and breadth.

[0110] The webpage content corresponding to the target message obtained in each exploration is purified to obtain the effective search content corresponding to the target message, achieving high-precision content separation. After the exploration stops, a fusion report is generated based on the preset report template and the access address, effective search content, and semantic relevance of all target messages. This realizes the automated generation of standardized reports from multi-source information, compressing the research work of several hours to several days to be completed in minutes, significantly reducing research time costs. It can simulate human experts to automatically complete intelligent navigation, dynamic collection, semantic understanding, deep integration, and report generation, achieving end-to-end automation from user natural language requests to in-depth reports.

[0111] Optionally, the network information fusion device 400 also includes a scheduling control module 440, which is used to implement parallel crawling using multi-threaded or asynchronous programming, integrate resource monitoring functions (such as time and bandwidth), dynamically adjust the crawling frequency and concurrency, and record the thought process containing the entire access path and scoring data, so as to facilitate technical traceability, fault debugging and result verification.

[0112] Optionally, the semantic navigation module 410 is used to determine the semantic relevance score, authority score, accessibility score, and timeliness score of each initial access link based on the user request and each initial access link; to perform a weighted summation of the semantic relevance score, authority score, accessibility score, and timeliness score of each initial access link to obtain the value score of each initial access link; and to determine the initial access link corresponding to the value score that meets the high value standard as the target link.

[0113] Optionally, key information includes a title, summary, and body. The semantic navigation module 410 evaluates the semantic relevance of each message's title, summary, and body to the user request, obtaining the semantic relevance score for each message; messages with semantic relevance scores exceeding a relevance threshold are identified as target messages.

[0114] Optionally, key information includes title, summary, body, and access address. The semantic navigation module 410 is used to evaluate the semantic relevance of the title, summary, and body of each sub-message in the target message to the user request, obtaining the semantic relevance of each sub-message; evaluate the similarity between the access address of each sub-message and the access addresses of already visited messages, obtaining the URL similarity of each sub-message; determine the content quality score of each sub-message based on the number of images contained in the webpage content corresponding to each sub-message; and determine whether to identify new target links to continue exploration based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message.

[0115] Optionally, the semantic navigation module 410 is used to determine the exploration score of each sub-message based on its semantic relevance, exploration depth, URL similarity, and content quality score. If the exploration depth is less than the depth threshold and the semantic relevance of the sub-message with the highest exploration score is not lower than the relevance exploration threshold and the content quality score of the sub-message with the highest exploration score is not lower than the quality exploration threshold, then it is determined to continue exploring and the link corresponding to the sub-message with the highest exploration score is determined as the new target link. If the exploration depth is not less than the depth threshold or the semantic relevance of the sub-message with the highest exploration score exceeds the relevance exploration threshold or the content quality score of the sub-message with the highest exploration score exceeds the quality exploration threshold, then it is determined to stop exploring.

[0116] Optionally, the deep acquisition module 420 is used to divide the webpage content corresponding to the target message into multiple text blocks; determine the core content identifier, text density, and noise similarity of each text block based on each text block and preset noise keywords; determine the content confidence of each text block based on the core content identifier, text density, noise similarity, and total character length; determine the text blocks whose content confidence exceeds the confidence threshold as valid text blocks; and concatenate all valid text blocks corresponding to the target message to obtain valid search content.

[0117] Optionally, the depth acquisition module 420 is used to identify the core content if the text block is located in the main text, and to identify the core content if the text block is not located in the main text, and to identify the core content if the text block is not located in the main text; to determine the text density of the text block by the ratio of the number of text characters in the text block to the total number of characters; and to determine the noise similarity of the text block based on the text block and preset noise keywords.

[0118] Please refer to Figure 4 This is a block diagram illustrating an electronic device 500 provided in an embodiment of the present invention. The electronic device 500 includes, but is not limited to, a personal computer (PC), a personal digital assistant (PDA), a laptop computer, a tablet computer, and a server. The electronic device 500 includes a memory 510, a processor 520, and a communication module 530. The memory 510, processor 520, and communication module 530 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0119] The memory 510 is used to store programs or data. The memory 510 may be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable read-only memory, electrically erasable read-only memory, etc.

[0120] The processor 520 is used to read / write data or programs stored in the memory 510 and perform corresponding functions. For example, when a computer program stored in the memory 510 is executed by the processor 520, the network information fusion method disclosed in the above embodiments can be implemented.

[0121] The communication module 530 is used to establish a communication connection between the electronic device 500 and other communication terminals via a network, and to send and receive data via the network.

[0122] It should be understood that, Figure 4 The structure shown is only a schematic diagram of the electronic device 500. The electronic device 500 may also include components that are larger than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0123] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor 520, implements the network information fusion method disclosed in the above embodiments.

[0124] This invention also provides a program product that, when executed by processor 520, implements the network information fusion method disclosed in the above embodiments.

[0125] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0126] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0127] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0128] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A network information fusion method, characterized in that, The method includes: Filter multiple high-value target links from the initial access links based on user requests; Extract key information from each message in the target link; Target messages are filtered from each message based on the semantic relevance of the key information of each message to the user request; Based on the key information of each sub-message in the target message, the key information of the user request and the accessed message, determine whether to identify a new target link and continue exploring; The webpage content corresponding to the target message obtained in each exploration is purified to obtain the effective search content corresponding to the target message; After the exploration stops, a fusion report is generated based on the preset report template and the access addresses, valid search content, and semantic relevance of all the target messages.

2. The network information fusion method according to claim 1, characterized in that, The process of filtering multiple high-value target links from the initial access links based on user requests includes: Based on the user request and each of the initial access links, determine the semantic relevance score, authority score, accessibility score, and timeliness score of each initial access link; The value score of each initial access link is obtained by weighted summation of its semantic relevance score, authority score, accessibility score, and timeliness score. The initial access links corresponding to the value scores that meet the high-value criteria are identified as target links.

3. The network information fusion method according to claim 1, characterized in that, The key information includes a title, summary, and body; the process of filtering target messages from each message based on the semantic relevance of the key information of each message to the user request includes: The semantic relevance of the title, summary, and body of each message to the user request is evaluated to obtain the semantic relevance of each message; Messages whose semantic relevance exceeds the relevance threshold are identified as the target messages.

4. The network information fusion method according to claim 1, characterized in that, The key information includes the title, summary, body, and access address; the step of determining whether to identify a new target link and continue exploring based on the key information of each sub-message in the target message, the user request, and the key information of the already accessed messages includes: The semantic relevance of the title, summary, and body of each sub-message in the target message to the user request is evaluated to obtain the semantic relevance of each sub-message; The similarity between the access address of each sub-message and the access address of the accessed messages is evaluated to obtain the URL similarity of each sub-message; The content quality score of each sub-message is determined based on the number of images contained in the webpage content corresponding to each sub-message; The decision to determine whether to identify new target links and continue exploration is based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message.

5. The network information fusion method according to claim 4, characterized in that, The step of determining whether to identify a new target link to continue exploring based on the semantic relevance, exploration depth, URL similarity, and content quality score of each sub-message includes: The exploration score of each sub-message is determined based on its semantic relevance, exploration depth, URL similarity, and content quality score. If the exploration depth is less than the depth threshold and the semantic relevance of the sub-message with the highest exploration score is not lower than the relevance exploration threshold and the content quality score of the sub-message with the highest exploration score is not lower than the quality exploration threshold, then it is determined to continue exploration and the link corresponding to the sub-message with the highest exploration score is determined as a new target link. If the exploration depth is not less than the depth threshold, or the semantic relevance of the sub-message with the highest exploration score exceeds the relevance exploration threshold, or the content quality score of the sub-message with the highest exploration score exceeds the quality exploration threshold, then the exploration is stopped.

6. The network information fusion method according to claim 1, characterized in that, The step of purifying the webpage content corresponding to the target message obtained in each exploration to obtain the effective search content corresponding to the target message includes: The webpage content corresponding to the target message is divided into multiple text blocks; The core content identifier, text density, and noise similarity of each text block are determined based on each text block and preset noise keywords; The content confidence of each text block is determined based on its core content identifier, text density, noise similarity, and total character length. Text blocks whose content confidence exceeds the confidence threshold are identified as valid text blocks; The valid text blocks corresponding to the target message are concatenated to obtain the valid search content.

7. The network information fusion method according to claim 6, characterized in that, The step of determining the core content identifier, text density, and noise similarity of each text block based on each text block and preset noise keywords includes: If the text block is located in the main text, then the core content identifier indicates that the text block belongs to the core content; If the text block is not located in the main text, then the core content identifier indicates that the text block does not belong to the core content; The ratio of the number of text characters in the text block to the total number of characters is determined as the text density of the text block; The noise similarity of the text block is determined based on the text block and preset noise keywords.

8. A network information fusion device, characterized in that, The device includes: The semantic navigation module is used to filter multiple high-value target links from the initial access links based on user requests; extract key information of each message in the target links; filter target messages from each message based on the semantic relevance of the key information of each message to the user request; and determine whether to identify new target links to continue exploring based on the key information of each sub-message in the target messages, the user request, and the key information of the accessed messages. The deep acquisition module is used to clean up the web page content corresponding to the target message obtained in each exploration to obtain the effective search content corresponding to the target message; The intelligent analysis module is used to generate a fusion report based on a preset report template and the access addresses, valid search content, and semantic relevance of all the target messages after the exploration stops.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor to implement the network information fusion method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the network information fusion method as described in any one of claims 1-7.

Images