Data acquisition method and device, computer equipment and computer program product

Abstract

The invention provides a data acquisition method and device, computer equipment and a computer program product, and relates to the technical field of data processing. The data acquisition method disclosed by the invention comprises the following steps: acquiring a data query request; determining a data acquisition tool and data acquisition parameters matched with the data query request based on the data query request, wherein the data acquisition parameters comprise an access address and an authentication type of a data resource corresponding to the data query request; obtaining an access credential for accessing the data resource based on the authentication type; using the data acquisition tool to acquire initial content data from the data resource based on the access credential and the access address of the data resource; based on semantic analysis of the initial content data, content recombination is conducted on the initial content data, target data are obtained, and the target data are used for generating data query feedback information corresponding to the data query request.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a data acquisition method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] In recent years, Large Language Models (LLMs) have developed rapidly, and technologies for cross-platform data acquisition based on LLMs have also attracted much attention. However, there are currently many problems in the process of cross-platform data acquisition based on LLMs, such as restricted access, low security of the data acquisition process, and low quality of the acquired data, which limit the capabilities of LLMs in cross-platform data acquisition and result in low data acquisition efficiency.

[0003] In view of the above, some embodiments of this specification provide a data acquisition method, apparatus, computer device, computer-readable storage medium, and computer program product, which aim to improve data access capabilities, enhance the security, data quality, and data acquisition efficiency of cross-platform data acquisition. Summary of the Invention

[0004] This specification provides one or more embodiments of a data acquisition method, the method comprising: acquiring a data query request; determining a data acquisition tool and data acquisition parameters matching the data query request based on the data query request, the data acquisition parameters including the access address and authentication type of the data resource corresponding to the data query request; acquiring access credentials for accessing the data resource based on the authentication type; using the data acquisition tool to acquire initial content data from the data resource based on the access credentials and the access address of the data resource; and performing content reorganization on the initial content data based on semantic analysis to obtain target data, the target data being used to generate data query feedback information corresponding to the data query request.

[0005] According to one or more embodiments of this specification, a method for obtaining access credentials for accessing data resources based on an authentication type includes: determining whether the authentication type is in a preset authentication type list; when the authentication type is in the preset authentication type list, determining whether there is a valid access credential; and when there is no valid access credential, obtaining access credentials for accessing data resources based on the authentication type.

[0006] According to one or more embodiments of this specification, a method for obtaining access credentials for accessing data resources based on an authentication type includes: when the authentication type is a first preset authentication type, providing an authentication page corresponding to the access address of the data resource, and obtaining access credentials based on the input information of the authentication page; when the authentication type is a second preset authentication type, providing an authentication page corresponding to the access address of the data resource, and obtaining access credentials based on the authorization operation of the authentication page.

[0007] According to one or more embodiments of this specification, the method for determining a data acquisition tool and data acquisition parameters matching a data query request based on the data query request includes: determining a data acquisition tool and data acquisition parameters matching a data query request based on the data query request and tool configuration information, wherein the tool configuration information includes a data acquisition tool identifier and tool parameters.

[0008] According to one or more embodiments of this specification, a method for reorganizing initial content data based on semantic analysis of initial content data includes: performing structuring processing on the initial content data to obtain structured initial content data; and reorganizing the structured initial content data based on semantic analysis of the structured initial content data.

[0009] According to one or more embodiments of this specification, a method for reorganizing structured initial content data based on semantic analysis includes: performing semantic analysis on the structured initial content data to determine semantic analysis information, wherein the semantic analysis information includes at least one of the following: the semantic type of each content node in the structured initial content data, the importance of each content node in the structured initial content data, the relationship type between content nodes in the structured initial content data, and the relationship strength between content nodes in the structured initial content data; wherein the semantic type is used to represent the content classification of the content nodes; and reordering each content node based on the semantic analysis information.

[0010] According to one or more embodiments of this specification, the semantic types include concept types, example types, suggestion types, and performance optimization types according to their priority from high to low; the content nodes are reordered based on semantic analysis information, including: reordering the content nodes based on the priority order of their semantic types.

[0011] According to one or more embodiments of this specification, the method for reordering each content node based on semantic analysis information includes: determining the importance of each content node based on its semantic type, its position in the structured initial content data, and its semantic weight information; and reordering the content nodes based on their importance.

[0012] According to one or more embodiments of this specification, the relationship types between content nodes include hierarchical dependencies and semantic dependencies. Hierarchical dependencies are determined based on the structural relationships of each content node in the structured initial content data, and semantic dependencies are determined based on the semantic relevance of each content node in the structured initial content data. The method reorders each content node based on semantic analysis information, including: determining a topic block based on hierarchical dependencies, where each topic block includes one or more content nodes; and reordering each content node within the topic block based on semantic dependencies and / or the strength of relationships between content nodes.

[0013] According to one or more embodiments of this specification, the method for reorganizing structured initial content data based on semantic analysis of the structured initial content data further includes: performing at least one of the following processes based on semantic analysis information: organizing two or more content nodes into semantic units and reordering the semantic units as a whole; adding transition descriptions between content nodes; and adding supplementary descriptions to content nodes.

[0014] The method provided according to one or more embodiments of this specification further includes: converting the format of the initial content data after content reorganization; using a large language model to perform quality assessment on the initial content data after format conversion based on a preset evaluation dimension, and obtaining a quality assessment result; and using a large language model to optimize the initial content data after format conversion based on the quality assessment result.

[0015] According to one or more embodiments of this specification, a method is used to perform quality assessment on initial content data after format conversion using a large language model based on preset evaluation dimensions, and to obtain quality assessment results. This includes: performing quality assessment on the initial content data after format conversion using a large language model based on preset evaluation dimensions, and determining the score corresponding to each preset evaluation dimension; determining a comprehensive quality score for the initial content data after format conversion based on the scores corresponding to each preset evaluation dimension and the preset weights corresponding to each preset evaluation dimension; wherein the preset evaluation dimensions for quality assessment include at least one of the following dimensions: structural integrity, semantic accuracy, format standardization, and comprehensive intelligent assessment; the comprehensive intelligent assessment is used to perform an overall assessment of the initial content data after format conversion.

[0016] One or more embodiments of this specification also provide a data acquisition device, the device comprising: a request acquisition module for acquiring a data query request; a tool and information determination module for determining a data acquisition tool and data acquisition parameters matching the data query request based on the data query request, the data acquisition parameters including the access address and authentication type of the data resource corresponding to the data query request; an access credential acquisition module for acquiring access credentials for accessing the data resource based on the authentication type; a data acquisition module for acquiring initial content data from the data resource using the data acquisition tool based on the access credentials and the access address of the data resource; and a content reorganization module for reorganizing the initial content data based on semantic analysis to obtain target data, the target data being used to generate data query feedback information corresponding to the data query request.

[0017] One or more embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it is able to implement the data acquisition method described in some embodiments of this specification.

[0018] This specification also provides a computer-readable storage medium in one or more embodiments, which stores computer instructions that, when executed by a processor, enable the data acquisition method described in some embodiments of this specification.

[0019] One or more embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement the data acquisition method described in some embodiments of this specification.

[0020] The beneficial effects that the embodiments of this specification may bring include, but are not limited to: obtaining a data query request, determining a data acquisition tool and data acquisition parameters matching the data query request based on the data query request, the data acquisition parameters including the access address and authentication type of the data resource corresponding to the data query request, further obtaining access credentials for accessing the data resource based on the authentication type, and using the data acquisition tool to obtain initial content data from the data resource based on the access credentials and the access address of the data resource, thereby improving data access capabilities, enhancing the security of cross-platform data acquisition, and thus improving data acquisition efficiency. By performing content reorganization on the initial content data based on semantic analysis, target data is obtained, which is used to generate data query feedback information corresponding to the data query request, thereby improving the quality of acquired data. By determining the authentication type and obtaining access credentials for accessing data resources based on the authentication type, unified authentication management of multiple authentication methods is achieved, enhancing the security of cross-platform data acquisition, and enabling authentication mechanisms to be implemented even when using AI agents to acquire data, allowing information requiring authentication to be acquired based on a large language model. By performing semantic analysis on the structured initial content data, semantic analysis information is determined. Based on this information, the content nodes are reordered, thereby improving the content logic, tightly integrating related content, and reducing the difficulty of understanding. This allows the large language model to generate more accurate data query feedback information based on the target data. By organizing two or more content nodes into semantic units based on semantic analysis information and reordering these units as a whole, or by adding transitional or supplementary explanations to content nodes, the readability of the target data is enhanced, facilitating the large language model's understanding of the data content. Furthermore, by performing quality assessment and optimization on the initial content data, the format of the target data is further standardized, improving its quality. This allows the large language model to more accurately understand the content of the target data and generate more accurate data query feedback information. It should be noted that different embodiments may produce different beneficial effects. In different embodiments, the beneficial effects may be any one or a combination of the above, or any other possible beneficial effects. Attached Figure Description

[0021] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. The same numbers in the drawings denote the same structures or steps.

[0022] Figure 1 This is an exemplary flowchart of a data acquisition method according to some embodiments of this specification.

[0023] Figure 2 This is a schematic diagram of the data acquisition process according to some embodiments of this specification.

[0024] Figure 3 This is an exemplary flowchart of a content reorganization method according to some embodiments of this specification.

[0025] Figure 4 This is an exemplary flowchart of another content reorganization method according to some embodiments of this specification.

[0026] Figure 5 This is an exemplary flowchart illustrating a method for obtaining target data according to some embodiments of this specification.

[0027] Figure 6 This is a schematic diagram illustrating a comprehensive quality score determination method according to some embodiments of this specification.

[0028] Figure 7 This is an exemplary block diagram of a data acquisition apparatus according to some embodiments of this specification. Detailed Implementation

[0029] To more clearly illustrate the technical solutions of the embodiments in this specification, the embodiments will be described in detail below with reference to the accompanying drawings. Obviously, the content described below are some examples or embodiments of this specification. For those skilled in the art, without creative effort, the technical solutions or means disclosed in this specification can be applied to other scenarios based on this technical content.

[0030] It should be understood that the terms "system," "device," "unit," and / or "module" used in this specification are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0031] Unless otherwise specified, the technical terms used to describe components, elements, etc. in this specification are not singular but may include plural. Generally speaking, terms such as "comprising" or "including" only indicate that explicitly identified steps, elements, or components are included, and these steps, elements, and components do not constitute an exclusive list, as the described method or apparatus may also include other steps or components.

[0032] This specification uses flowcharts to illustrate the operational steps performed by the apparatus or system of related embodiments. However, unless otherwise specified, the order in which these steps are described should not be construed as a limitation on the order of execution. Those skilled in the art can adjust the order of these steps based on the knowledge and information conveyed by the embodiments in this specification. Such adjustments include, but are not limited to, reversing the order of steps, merging multiple steps, and splitting a step.

[0033] Large Language Models (LLMs) are deep learning-based natural language processing systems used to process and generate natural language text. By learning from large amounts of text data, they can understand the syntax, semantics, and contextual relationships of a language, and can be used for tasks such as answering questions, writing assistance, translation, and dialogue.

[0034] In recent years, Large Language Models (LLMs) have developed rapidly, and technologies for cross-platform data acquisition based on LLMs have attracted much attention. In some embodiments, data can be acquired across platforms by hard-coding external APIs (Application Programming Interfaces), but this requires writing specific calling code for each data source, resulting in low development efficiency and high maintenance costs. In other embodiments, dedicated tools developed for specific needs, such as web crawlers and document converters, can be used to acquire data across platforms. However, this approach has limited functionality, lacks unified management of dedicated tools, cannot flexibly support the acquisition of multiple data sources, and has poor scalability. In still other embodiments, cross-platform data acquisition can be achieved based on the standard Model Context Protocol (MCP). However, this method only provides simple tool calling functionality and does not perform intelligent transformation processing on the acquired data, resulting in low data quality and a lack of authentication mechanisms that restrict data access, leading to low data acquisition efficiency.

[0035] To address this, some embodiments of this specification propose a data acquisition method. This method includes: acquiring a data query request; determining a data acquisition tool and data acquisition parameters matching the data query request based on the data query request, wherein the data acquisition parameters include the access address and authentication type of the data resource corresponding to the data query request; further, acquiring access credentials for accessing the data resource based on the authentication type; and using the data acquisition tool to acquire initial content data from the data resource based on the access credentials and the access address of the data resource. This improves data access capabilities, enhances the security of cross-platform data acquisition, and thus improves data acquisition efficiency. Additionally, semantic analysis of the initial content data can be used to reorganize the initial content data to obtain target data, which is used to generate data query feedback information corresponding to the data query request, thereby improving the quality of the acquired data.

[0036] Figure 1 This is an exemplary flowchart of a data acquisition method according to some embodiments of this specification. Figure 1The illustrated process 100 can be executed by a computer device, for example, by a data acquisition device 700 deployed on the computer device. In some embodiments, the computer device can be a server. The server can include a local server or a cloud server. Depending on different service requirements, a local server corresponding to a specific region can be deployed in one or more regions. In some embodiments, the server can be a single computer or a computing cluster composed of multiple computers, thereby providing more powerful computing power and responding to user service requests more efficiently. In other embodiments, the computer device can be a terminal device, including but not limited to desktop computers, smartphones, laptops, tablets, etc. In some embodiments, part of process 100 can be executed by a computer device acting as a user end, while another part can be executed by a computer device acting as a server end. Figure 1 As shown, process 100 may include the following steps.

[0037] Step 110: Obtain a data query request. In some embodiments, step 110 can be implemented by the request acquisition module 710.

[0038] In some embodiments, a computer device may include an AI agent. The AI ​​agent can be a computing system capable of autonomously perceiving its environment and achieving its goals through reasoning, action, and learning. For example, the AI ​​agent can receive input information from the environment (e.g., an interactive interface, an API interface), use a large language model to reason and analyze the received information, interact with external computing resources by invoking tools, and return results.

[0039] In some embodiments, a data query request can be natural language input by the user through an AI agent interactive interface (e.g., a text box in the interactive interface). For example, the user can input "Please give me a demo example of the **front-end component library button component" in a text box of the AI ​​agent interactive interface, and the AI ​​agent can obtain the data query request.

[0040] Step 120: Determine the data acquisition tool and data acquisition parameters that match the data query request based on the data query request. In some embodiments, step 120 can be implemented by the tool and information determination module 720.

[0041] In some embodiments, a data acquisition tool may be a functional module that encapsulates specific data access logic for extracting content data from data resources. For example, a data acquisition tool may include a web page content acquisition tool "get_**_text", which can extract the main content data from a page containing **. A data acquisition tool may also include a local file reading tool, which can read file content from the local file system of a server or user device.

[0042] In some embodiments, data acquisition parameters may include the access address and authentication type of the data resource corresponding to the data query request. The access address of the data resource may be a Uniform Resource Locator (URL) or a local file system path. For example, the access address of the data resource corresponding to the data query request "Please give me a demo example of the ** front-end component library button component" may be https: / / design.**.com / docs / button. The data resource may be the specific data content corresponding to this access address, such as the specific webpage content corresponding to https: / / design.**.com / docs / button. The authentication type may include a first preset authentication type and a second preset authentication type. The first preset authentication type may be an authentication type where the authentication method is based on user input information, such as cookie authentication or token authentication. The second preset authentication type may be an authentication type where the user completes the authentication through an authorized third-party platform, such as OAuth 2.0 authentication.

[0043] In some embodiments, the computer device can determine the data acquisition tool and data acquisition parameters that match the data query request based on the data query request and tool configuration information. For example, the computer device can perform intent analysis based on the data query request to determine the data acquisition tool identifier that matches the data query request, invoke the data acquisition tool based on the data acquisition tool identifier, and extract the data acquisition parameters based on the tool configuration information. Figure 2 This is a schematic diagram illustrating the data acquisition process according to some embodiments of this specification. The following is in conjunction with... Figure 2 The specific description outlines how to determine the data acquisition tools and parameters that match the data query request.

[0044] In some embodiments, such as Figure 2As shown, after receiving a data query request, the AI ​​agent can forward it to a large language model in the computer device. The large language model can perform intent recognition on the data query request and determine the data acquisition tool identifier corresponding to the data query request based on the intent recognition result and a pre-configured list of tool descriptions (e.g., a list of tool function descriptions). The tool description list can pre-configure various data acquisition tool identifiers and their corresponding tool descriptions. The large language model can perform semantic matching based on the intent recognition result and the tool description information to determine the data acquisition tool identifier corresponding to the matched tool description information. The large language model can also determine key parameters based on the data query request. In some embodiments, key parameters may include the access address corresponding to the data query request. For example, the large language model can perform intent recognition on the data query request "Please give me a demo example of the ** front-end component library button component" and query the knowledge base to find that the access address corresponding to the data query request is https: / / design.**.com / docs / button. In other embodiments, key parameters may include key data extracted from the data query request for generating the access address corresponding to the data query request. For example, a large language model can perform intent recognition on a data query request "Please search for information about monkeys", parse the data query request and extract the key data "monkey". The key data "monkey" can then be used to generate an access address corresponding to the data query request, such as http: / / www.**.com?search=monkey.

[0045] In some embodiments, the large language model can generate tool call instructions based on the data acquisition tool identifier and key parameters. For example, as shown in the tool call instruction "tool_calls", the tool call instruction may include the data acquisition tool identifier "get_**_text" and the key parameter "url": https: / / design.****.com / docs / button. The tool call instruction may also include other information, such as the tool call instruction id "call_abc123", the call type "function", and the description information "I am a method to retrieve **XXX document", etc. The tool call instruction "tool_calls" is as follows: { "tool_calls":[ { "id": "call_abc123", "type": "function", "function": { "name": "get_**_text", "arguments": "{\"url\": \"https: / / design.****.com / docs / button\"}" }, "desc": "This is the method for retrieving the **XXX document" } ] }

[0046] In some embodiments, such as Figure 2 As shown, the large language model can send the generated tool invocation command to the AI ​​agent. The AI ​​agent extracts the data acquisition tool identifier and key parameters from the tool invocation command, constructs an HTTP request based on the data acquisition tool identifier and key parameters, and sends the HTTP request to the HTTP interface layer in the computer device. For example, the HTTP request may include the data acquisition tool identifier "get_**_text" and the key parameter "url": "https: / / design.****.com / docs / button". The HTTP request may also include other information, such as the request method "POST", the request target " / api / v1 / mcp / call", and request headers.

[0047] In some embodiments, the HTTP interface layer can parse the HTTP request, extract the data acquisition tool identifier and key parameters, look up the tool configuration information based on the data acquisition tool identifier, and determine the tool parameters corresponding to the data acquisition tool identifier. The tool configuration information may include the data acquisition tool identifier and tool parameters, and the tool parameters may include an address template, request method, authentication type, etc. The address template can be a complete access address, such as https: / / design.**.com / docs / button, which can be used as the access address for the data resource. The address template can also be an address template containing placeholders, such as https: / / design.**.com / docs / {placeholder}, where a complete access address can be constructed based on key parameters (e.g., key data extracted from the data query request) and the address template containing placeholders, and this constructed complete access address can be used as the access address for the data resource. The address template can also be an empty template, where key parameters (e.g., the complete access address corresponding to the data query request) can be filled in and used as the access address for the data resource. The request method can be used to instruct the data acquisition tool how to interact with the data resource. For example, the request method can be the GET method, the POST method, etc. The GET method is used to request and retrieve a specified resource. The POST method is used to submit data to a specified resource, for example, to submit a tool call request to a specified resource. The authentication type can include a first preset authentication type (such as cookie authentication or token authentication) and a second preset authentication type (such as OAuth 2.0 authentication).

[0048] Step 130: Obtain access credentials for accessing data resources based on the authentication type. In some embodiments, step 130 can be implemented by the access credential acquisition module 730.

[0049] In some embodiments, the computer device can determine whether the authentication type is in a preset authentication type list. If the authentication type is in the preset authentication type list, it can determine whether a valid access credential exists. If no valid access credential exists, it can obtain an access credential for accessing the data resource based on the authentication type. The following continues in conjunction with... Figure 2 This explains how to obtain access credentials for accessing data resources based on authentication type.

[0050] In some embodiments, such as Figure 2As shown, the HTTP interface layer can send the parsed access address, authentication type, and data acquisition tool identifier of the data resource to the dynamic service scheduling engine in the computer device. The dynamic service scheduling engine can determine whether the authentication type is in the preset authentication type list. When the authentication type is in the preset authentication type list, the dynamic service scheduling engine constructs an authentication request based on the access address, authentication type, and data acquisition tool identifier of the data resource and sends it to the authentication module. The authentication module can check whether there is a valid access credential in the cache or local storage. When there is no valid access credential (e.g., the access credential has expired or there is no access credential in the cache), it can use different authentication strategies based on the authentication type to obtain access credentials for accessing the data resource.

[0051] In some embodiments, obtaining access credentials for accessing data resources based on different authentication strategies according to the authentication type can be achieved through the following scheme: When the authentication type is a first preset authentication type, an authentication page corresponding to the access address of the data resource can be provided, and access credentials can be obtained based on the input information on the authentication page; when the authentication type is a second preset authentication type, an authentication page corresponding to the access address of the data resource can be provided, and access credentials can be obtained based on the authorization operation on the authentication page. The first preset authentication type can be, for example, cookie authentication or token authentication, and the second preset authentication type can be, for example, OAuth 2.0 authentication. For example, when the authentication type is cookie authentication or token authentication, a browser tool can be invoked to access the access address of the data resource and display the returned authentication page. The user can enter cookie or token information on the authentication page, or generate cookie or token information based on the information entered by the user on the authentication page (such as account and password). The AI ​​agent obtains the cookie or token information as access credentials. When the authentication type is OAuth 2.0, the AI ​​can access the data resource's address using browser tools and display the returned authorization page. Users can then authorize the access, and by monitoring the authorization information in the browser tools, access credentials can be obtained. By determining the authentication type and obtaining access credentials for accessing data resources based on that type, unified authentication management across multiple authentication methods is achieved, improving the security of cross-platform data acquisition. Currently, solutions for accessing data based on large language models typically lack authentication mechanisms, causing the model to be unable to access data requiring authentication when attempting to acquire data across platforms, resulting in significant limitations. Introducing an authentication mechanism overcomes these limitations, and unified authentication management also improves the security of acquired data.

[0052] Step 140: Using a data acquisition tool, initial content data is obtained from the data resource based on access credentials and the access address of the data resource. In some embodiments, step 140 can be implemented by the data acquisition module 740.

[0053] In some embodiments, the acquired access credentials can be sent to the dynamic service scheduling engine. The dynamic service scheduling engine then sends the access credentials, the access address of the data resource, and the data acquisition tool identifier to the MCPServer (Model Context Protocol Service) in the computer device. This allows the MCPServer to use the data acquisition tool to retrieve initial content data from the data resource. The MCPServer can be a server-side program implementing the Model Context Protocol, used to encapsulate and manage the specific access logic to external data sources. The data acquisition tool can be a function within the MCPServer used to complete a specific data acquisition task. The data acquisition tool identifier can be a string used within the MCPServer to identify the data acquisition tool. For example, the MCPServer can access the access address of the data resource using a built-in HTTP client based on the access credentials, determine the data acquisition tool based on the data acquisition tool identifier, and use the data acquisition tool to retrieve initial content data from the data resource corresponding to the access address. Further, the MCPServer can return this initial content data to the dynamic service scheduling engine, which forwards it to the conversion module for subsequent processing. The initial content data can be unprocessed raw content extracted from the data resource; for example, the initial content data can be HTML formatted content data. The initial content data may contain many invalid elements and lack semantic characteristics in its structure; for example, it may not contain semantic information such as core concepts or code examples. The logic and layout of the data content in the initial content data may be chaotic, resulting in low data quality. It is difficult for a large language model to parse directly based on the initial content data, and it is prone to comprehension errors. Therefore, further processing of the initial content data is required to improve its quality.

[0054] Step 150: Based on semantic analysis of the initial content data, the initial content data is reorganized to obtain the target data. In some embodiments, step 150 can be implemented by the content reorganization module 750.

[0055] In some embodiments, such as Figure 2As shown, the conversion module in the computer device can perform structuring processing on the initial content data to obtain structured initial content data. Based on semantic analysis of the structured initial content data, it can reorganize the structured initial content data to obtain the target data. The structuring processing can be the process of converting the initial content data into structured data that reflects its internal logical hierarchy and structural relationships. Further explanation of structuring processing can be found in step 520 below.

[0056] In some embodiments, the computer device can perform semantic analysis on the structured initial content data to determine semantic analysis information, and reorder the content nodes based on the semantic analysis information, thereby improving the content logic, tightly integrating related content, reducing the difficulty of understanding, and enabling the large language model to subsequently generate more accurate data query feedback information based on the target data. This part will be described in detail in process 300 later.

[0057] In some embodiments, the computer device may also organize two or more content nodes into semantic units based on semantic analysis information, reorder the semantic units as a whole, or add transitional or supplementary explanations to the content nodes, thereby enhancing the readability of the target data and facilitating the understanding of the data content by the large language model. This part will be described in detail in process 400 later.

[0058] In some embodiments, target data can be used to generate data query feedback information corresponding to the data query request. For example, if the data query request is "Please give me a demo example of the ** front-end component library button component", the large language model can perform in-depth interpretation and information extraction based on the target data, generate a demo example of the button component, and return it as data query feedback information to the AI ​​agent, which then displays the data query feedback information.

[0059] Figure 3 This is an exemplary flowchart of a content reorganization method according to some embodiments of this specification. Figure 3 The illustrated process 300 can be executed by a computer device, for example, by a content reconstruction module 750 in a data acquisition device 700 deployed on a computer device. Figure 3 As shown, in some embodiments, process 300 may include the following steps.

[0060] Step 310: Perform semantic analysis on the structured initial content data to determine semantic analysis information.

[0061] In some embodiments, semantic analysis information may include at least one of the following: the semantic type of each content node in the structured initial content data, the importance of each content node in the structured initial content data, the relationship type between content nodes in the structured initial content data, and the relationship strength between content nodes in the structured initial content data. A content node may be the smallest unit in the structured initial content data, such as a document title node, chapter title node, paragraph node, list node, table node, code example node, etc.

[0062] In some embodiments, a computer device can utilize a large language model to perform semantic analysis on the content of each content node and determine the semantic type corresponding to the content node based on a preset semantic type. The semantic type can be used to represent the content classification of the content node, such as conceptual, example, suggestion, and performance optimization categories. Conceptual categories can include content nodes used to describe technical concepts, definitions, and basic principles, such as document title nodes and chapter title nodes. Example categories can include content nodes used to provide code examples, such as code example nodes. Suggestion categories can include content nodes used to provide operational suggestions, specifications, or precautions, such as paragraph nodes providing suggested content. Performance optimization categories can include content nodes used to provide resource optimization, efficiency improvement, or application performance enhancement, such as table nodes displaying comparative information.

[0063] In some embodiments, the importance of content nodes is determined based on their semantic type, position within the structured initial content data, and semantic weight information. The semantic weight information can be determined based on the functional classification of content nodes, which may include core concept categories, supplementary explanation categories, and example description categories. Core concept categories may include content nodes used to define key terms, principles, or fundamental knowledge within a technical field. Supplementary explanation categories may include content nodes used to provide additional background information, explanations, expansions, or articulations of viewpoints. Example description categories may include content nodes that introduce, guide, or summarize examples. For example, a large language model can perform semantic analysis on the content of content nodes from a functional dimension to determine their corresponding functional classification. Content nodes in different functional classifications have different semantic weight information; for example, the semantic weight of core concept categories is higher than that of supplementary explanation categories, and the semantic weight of supplementary explanation categories is higher than that of example description categories.

[0064] In some embodiments, the position of each content node in the structured initial content data can indicate the importance of that content node. For example, a content node located in a title position is more important than a content node located in the body text, and a content node in a document title is more important than a content node in a chapter title. The weight information of a content node can be determined based on its position. For example, the weight information of a content node in a document title can be 1, the weight information of a content node in a chapter title can be 0.9, and the weight information of a content node located in the body text can be 0.8.

[0065] In some embodiments, the computer device can perform a weighted calculation based on the weight information corresponding to the semantic type of the content node, the positional weight information of the content node in the structured initial content data, and the semantic weight information of the content node to determine the importance score of the content node. For example, if a content node is a chapter title node, its corresponding semantic type is concept class, and the weight information corresponding to concept class can be 0.91; the positional weight information of the content node in the structured initial content data can be 0.9; and the functional classification of the content node can be core concept class, and the semantic weight information corresponding to core concept class can be 0.93, then the importance score corresponding to the content node can be p = 0.91 × α + 0.9 × β + 0.93 × γ, where α, β, and γ can be preset weight coefficients.

[0066] In some embodiments, the relationship types between content nodes may include hierarchical dependencies and semantic dependencies. Hierarchical dependencies can be determined based on the structural relationships between content nodes in the structured initial content data, while semantic dependencies can be determined based on the semantic relationships between content nodes in the structured initial content data. Hierarchical dependencies and semantic dependencies are described in detail below.

[0067] In some embodiments, the computer device may treat the nesting relationships of content nodes in the structured initial content data as hierarchical dependencies. For example, parent content nodes and child content nodes in the structured initial content data may form a parent-child relationship, and a chapter title node (parent content node) and multiple child content nodes under that chapter title node may form an inclusion relationship.

[0068] In some embodiments, a computer device can utilize a large language model to perform topic identification and keyword extraction on the content of each content node in structured initial content data. For example, the computer device can identify the topic of the content node in the structured initial content data as a technical document main title, a basic concept section, a concept definition, a code example, etc. For example, the computer device can extract keywords such as "React," "component," and "function" based on the content of the content node. The computer device can also use a large language model to perform semantic analysis on the content of the content node, and combine the topics and keywords extracted based on the content node to perform relational reasoning between the content nodes, thereby determining the semantic dependencies between the content nodes. For example, for content node A (about a concept definition) and content node B (a piece of code), the large language model can determine the matching semantic dependencies from a preset semantic dependency relationship based on the semantic analysis results and the topics and keywords extracted based on the content node. The preset semantic dependencies may include, for example, concept-example relationships, theory-practice relationships, precondition relationships, and comparison-reference relationships. Among them, a concept-example relationship can be a dependency relationship between a content node describing a technical concept, definition, or principle and a content node providing specific example code. A theory-practice relationship can be a dependency between a content node that explains a theoretical method and a content node that provides a practical example. A precondition relationship can be used to indicate that one content node is a prerequisite for another. A contrast-reference relationship can occur when two content nodes describe two related, opposing, or contrasting technical solutions. For example, a paragraph node defining a "React functional component" and a code example node demonstrating its specific code implementation can form a concept-example relationship.

[0069] In some embodiments, the computer device can also utilize a large language model to determine the strength of relationships between content nodes. For example, the large language model can encode the content of each content node, convert it into semantic vectors, and calculate the cosine similarity between the semantic vectors corresponding to different content nodes, determining the strength of relationships between content nodes based on the cosine similarity. The higher the cosine similarity, the stronger the relationship between content nodes.

[0070] Step 320: Reorder each content node based on semantic analysis information.

[0071] In some embodiments, the computer device may reorder the content nodes based on the priority order of their semantic types. The semantic types, ranked from highest to lowest priority, may include concept types, example types, suggestion types, and performance optimization types. Further description of semantic types can be found in step 310 above. For example, the computer device may reorder the content nodes within a chapter in the order of concept type, example type, suggestion type, and performance optimization type.

[0072] In some embodiments, the computer device can determine the importance of each content node based on its semantic type, its position in the structured initial content data, and its semantic weight information, and then reorder the content nodes based on their importance. Explanations regarding how to determine the importance of content nodes and their positions in the structured initial content data can be found in step 310 above, and will not be repeated here. For example, within the same chapter, if a concept-type content node has an importance score of 0.95 and a suggestion-type content node has an importance score of 0.8, then the concept-type content node with an importance score of 0.95 can be ranked before the suggestion-type content node with an importance score of 0.8.

[0073] In some embodiments, the computer device can determine topic blocks based on hierarchical dependencies, and a topic block may include one or more content nodes. The computer device can reorder the content nodes within a topic block based on semantic dependencies and / or the strength of relationships between content nodes. For a description of hierarchical and semantic dependencies, please refer to step 310 above; it will not be repeated here. For example, the computer device can divide the structured initial content data into multiple logically independent topic blocks based on hierarchical dependencies. For instance, a chapter title node and multiple sub-content nodes under that chapter title node can be determined as a topic block. In some embodiments, within the same topic block, the computer device can reorder the content nodes based on the semantic dependencies between them. For example, within the same topic block, if the semantic dependency between a concept node and a code example node is a concept-example relationship, the code example node can be moved after the concept node. For example, if content node A is a prerequisite for understanding content node B, and their semantic dependency is a precondition relationship, then when reordering the content nodes, content node A is placed before content node B.

[0074] In other embodiments, the computer device may also reorder the content nodes based on the semantic dependencies and correspondence strengths between them. For example, when the relationship strength between two content nodes (e.g., a concept node and a code example node) is greater than a preset strength threshold, and the semantic dependency is a concept-example relationship, the code example node can be moved after the concept node. In still other embodiments, the computer device may also reorder the content nodes based on the relationship strengths between them. For example, when the relationship strength between two content nodes is greater than a preset strength threshold, the order of these two content nodes can be adjusted to make them adjacent.

[0075] By performing semantic analysis on the structured initial content data, semantic analysis information is determined, and the content nodes are reordered based on the semantic analysis information, thereby improving the logical consistency of the content, tightly integrating related content, reducing the difficulty of understanding, and enabling the large language model to generate more accurate data query feedback information based on the target data.

[0076] Figure 4 This is an exemplary flowchart of another content reorganization method according to some embodiments of this specification. Figure 4 The illustrated process 400 can be executed by a computer device, for example, by a content reconstruction module 750 in a data acquisition device 700 deployed on a computer device. Figure 4 As shown, in some embodiments, process 400 may include the following steps.

[0077] Step 410: Perform semantic analysis on the structured initial content data to determine semantic analysis information.

[0078] The explanation for this step is similar to that in the previous text. For details, please refer to the description in step 310 above. It will not be repeated here.

[0079] Step 420: Based on semantic analysis information, organize the two or more content nodes into semantic units, and reorder the semantic units as a whole.

[0080] In some embodiments, a computer device can organize two or more content nodes into semantic units based on the relationship type and / or relationship strength between content nodes in structured initial content data. For example, when the relationship type between two or more content nodes is a preset relationship type, and / or the relationship strength between the content nodes is greater than a preset strength threshold, the positions of the two or more content nodes are adjusted to make them adjacent to each other, and then a new parent node is created for them, with the label "semantic unit" created for this new parent node. When rearranging content nodes, the parent node with the "semantic unit" label and all child nodes under that parent node can be moved as a whole. Furthermore, the computer device can also analyze the semantic analysis information corresponding to the parent node to reorder the parent node based on the semantic analysis information.

[0081] For example, if the semantic dependency between a paragraph node and a code example node is a concept-example relationship, then after adjusting the code example node to the paragraph node, the paragraph node and the code example node can be organized into a semantic unit, and a new parent node can be created for it. When reordering the parent node, the parent node and its child nodes (between the paragraph node and the code example node) can be moved as a whole.

[0082] Step 430: Add transition descriptions between content nodes based on semantic analysis information.

[0083] In some embodiments, the computer device may determine whether to add a transition description between content nodes based on the relationship type and / or relationship strength between content nodes. The transition description can be explanatory content that connects two related content nodes and serves a connecting and transitional function. For example, the transition description could be "After understanding the theoretical basis, let's look at an example in a practical application." For instance, when the semantic dependency relationship between two content nodes is a concept-example relationship or a theory-practice relationship, a transition description serving a connecting and transitional function can be added between the two content nodes. For instance, when the semantic dependency relationship between two content nodes is a concept-example relationship or a theory-practice relationship, and the corresponding relationship strength is greater than a preset strength threshold, a transition description can be added between the two content nodes.

[0084] In some embodiments, the computer device may use a large language model to perform semantic analysis on the content of the content node to which transition descriptions need to be added, and combine the topic and keywords corresponding to the content node to determine the specific content of the transition descriptions to be added.

[0085] Step 440: Add supplementary descriptions to the content nodes based on semantic analysis information.

[0086] In some embodiments, the computer device may determine whether to add supplementary descriptions to content nodes based on the relationship type and / or relationship strength between content nodes. For example, when the semantic dependency between two content nodes is a specific semantic dependency, supplementary descriptions can be added to these two content nodes. These supplementary descriptions can be additional content added to content nodes that are incomplete, unclear, or lack necessary content. For example, if a code example content node lacks descriptions of the execution results, supplementary descriptions can be added to that code example content node. Example of component usage: <welcome name="张三" / > Render as <h1> Hello, Zhang San!< / h1> In some embodiments, when the semantic dependency between two content nodes is a specific semantic dependency and the corresponding relationship strength is greater than a preset strength threshold, supplementary explanations can be added to the two content nodes. For example, if the semantic dependency between content node C and content node D is a comparison-reference relationship, the computer device can place the two content nodes side by side and generate a summary comparison table as supplementary explanations to enhance the readability and clarity of the content nodes.

[0087] In some embodiments, the computer device may determine whether to add supplementary descriptions to content nodes based on their importance. For example, if the average importance score of content nodes in a certain area (e.g., an area containing a preset number of adjacent content nodes) is lower than a preset score, the computer device may add supplementary descriptions to the content nodes in that area.

[0088] In some embodiments, the computer device may use a large language model to perform semantic analysis on the content of the content node that needs to be supplemented with additional explanations, and combine the topic and keywords corresponding to the content node to determine the specific content of the supplementary explanations that need to be added.

[0089] By organizing two or more content nodes into semantic units based on semantic analysis information, and then reordering the semantic units as a whole, or by adding transitional or supplementary explanations to the content nodes, the readability of the target data is enhanced, making it easier for large language models to understand the data content.

[0090] In some embodiments, steps 420, 430 and 440 are parallel, and the computer device may choose to execute one of them or execute any number of them, and there is no order restriction.

[0091] In some embodiments, converting initial content data into target data may include multiple processing stages such as preprocessing, structure recognition, semantic processing, format conversion, quality assessment, and optimization of the initial content data, thereby achieving high-precision data conversion. Figure 5 This is an exemplary flowchart illustrating a method for obtaining target data according to some embodiments of this specification. Figure 5 The process 500 shown can be executed by a computer device, for example, by a data acquisition device 700 deployed on a computer device. Figure 5 As shown, in some embodiments, process 500 may include the following steps.

[0092] Step 510: Preprocess the initial content data. In some embodiments, step 510 may be implemented by the content reorganization module 750.

[0093] In some embodiments, the computer device can perform a series of preprocessing operations on the initial content data, including input validation, tag cleaning, format standardization, noise filtering, and encoding. Input validation may include checking the validity of the content, its encoding format, length, and complexity. Tag cleaning may include removing useless tags from the initial content data while retaining core content structure tags. Format standardization may include converting various forms of line breaks, spaces, and tabs into a unified standard format. Noise filtering may include identifying and removing irrelevant content such as advertisements and navigation menus from the initial content data. Encoding may include correctly parsing and displaying entities and special characters in the initial content data to prevent garbled text. By preprocessing the initial content data, invalid information can be removed, the format standardized, and the quality of the initial content data improved. In some embodiments, the initial content data may be HTML formatted data.

[0094] Step 520: Perform structuring processing on the preprocessed initial content data to obtain structured initial content data. In some embodiments, step 520 can be implemented by the content reorganization module 750.

[0095] In some embodiments, structured processing may include a series of structured processing operations such as DOM tree construction, heading hierarchy analysis, list and table recognition, code reference recognition, paragraph extraction, and structure tree construction. DOM tree construction may include parsing the preprocessed initial content data into a DOM tree using an HTML parser. The DOM tree can be a tree-like data structure containing hierarchical relationships such as parent-child and sibling relationships. Heading hierarchy analysis may include identifying heading tags in the DOM tree and establishing a hierarchical structure, for example, traversing the DOM tree and identifying heading tags within the DOM tree. <h1> to< / h1> <h6>Title tags, etc., and establish a clear hierarchical structure based on tag levels. List and table recognition can include recognizing ordered lists, unordered lists, and table structures within the DOM tree. For example, recognizing... 、 、 List tags and 、 、 This includes identifying table tags and reconstructing their complete nesting and row / column structure. Code reference recognition can include identifying code blocks and referenced blocks in the DOM tree. For example, identifying... <pre>、 <code>、 <blockquote>Tags such as `<paragraph>` are used to identify and categorize paragraph content within the DOM tree as code blocks or quotation blocks. Paragraph extraction can include recognizing and classifying paragraph content within the DOM tree. For example, recognizing... Paragraph tags are used to categorize continuous text content and extract it into independent paragraph units. Structure tree construction can include integrating the above recognition results (including heading hierarchy analysis, list / table recognition, code citation recognition, and paragraph extraction) onto the original DOM tree to obtain structured initial content data.

[0096] Step 530: Based on semantic analysis of the structured initial content data, the structured initial content data is reorganized. In some embodiments, step 530 can be implemented by the content reorganization module 750.

[0097] In some embodiments, the computer device may perform semantic analysis on the structured initial content data to determine semantic analysis information, and reorder the content nodes based on the semantic analysis information. For a description of this part, please refer to the preceding process 300; it will not be repeated here.

[0098] In some embodiments, the computer device may perform semantic analysis on the structured initial content data to determine semantic analysis information. Further, based on the semantic analysis information, two or more content nodes may be organized into semantic units, and the semantic units may be reordered as a whole; alternatively, transitional descriptions may be added between content nodes based on the semantic analysis information; or, supplementary descriptions may be added to content nodes based on the semantic analysis information. For a description of this part, please refer to the description in process 400 above, and it will not be repeated here.

[0099] Step 540 involves converting the format of the initial content data after content reorganization. In some embodiments, step 540 can be implemented by the optimization processing module 760.

[0100] In some embodiments, the computer device can convert the format of the reorganized initial content data into an easy-to-write plain text markup format to facilitate reading and understanding by a large language model. For example, the reorganized initial content data can be converted from HTML format to Markdown format.

[0101] In some embodiments, a computer device can convert titles, paragraphs, lists, tables, code blocks, and other content in the initial content data after content reorganization into an easy-to-write plain text markup format, and merge the format-converted content according to the original logical order in the initial content data after content reorganization.

[0102] For example, a computer device can identify the initial content data after content reorganization. <h1> to< / h1> <h6>The text is identified as a heading by tags or styles (such as font-size, font-weight) and converted to Markdown format. For example, the number of # symbols indicates different heading levels.

[0103] Step 550: The initial content data after format conversion is evaluated using a large language model based on preset evaluation dimensions to obtain a quality evaluation result. In some embodiments, step 550 can be implemented by the optimization processing module 760.

[0104] In some embodiments, a computer device can utilize a large language model to perform quality assessment on the initial content data after format conversion based on preset evaluation dimensions, determine the score corresponding to each preset evaluation dimension, and determine the comprehensive quality score of the initial content data after format conversion based on the scores corresponding to each preset evaluation dimension and the preset weights corresponding to each preset evaluation dimension. The preset evaluation dimensions for quality assessment include at least one of the following dimensions: structural integrity, semantic accuracy, format standardization, and comprehensive intelligent assessment; the comprehensive intelligent assessment is used to perform an overall assessment of the initial content data after format conversion.

[0105] In some embodiments, sub-dimensions can be further subdivided within each preset evaluation dimension. For example, the evaluation of structural integrity can be conducted according to sub-dimensions such as titles, lists, and tables; the evaluation of semantic accuracy can be conducted according to sub-dimensions such as content relevance, contextual consistency, and link integrity; the evaluation of format conformity can be conducted according to sub-dimensions such as grammatical conformity and code format conformity; and the comprehensive intelligence evaluation can be conducted according to sub-dimensions such as content logic, content readability, and content professionalism.

[0106] Figure 6 This is a schematic diagram illustrating a comprehensive quality scoring method according to some embodiments of this specification. For example, as shown... Figure 6 As shown, for assessing structural integrity, computer devices can use large language models to evaluate whether the heading hierarchy is coherent and whether the structure of lists and tables is complete and correctly nested in the initial content data after format conversion. The large language model can also provide scores for each sub-dimension based on the assessment results, such as heading scores, list scores, and table scores, and determine the structural integrity score based on the scores of each sub-dimension and their corresponding weights. For example, as... Figure 6 As shown, the structural integrity score = title score × 0.4 + list score × 0.3 + table score × 0.3. For semantic accuracy assessment, the large language model can evaluate whether the content of the format-converted initial content data is relevant to the original content data, whether the logical order and relationships of the context remain consistent, and whether hyperlinks are fully preserved. The large language model can also provide scores for each sub-dimension based on the assessment results, such as content relevance score, context preservation score, and link integrity score, and determine the semantic accuracy score based on the scores of each sub-dimension and their corresponding weights. For example, as... Figure 6 As shown, the semantic accuracy score = content relevance score × 0.5 + context preservation score × 0.3 + link integrity score × 0.2. For formatting compliance assessment, the large language model can evaluate whether the syntax and code format of the initial content data are standardized after formatting conversion. The large language model can also provide scores for each sub-dimension based on the assessment results, such as Markdown syntax scores and code format scores, and determine the formatting compliance score based on the scores of each sub-dimension and their corresponding weights. For example, as... Figure 6 As shown, the formatting compliance score = Markdown syntax score × 0.6 + code formatting score × 0.4. For comprehensive intelligent evaluation, the large language model can assess the overall content logic, overall readability, and overall professionalism of the initial content data after formatting conversion. The large language model can also provide scores for each sub-dimension based on the evaluation results, such as logic score, readability score, and professionalism score, and determine the comprehensive intelligent evaluation score based on the scores of each sub-dimension and their corresponding weights. For example, as... Figure 6 As shown, the comprehensive intelligent evaluation score = logicality score × 0.4 + readability score × 0.3 + professionalism score × 0.3. Furthermore, the large language model can determine the comprehensive quality score of the initial content data after format conversion based on the scores corresponding to each preset evaluation dimension, the preset weights corresponding to each preset evaluation dimension, and the evaluation integrity factor. The evaluation integrity factor can be determined based on the evaluation results of the large language model for each preset evaluation dimension. For example, if there are n preset evaluation dimensions, and the large language model fails to evaluate m evaluation dimensions, then the evaluation integrity factor = (nm) / n. For example, as... Figure 6 As shown, the overall quality score = (structural integrity score × 0.25 + semantic accuracy score × 0.3 + format standardization score × 0.2 + overall intelligent evaluation score × 0.25) × evaluation integrity factor.

[0107] Step 560: Optimize the initial content data after format conversion using a large language model based on the quality assessment results to obtain the target data. In some embodiments, step 560 can be implemented by the optimization processing module 760.

[0108] In some embodiments, the computer device may use a large language model to determine whether the initial content data after format conversion needs to be optimized based on a comprehensive quality score. For example, when the comprehensive quality score is lower than a first preset score threshold, it is determined that the initial content data after format conversion needs to be optimized.

[0109] In some embodiments, the computer device can use a large language model to determine the content that needs optimization based on the scores corresponding to each preset evaluation dimension. For example, when the structural integrity score is lower than a second preset score threshold, the large language model is used to optimize the structure of the initial content data after format conversion.

[0110] In some embodiments, the computer device also interacts with other models via the MCP protocol to obtain optimization suggestions, and optimizes the initial content data after format conversion according to the optimization suggestions to obtain target data.

[0111] By conducting quality assessment and optimization of the initial content data, the format of the target data is further standardized and its quality is improved, so that the subsequent large language model can more accurately understand the content of the target data and generate more accurate data query feedback information.

[0112] This specification also provides a data acquisition device. Figure 7 This is an exemplary block diagram of a data acquisition apparatus according to some embodiments of this specification. In some embodiments, the data acquisition apparatus 700 may be deployed on a computer device. In some embodiments, the computer device may be a server. The server may include a local server or a cloud server, and depending on different service needs, a local server corresponding to a region may be deployed in one or more regions. In some embodiments, the server may be a single computer or a computing cluster composed of multiple computers, thereby providing more powerful computing power and responding to user service requests more efficiently. In other embodiments, the computer device may be a terminal device, including but not limited to desktop computers, smartphones, laptops, tablets, etc. In some embodiments, the computer device may further include a terminal device and a server. Figure 7 As shown, in some embodiments, the data acquisition device 700 may include a request acquisition module 710, a tool and information determination module 720, an access credential acquisition module 730, a data acquisition module 740, and a content reconstruction module 750.

[0113] The request retrieval module 710 can be used to retrieve data query requests.

[0114] The tool and information determination module 720 can be used to determine the data acquisition tool and data acquisition parameters that match the data query request based on the data query request. The data acquisition parameters include the access address and authentication type of the data resource corresponding to the data query request.

[0115] The access credential acquisition module 730 can be used to acquire access credentials for accessing data resources based on the authentication type.

[0116] The data acquisition module 740 can be used to acquire initial content data from data resources based on access credentials and the access address of data resources using a data acquisition tool.

[0117] The content reorganization module 750 can be used to reorganize the initial content data based on semantic analysis to obtain target data. The target data is used to generate data query feedback information corresponding to the data query request.

[0118] In some optional embodiments, the access credential acquisition module 730 can also be used to determine whether the authentication type is in the preset authentication type list; when the authentication type is in the preset authentication type list, determine whether there is a valid access credential; when there is no valid access credential, acquire an access credential for accessing data resources based on the authentication type.

[0119] In some optional embodiments, the access credential acquisition module 730 can also be used to: when the authentication type is a first preset authentication type, provide an authentication page corresponding to the access address of the data resource, and acquire access credentials based on the input information of the authentication page; when the authentication type is a second preset authentication type, provide an authentication page corresponding to the access address of the data resource, and acquire access credentials based on the authorization operation of the authentication page.

[0120] In some optional embodiments, the tool and information determination module 720 can also be used to determine, based on the data query request and tool configuration information, a data acquisition tool and data acquisition parameters that match the data query request. The tool configuration information includes the data acquisition tool identifier and tool parameters.

[0121] In some optional embodiments, the content reorganization module 750 can also be used to perform structuring processing on the initial content data to obtain structured initial content data; and to reorganize the structured initial content data based on semantic analysis of the structured initial content data.

[0122] In some optional embodiments, the content reorganization module 750 can also be used to perform semantic analysis on the structured initial content data to determine semantic analysis information, which includes at least one of the following: the semantic type of each content node in the structured initial content data, the importance of each content node in the structured initial content data, the relationship type between content nodes in the structured initial content data, and the relationship strength between content nodes in the structured initial content data; wherein the semantic type is used to represent the content classification of the content node; and the content nodes are reordered based on the semantic analysis information.

[0123] In some optional embodiments, the semantic types include concept types, example types, suggestion types, and performance optimization types according to their priority from high to low. The content reorganization module 750 can also be used to reorder each content node based on semantic analysis information, including: reordering each content node based on the priority order of its semantic type.

[0124] In some optional embodiments, the content reorganization module 750 may also be used to determine the importance of each content node based on the semantic type of each content node, the position of each content node in the structured initial content data, and the semantic weight information of each content node; and to reorder each content node based on its importance.

[0125] In some optional embodiments, the relationship types between content nodes include hierarchical dependencies and semantic dependencies. Hierarchical dependencies are determined based on the structural relationships between content nodes in the structured initial content data, while semantic dependencies are determined based on the semantic relevance between content nodes in the structured initial content data. The content reorganization module 750 can also be used to determine a topic block based on hierarchical dependencies, the topic block including one or more content nodes; and to reorder the content nodes within the topic block based on semantic dependencies and / or the strength of relationships between content nodes.

[0126] In some optional embodiments, the content reorganization module 750 may also be used to perform at least one of the following processes based on semantic analysis information: organizing two or more content nodes into semantic units and reordering the semantic units as a whole; adding transition descriptions between content nodes; and adding supplementary descriptions to content nodes.

[0127] In some optional embodiments, the data acquisition device 700 may further include an optimization processing module 760, which is used to convert the format of the initial content data after content reorganization; use a large language model to perform quality assessment on the initial content data after format conversion based on a preset evaluation dimension to obtain a quality assessment result; and use a large language model to perform optimization processing on the initial content data after format conversion based on the quality assessment result.

[0128] In some optional embodiments, the optimization processing module 760 can also be used to: use a large language model to perform quality assessment on the initial content data after format conversion based on preset evaluation dimensions, and determine the score corresponding to each preset evaluation dimension; determine the comprehensive quality score of the initial content data after format conversion based on the score corresponding to each preset evaluation dimension and the preset weight corresponding to each preset evaluation dimension; wherein, the preset evaluation dimensions of quality assessment include at least one of the following dimensions: structural integrity, semantic accuracy, format standardization, and comprehensive intelligent assessment; the comprehensive intelligent assessment is used to perform an overall assessment of the initial content data after format conversion.

[0129] For more information on each module, please refer to [link / reference]. Figures 1-6 The relevant explanations will not be repeated here. It should be understood that... Figure 7 The apparatus and modules shown can be implemented in various ways. For example, in some embodiments, they can be implemented by hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the methods and apparatus described above can be implemented using computer-executable instructions and / or included in the control code of a processor, such as on a media such as a disk, CD, or DVD-ROM, or in the memory of a programmable device. The apparatus and modules described in this specification can be implemented not only by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips or transistors, or programmable hardware devices such as field-programmable gate arrays or programmable logic devices, but also by software executed by various types of processors, or by a combination of the aforementioned hardware circuits and software (e.g., firmware).

[0130] It should be noted that the above description of the device and modules is for convenience only and should not be construed as limiting this specification to the embodiments described. It is understood that those skilled in the art, after understanding the principle of the device, can arbitrarily combine the various modules without departing from this principle to form sub-devices connected to other modules. Alternatively, some modules can be split to obtain more modules or multiple units under a single module. Such modifications are all within the scope of this specification.

[0131] Some embodiments of this specification also provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can implement this specification. Figures 1-6 The method shown.

[0132] Some embodiments of this specification also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, can implement this specification. Figures 1-6 The method shown.

[0133] Some embodiments of this specification also provide a computer program product, including a computer program that, when at least a portion of the computer program is executed by a processor, can implement this specification. Figures 1-6 The method is illustrated. In some embodiments, the computer program product may involve only a computer program, which may be carried on a storage medium or a processing device. In other embodiments, the computer program product may also be a storage medium or a processing device containing the aforementioned computer program. The processing device may include one or more processors, and the storage medium.

[0134] In some embodiments, the processor may be a combination of one or more of the following processors: central processing unit (CPU), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), graphics processing unit (GPU), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), programmable logic device (PLD), programmable logic controller (PLC), reduced instruction set computer (RISC), and microprocessor.

[0135] In some embodiments, the storage medium may include one or more combinations of the following: mass storage, removable storage, volatile read-write memory, and read-only memory (ROM). Exemplary mass storage may include disks, optical disks, solid-state drives, etc. Exemplary removable storage may include flash drives, floppy disks, optical disks, memory cards, compressed hard disks, magnetic tapes, etc. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAM may include dynamic random access memory (DRAM), dual data rate synchronous dynamic random access memory (DDRSDRAM), static random access memory (SRAM), silicon controlled retrieval memory (T-RAM), and zero-capacitance memory (Z-RAM), etc. Exemplary read-only memory may include masked read-only memory (MROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compressed hard disk read-only memory (CD-ROM), and digital multifunction hard disk read-only memory, etc.

[0136] The basic concepts have been described above. It is obvious that the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, various modifications, improvements, and corrections may be made to this specification by those skilled in the art. Such modifications, improvements, and corrections are taught in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.< / h6> < / blockquote> < / code> < / pre> < / h6>

Claims

1. A data acquisition method, characterized in that, The method includes: Get data query request; Based on the data query request, determine the data acquisition tool and data acquisition parameters that match the data query request. The data acquisition parameters include the access address and authentication type of the data resource corresponding to the data query request. Based on the authentication type, obtain access credentials for accessing the data resources; Using the data acquisition tool, initial content data is obtained from the data resource based on the access credentials and the access address of the data resource; Based on semantic analysis of the initial content data, the initial content data is reorganized to obtain target data, which is used to generate data query feedback information corresponding to the data query request.

2. The method according to claim 1, characterized in that, The step of obtaining access credentials for accessing the data resources based on the authentication type includes: Determine whether the authentication type is in the preset authentication type list; When the authentication type is in the preset authentication type list, determine whether there is a valid access credential; When no valid access credential exists, the access credential for accessing the data resource is obtained based on the authentication type.

3. The method according to claim 2, characterized in that, The step of obtaining the access credential for accessing the data resource based on the authentication type includes: When the authentication type is the first preset authentication type, an authentication page corresponding to the access address of the data resource is provided, and the access credential is obtained based on the input information of the authentication page; When the authentication type is the second preset authentication type, an authentication page corresponding to the access address of the data resource is provided, and the access credential is obtained based on the authorization operation of the authentication page.

4. The method according to claim 1, characterized in that, The step of determining the data acquisition tool and data acquisition parameters matching the data query request based on the data query request includes: Based on the data query request and tool configuration information, the data acquisition tool and data acquisition parameters that match the data query request are determined. The tool configuration information includes the data acquisition tool identifier and tool parameters.

5. The method according to any one of claims 1-4, characterized in that, The semantic analysis based on the initial content data to reorganize the initial content data includes: The initial content data is processed in a structured manner to obtain structured initial content data; Based on semantic analysis of the structured initial content data, the structured initial content data is reorganized.

6. The method according to claim 5, characterized in that, The semantic analysis based on the structured initial content data, and the content reorganization of the structured initial content data, includes: Semantic analysis is performed on the structured initial content data to determine semantic analysis information, which includes at least one of the following: the semantic type of each content node in the structured initial content data, the importance of each content node in the structured initial content data, the relationship type between content nodes in the structured initial content data, and the relationship strength between content nodes in the structured initial content data; wherein, the semantic type is used to represent the content classification of the content node; The content nodes are reordered based on the semantic analysis information.

7. The method according to claim 6, characterized in that, The semantic types, ranked from highest to lowest priority, include concept type, example type, suggestion type, and performance optimization type; The reordering of the content nodes based on the semantic analysis information includes: Based on the priority order of the semantic types of each content node, the content nodes are reordered.

8. The method according to claim 6, characterized in that, The reordering of the content nodes based on the semantic analysis information includes: The importance of each content node is determined based on its semantic type, its position in the structured initial content data, and its semantic weight information. The content nodes are reordered based on their importance.

9. The method according to claim 6, characterized in that, The relationship types between the content nodes include hierarchical dependency relationships and semantic dependency relationships. The hierarchical dependency relationship is determined based on the structural relationship between each content node in the structured initial content data, and the semantic dependency relationship is determined based on the semantic association between each content node in the structured initial content data. The reordering of the content nodes based on the semantic analysis information includes: Thematic blocks are determined based on the hierarchical dependency relationship, and each thematic block includes one or more of the aforementioned content nodes; The content nodes within the topic block are reordered based on the semantic dependencies and / or the strength of the relationships between the content nodes.

10. The method according to claim 6, characterized in that, The semantic analysis based on the structured initial content data, and the content reorganization of the structured initial content data, further includes: performing at least one of the following processes based on the semantic analysis information: Organize two or more of the content nodes into semantic units, and reorder the semantic units as a whole; Add transition descriptions between the content nodes; Add supplementary descriptions to the content nodes.

11. The method according to claim 1, characterized in that, The method further includes: The initial content data for content reorganization is formatted and converted. The quality of the initial content data after format conversion is evaluated using a large language model based on preset evaluation dimensions, and the quality evaluation results are obtained. The large language model is used to optimize the initial content data after the format conversion based on the quality assessment results.

12. The method according to claim 11, characterized in that, The process involves using a large language model to perform a quality assessment on the initial content data after format conversion based on preset evaluation dimensions, and obtaining the quality assessment results, including: The large language model is used to evaluate the quality of the initial content data after format conversion based on the preset evaluation dimensions, and the scores corresponding to each preset evaluation dimension are determined. The overall quality score of the initial content data after format conversion is determined based on the scores corresponding to each preset evaluation dimension and the preset weights corresponding to each preset evaluation dimension. The preset evaluation dimensions of the quality assessment include at least one of the following dimensions: structural integrity, semantic accuracy, format standardization, and comprehensive intelligent assessment; the comprehensive intelligent assessment is used to perform an overall evaluation of the initial content data after the format conversion.

13. A data acquisition device, characterized in that, The device includes: The request retrieval module is used to retrieve data query requests; The tool and information determination module is used to determine the data acquisition tool and data acquisition parameters that match the data query request based on the data query request. The data acquisition parameters include the access address and authentication type of the data resource corresponding to the data query request. The access credential acquisition module is used to acquire access credentials for accessing the data resources based on the authentication type. The data acquisition module is used to acquire initial content data from the data resource using the data acquisition tool based on the access credentials and the access address of the data resource; The content reorganization module is used to reorganize the initial content data based on semantic analysis to obtain target data. The target data is used to generate data query feedback information corresponding to the data query request.

14. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it is able to implement the method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, enable the implementation of the method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes a computer program that, when at least a portion of the computer program is executed by a processor, enables the implementation of the method as described in any one of claims 1 to 12.

Images