Resource acquisition method and device, reinforcement learning method and device, storage medium and program product

Abstract

The embodiment of the invention provides a resource acquisition and reinforcement learning method and device, a storage medium and a program product. In the method, a resource acquisition demand is acquired; aiming at the resource acquisition demand, generating a first task queue, the first task queue being used for storing a uniform resource identifier (URI) to be explored; a resource obtaining module obtains the first URI from the first task queue and initiates a resource access request based on the first URI to obtain a first network resource; acquiring a first reference URI (Uniform Resource Identifier) embedded in the first network resource; according to a resource acquisition state corresponding to the first URI, determining whether the first reference URI needs to be explored by using a decision model obtained based on reinforcement learning optimization; and when the first reference URI needs to be explored, the first reference URI is added to the first task queue for subsequent acquisition of corresponding network resources.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a resource acquisition, reinforcement learning method, device, storage medium, and program product. Background Technology

[0002] In today's era of deep integration of artificial intelligence and big data, data has become a key resource for enterprise competition as a core production factor. Data is central to all stages of large language model training—including data collection, preprocessing, training, and validation.

[0003] The efficiency and quality of data acquisition directly impact the effectiveness of data value mining and application. However, traditional data acquisition systems face challenges such as difficulty in acquiring dynamic web page content and complex anti-acquisition strategies, making them ill-suited for the effective acquisition of massive amounts of heterogeneous data.

[0004] Therefore, there is an urgent need to provide a data acquisition solution that can improve data acquisition efficiency. Summary of the Invention

[0005] This specification provides, in various aspects, a resource acquisition method, a device, a storage medium, and a program product for improving resource acquisition efficiency by introducing reinforcement learning to optimize resource exploration paths.

[0006] The first aspect of this specification provides a method for obtaining resources, including: Acquire resources and acquire requirements; To meet the resource acquisition requirements, a first task queue is generated, which is used to store the Uniform Resource Identifiers (URIs) to be explored. The resource acquisition module obtains a first URI from the first task queue and initiates a resource access request based on the first URI to obtain a first network resource; Obtain the first reference URI embedded in the first network resource; Based on the resource acquisition status corresponding to the first URI, a decision model optimized by reinforcement learning is used to determine whether it is necessary to explore the first reference URI. When the first reference URI needs to be explored, the first reference URI is added to the first task queue for subsequent acquisition of the corresponding network resources.

[0007] The second aspect of this specification provides a reinforcement learning method, including: Create a second task queue; The resource acquisition module obtains the second URI from the second task queue and acquires the second network resource based on the second URI; Obtain the second reference URI embedded in the second network resource; Based on the resource acquisition status corresponding to the second URI, the decision model to be optimized is used to determine whether it is necessary to explore the second reference URI; When the second reference URI needs to be explored, the second reference URI is added to the second task queue for subsequent acquisition of the corresponding network resources; The reward is determined based on the success rate of resource retrieval corresponding to the second reference URI; Based on the reward, the decision model to be optimized is optimized.

[0008] A third aspect of this specification provides a resource acquisition device, comprising: The first acquisition module is used to acquire resource acquisition requirements; The generation module is used to generate a first task queue for the resource acquisition requirements, and the first task queue is used to store the Uniform Resource Identifiers (URIs) to be explored. The resource acquisition module is used to obtain a first URI from the first task queue and initiate a resource access request based on the first URI to obtain a first network resource; The second acquisition module is used to acquire the first reference URI embedded in the first network resource; The determination module is used to determine whether it is necessary to explore the first reference URI based on the resource acquisition status corresponding to the first URI and a decision model optimized by reinforcement learning. An add module is used to add the first reference URI to the first task queue when the first reference URI needs to be explored, so that the corresponding network resources can be obtained later.

[0009] A fourth aspect of this specification provides a reinforcement learning device, comprising: Create a module to create a second task queue; The resource acquisition module is used to obtain a second URI from the second task queue and obtain a second network resource based on the second URI; The acquisition module is used to acquire the second reference URI embedded in the second network resource; The first determining module is used to determine whether it is necessary to explore the second referencing URI based on the resource acquisition status corresponding to the second URI and using the decision model to be optimized. An add module is used to add the second reference URI to the second task queue when the second reference URI needs to be explored, so that the corresponding network resources can be obtained later. The second determining module is used to determine the reward based on the success rate of resource acquisition corresponding to the second reference URI; An optimization module is used to optimize the decision model to be optimized based on the reward.

[0010] A fifth aspect of this specification provides an electronic device, comprising: a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0011] A sixth aspect of this specification provides a computer-readable storage medium storing a computer program that, when executed by a computer, enables the implementation of any of the methods described above.

[0012] A seventh aspect of this specification provides a computer program product including a computer program that, when executed by a processor, implements the method described in any of the preceding descriptions.

[0013] In the technical solution provided in the embodiments of this specification, a task queue is created to store URIs to be explored, addressing resource acquisition needs. The resource acquisition module retrieves URLs from the task queue and initiates resource access requests based on these URLs to obtain network resources. Then, based on the resource acquisition status corresponding to the URL, a decision model based on reinforcement learning is used to determine whether a reference URL embedded in the network resource is worth exploring, thereby deciding whether to add the reference URL to the task queue for further exploration. It is evident that this solution, through reinforcement learning, can optimize the resource exploration path / direction based on real-time resource acquisition status, exhibiting good adaptability and the ability to cope with dynamically changing network environments, such as dynamically changing web pages, dynamically changing anti-acquisition strategies, and dynamically changing network connection quality. This reduces inefficient or even ineffective resource acquisition behavior, thereby improving resource acquisition efficiency. Attached Figure Description

[0014] The accompanying drawings, which are provided to further illustrate this specification, form part of this specification.

[0015] Figure 1 A schematic diagram of the structure of a resource acquisition system provided in an exemplary embodiment of this specification; Figure 2 A flowchart illustrating a resource acquisition method provided for an exemplary embodiment of this specification; Figure 3 A flowchart illustrating an exemplary embodiment of the reinforcement learning method provided in this specification; Figure 4 A flowchart illustrating an exemplary embodiment of the reinforcement learning method provided in this specification; Figure 5 This is a schematic diagram of the structure of an electronic device provided as another exemplary embodiment of this specification. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments, and not all of the embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0017] It should be noted that, in the cases involving user information in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this specification (including but not limited to language models or large models) comply with relevant laws and standards.

[0018] Before introducing the information display and information processing methods provided in the embodiments of this specification, the system architecture involved in the information display and information processing methods provided in the embodiments of this specification will be described. For example... Figure 1 As shown, the information processing system includes: server 1 and client 2. Among them, Before introducing the specific details of each embodiment in this specification, a brief explanation of the technical terms mentioned herein will be provided.

[0019] Reinforcement learning (RL) is a machine learning method. Its fundamental framework is the Markov decision process, which allows an agent to learn better policies through trial and error in its interactions with the environment. The agent performs actions in the environment and receives feedback, or rewards, based on the outcomes of those actions. These reward signals guide the agent to adjust its policy to maximize long-term cumulative rewards.

[0020] A pre-trained language model (LM) is a deep learning model trained on data, typically possessing broad task adaptability and inference capabilities, such as the Transformer architecture. These models can understand complex contextual information and generate accurate predictions and suggestions in scenarios such as data analysis and decision support. The embodiments in this specification do not limit the number of model parameters supported by the language model, aiming to meet application requirements. If the model has relatively more parameters, the language model will be relatively larger and its performance will be relatively better; however, it will consume more time and resources during inference and training. If the model has relatively fewer parameters, the language model will be relatively smaller and, while meeting performance requirements, more lightweight, consuming less time and resources during inference and training. When the number of model parameters in a pre-trained language model is greater than or equal to a preset threshold, the pre-trainable language model is called a Large Language Model (LLM).

[0021] A Uniform Resource Identifier (URI) is a compact string used to identify abstract or physical resources. URIs can be further divided into two forms: Uniform Resource Locators (URLs) and Uniform Resource Names (URNs). A URL is the standard address format used on the Internet to uniquely identify and locate a resource. Simply put, a URL is the "web address" you enter in your browser's address bar; it tells the computer "where to go, how to use it, and what resource to obtain." A URN is a persistent, location-independent resource identifier. In practical applications, a URN is resolved using a pre-defined resolution service to obtain its corresponding URL. Therefore, before accessing the network resource corresponding to a URN, a pre-defined resolution service can be used to convert the URN into a URL.

[0022] Traditional web resource acquisition systems typically rely on preset rules or fixed strategies for resource retrieval. However, in practical applications, changes to the HTML (Hypertext Markup Language) source code, Document Object Model (DOM) nodes, or page layout can render previously written data parsing / extraction rules ineffective, preventing the correct location and extraction of target information, thus leading to parsing failures. Furthermore, websites, to protect data and reduce server load, commonly employ increasingly complex anti-retrieval mechanisms such as IP frequency restrictions, CAPTCHAs, and request header verification, significantly increasing the probability of resource acquisition failures. In addition, network connection instability can also lead to inefficient resource acquisition. Therefore, traditional web resource acquisition systems suffer from low efficiency and poor adaptability.

[0023] In some web resource acquisition systems, the resource acquisition engine is responsible for acquiring and parsing web page resources, using SQL relational databases to store structured data. Simultaneously, a NoSQL database is introduced to temporarily store the raw, semi-structured data, such as complete HTML pages or JSON responses, for subsequent processing and troubleshooting. In-memory databases like Redis are used for task queue management, and their collection data structures are used to deduplicate visited URLs, preventing repeated access. To counter the anti-access mechanisms of target websites, third-party paid proxy services are typically integrated to build a proxy IP pool, enabling the rotation and masquerading of request IPs. Middleware is also written to dynamically switch User-Agent strings in the request headers, simulating different browsers or devices. The raw data is cleaned and transformed using the Pandas library, and regular expressions are used for pattern matching and information extraction. The overall system startup and operation rely on operating system-level scheduled task tools, triggering crawler tasks periodically at preset times.

[0024] A web crawling task typically begins with a manually configured initial list of URLs (seed URLs). During the crawling process, the system dynamically discovers and collects new, rule-compliant links by parsing the HTML content of downloaded pages, adding them to the crawling queue to achieve a certain degree of automated expansion. The core countermeasures include two aspects: first, frequently switching the exit IP address through the aforementioned paid proxy pool to distribute the request pressure on a single IP; second, randomly rotating a predefined list of User-Agents when sending HTTP requests to simulate the access behavior of a real user's browser. This architecture utilizes the asynchronous processing mechanism built into the Scrapy framework to achieve multi-threaded concurrent crawling on a single server, thereby improving data crawling efficiency on a single machine.

[0025] While the above solutions can meet resource acquisition needs on a certain scale, in practical applications, especially in scenarios with large amounts of data and complex requirements, the following main drawbacks have been revealed: This architecture is essentially a single-machine multi-threaded model. When the number of website domains that need to be accessed simultaneously surges, the number of target links reaches a massive scale, or multiple heterogeneous acquisition tasks need to be executed in parallel, the system's performance will be severely constrained by the CPU, memory, and network bandwidth resources of a single server, making it difficult to improve overall throughput through horizontal scaling.

[0026] Data extraction relies entirely on manually written page parsing rules (such as DOM-based or regular expression-based location rules) for each target website. For example, the front-end page structure of e-commerce websites is frequently redesigned or adjusted. Once a change occurs, the corresponding parsing rules must be manually identified and modified, resulting in heavy system maintenance work and slow response times.

[0027] The system lacks a unified and intelligent processing capability for different data sources (such as web pages with different structures). For each new website structure or data type, it is necessary to reset and configure specific crawling and parsing rules. Furthermore, various tasks can only be executed sequentially or compete for resources within a single machine, resulting in poor data collection flexibility and low overall efficiency.

[0028] However, the era of big data and large-scale artificial intelligence (AI) models presents new challenges to the acquisition of network resources. With the rapid development of AI and large-scale model technologies, high-quality data and rapid data acquisition have become key factors influencing technological development. Large-scale model training requires a large amount of high-quality data to ensure the model's complexity and predictive ability. Especially in the data collection and preprocessing stages, the diversity and completeness of the data determine the foundation of the model. Data in the training phase needs to undergo rigorous screening and optimization to ensure the model can accurately fit the model and avoid overfitting. The validation phase requires a comprehensive evaluation of the model; the depth and usability of the data directly affect the model's final performance.

[0029] To address or partially address the aforementioned problems, this specification proposes an adaptive network resource acquisition scheme based on reinforcement learning. Since traditional network resource acquisition schemes cannot cope with dynamically changing network environments, this scheme introduces reinforcement learning to optimize resource exploration paths, reduce inefficient or even ineffective resource acquisition behaviors, and thus improve resource acquisition efficiency.

[0030] Before introducing the resource acquisition method provided in the embodiments of this specification, the system architecture on which the implementation of this method depends will be introduced. Figure 1 This is a schematic diagram of the structure of the resource acquisition system provided in the embodiments of this specification. Figure 1As shown, the resource acquisition system may include: a task queue 2, a task execution cluster 3, a parsing and decision module 4, and a database (DB) 5.

[0031] The message producer-message queue-message consumer architecture is comprised of task queue 2, task execution cluster 3, and parsing and decision-making module 4. Task queue 2 is the message queue, the worker nodes in task execution cluster 3 are message consumers, and parsing and decision-making module 4 is the message producer.

[0032] The task queue 2 mentioned above is used to store URIs to be explored. Each URI can be considered as a message or a resource retrieval task.

[0033] In some embodiments, multiple seed URIs can be constructed based on the seed list 1 using the URI constructor, and these multiple seed URIs can be added to the task queue 2.

[0034] Task execution cluster 3 consists of multiple worker nodes, meaning that there are multiple message consumers for task queue 2. The working mode between task execution cluster 3 and task queue 2 is point-to-point, meaning that a specific message can only be consumed by one consumer. In other words, when a message is being processed by one consumer, it will be deleted or locked in the queue, preventing other consumers from processing it.

[0035] Each of the multiple worker nodes is used to obtain the URI from task queue 2 and initiate a resource access request based on the URI to obtain the corresponding network resources.

[0036] The parsing and decision-making module 4 is used to parse the network resources acquired by the worker nodes in the task execution cluster 3, obtaining the resource subject and the reference URI embedded in the network resource. The reference URI refers to a URI that appears as a string in the acquired network resource and points to other network resources. Taking the scenario of acquiring online articles as an example, the resource subject may include information such as the article title and article body. The parsing and decision-making module 4 is also used to determine whether to store the resource subject in the database 5 and, based on a reinforcement learning model (i.e., a decision model optimized through reinforcement learning), whether to add the reference URI to the task queue for subsequent acquisition of the corresponding network resources.

[0037] Task execution cluster 3 continuously retrieves URLs from task queue 2 until there are no more URIs to be explored in the task queue or until the number of resources stored in the database meets the preset requirements.

[0038] In this embodiment, the resource acquisition system is a distributed processing system. The task execution cluster is a module within the distributed processing system that executes the actual resource acquisition tasks. Its main function is to retrieve resource acquisition tasks from the URL queue, initiate resource acquisition requests to the target website using distributed or parallel processing, and acquire the corresponding network resources. Furthermore, to avoid being blocked or restricted by the target website, the task execution cluster includes randomly generated client information in each resource acquisition request, reducing the probability of being identified as an automated program by the target website.

[0039] The embodiments in this specification utilize a distributed architecture design to not only solve the problems of limited single-machine resources and insufficient task distribution throughput, achieving elastic scaling and effective task processing, but also to a certain extent counteract anti-acquisition strategies that rely on IP frequency restrictions.

[0040] The specific implementation details of each component of the above system will be described in the following embodiments.

[0041] Figure 2 This is a schematic flowchart illustrating a resource acquisition method provided in one embodiment of this specification. Figure 2 As shown, the method includes the following steps: 200. Obtaining resources and meeting requirements.

[0042] 202. Generate a first task queue in response to the resource acquisition requirements.

[0043] 204. The resource acquisition module obtains the first URI from the first task queue and initiates a resource access request based on the first URI to obtain the first network resource.

[0044] 206. Obtain the first reference URI embedded in the first network resource.

[0045] 208. Based on the resource acquisition status corresponding to the first URI, determine whether it is necessary to explore the first reference URI using a decision model optimized by reinforcement learning.

[0046] 210. When the first reference URI needs to be explored, the first reference URI is added to the first task queue for subsequent acquisition of the corresponding network resources.

[0047] In the above 200, resource acquisition requirement refers to a description of the intention to acquire resources, which may include: attribute information of the resources to be acquired. The attribute information may include the relevant field. For example, the resource acquisition requirement is "to acquire articles related to architecture".

[0048] In step 202 above, the first task queue is used to store Uniform Resource Identifiers (URIs) to be explored. The resource identified by the URI may include: a webpage (e.g., an HTML webpage).

[0049] In practical applications, multiple initial URLs can be obtained to meet resource acquisition needs, and a first task queue can be created to add these initial URLs. These initial URLs can be manually entered or automatically generated.

[0050] A seed list can be generated based on resource acquisition needs. The seed list may include keywords and / or initial URIs. The number of keywords and initial URIs can be one or more.

[0051] For example, based on the attribute information of the resource to be obtained, keywords or URIs related to the attribute information are searched using an internet search engine.

[0052] For example, based on the attribute information of the resource to be obtained, a pre-trained language model can be used to provide keywords or URLs related to that attribute information.

[0053] After obtaining the seed list, you can use an internet search engine to search for online resources related to the keywords in the seed list, and use the URI of the corresponding online resource as the seed URI. If the seed list contains an initial URI, the initial URI can be used directly as the seed URI.

[0054] After obtaining the seed URI, a first task queue is generated based on the resource acquisition needs, and the seed URI is added to the first task queue, which means that the seed URI is published to the task queue.

[0055] In the above 204, the resource acquisition module, as a consumer of the first task queue, is used to obtain the first URI from the first task queue and initiate a resource access request to the target website based on the URI to obtain the first network resource.

[0056] In some embodiments, the resource acquisition module may be derived from the above. Figure 1 It is composed of task execution clusters to achieve a distributed processing architecture.

[0057] In the above 206, the resource acquisition module obtains the first network resource after initiating a resource access request based on the URI.

[0058] The first network resource can be parsed to obtain a first reference URI embedded in the first network resource. There can be one or more first reference URIs embedded in the first network resource. Each first reference URI points to a different network resource than the first network resource.

[0059] In the above 208, the resource acquisition status corresponding to the first URI refers to the status related to resource acquisition based on the first URL.

[0060] In some embodiments, the resource acquisition status corresponding to the first URI includes one or more of the following: resource acquisition response information corresponding to the first URI, exploration depth of the first URL, website to which the first URL belongs, access frequency for the website, operating status of the resource acquisition module, historical decision information of the decision model for the website, and content semantic score of the resource subject parsed from the first network resource.

[0061] The resource acquisition response information corresponding to the first URI may include one or more of the following: response status code, response time, response content size, and content parsing quality. The response status code indicates the processing result of the request and may include Hypertext Transfer Protocol (HTTP) status codes. The response time refers to the time consumed from when the resource acquisition module initiates a resource acquisition request based on the first URI to when it receives the response content. The response content size refers to the data size of the response content. The content parsing quality is determined based on the accuracy and completeness of the data obtained by parsing the response content (i.e., the first network resource) obtained based on the first URI. The content semantic score of the resource subject is determined based on the matching degree between the resource subject and the resource acquisition request and / or the redundancy degree between the resource subject and resource subjects already acquired for the resource acquisition request.

[0062] The exploration depth of the first URI refers to the link distance or number of links of the first URI relative to the seed URI during the resource acquisition process performed in response to the aforementioned resource acquisition needs. Specifically, the depth of the seed URI can be defined as 0 (or 1); for any other URI, its depth is equal to the depth of the parent page that discovered the URI plus one. For example, the depth of page 1 is n, and the depth of the URIs discovered in page 1 is n+1.

[0063] The website to which the first URL belongs refers to the website to which the webpage corresponding to the first URL belongs.

[0064] The operating status of the resource acquisition module may include information such as CPU utilization and memory utilization.

[0065] The decision model's historical decision information for the website may include: decision actions performed by the decision model for web pages on the website and the corresponding rewards for those actions. Decision actions determine whether a reference URL within the web page needs to be explored. The reward for the decision action can be determined based on the resource acquisition success rate, computational cost, and / or content semantic score corresponding to the reference URL. The specific determination method will be described in detail in the following embodiments.

[0066] A state vector can be generated based on the resource acquisition status corresponding to the first URI. The state vector is then input into a decision model optimized by reinforcement learning, so that the decision model can determine whether the first reference URI needs to be explored.

[0067] Specifically, the resource acquisition state is encoded into a state vector and input into a decision model. The decision model then outputs a binary action indicating whether the first referencing URI needs to be explored. For example, a positive binary action means the first referencing URI needs to be explored, while a negative binary action means the first referencing URI does not need to be explored.

[0068] The reinforcement learning process of the decision model will be described in detail in the following examples.

[0069] In step 210 above, when the first reference URI needs to be explored, the first reference URI is added to the first task queue for subsequent acquisition of the corresponding network resources.

[0070] If the first referencing URI does not need to be explored, discard the first referencing URI.

[0071] In the technical solution provided in the embodiments of this specification, a task queue is created to store URIs to be explored, addressing resource acquisition needs. The resource acquisition module retrieves URLs from the task queue and initiates resource access requests based on these URLs to obtain network resources. Then, based on the resource acquisition status corresponding to the URL, a decision model based on reinforcement learning is used to determine whether a reference URL embedded in the network resource is worth exploring, thereby deciding whether to add the reference URL to the task queue for further exploration. It is evident that this solution, through reinforcement learning, can optimize the resource exploration path / direction based on real-time resource acquisition status, exhibiting good adaptability and the ability to cope with dynamically changing network environments, such as dynamically changing web pages, dynamically changing anti-acquisition strategies, and dynamically changing network quality. This reduces inefficient or even ineffective resource acquisition behavior, thereby improving resource acquisition efficiency.

[0072] In some embodiments, the step of "generating a first task queue in response to the resource acquisition requirement" in step 202 above can be implemented by the following steps: 2020. Obtain multiple keywords related to the resource acquisition requirements.

[0073] 2022. Based on the aforementioned keywords, determine multiple seed URIs.

[0074] 2024. Create a first task queue and add the multiple seed URIs to the first task queue.

[0075] In one optional implementation of the above-mentioned 2020, the resource acquisition requirement can be input to the pre-trained language model to request the pre-trained language model to output multiple keywords related to the resource acquisition requirement.

[0076] In one optional implementation of the above-mentioned 2022, multiple keywords can be input into a search engine to obtain search results returned by the search engine. The search results include web pages, and the URI of the web page can be used as a seed URI.

[0077] In another alternative implementation, a URI database can be pre-set, storing multiple URIs, each indexed with one or more keywords extracted from the corresponding network resource. Based on these keywords related to the resource acquisition requirement, a match can be made within the pre-set URI database to obtain a URI, which is then used as a seed URI. Specifically, keywords related to the resource acquisition requirement can be matched against the index of each URI in the URI database; once a match is found, the matched URI is used as the seed URI.

[0078] In the above 2024, the first task queue is a first-in-first-out queue. The multiple seed URIs are added to the first task queue sequentially.

[0079] In this specification, different keywords can be used to approach the target domain from different dimensions and subjects, thereby increasing the diversity of seed sources and avoiding the initial URIs being too concentrated on one aspect of the target domain. This helps to improve the breadth and diversity of the initial coverage.

[0080] The reward function used by the decision model in the reinforcement learning phase is configured to generate a reward based on the resource acquisition success rate of the second reference URI, the computing power consumption cost of initiating access based on the second reference URI, and / or the content semantic score of the network resources obtained based on the second reference URI; the second reference URI refers to the reference URI that the decision model determines needs to explore in the reinforcement learning phase.

[0081] For example, the mathematical expression for the reward function is: (1) Where P is the resource acquisition success rate, Q is the content semantic score, and E is the computing power consumption cost. α1, α2, and α3 are weight coefficients, all of which are positive numbers.

[0082] The reinforcement learning process of the decision-making model will be described in detail below.

[0083] In some embodiments, the reinforcement learning process of the above decision model includes the following steps: 212. Create a second task queue.

[0084] 214. The resource acquisition module obtains a second URI from the second task queue and acquires a second network resource based on the second URI.

[0085] 216. Obtain the second reference URI embedded in the second network resource.

[0086] 218. Based on the resource acquisition status corresponding to the second URI, determine whether it is necessary to explore the second reference URI using the decision model to be optimized.

[0087] 220. When the second reference URI needs to be explored, add the second reference URI to the second task queue for subsequent acquisition of the corresponding network resources.

[0088] 222. Determine the reward based on the resource retrieval success rate corresponding to the second reference URI.

[0089] 224. Based on the reward, optimize the decision model to be optimized.

[0090] In some embodiments of 212 above, a seed list can be initialized for multiple different domains. The seed list may contain keywords for multiple domains. Based on the seed list, multiple initial URIs are generated. The multiple initial URIs are added to the second task queue to complete the initialization of the second task queue.

[0091] The second task queue is used to store URIs to be explored.

[0092] In step 214 above, the resource acquisition module obtains the second URI from the second task queue and initiates a resource access request to the target website based on the second URI to obtain the second network resource.

[0093] In step 216 above, the obtained second network resource can be parsed to obtain a second reference URI embedded in the second network resource. The second application URI can be one or more.

[0094] In step 218 above, a state vector can be generated based on the resource acquisition status corresponding to the second URI, and the state vector can be input into the decision model to be optimized so that the decision model to be optimized can determine whether it is necessary to explore the second reference URI based on the state vector.

[0095] The resource acquisition status may include: the resource acquisition status corresponding to the second URI includes one or more of the following: the resource acquisition response information corresponding to the first URI, the exploration depth of the second URI, the website to which the second URI belongs, the access frequency of the website during this resource acquisition process, the operating status of the resource acquisition module, the historical decision information of the decision model to be optimized for the website, and the content semantic score of the resource subject parsed from the second network resource. In other words, the state space of the decision model to be optimized includes the following features: the resource acquisition status corresponding to the second URI includes one or more of the following: the resource acquisition response information corresponding to the first URI, the exploration depth of the second URI, the website to which the second URI belongs, the access frequency of the website during this resource acquisition process, the operating status of the resource acquisition module, the historical decision information of the decision model to be optimized for the website, and the content semantic score of the resource subject parsed from the second network resource.

[0096] The action space of the decision model to be optimized can include: URIs that need to be explored and URIs that do not need to be explored.

[0097] The decision model to be optimized may include a policy network built based on a Deep Q-Network (DQN). The core idea of ​​a Deep Q-Network is to combine the Q-Learning algorithm in reinforcement learning with a deep neural network, using a neural network to approximate the value function of a high-dimensional state space. A neural network containing multiple activation functions can achieve nonlinear function approximation, making it a very powerful function approximator.

[0098] The policy network includes a state encoding module and a Q-value output module. The state encoding module encodes the resource acquisition state into a fixed-dimensional state feature vector, and the Q-value output module takes the aforementioned state feature vector as input and outputs the Q-value of each candidate action in the action space. The state encoding module contains a convolutional neural network (CNN) layer or a recurrent neural network (RNN) layer, and the Q-value output module is a fully connected neural network layer.

[0099] In step 220 above, when the decision model to be optimized determines that the second reference URI needs to be explored, the second application URI is added to the second task queue for subsequent acquisition of the corresponding network resources.

[0100] In the embodiments described in this specification, the resource acquisition module continuously obtains URIs from the second task queue and initiates resource acquisition requests based on the URIs to obtain the corresponding network resources until the decision model to be optimized reaches a convergence state.

[0101] In step 222 above, the second application URI stored in the second task queue is obtained by the resource acquisition module, and a resource access request is initiated based on the second application URI. In this way, the resource acquisition success rate corresponding to the second application URI can be calculated, and the reward can be determined based on the resource acquisition success rate. The reward can be calculated based on a preset reward function.

[0102] In step 224 above, when optimizing the decision model to be optimized, the rewards provided by the environment are combined with the model's predictions of future returns to calculate a more forward-looking target Q value. Then, the internal parameters of the model are adjusted by continuously narrowing the gap between the model's predicted Q value and this target Q value. This process enables the model to increasingly favor actions that not only offer high immediate rewards but also bring substantial long-term returns when making decisions.

[0103] Through numerous iterations, the parameters of the decision model are continuously adjusted, and its output action value distribution or action probability distribution becomes increasingly accurate. This optimized decision model can then consistently select actions with higher expected returns. In some embodiments, the decision model can employ an online interactive training mode to continuously optimize and adjust its strategy, adapting to environmental changes.

[0104] During the reinforcement learning phase, an ε-greedy exploration strategy can be adopted. For example, the initial value of ε is set to 1.0 to encourage full exploration in the initial stage; after every 1000 decisions, ε decreases linearly by 0.05 until it reaches a minimum value of 0.1, which is used to ensure sufficient exploration in the early stage and policy convergence in the later stage.

[0105] In some embodiments, the above method may further include: 226. Obtain the resource retrieval response information corresponding to the second referenced URI.

[0106] 228. Determine the resource retrieval success rate based on the resource retrieval response information corresponding to the second reference URI.

[0107] In section 226 above, the resource acquisition response information may include at least two of the following: response status code, response time, response content size, and content parsing quality.

[0108] In some optional implementations of 228 above, the resource acquisition response information may include: response status code, response time, response content size, and content parsing quality. The resource acquisition success rate refers to the ratio of the number of reference URIs that successfully acquire available network resources out of N second application URIs to N. Here, N is a positive integer. That is, whether a request is successful depends not only on the response status code, but also on a comprehensive judgment based on the response time, response content size, and content parsing quality. Specifically, the response time reflects whether the response is timely, the response content size is used to determine whether the returned content size is within a reasonable range, and the content parsing quality is used to determine whether the content is structurally correct and complete.

[0109] Whether a usable network resource has been successfully obtained based on a specific reference URI can be determined using one or more of the following methods: Method 1: Determine if the status code is 200; determine if the response time is less than or equal to a preset time threshold; determine if the response content size is within a preset reasonable range; determine if the content parsing quality is greater than or equal to a preset quality threshold. When the status code is 200, the response time is less than or equal to the preset time threshold, the response content size is within a preset reasonable range, and the content parsing quality is greater than or equal to the preset quality threshold, it is determined that an available network resource has been successfully obtained based on the reference URI.

[0110] Method 1 uses a strict mode, which means that all dimensions must be met for it to be considered successful.

[0111] Method 2: When the status code is 200, the status code score is 1; otherwise, the status code score is 0. The response time score is determined based on the response time. For example, when the response time is less than or equal to a preset time threshold, the response time score is inversely proportional to the response time; when the response time is greater than the preset time threshold, the response time score is 0. The response content size score is determined based on the response content size. For example, when the response content size is within a preset reasonable range, the response content size score is 1; otherwise, the response content size score is 0. Then, the status code score, response time score, response content size score, and content parsing quality are weighted and summed according to pre-configured weights to obtain a comprehensive score. When the comprehensive score is greater than or equal to a preset score threshold, it is determined that an available network resource has been successfully obtained based on the reference URI.

[0112] In Method 2, the weights can be set according to actual needs, and the embodiments in this specification do not impose specific limitations on this.

[0113] In the embodiments of this specification, the accuracy of resource acquisition success rate assessment can be improved by using the multi-dimensional resource acquisition response information corresponding to the second reference URI.

[0114] In some embodiments, the reward can be determined by combining the resource acquisition success rate corresponding to the second reference URI, the computing power consumption corresponding to the second reference URI, and / or the content semantic score corresponding to the second reference URI. The computing power consumption cost corresponding to the second reference URI refers to the computational resource consumption cost (i.e., computing power cost) incurred by the resource acquisition module initiating a resource access request based on the second reference URI, including CPU consumption cost and memory consumption cost. The content semantic score is determined based on the matching degree between the second network resource and the resource acquisition request and / or the duplication degree between the second network resource and the acquired network resources. The content semantic score is directly proportional to the matching degree and inversely proportional to the duplication degree.

[0115] In one specific implementation, the reward is determined based on the resource acquisition success rate and computing power consumption cost corresponding to the second reference URI. Optionally, the reward can be obtained by weighted summing the reciprocal / negative of the resource acquisition success rate and computing power consumption cost according to a pre-set weight.

[0116] In another specific implementation, the reward is determined based on the resource acquisition success rate and content semantic score corresponding to the second reference URI. Optionally, the reward can be obtained by weighted summing of the resource acquisition success rate and content semantic score according to preset weights.

[0117] In another specific implementation, the reward is determined based on the resource acquisition success rate, computing power consumption cost, and content semantic score corresponding to the second reference URI. Optionally, the reward can be obtained by weighted summing the resource acquisition success rate, the reciprocal / negative of the computing power consumption cost, and the content semantic score according to pre-set weights.

[0118] In the embodiments of this specification, indicators such as resource acquisition success rate, resource consumption, and content semantic score are jointly optimized to avoid the local optima trap caused by a single objective. The decision model trained using the training method provided in the embodiments of this specification can achieve a good balance between efficiency, cost, and quality.

[0119] In some embodiments, the above method may further include: 230. Using preset cleaning rules, perform cleaning processing on the obtained first network resources to obtain cleaned resources.

[0120] 232. Use a pre-trained language model to parse the cleaned resources to obtain the first resource subject and the first reference URI embedded in the first network resource.

[0121] In section 230 above, the cleaning rules can be designed according to actual needs, as long as they ensure that only deterministic noise (such as scripts, styles, etc.) is deleted, and that the required content is not deleted.

[0122] In some embodiments, the cleaning rules may include a set of preset tags to be deleted, which can be used to delete the tags and their contents from the first network resource. In practical applications, some basic tags usually exist regardless of how the website version is updated. Therefore, these commonly existing tags that are irrelevant to the parsing results can be recorded in the cleaning rules, and these irrelevant noises can be removed subsequently.

[0123] In practical applications, preliminary cleaning can reduce a large amount of irrelevant data, thereby reducing the number of tokens input to the pre-trained language model and improving the parsing accuracy of the pre-trained language model.

[0124] In step 232 above, the cleaned resources can be input into a pre-trained language model so that the pre-trained language model can parse the cleaned resources to obtain the first resource subject and the first reference URI embedded in the first network resource.

[0125] For example, a pre-set prompt can be obtained, such as "I'm giving you the original text of a webpage. I've already cleaned this original text to some extent, but you need to remove all front-end language characters (HTML tags, JS scripting language configuration classes) and delete navigation information unrelated to the article, thereby extracting the article body and the embedded reference URI." The prompt and the cleaned resource are input into a pre-trained language model, which processes the cleaned resource under the prompt, thereby obtaining the resource body and the first reference URI embedded in the first network resource.

[0126] In the embodiments of this specification, the language understanding and reasoning capabilities of a pre-trained language model can improve the adaptability of the resource acquisition system to website redesigns, thereby improving the quality of content parsing and thus increasing resource acquisition efficiency.

[0127] In some embodiments, the above method may further include: 234. Based on the matching degree between the resource subject and the resource acquisition requirement, and the degree of overlap between the resource subject and other resource subjects already acquired for the resource acquisition requirement, determine whether to save the resource subject. Here, "other resource subjects already acquired" refers to resource subjects already stored in the database.

[0128] The matching degree between the resource subject and the resource acquisition demand can be semantic similarity. The similarity between the semantic vector of the resource subject and the semantic vector of the resource acquisition demand can be calculated, and this similarity is used as the semantic similarity.

[0129] The similarity between the semantic vector of a resource subject and the semantic vectors of other resource subjects that have been acquired in response to the resource acquisition requirement can be calculated, and this similarity can be determined as the repetition rate.

[0130] When the matching degree is greater than or equal to a preset matching degree threshold and the repetition degree is less than or equal to a preset repetition threshold, the resource subject is determined to be saved; when the matching degree is less than the preset matching degree or the repetition degree is greater than the preset repetition threshold, the resource subject is determined not to be saved. The preset matching degree threshold and preset repetition threshold can be set according to actual needs, and this specification does not specifically limit them in this way.

[0131] In some embodiments, a Transformer encoder, which has been pre-trained on a general web page corpus for masked language modeling, is used to semantically vectorize the resource subject (i.e., the page text) to obtain semantic vectors.

[0132] Once the resource subject is determined, it can be stored in the database.

[0133] In the embodiments of this specification, the matching degree between the resource subject and the resource acquisition requirement, and the duplication degree between the resource subject and other acquired resource subjects are analyzed to determine whether to save the resource subject in the database, thus ensuring the quality of the final resource acquisition result.

[0134] Figure 3 This is a flowchart illustrating a reinforcement learning method provided in one embodiment of this specification. Figure 3 As shown, the method may include: 300. Create a second task queue.

[0135] 302. The resource acquisition module obtains a second URI from the second task queue and obtains a second network resource based on the second URI.

[0136] 304. Obtain the second reference URI embedded in the second network resource.

[0137] 306. Based on the resource acquisition status corresponding to the second URI, determine whether it is necessary to explore the second reference URI using the decision model to be optimized.

[0138] 308. When the second reference URI needs to be explored, add the second reference URI to the second task queue for subsequent acquisition of the corresponding network resources.

[0139] 310. Determine the reward based on the resource retrieval success rate corresponding to the second reference URI.

[0140] 312. Based on the reward, optimize the decision model to be optimized.

[0141] The specific implementation of steps 300 to 312 above can be found in the corresponding contents of the above embodiments, and will not be repeated here.

[0142] In the technical solution provided in the embodiments of this specification, a task queue is created to store URIs to be explored, addressing resource acquisition needs. The resource acquisition module retrieves URLs from the task queue and initiates resource access requests based on these URLs to obtain network resources. Then, based on the resource acquisition status corresponding to the URL, a decision model based on reinforcement learning is used to determine whether a reference URL embedded in the network resource is worth exploring, thereby deciding whether to add the reference URL to the task queue for further exploration. It is evident that this solution, through reinforcement learning, can optimize the resource exploration path / direction based on real-time resource acquisition status, exhibiting good adaptability and the ability to cope with dynamically changing network environments, such as dynamically changing web pages, dynamically changing anti-acquisition strategies, and dynamically changing network connection quality. This reduces inefficient or even ineffective resource acquisition behavior, thereby improving resource acquisition efficiency.

[0143] It should be noted that any steps in the methods provided in the embodiments of this specification that are not fully described in detail can be found in the corresponding content of the above embodiments, and will not be repeated here. Furthermore, the methods provided in the embodiments of this specification may include other parts or all of the steps in the above embodiments in addition to the steps described above; for details, please refer to the corresponding content of the above embodiments, and will not be repeated here.

[0144] The following will combine Figure 4 The reinforcement learning methods provided in the embodiments of this specification are described in detail below: like Figure 4 As shown, the system involved in this reinforcement learning method includes: a URI queue 11, a task execution cluster 12, a parsing unit 13, a database 14, and a reinforcement learning driving unit 15. The URI queue is also the second task queue mentioned above. This reinforcement learning method includes the following steps: Step 1: The worker nodes in the task execution cluster 12 continuously retrieve URI_1 (that is, the second URI mentioned above) from the URI queue 11.

[0145] At the beginning, for this training, a seed list is initialized, which can contain keywords from multiple domains. Based on the seed list, multiple initial URIs are generated. These initial URIs are then added to the URI queue to complete the initialization of the URI queue.

[0146] After the worker node in the task execution cluster obtains URI_1 from the URI queue, it will send a resource access request to the target website based on URI_1 in order to obtain the response body returned by the target website.

[0147] URL queues enable important functions such as balanced task distribution, task scheduling detection, frequency control, and URI deduplication.

[0148] Among them, the task execution cluster supports distributed parallel task execution, decouples task nodes, and improves system throughput.

[0149] Step 2: The worker nodes in the task execution cluster 12 send the response body to the parsing unit 13.

[0150] Parsing unit 13 can use a pre-trained language model to parse the response body, obtaining the resource body and the reference URI embedded in the response body. The resource body can exist in a structured format.

[0151] Furthermore, after receiving the response body, the parsing unit 13 collects the resource response status corresponding to URI_1. This resource response status includes: resource acquisition response information corresponding to URI_1, exploration depth of URI_1, the website to which URI_1 belongs, the access frequency to that website, the running status of the task execution cluster, and the historical decision information of the policy network for that website. The resource acquisition response information corresponding to URI_1 includes: response status code, response time, response content size, and content parsing quality. The access frequency to that website refers to the frequency of access to that website by the task execution cluster after starting this training. The running status of the task execution cluster refers to its current running status, which may include information such as CPU utilization and memory utilization. The historical decision information of the policy network for that website refers to the decision actions and corresponding rewards performed by the policy network for web pages under that website after starting this training. Specifically, the decision actions performed by the policy network for web pages under that website are used to determine whether the reference URI embedded under that web page needs to be explored. The corresponding reward is determined based on a comprehensive assessment of the resource acquisition success rate, computing power consumption, and content semantic score corresponding to that reference URI. Optionally, the resource response status corresponding to URI_1 may also include: the content semantic score of the aforementioned resource subject. This content semantic score may be based on the matching degree between the resource subject and the keywords in the seed list and / or the duplication degree between the resource subject and the resource subjects already stored in the database DB14.

[0152] In the early stages of this training, the historical decision information in the resource response status is likely to be empty. As time goes on and more websites are visited, a large amount of historical decision information will be accumulated, increasing the probability that the historical decision information in the resource response status is not empty.

[0153] Step 3: The parsing unit 13 sends the resource response status corresponding to URI_1 to the reinforcement learning driving unit 15.

[0154] In the embodiments of this specification, before performing reinforcement learning, a state space, action space, and reward function are defined for the policy network (i.e., the decision model). The state space describes the environment, representing the situation in which the decision model exists. It describes all possible environmental states, and the set of states constitutes the state space, typically denoted as s. Actions describe the behavior of the decision model. The decision model can influence the environment by choosing actions, and the set of actions constitutes the action space, typically denoted as a. The action space describes all possible behaviors. The reward function determines the reward given to the decision model by the environment after the model performs action a in state s.

[0155] The state space contains states that correspond one-to-one with the resource response state corresponding to URI_1. The action space contains two types of actions: those that need to be explored and those that do not need to be explored.

[0156] Step 4: The reinforcement learning driving unit 15 determines the state vector based on the resource response state corresponding to URI_1 and inputs the state vector into the policy network.

[0157] The policy network can be implemented based on DQN.

[0158] Step 5: Strategy network outputs actions.

[0159] This action is used to indicate whether the reference URI embedded in the webpage corresponding to URI_1 needs to be explored.

[0160] Step 6: The reinforcement learning driving unit 15 returns the action output by the policy network to the parsing unit.

[0161] Step 7: The parsing unit 13 executes the actions output by the strategy network.

[0162] When the action is to explore, the reference URI embedded in the webpage corresponding to URI_1 is added to the URI queue for subsequent retrieval of the corresponding network resources.

[0163] When the action is "no need to explore", discard the reference URI embedded in the webpage corresponding to URI_1, that is, do not add the reference URI embedded in the webpage corresponding to URI_1 to the URI queue.

[0164] Parsing unit 13, combined with a pre-trained language model, can handle network resources of various data formats and supports dynamic URL reflow, forming a self-looping resource acquisition mechanism.

[0165] Step 8: The parsing unit determines whether to persist the resource subject in the database DB14 based on the quality score of the parsed resource subject.

[0166] The quality score of a resource subject can be determined based on the matching degree between the resource subject and the keywords in the seed list obtained during this training, and the duplication degree between the resource subject and the resource subjects already stored in the database.

[0167] Step 8 can be executed after the parsing unit 13 receives the response body sent by the task execution cluster.

[0168] Step 9, Action Feedback.

[0169] If the reference URI embedded in the webpage corresponding to URI_1 is added to the URI queue, the task execution cluster 12 retrieves the reference URI from the URI queue and initiates a resource access request based on the reference URI. After that, the parsing unit can collect the resource acquisition success rate, computing power consumption cost and content semantic score corresponding to the reference URI, and send them to the reinforcement learning driving unit.

[0170] The reinforcement learning-driven unit 15 calculates the reward by inputting the resource acquisition success rate, computational cost, and content semantic score corresponding to the referenced URI into the reward function. The reward in this paper can be understood as a short-term reward.

[0171] Step 10: Optimization.

[0172] The reinforcement learning-driven unit 15 optimizes the policy network based on rewards.

[0173] In practical applications, the reinforcement learning driving unit 15 can continuously optimize the policy network based on the continuously generated rewards until the final reward converges or the training time reaches a preset duration. The final reward convergence means that the rewards generated multiple times consecutively fluctuate within a preset interval. The preset interval can be designed according to actual needs; this specification does not specifically limit it in the embodiments.

[0174] Through trial and error training, we continuously explore optimal strategies for resource acquisition tasks, achieving dynamic adjustment and resource allocation optimization.

[0175] In the embodiments described in this specification, reinforcement learning algorithms are used to achieve intelligent and adaptive optimization of key decisions (including URL backflow decisions) for resource acquisition tasks. By combining real-time operational data and historical feedback data from the entire system, the acquisition strategy is dynamically adjusted to improve overall system efficiency and the quality of acquisition results. A distributed architecture design is used to solve the problems of limited single-machine resources and insufficient task distribution throughput, achieving elastic scaling and effective task processing. The reinforcement learning-driven unit enables dynamic optimization of the parsing process, overcoming the drawbacks of manual rule maintenance, improving the system's adaptability, and reducing cumbersome manual costs. By optimizing the URL backflow crawling strategy through reinforcement learning, and combining it with the automated parsing capabilities of the unified parsing layer for different data sources, the resource acquisition system can dynamically adjust its response to different types of data sources, avoiding inefficient task execution operations.

[0176] This patent utilizes reinforcement learning to optimize key acquisition strategies (such as URL backflow decisions), eliminating the need for traditional manual rule configuration and improving the adaptability of resource acquisition solutions. By combining state and action spaces, intelligent task execution is achieved. A short-term reward function is designed to optimize acquisition efficiency, and decision-making behavior is dynamically optimized based on URL value to avoid inefficient or redundant crawling actions. This patent proposes a reinforcement learning-driven distributed adaptive system that, through intelligent dynamic optimization and distributed collaboration, overcomes efficiency and quality bottlenecks, enhances data acquisition capabilities, and can effectively acquire massive amounts of heterogeneous data.

[0177] The embodiments in this specification solve the scalability and adaptability problems of traditional rule-driven resource acquisition by combining distributed architecture and reinforcement learning, which has technological innovation value.

[0178] One embodiment of this specification also provides a resource acquisition device, including: The first acquisition module is used to acquire resource acquisition requirements; The generation module is used to generate a first task queue for the resource acquisition requirements, and the first task queue is used to store the Uniform Resource Identifiers (URIs) to be explored. The resource acquisition module is used to obtain a first URI from the first task queue and initiate a resource access request based on the first URI to obtain a first network resource; The second acquisition module is used to acquire the first reference URI embedded in the first network resource; The determination module is used to determine whether it is necessary to explore the first reference URI based on the resource acquisition status corresponding to the first URI and a decision model optimized by reinforcement learning. An add module is used to add the first reference URI to the first task queue when the first reference URI needs to be explored, so that the corresponding network resources can be obtained later.

[0179] One embodiment of this specification also provides a reinforcement learning device, comprising: Create a module to create a second task queue; The resource acquisition module is used to obtain a second URI from the second task queue and obtain a second network resource based on the second URI; The acquisition module is used to acquire the second reference URI embedded in the second network resource; The first determining module is used to determine whether it is necessary to explore the second referencing URI based on the resource acquisition status corresponding to the second URI and using the decision model to be optimized. An add module is used to add the second reference URI to the second task queue when the second reference URI needs to be explored, so that the corresponding network resources can be obtained later. The second determining module is used to determine the reward based on the success rate of resource acquisition corresponding to the second reference URI; An optimization module is used to optimize the decision model to be optimized based on the reward.

[0180] It should be noted that the devices provided in the above embodiments can implement the technical solutions described in the corresponding method embodiments above. The specific implementation principles and corresponding beneficial effects of the above modules or units can be found in the corresponding content of the above method embodiments, and will not be repeated here.

[0181] This specification also provides an electronic device according to one embodiment. For example... Figure 5 As shown, the electronic device includes a processor 42 and a memory 41. The memory 41 stores one or more computer programs (or instructions); the processor 42 is coupled to the memory 41 and is used for the at least one or more computer programs to implement the steps in the methods provided in the embodiments of this specification.

[0182] Furthermore, the electronic device also includes other components such as a communication component 43, a display 44, a power supply component 45, and an audio component 46. Only some components are shown here for illustrative purposes, and it is not intended that the electronic device contains only these components.

[0183] The methods in this specification can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. Therefore, this specification also provides a computer program product. This computer program product includes a computer program / instructions that, when executed by an electronic component such as a processor, can perform, in whole or in part, the steps or functions of the methods provided in the embodiments of this specification. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, or other programmable device.

[0184] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0185] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0186] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0187] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0188] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium.

[0189] Accordingly, embodiments of this specification also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0190] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0191] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for embodiments such as devices, electronic devices, storage media, and program products, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0192] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0193] The above are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A method for acquiring resources, characterized in that, include: Acquire resources and acquire requirements; To meet the resource acquisition requirements, a first task queue is generated, which is used to store the Uniform Resource Identifiers (URIs) to be explored. The resource acquisition module obtains a first URI from the first task queue and initiates a resource access request based on the first URI to obtain a first network resource; Obtain the first reference URI embedded in the first network resource; Based on the resource acquisition status corresponding to the first URI, a decision model optimized by reinforcement learning is used to determine whether it is necessary to explore the first reference URI. When the first reference URI needs to be explored, the first reference URI is added to the first task queue for subsequent acquisition of the corresponding network resources.

2. The method according to claim 1, characterized in that, The reward function used by the decision model in the reinforcement learning phase is configured to generate a reward based on the resource acquisition success rate of the second reference URI, the computing power consumption cost of initiating access based on the second reference URI, and / or the content semantic score of the network resources obtained based on the second reference URI; the second reference URI refers to the reference URI that the decision model determines needs to explore in the reinforcement learning phase.

3. The method according to claim 1, characterized in that, The reinforcement learning process of the decision model includes: Create a second task queue; The resource acquisition module obtains a second URI from the second task queue and initiates a resource access request based on the second URI to obtain a second network resource; Obtain the second reference URI embedded in the second network resource; Based on the resource acquisition status corresponding to the second URI, the decision model to be optimized is used to determine whether it is necessary to explore the second reference URI; When the second reference URI needs to be explored, the second reference URI is added to the second task queue for subsequent acquisition of the corresponding network resources; The reward is determined based on the resource retrieval success rate of the second reference URI; Based on the reward, the decision model to be optimized is optimized.

4. The method according to claim 3, characterized in that, Also includes: Obtain the resource retrieval response information corresponding to the second reference URI. The resource retrieval response information includes at least two of the following: response status code, response time, response content size, and content parsing quality. The success rate of resource retrieval is determined based on the resource retrieval response information corresponding to the second reference URI.

5. The method according to claim 3, characterized in that, The reward is determined based on the resource retrieval success rate of the second reference URI, including: The reward is determined based on the resource acquisition success rate of the second reference URI and the computing power consumption cost of initiating access based on the second reference URI.

6. The method according to claim 3, characterized in that, The reward is determined based on the resource retrieval success rate corresponding to the second reference URI, including: The reward is determined based on the resource retrieval success rate corresponding to the second reference URI and the content semantic score of the second network resource; The content semantic score is determined based on the matching degree between the second network resource and the resource acquisition requirement and / or the duplication degree between the second network resource and the acquired network resource.

7. The method according to any one of claims 1 to 6, characterized in that, The resource acquisition status corresponding to the first URI includes one or more of the following: resource acquisition response information corresponding to the first URI, exploration depth of the first URI, website to which the first URI belongs, access frequency for the website, operating status of the resource acquisition module, historical decision information of the decision model for the website, and content semantic score of the resource subject parsed from the first network resource.

8. The method according to any one of claims 1 to 6, characterized in that, To address the resource acquisition requirement, a first task queue is generated, including: Obtain multiple keywords related to the resource acquisition requirement; Based on the aforementioned keywords, multiple seed URIs are determined; Create a first task queue and add the multiple seed URIs to the first task queue.

9. The method according to any one of claims 1 to 6, characterized in that, Also includes: Using preset cleaning rules, the first network resource is cleaned to obtain cleaned resources; The cleaned resources are parsed using a pre-trained language model to obtain the resource subject and the first reference URI embedded in the first network resource.

10. The method according to claim 9, characterized in that, Also includes: Whether to save the resource subject is determined based on the matching degree between the resource subject and the resource acquisition requirement and the degree of overlap between the resource subject and other resource subjects already acquired for the resource acquisition requirement.

11. The method according to any one of claims 1 to 6, characterized in that, The resource acquisition module includes a cluster, which includes multiple worker nodes that act as consumers of the first task queue.

12. A reinforcement learning method, characterized in that, include: Create a second task queue; The resource acquisition module obtains the second URI from the second task queue and acquires the second network resource based on the second URI; Obtain the second reference URI embedded in the second network resource; Based on the resource acquisition status corresponding to the second URI, the decision model to be optimized is used to determine whether it is necessary to explore the second reference URI; When the second reference URI needs to be explored, the second reference URI is added to the second task queue for subsequent acquisition of the corresponding network resources; The reward is determined based on the success rate of resource retrieval corresponding to the second reference URI; Based on the reward, the decision model to be optimized is optimized.

13. A resource acquisition device, characterized in that, include: The first acquisition module is used to acquire resource acquisition requirements; The generation module is used to generate a first task queue for the resource acquisition requirements, and the first task queue is used to store the Uniform Resource Identifiers (URIs) to be explored. The resource acquisition module is used to obtain a first URI from the first task queue and initiate a resource access request based on the first URI to obtain a first network resource; The second acquisition module is used to acquire the first reference URI embedded in the first network resource; The determination module is used to determine whether it is necessary to explore the first reference URI based on the resource acquisition status corresponding to the first URI and a decision model optimized by reinforcement learning. An add module is used to add the first reference URI to the first task queue when the first reference URI needs to be explored, so that the corresponding network resources can be obtained later.

14. A reinforcement learning device, characterized in that, include: Create a module to create a second task queue; The resource acquisition module is used to obtain a second URI from the second task queue and obtain a second network resource based on the second URI; The acquisition module is used to acquire the second reference URI embedded in the second network resource; The first determining module is used to determine whether it is necessary to explore the second referencing URI based on the resource acquisition status corresponding to the second URI and using the decision model to be optimized. An add module is used to add the second reference URI to the second task queue when the second reference URI needs to be explored, so that the corresponding network resources can be obtained later. The second determining module is used to determine the reward based on the success rate of resource acquisition corresponding to the second reference URI; An optimization module is used to optimize the decision model to be optimized based on the reward.

15. An electronic device, characterized in that, include: Memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method of any one of claims 1 to 12.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a computer, it can implement the method of any one of claims 1 to 12.

17. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 12.

Images