Small language webpage self-adaptive collection method and device based on large language model

Abstract

This application relates to the field of artificial intelligence technology and discloses an adaptive acquisition method and apparatus for minority language web pages based on a large language model. The method includes: loading and rendering a target minority language web page through a headless browser to obtain a document object model tree structure; parsing and semantically annotating the document object model tree structure based on the large language model, identifying dynamic content nodes and generating adaptive content location rules; semantically vectorizing the content of at least two consecutively acquired web pages based on the adaptive content location rules and calculating semantic similarity; dynamically adjusting the initiation frequency of subsequent acquisition requests according to a preset frequency mapping rule based on semantic similarity to generate an acquisition strategy; executing the acquisition strategy to access the target web page, and injecting an operation sequence to simulate human interaction behavior during the acquisition process. This application can achieve complete acquisition of dynamic content of minority language web pages and ensure the continuous and stable acquisition process.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method and apparatus for adaptive collection of web pages in minority languages ​​based on a large language model. Background Technology

[0002] With the deepening of global industrial expansion overseas, the commercial data value of markets with less common languages ​​such as Southeast Asia and the Middle East continues to be released. Web crawling has become a core technical means to obtain such data across regions and is widely used in market research, business analysis and other scenarios.

[0003] Currently, existing web crawlers generally use preset fixed location rules to crawl web page content, and the crawling frequency is relatively fixed, lacking flexibility in operation. However, in the scenario of crawling web pages in less commonly spoken languages, existing technologies have some shortcomings. These web pages often present their core content through dynamic loading technology, and their Document Object Model (DOM) structure has dynamic and diverse characteristics. Fixed location rules are difficult to adapt to this type of structure, resulting in incomplete crawling of dynamic content. At the same time, the fixed-frequency crawling mode is prone to triggering the anti-crawling mechanism of the target website, and the lack of human-like operation behavior further increases the risk of IP blocking, affecting the continuity of crawling. It is evident that existing technologies cannot simultaneously achieve complete crawling of dynamic content from web pages in less commonly spoken languages ​​and a continuous and stable crawling process, making it difficult to meet the actual needs of data collection in the less commonly spoken language market.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] This application provides an adaptive collection method and apparatus for minority language web pages based on a large language model. It can achieve complete collection of dynamic content of minority language web pages, and at the same time reduce the risk of anti-crawling triggering by dynamically adjusting the collection frequency and simulating human interaction behavior, thus ensuring the continuous and stable collection process.

[0006] In a first aspect, embodiments of this application provide an adaptive data collection method for minority language web pages based on a large language model, including:

[0007] Load and render the target language webpage using a headless browser, and obtain the corresponding document object model tree structure;

[0008] Based on the large language model, the document object model tree structure is parsed and semantically annotated to identify dynamic content nodes and generate corresponding adaptive content positioning rules.

[0009] Based on the adaptive content location rule, semantic vectorization is performed on at least two consecutively collected web page contents, and the corresponding semantic similarity is calculated.

[0010] Based on the preset frequency mapping rules of the semantic similarity query, the frequency of subsequent collection requests is dynamically adjusted to generate a collection strategy;

[0011] The acquisition strategy is executed to access the target webpage, and an operation sequence for simulating human interaction behavior is injected during the acquisition process.

[0012] Furthermore, in some embodiments of this application, the step of parsing and semantically annotating the document object model tree structure based on a large language model, identifying dynamic content nodes, and generating corresponding adaptive content positioning rules includes:

[0013] The document object model tree structure is input into a pre-trained multilingual large language model;

[0014] The structured description and semantic annotation information of the nodes in the document object model tree are obtained through the large language model. The semantic annotation information includes at least the node function category and content language tag.

[0015] Based on the structured description and the semantic annotation information, identify target nodes related to dynamic content loading;

[0016] Based on the path information and attribute characteristics of the target node, adaptive rules for locating the content of the target node are generated or updated.

[0017] Furthermore, in some embodiments of this application, the generation or updating of adaptive rules for locating target node content includes:

[0018] Based on the path information of the target node, construct an initial location path expression;

[0019] Based on the node function categories in the semantic annotation information, the initial localization path expression is generalized or specialized to form the adaptive rule;

[0020] The adaptive rules and the content language tags are associated and stored in the rule base.

[0021] Furthermore, in some embodiments of this application, the step of semantically vectorizing at least two consecutively collected webpage contents based on the adaptive content localization rule and calculating the corresponding semantic similarity includes:

[0022] The continuously collected web page content is input into the cross-language semantic coding model;

[0023] The cross-language semantic coding model is used to obtain vector representations of the content of each webpage in a unified semantic space;

[0024] Calculate the cosine distance between at least two of the vector representations, and use the cosine distance as a measure of the semantic similarity.

[0025] Furthermore, in some embodiments of this application, the step of dynamically adjusting the initiation frequency of subsequent collection requests and generating a collection strategy based on the preset frequency mapping rules of the semantic similarity query includes:

[0026] At least two semantic similarity threshold intervals are preset, and each threshold interval is associated with a collection frequency level;

[0027] The calculated semantic similarity is matched with the threshold range to determine the collection frequency level to which the semantic similarity belongs;

[0028] Based on the determined collection frequency level, the request time interval for the next batch of collection tasks is adjusted to generate the collection strategy.

[0029] Furthermore, in some embodiments of this application, the execution of the collection strategy to access the target webpage, and the injection of an operation sequence to simulate human interaction behavior during the collection process, includes:

[0030] Based on the request time interval in the acquisition strategy, a acquisition schedule containing random delays is generated;

[0031] Construct a set of script instructions that include at least one of mouse movement events, scrolling events, or click events;

[0032] After the headless browser initiates a request and loads the page according to the collection schedule, it executes the script instruction set to extract the corresponding page content.

[0033] Furthermore, in some embodiments of this application, the step of executing the collection strategy to access the target webpage and injecting an operation sequence to simulate human interaction behavior during the collection process further includes:

[0034] The jurisdiction of the target webpage is determined based on its domain name or Internet Protocol address, and the corresponding set of data compliance clauses is loaded.

[0035] Parse the set of data compliance clauses and extract the defined sensitive data patterns;

[0036] The collected raw data is matched with the sensitive data pattern in real time, and the sensitive data that is successfully matched is filtered or de-identified.

[0037] Furthermore, in some embodiments of this application, the step of performing real-time matching between the collected raw data and the sensitive data pattern, and filtering or de-identifying the successfully matched sensitive data, includes:

[0038] Regular expressions or keyword matching algorithms are used to compare the text fields in the original data with the sensitive data patterns.

[0039] When the comparison is successful, the corresponding field is processed according to the processing requirements in the data compliance clause set. The processing operation includes at least one of the following: replacement operation, masking operation, or deletion operation.

[0040] Record the operation type, target field, and timestamp of the disposal operation to generate a compliance audit log.

[0041] Furthermore, in some embodiments of this application, before parsing and semantically annotating the document object model tree structure based on the large language model, identifying dynamic content nodes, and generating corresponding adaptive content positioning rules, the method further includes:

[0042] Obtain a training set of web page corpus for the target minority language, wherein the training set of web page corpus includes web page text and the structural annotations of the corresponding basic document object model;

[0043] The basic multilingual large language model is incrementally trained using the web page corpus training set to optimize the semantic understanding ability and node classification ability of the target minority language web page structure of the basic multilingual large language model, thereby obtaining an enhanced large language model.

[0044] The document object model tree structure is input into the enhanced large language model.

[0045] Secondly, embodiments of this application provide a device for adaptively collecting web pages in minority languages ​​based on a large language model, including:

[0046] The web page rendering module is used to load and render target language web pages through a headless browser and obtain the corresponding document object model tree structure.

[0047] The rule generation module is used to parse and semantically annotate the document object model tree structure based on the large language model, identify dynamic content nodes, and generate corresponding adaptive content positioning rules.

[0048] The semantic analysis module is used to perform semantic vectorization on at least two consecutively collected web page contents based on the adaptive content positioning rules, and calculate the corresponding semantic similarity.

[0049] The strategy generation module is used to dynamically adjust the initiation frequency of subsequent collection requests based on the semantic similarity query preset frequency mapping rules, and generate a collection strategy.

[0050] The collection module is used to execute the collection strategy to access the target webpage and inject operation sequences to simulate human interaction behavior during the collection process.

[0051] This application provides an adaptive content acquisition method and apparatus for minority language web pages based on a large language model. First, a headless browser loads and renders the target minority language web page and obtains its document object model (DOM) tree structure. Combining the large language model's analysis and semantic annotation of this structure, dynamic content nodes can be accurately identified and adaptive content location rules can be generated. This overcomes the limitation of traditional fixed rules being difficult to adapt to the dynamic structure of minority language web pages, effectively improving the completeness of dynamic content acquisition. Second, based on the adaptive location rules, semantic vectorization and similarity calculation are performed on continuously acquired web page content, and the acquisition frequency is dynamically adjusted to avoid anti-crawling triggers caused by high-frequency invalid requests. Simultaneously, by injecting operation sequences simulating human interaction behavior during the acquisition process, the probability of triggering the target website's anti-crawling mechanism is further reduced, minimizing the risk of IP blocking. In summary, this application uses a headless browser to render the target web page to obtain its DOM tree structure, parses this structure using a large language model to generate adaptive content location rules, dynamically adjusts the acquisition frequency based on semantic similarity calculation, and injects operation sequences simulating human interaction during the acquisition process. Ultimately, this achieves complete crawling of dynamic content from minority language web pages, effectively reducing the risk of anti-crawling triggers and ensuring continuous and stable acquisition. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an application environment diagram of the adaptive webpage acquisition method for minority languages ​​based on a large language model provided in the embodiments of this application;

[0054] Figure 2 This is a flowchart illustrating the adaptive acquisition method for minority language web pages based on a large language model provided in this application embodiment;

[0055] Figure 3 This is a schematic diagram of the structure of the adaptive webpage acquisition device for minority languages ​​based on a large language model provided in this application embodiment;

[0056] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of systems and methods consistent with those detailed in the appended claims or with some aspects of this application.

[0058] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover descriptions such as non-exclusive inclusion, so that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0059] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0060] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0061] To address the aforementioned technical problems and overcome the shortcomings of existing technologies, this application provides an adaptive data collection method and apparatus for minority language web pages based on a large language model. This method enables the complete collection of dynamic content from minority language web pages and reduces the risk of anti-crawling triggers by dynamically adjusting the collection frequency and simulating human interaction behavior, thereby ensuring the continuous and stable collection process.

[0062] Figure 1 This is an application environment diagram of a method for adaptive webpage acquisition in minority languages ​​based on a large language model, as illustrated in one embodiment. (Refer to...) Figure 1This adaptive acquisition method for minority language web pages based on a large language model is applied to a small language web page adaptive acquisition system based on a large language model. The system includes a terminal 110 and a server 120. The terminal 110 and server 120 are connected via a network. The terminal 110 can be a desktop terminal or a mobile terminal, specifically a mobile phone, tablet, or laptop. The server 120 can be a standalone server or a server cluster consisting of multiple servers. The server 120 is configured to execute the aforementioned adaptive acquisition method for minority language web pages based on a large language model, including: loading and rendering the target minority language web page through a headless browser to obtain the corresponding document object model tree structure; parsing and semantically annotating the document object model tree structure based on the large language model, identifying dynamic content nodes and generating corresponding adaptive content positioning rules; semantically vectorizing at least two consecutively acquired web page contents based on the adaptive content positioning rules and calculating the corresponding semantic similarity; querying a preset frequency mapping rule based on the semantic similarity to dynamically adjust the initiation frequency of subsequent acquisition requests and generate an acquisition strategy; executing the acquisition strategy to access the target web page, and injecting an operation sequence to simulate human interaction behavior during the acquisition process.

[0063] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for adaptively collecting web pages in minority languages ​​based on a large language model, according to an embodiment of this application. This embodiment primarily uses the application of this method to a computer device as an example. Specifically, the method for adaptively collecting web pages in minority languages ​​based on a large language model can include the following steps:

[0064] S1. Load and render the target language webpage using a headless browser, and obtain the corresponding document object model tree structure;

[0065] Specifically, for step S1, the headless browser is a web browsing tool that does not require a visual interface. It can simulate the web page loading process of a regular browser, including parsing HTML, CSS, and JavaScript code and handling dynamic interaction logic. In scenarios involving the collection of web pages in less commonly spoken languages, such as Thai and Vietnamese, after initiating an access request through this browser, it will fully load all resources of the web page, including dynamically loading the required script files and data interface response content. It will also organize the structure of the web page into a Document Object Model (DOM) tree. The DOM tree presents all elements of the web page in the form of hierarchical nodes, with each node corresponding to a component on the web page (such as a title, button, product list, text content, etc.). For example, when accessing a product category page on a Vietnamese e-commerce platform, the headless browser will automatically load the initial content of the page and the subsequent product data dynamically obtained through Ajax requests, ultimately generating a complete DOM tree containing "product name nodes," "price nodes," "view more button nodes," etc., ensuring that all structural information of the web page can be captured subsequently.

[0066] S2. Based on the large language model, the document object model tree structure is parsed and semantically annotated to identify dynamic content nodes and generate corresponding adaptive content positioning rules;

[0067] Specifically, for step S2, the large language model possesses the ability to parse structured data and understand cross-language semantics. After inputting the acquired DOM tree, the model analyzes the node attributes (such as ID, class name, and tag type), hierarchical relationships, and associated text content level by level. Simultaneously, it semantically annotates the nodes, clarifying the functional category and content language attributes of each node. Functional categories include dynamic content triggering nodes, product detail display nodes, and promotional information nodes, while content language attributes include Thai content nodes and Vietnamese content nodes. Based on the above parsing and annotation results, the model accurately identifies target nodes related to dynamic content loading, such as scrolling nodes that trigger lazy loading or click button nodes that trigger loading more content. Combining the node's path features, attribute features, and semantic information, it generates adaptive content positioning rules. These rules differ from fixed positioning expressions and can adapt to the dynamic changes in the DOM structure of minority language web pages, eliminating the need for manual reconfiguration. For example, when dealing with dynamic content related to "promotions" on Thai e-commerce web pages, the large language model, after parsing the DOM tree, will identify the "promotion information list node" and the "view more" button node that triggers the loading of more promotional content. It will then generate an adaptive positioning rule that includes node semantic features and hierarchical paths. Even if the web page DOM structure is slightly adjusted, this rule can still accurately locate the promotion-related dynamic content.

[0068] S3. Based on adaptive content location rules, semantic vectorize the content of at least two consecutively collected web pages and calculate the corresponding semantic similarity.

[0069] Specifically, for step S3, firstly, based on the generated adaptive content positioning rules, core content, such as product descriptions, promotional information, and service introductions, is accurately extracted from multiple continuously collected web pages in various languages, ensuring the relevance and effectiveness of the extracted content. Then, semantic encoding technology is used to convert the extracted language text content into a computer-computable vector form. This vector maps the semantic features of the text, with different semantic content corresponding to different vector space positions. Finally, a vector distance calculation method is used to compare the vectors corresponding to at least two continuously collected web page contents to obtain a semantic similarity value. This value directly reflects the degree of semantic association between the two pieces of content; the higher the value, the more similar the content semantics; the lower the value, the greater the semantic differences. For example, two product pages from a Thai e-commerce platform are continuously collected, and the core content extracted using adaptive positioning rules is "new product, red, low price" and "new product, pink, promotion," respectively. After semantic vectorization, the calculated similarity is 82%, indicating that the two pieces of content are highly semantically related.

[0070] S4. Based on the semantic similarity query, the preset frequency mapping rules are used to dynamically adjust the frequency of subsequent collection requests and generate a collection strategy;

[0071] Specifically, for step S4, the preset frequency mapping rule is an association rule based on the characteristics of anti-crawling mechanisms for minority language web pages. It pre-defines the matching relationship between different semantic similarity intervals and corresponding collection frequencies. For example, when semantic similarity is high, the collection frequency is reduced to avoid invalid high-frequency requests; when semantic similarity is low, the collection frequency is maintained or moderately increased to ensure coverage of more differentiated content. After obtaining the semantic similarity, the preset rule is queried to determine the collection frequency level corresponding to the current similarity. This allows for adjustments to parameters such as the initiation time interval of subsequent collection requests and the number of requests per batch, forming a specific and executable collection strategy. This strategy can dynamically adapt to the semantic association of web page content, rather than using a fixed collection frequency. For example, if the preset rule "semantic similarity ≥ 80%" corresponds to a collection frequency of 10 times / minute, and the semantic similarity between two consecutive web pages is calculated to be 82%, then after querying the rule, the subsequent collection frequency is adjusted to 10 times / minute, generating a specific strategy of "initiating a collection request every 6 seconds, collecting 5 web pages per batch."

[0072] S5. Execute the collection strategy to access the target webpage, and inject an operation sequence to simulate human interaction behavior during the collection process;

[0073] Specifically, for step S5, according to the generated collection strategy, headless browsers initiate access requests to target language web pages at an adjusted frequency to ensure that the collection behavior conforms to the time interval and batch requirements set by the strategy. Simultaneously, during web page loading and content extraction, a sequence of operations simulating real human behavior is injected. This sequence simulates typical user browsing behavior, making the collection process closer to human operation patterns and reducing the probability of being detected by anti-scraping systems. For example, when executing a collection strategy targeting a Middle Eastern Arabic e-commerce platform, access is initiated at a frequency of 15 times per minute. During page loading, an operation sequence of "randomly moving the mouse (irregular trajectory) → pausing for 1-2 seconds → scrolling down 30% of the page → pausing for 0.8 seconds → clicking the pagination button" is injected to simulate the browsing interaction process of a real user, avoiding triggering the platform's anti-scraping mechanism.

[0074] This embodiment can accurately capture dynamic content from web pages in less commonly spoken languages, solving the problem that traditional collection methods are difficult to adapt to dynamic DOM structures. Furthermore, through semantically driven frequency adjustment and anthropomorphic interactive simulation, it can effectively reduce the risk of anti-crawling mechanisms being triggered, ensure the continuous and stable collection process, and meet the core needs of data collection in the less commonly spoken language market.

[0075] Furthermore, in some embodiments, step S2, "parse and semantically annotate the document object model tree structure based on the large language model, identify dynamic content nodes, and generate corresponding adaptive content positioning rules," may specifically include:

[0076] S21. Input the document object model tree structure into the pre-trained multilingual large language model;

[0077] Specifically, for step S21, the pre-trained multilingual large language model possesses cross-language processing capabilities and structured data parsing capabilities, enabling it to be compatible with the semantic features and webpage structural features of various minority languages ​​such as Thai, Vietnamese, and Arabic. The Document Object Model (DOM) tree is input into the model in a structured data format (such as hierarchical node data, attribute key-value pair sets, etc.). The model can directly read the hierarchical relationships, tag types, attribute information (such as ID, class name, text content), and other core data of the nodes without additional format conversion for subsequent analysis. For example, for the DOM tree of a product details page on a Thai e-commerce platform, which contains hierarchical structures such as "product image nodes," "price nodes," "promotional information nodes," and "view more button nodes," inputting this DOM tree into the pre-trained multilingual large language model in a standardized structured data format allows the model to quickly identify the basic information and relationships of each node.

[0078] S22. Obtain the structured description and semantic annotation information of the nodes in the document object model tree through the large language model. The semantic annotation information shall include at least the node function category and content language label.

[0079] Specifically, for step S22, after the large language model performs node-by-node deep parsing of the input DOM tree, it outputs two core pieces of information. The first is a structured description of the node, which clearly presents the node's hierarchical path in the DOM tree, attribute features (such as node ID, class name, tag type, associated JavaScript events), and other structured data that can be used for location. The second is semantic annotation information, in which the node function category is used to clarify the role of the node, such as a dynamic content loading trigger node, a product core information display node, a pagination navigation node, etc., and the content language tag is used to identify the language type of the content carried by the node, such as Thai, Vietnamese, Arabic, etc.

[0080] For example, after parsing the DOM tree of a Vietnamese e-commerce webpage, the model outputs a structured description for a specific button node as "body>div[class='goods-list']>button[class='load-more']", with semantic annotation information as "node function category: dynamic content loading trigger node; content language tag: Vietnamese", clearly defining the node's positioning information and core attributes.

[0081] S23. Identify target nodes related to dynamic content loading based on structured descriptions and semantic annotation information;

[0082] Specifically, for step S23, combining the acquired structured description and semantic annotation information, the target node is accurately identified through logical filtering. First, based on the node function category in the semantic annotation information, candidate nodes related to dynamic content, such as nodes marked as dynamic content loading trigger nodes or dynamic data receiving nodes, are initially filtered out. Then, the hierarchical path and attribute features in the structured description are further verified, such as whether it is associated with Ajax request events or whether it is bound to a scroll loading function, to eliminate misjudged function annotations or irrelevant nodes, and finally determine the target node directly related to dynamic content loading.

[0083] For example, in the DOM tree of a Vietnamese webpage, multiple nodes are marked as nodes that trigger dynamic content loading. Through structured description analysis, it was found that one of the nodes has the path "body>div[class='content']>button" and is associated with the JavaScript event "onclick=loadMoreGoods()". The other candidate nodes are not bound to dynamic loading-related events. Therefore, this node is determined to be the target node related to dynamic content loading.

[0084] S24. Based on the path information and attribute characteristics of the target node, generate or update adaptive rules for locating the content of the target node;

[0085] Specifically, in step S24, the identified target node is the core element, and its key features are extracted for rule construction. These features include: first, path information, i.e., the node's hierarchical path in the DOM tree, which serves as the basic basis for rule localization; and second, attribute features, including the node's ID, class name, functional semantic tags, and associated events, which serve as supplementary adaptation criteria for the rules. Rules generated based on these features have adaptive capabilities. When the webpage DOM structure undergoes slight adjustments (such as minor changes in the hierarchical path or minor adjustments to the class name), the rules can still accurately locate the target node through attribute features and semantic associations. If changes to the target node's features are detected during subsequent data collection, the rules will be updated based on the new path information and attribute features to ensure effective localization.

[0086] For example, for a specific Vietnamese webpage's target node for loading more products, its path information is "body>div[class='content']>button", and its attribute features are "class='load-more', functional semantic tag: dynamic content loading trigger, and associated event: loadMoreGoods()". The adaptive rule generated based on this information will include the core path fragment, key attributes, and semantic features. If the path of this node changes to "body>div[class='new-content']>button" after the webpage is subsequently updated, but the core attributes and semantic features remain unchanged, the rule can automatically adapt to the new path and still accurately locate the node. If the attribute features change at the same time, the corresponding feature items in the rule will be updated.

[0087] This embodiment, through precise parsing and semantic annotation of a multilingual large language model, can efficiently identify target nodes related to dynamic content in web pages in less commonly spoken languages. The generated adaptive positioning rules can adapt to the dynamic changes in the web page DOM structure, improving the accuracy and adaptability of dynamic content positioning in web pages in less commonly spoken languages, and providing a reliable guarantee for the subsequent complete collection of dynamic content.

[0088] Furthermore, in some embodiments, step S24, "generating or updating adaptive rules for locating the content of the target node," may specifically include:

[0089] S241. Construct an initial location path expression based on the path information of the target node;

[0090] Specifically, for step S241, the path information of the target node refers to its complete hierarchical trajectory in the Document Object Model (DOM) tree, including the hierarchical relationship of all parent and child nodes from the root node to the target node, as well as node attributes (such as tag type, class name, ID, etc.). The path expression is a structured syntax used to precisely point to the target node, capable of locking the node position based on hierarchical relationships and attribute characteristics. When constructing the initial expression based on the above path information, the core hierarchical structure and key attributes of the target node are fully preserved, ensuring that the expression can directly locate the target node in the current DOM tree.

[0091] S242. Based on the node function categories in the semantic annotation information, the initial localization path expression is generalized or specialized to form adaptive rules;

[0092] Specifically, for step S242, the node function category defines the core function of the target node, such as a product dynamic list loading node, a promotional information triggering node, and a pagination navigation loading node. The corresponding processing method is selected based on this category, including generalization or specialization. When the node function category is general, meaning that the same type of functional node may have minor adjustments to its hierarchical path and changes in non-core attributes on different language web pages, but the essential function remains the same, easily variable non-core constraints in the initial expression are removed, such as secondary class names and non-critical hierarchical nodes, while retaining the core functional association characteristics, allowing the rule to adapt to more web pages with the same function but different structures. When the node function category is specific, meaning that the node function only exists on a specific platform and on a specific type of web page, and its path and attribute characteristics are unique, specific constraints are added to the initial expression, such as a platform-specific ID and unique semantic association attributes, to enhance the accuracy of the rule and avoid positioning confusion.

[0093] S243. Associate and store the adaptive rules and content language tags in the rule base;

[0094] Specifically, in step S243, the content language tag is used to specify the language type of the content carried by the target node, such as Thai, Vietnamese, or Arabic. Associated storage refers to binding the formed adaptive rules with this language tag, so that each rule in the rule base corresponds to a specific applicable language scenario. The rule base has the functions of categorized storage and fast retrieval, and can be indexed by language type and node function category, making it convenient to quickly call the matching rules when collecting web pages in corresponding minority languages. For example, the generalized Vietnamese rule " / / button[contains(@class,'load-more')]" for "product dynamic list loading node" is associated with the "Vietnamese" tag and stored in the "Vietnamese - Dynamic Loading Class" directory of the rule base; the specialized Thai rule for "limited-time promotion loading node" is associated with the "Thai" tag and stored in the "Thai - Exclusive Promotion Class" directory, so that it can be directly retrieved and called when collecting web pages in the corresponding language and type.

[0095] This embodiment accurately constructs the initial positioning path expression and performs targeted generalization or specialization processing based on the node function category. This enables the generated adaptive rules to have both cross-scene adaptability and ensure positioning accuracy. At the same time, it achieves efficient reuse of rules through language tag association storage, providing flexible and reliable support for the stable positioning of dynamic content on web pages in less common languages.

[0096] Furthermore, in some embodiments, step S3, "based on adaptive content localization rules, semantically vectorizing the content of at least two consecutively collected web pages and calculating the corresponding semantic similarity," may specifically include:

[0097] S31. Input the continuously collected web page content into the cross-language semantic coding model respectively;

[0098] Specifically, in step S31, the continuously collected webpage content refers to the core effective information extracted from the target minority language webpage based on adaptive content positioning rules, such as product descriptions, promotional copy, and service instructions. This content can involve different minority languages ​​such as Thai, Vietnamese, and Arabic. The cross-language semantic coding model has the processing capability to overcome language barriers, is compatible with the semantic features of multiple minority languages, and does not require additional language conversion preprocessing. It can directly receive the original text content of different languages ​​as input, laying the foundation for subsequent vector conversion of a unified semantic space. For example, continuously collecting two product pages from a Thai e-commerce platform, the extracted core content is Thai phrases such as "new product, red, warranty, 1 year" and "new product, pink, free shipping." These two Thai texts are directly input into the cross-language semantic coding model.

[0099] S32. Obtain the vector representation of each webpage's content in a unified semantic space through a cross-language semantic coding model;

[0100] Specifically, in step S32, the cross-language semantic coding model performs deep semantic analysis on the input minority language text, extracting the core semantic features of the text, such as the product status features corresponding to "new product", the color features corresponding to "red" and "pink", and the service features corresponding to "warranty" and "free shipping". These semantic features are then mapped into a unified high-dimensional semantic space, forming a fixed-dimensional vector representation. This vector is the digital carrier of semantic features. The semantic differences between different texts are reflected through the differences in the position of the vectors in the space. Moreover, regardless of the original language of the text, semantically similar content will form vectors with similar positions in the unified semantic space. For example, after the two Thai texts above are processed by the model, vectors A ([0.82,0.15,0.79,0.33,...]) and B ([0.81,0.17,0.21,0.85,...]) are generated respectively. Among them, the dimension related to "warranty" in vector A has a higher value, and the dimension related to "free shipping" in vector B has a higher value. However, the dimension related to "new product" in both is similar, which reflects partial semantic correlation.

[0101] S33. Calculate the cosine distance between at least two vector representations and use the cosine distance as a measure of semantic similarity;

[0102] Specifically, for step S33, cosine distance is the core indicator for measuring the directional similarity between two vectors in a unified semantic space. Its calculation is based on the ratio of the dot product to the magnitude of the vectors, with the result ranging from [0,1]. Specifically, when two vectors are completely aligned (semantically identical), the cosine distance is 1; when two vectors are completely perpendicular (semantically unrelated), the cosine distance is 0. The closer the cosine distance value is to 1, the higher the semantic similarity of the corresponding webpage content; conversely, the greater the semantic difference, the closer it is to 1. Therefore, this value can be directly used as the quantitative result of semantic similarity. For example, calculating the cosine distance between vectors A and B yields a result of 0.83, indicating that the semantic similarity between the two Thai webpage contents is 83%, reflecting that both revolve around "new products," but differ in additional services (warranty / free shipping), indicating a high overall semantic correlation.

[0103] This embodiment utilizes a cross-language semantic coding model to achieve unified semantic mapping of web page content in different minority languages. It accurately quantifies semantic similarity through cosine distance, providing an objective and reliable basis for subsequent semantic-based adjustment of collection frequency, effectively solving the problem of difficulty in measuring similarity caused by semantic differences in minority languages.

[0104] Furthermore, in some embodiments, step S4, "dynamically adjusting the initiation frequency of subsequent collection requests based on semantic similarity queries and predefined frequency mapping rules, and generating a collection strategy," may specifically include:

[0105] S41. Preset at least two semantic similarity threshold intervals, each threshold interval being associated with a collection frequency level;

[0106] Specifically, for step S41, the semantic similarity threshold range is a pre-defined range of semantic similarity values ​​based on the characteristics of the content of web pages in minority languages ​​and the anti-crawling mechanisms of the target website. This range is used to distinguish the degree of semantic association between web page content. The collection frequency level is a collection request initiation density standard that corresponds one-to-one with the threshold range. The core logic is that the higher the semantic similarity, the stronger the repetition or association of the web page content, and the lower the collection frequency can be to avoid invalid high-frequency requests; the lower the semantic similarity, the greater the difference in web page content, and the collection frequency can be appropriately increased to cover more effective information. At least two ranges need to be pre-defined to ensure basic frequency differentiation adjustments can be achieved. Each range is bound to a specific collection frequency parameter, such as the number of requests per unit time. For example, for the Thai e-commerce platform data collection scenario, three semantic similarity threshold ranges and corresponding collection frequency levels are preset: ① Similarity ≥ 85% (high relevance range), corresponding to the collection frequency level of "low frequency" (10 times / minute); ② 60% ≤ similarity < 85% (medium relevance range), corresponding to the collection frequency level of "medium frequency" (30 times / minute); ③ Similarity < 60% (low relevance range), corresponding to the collection frequency level of "high frequency" (50 times / minute), forming a complete preset mapping relationship.

[0107] S42. Match the calculated semantic similarity with the threshold range to determine the collection frequency level to which the semantic similarity belongs;

[0108] Specifically, for step S42, the specific numerical value of the semantic similarity of the continuous web page content calculated in the previous step is first obtained. Then, this value is compared with each preset threshold interval to determine which interval the value falls into, thereby locking in the collection frequency level associated with that interval, completing the correspondence between semantic similarity and frequency level, and providing a clear basis for subsequent frequency adjustment. For example, if two web pages of a product category page in Vietnamese are continuously collected, and the calculated semantic similarity is 87%, after comparing this value with the preset interval, it is determined that it belongs to the high-association interval of "similarity ≥ 85%", and the corresponding collection frequency level is "low frequency" (10 times / minute).

[0109] S43. Based on the determined collection frequency level, adjust the request time interval of the next batch of collection tasks to generate a collection strategy;

[0110] Specifically, for step S43, the collection frequency level and the request time interval are inversely related: the higher the frequency, the shorter the time interval; the lower the frequency, the longer the time interval. Based on the determined frequency level, the time interval between two adjacent requests in the next batch of collection tasks is calculated and adjusted. At the same time, key parameters such as the number of web pages to be collected in the batch and the order in which requests are initiated are clarified to form a directly executable collection strategy. This strategy adapts the collection frequency to the semantic relevance of the web page content through dynamic adjustment of the time interval. For example, if the determined collection frequency level is "low frequency" (10 times / minute), the calculated request time interval is 60 seconds ÷ 10 times = 6 seconds, and the generated collection strategy is "collect 20 web pages in the next batch, initiate requests sequentially at a time interval of 6 seconds / time, and prioritize collecting the subpages not covered under the category page"; if the level is "high frequency" (50 times / minute), the time interval is adjusted to 1.2 seconds / time, and the corresponding strategy is "collect 50 web pages in the next batch, initiate requests at an interval of 1.2 seconds / time, and cover more differentiated product pages".

[0111] This embodiment achieves dynamic adaptation of collection frequency based on semantic similarity by associating preset threshold ranges with frequency levels. This effectively reduces invalid high-frequency requests, lowers the risk of triggering anti-crawling mechanisms, and ensures the collection efficiency of differentiated content while taking into account the stability and comprehensiveness of collecting web pages in less common languages.

[0112] Furthermore, in some embodiments, step S5, "execute a collection strategy to access the target webpage and inject an operation sequence to simulate human interaction behavior during the collection process," may specifically include:

[0113] S51. Generate a collection schedule including random delays based on the request time interval in the collection strategy;

[0114] Specifically, for step S51, the request time interval in the collection strategy is a preset base time parameter used to control the density of collection requests. To avoid the collection behavior exhibiting a mechanical regularity (easily identified by anti-scraping systems), a random delay is introduced on top of this time interval. The value range of the random delay needs to match the pausing habits of humans browsing web pages, ensuring that the overall collection rhythm conforms to the logic of human operation. The collection schedule is a specific execution plan that integrates the base time interval and the random delay, clearly defining the initiation time of each collection request, so that the request initiation time both follows the strategy framework and has uncertainty. For example, if the request time interval in the collection strategy is 6 seconds, and the random delay range is set to 0.5-2 seconds, then in the generated collection schedule, the initiation time of each request may be "first request initiated at 0 seconds → second request initiated at 6.8 seconds → third request initiated at 13.3 seconds → fourth request initiated at 19.1 seconds," with each interval adding a random delay on top of the 6-second interval to avoid triggering anti-scraping detection at fixed intervals.

[0115] S52. Construct a set of script instructions that includes at least one of the following events: mouse movement event, scrolling event, or click event;

[0116] Specifically, for step S52, the script instruction set is a collection of codes used to simulate typical human interaction behaviors when browsing web pages, with each event corresponding to the core characteristics of a real human operation. For simulating mouse movement events, the instructions will include irregular coordinate movement trajectories (non-linear uniform speed movement) to simulate the behavior of humans moving the mouse randomly; for simulating scrolling events, the instructions will set non-uniform scrolling speeds (such as slow at first and then fast, with pauses in the middle) to match the scrolling habits of humans when browsing long pages; for simulating click events, the instructions will include a brief pause before clicking and a feedback wait after clicking to simulate the operation logic of humans clicking after confirming the target. Based on the interaction requirements of the target web page, at least one event is selected to construct the instruction set to ensure the authenticity of the interaction behavior. For example, for the product list page of an Arabic e-commerce platform, the script instruction set is constructed as follows: "Move the mouse randomly from the top left corner of the page (100,200) to the coordinates (350,480) (the trajectory includes 3 small offsets) → scroll down the page at a speed of 150 pixels per second to the 50% position, stay for 0.7 seconds → click the "View More" button, stay for 1 second", which completely simulates the operation process of a human browsing and triggering dynamic content.

[0117] S53. After the headless browser initiates a request and loads the page according to the collection schedule, it executes the script instruction set to extract the corresponding page content;

[0118] Specifically, for step S53, the headless browser first strictly follows the time points in the data collection schedule to initiate webpage access requests. After the page completes the initial resource loading, which includes basic HTML, CSS, and core scripts, the constructed script instruction set is immediately executed. The execution of the script instructions triggers interactive responses on the webpage, such as dynamically loading hidden content and refreshing list data, so that the webpage fully presents all content, including dynamic data that requires interaction to display. After the page is fully rendered and stable, the target content (such as product information, promotional copy, etc.) is extracted from the DOM tree, ensuring the completeness of the extracted content. For example, the headless browser initiates a request to a category page of a Vietnamese e-commerce platform at 6.8 seconds according to the data collection schedule. After loading the initial page, it executes a script instruction set that includes scrolling the page and clicking the "load more" button, triggering an Ajax request to load the subsequent 20 product data items. After the page is rendered, a complete data set containing product name, price, and inventory status is extracted.

[0119] This embodiment optimizes the data collection rhythm by using random delays and simulates typical human interaction behaviors, significantly reducing the mechanical nature of the data collection process and effectively avoiding the identification of anti-scraping systems. At the same time, it ensures that the dynamic content of the webpage is fully rendered due to interaction triggers, providing reliable support for the comprehensive extraction of target content.

[0120] Furthermore, in some embodiments, step S5, "execute a collection strategy to access the target webpage and inject an operation sequence to simulate human interaction behavior during the collection process," may specifically include:

[0121] S54. Determine the jurisdiction of the target webpage based on its domain name or Internet Protocol address, and load the corresponding set of data compliance clauses;

[0122] Specifically, for step S51, the domain name or Internet Protocol (IP) address is the core basis for locating the region to which a webpage belongs. Domain name suffixes are usually strongly associated with countries or regions, such as .th corresponding to Thailand, .vn to Vietnam, and .ae to the UAE. IP addresses can be determined through a geolocation database to determine their registered region, thereby clarifying the jurisdiction of the webpage. The data compliance clause set is a standardized collection of laws, regulations, and industry standards related to data collection within the jurisdiction. It covers core content such as sensitive data definitions, collection restrictions, and processing requirements. It is pre-categorized and stored according to different jurisdictions and can be automatically loaded after the region is determined. For example, if the domain name of the target webpage is "sample.shop.th" (the .th suffix points to Thailand), or its IP address is found to belong to Bangkok, Thailand, then the jurisdiction of the webpage is determined to be Thailand. Subsequently, the data compliance clause set containing the core provisions of Thailand's Personal Data Protection Act (PDPA) is automatically loaded, clarifying the relevant requirements of Thai law for personal data collection.

[0123] S55. Parse the data compliance clause set and extract the defined sensitive data patterns;

[0124] Specifically, for step S55, text parsing technology is used to deeply analyze the loaded data compliance clause set, focusing on the sensitive data categories clearly defined in the clauses, such as personal identity information, financial information, and privacy data. The characteristic patterns of various sensitive data are extracted to form sensitive data patterns that can be used for data matching. These patterns include core identification criteria such as data format (e.g., number of digits, special symbol combinations), semantic features (e.g., keyword identifiers), and field attributes (e.g., field names such as "bank account" and "ID number"). For example, after parsing the aforementioned Thai data compliance clause set, two core sensitive data patterns were extracted: one is "Thai ID number," characterized by "13 consecutive digits without separators"; the other is "Thai bank account information," characterized by "10-12 digits (account holder) + 3 digits (bank code), which may contain hyphens as separators," clearly defining the sensitive data formats that require key control.

[0125] S56. The collected raw data and sensitive data patterns are matched in real time, and the sensitive data that is successfully matched is filtered or de-identified.

[0126] Specifically, for step S56, the collected raw data covers all text fields and data records extracted from the webpage, such as personal information in product reviews, contact information in user data entries, and transaction-related financial data. The matching process is carried out simultaneously with data collection. By comparing each field in the raw data with the extracted sensitive data patterns one by one, it is determined whether there is content that meets the characteristics of sensitive data. If the match is successful, the corresponding processing method is selected according to the requirements of the compliance clause set. Filtering means directly removing the sensitive data field and not including it in the final collection result; de-identification means retaining the main body of the field but hiding the core sensitive information, such as replacing some numbers with asterisks, to ensure data security. For example, in the raw data collected from a Thai e-commerce website, there was a user comment: "I registered with ID number 1234567890123, and the linked bank account is 1234567890-001". After real-time matching, "1234567890123" conforms to the Thai ID number pattern, and "1234567890-001" conforms to the Thai bank account pattern. In accordance with compliance requirements, the ID number was anonymized (becoming "123****890123"), and the bank account was filtered (the field was directly removed). The final output was the collection result without any illegal or sensitive data.

[0127] This embodiment achieves precise adaptation between the collection of web pages in less commonly spoken languages ​​and the compliance requirements of data in the target region. By automatically identifying the jurisdiction, extracting sensitive data rules, and processing them in real time, it effectively avoids the legal risks of illegal collection of sensitive data and ensures the compliance and security of the collected data.

[0128] Furthermore, in some embodiments, step S56, "matching the collected raw data and sensitive data patterns in real time, and filtering or de-identifying the successfully matched sensitive data," may specifically include:

[0129] S561. Use regular expressions or keyword matching algorithms to compare the text fields in the original data with the sensitive data patterns;

[0130] Specifically, for step S561, regular expressions are a highly efficient retrieval tool based on character pattern matching, suitable for sensitive data with fixed formats and well-structured structures, such as ID numbers with a specific number of digits or account numbers with fixed separators. The keyword matching algorithm, on the other hand, accurately identifies text fields containing preset sensitive keywords, including sensitive words corresponding to less common languages. During the comparison process, all text fields in the collected raw data (such as user comments, contact information, and transaction-related text on web pages) are traversed, and both algorithms are used to verify each sensitive data pattern individually, ensuring that no potentially sensitive data is missed. For example, for the raw data of a Vietnamese web page, if the sensitive data patterns are "Vietnamese ID number (9-12 digits)" and "Vietnamese bank account," then regular expressions are used to match fields in the 9-12 digit format, and the keyword "Vietnamese bank account" is used to match text fields containing this phrase, achieving a comprehensive comparison of the two types of sensitive data.

[0131] S562. When the comparison is successful, the corresponding field shall be processed in accordance with the processing requirements in the data compliance clause set. The processing operation includes at least one of the following: replacement operation, masking operation, or deletion operation.

[0132] Specifically, for step S562, a successful comparison indicates that the corresponding text field in the original data is sensitive data. At this point, it is necessary to strictly follow the explicit requirements of the data compliance clause set and select the appropriate handling operation. The replacement operation replaces the entire sensitive data with a unified non-sensitive identifier (such as "[sensitive information]"), preserving the field's existence but hiding the true content; the masking operation hides only the core part of the sensitive data (such as the middle few digits), preserving the beginning and end features, balancing privacy protection and data usability; the deletion operation directly removes the sensitive data field and does not retain this part of the content in the final collection result. For example, if a field in the collected Thai webpage original data matches the sensitive pattern of "Thai ID number (13 digits)" and the compliance clause requires "partial masking", then the masking operation is performed, processing "1234567890123" into "123****890123"; if a field is "Thai bank account 123-456-7890" and the compliance clause requires "prohibition of retention", then the deletion operation is performed, directly removing the field.

[0133] S563. Record the operation type, target field, and timestamp of the disposal operation to generate a compliance audit log;

[0134] Specifically, for step S563, after the processing operation is executed, key information must be recorded in real time to form a compliance audit log. The operation type includes replacement, masking, or deletion; the target field should indicate the specific category of sensitive data (such as "Thai ID number" or "Vietnamese bank account information"); the timestamp should be accurate to the second, recording the specific time of the operation execution. The log is stored in a structured format (such as a table), containing core entries such as "operation type - target field - timestamp - original data fragment - processed data fragment", ensuring that the entire sensitive data processing process is traceable and verifiable. For example, the log entry for the masking operation of the Thai ID number is "Operation type: Masking; Target field: Thai ID number; Timestamp: 2024-06-10 10:15:38; Original data fragment: 1234567890123; Processed data fragment: 123****890123"; the log entry for the deletion operation of the Vietnamese bank account is "Operation type: Deletion; Target field: Vietnamese bank account information; Timestamp: 2024-06-10 10:16:05; Original data fragment: 987654321; Processed data fragment: None".

[0135] This embodiment achieves efficient filtering and secure processing of sensitive data on minority language web pages through precise sensitive data matching algorithms and compliant handling operations. At the same time, the generated compliance audit logs provide complete traceability evidence for data collection activities, further enhancing the compliance and credibility of data collection.

[0136] Furthermore, in some embodiments, before step S2 "parses and semantically annotates the document object model tree structure based on the large language model, identifies dynamic content nodes, and generates corresponding adaptive content positioning rules", the method may further include:

[0137] S01. Obtain the webpage corpus training set for the target minority language. The webpage corpus training set includes the webpage text and the structural annotations of its corresponding basic document object model.

[0138] Specifically, for step S01, the target minority language refers to the specific minority language involved in the collection scenario, such as Thai, Vietnamese, Arabic, etc. The web page corpus training set is a sample set specifically constructed to optimize model performance. It contains two core components: first, web page text, which is various types of effective text information extracted from the target minority language web pages, such as product names, promotional copy, function button text, page description text, etc., covering different web page types, such as e-commerce list pages, product detail pages, and promotional activity pages; second, the corresponding basic document object model (DOM) structure annotation, which is the annotation of the structured information of each node in the web page DOM tree, including the node's hierarchical path, attribute features (such as class name, ID, tag type), and functional category (such as dynamic loading trigger node, product information display node), etc., forming a one-to-one correspondence between "web page text - DOM structure annotation", providing sample support for the model to learn the semantic and structural associations of minority language web pages. For example, to meet the data collection needs of Vietnamese e-commerce platforms, the obtained webpage corpus training set contains 1,000 Vietnamese e-commerce webpage samples. In each sample, the webpage text includes functional texts such as "View More" and "Price Reduction" as well as product description text. The corresponding DOM structure annotations clearly define the hierarchical path of the "View More" button node as "body>div [class='list']>button", the attribute "class='load-more'", and the functional category "dynamic loading trigger node", ensuring that the training set can fully present the textual features and structural patterns of Vietnamese e-commerce webpages.

[0139] S02. Use the web page corpus training set to incrementally train the basic multilingual large language model, optimize the semantic understanding ability and node classification ability of the target minority language web page structure of the basic multilingual large language model, and obtain the enhanced large language model.

[0140] Specifically, for step S02, the basic multilingual large language model possesses general cross-language processing capabilities, but its adaptability to the structural features and semantic association patterns of web pages in specific minority languages ​​is insufficient. Incremental training involves using the acquired web page corpus training set as training data to fine-tune the basic model without changing its core architecture. During training, the model learns the correspondence between the semantics of web page text and DOM structural features, gradually strengthening its semantic understanding of the target minority language web page structure (such as recognizing unique hierarchical organization methods in minority language web pages) and node classification capabilities (such as accurately distinguishing between dynamic and static content nodes), ultimately forming an enhanced large language model adapted to the target minority language web page acquisition scenario. For example, using the basic multilingual large language model as a blueprint, incremental training is performed using the aforementioned Vietnamese e-commerce web page corpus training set. During training, the model learned from a large number of samples that "the text 'price reduction' in Vietnamese is often associated with 'promotional information display nodes,' and these nodes are mostly located at the top of the product cards" and "the nodes corresponding to the text 'view more' are often bound to dynamic loading events." This significantly improved the semantic understanding accuracy of the DOM structure of Vietnamese web pages and the accuracy of node classification, ultimately resulting in an enhanced large language model optimized for Vietnamese e-commerce web pages.

[0141] S03. Input the document object model tree structure into the enhanced large language model;

[0142] Specifically, for step S03, the augmented large language model, after incremental training, already possesses the capability to enhance the processing of target minority language webpage structures. During the data collection process, the DOM tree structure of the target minority language webpage, obtained after loading and rendering via a headless browser, is input into the augmented model in a standardized structured data format. This provides a prerequisite for subsequent DOM tree parsing, semantic annotation, and dynamic content node recognition. Compared to the basic model, the augmented model can capture the core features of the target minority language webpage DOM tree more quickly and accurately, providing reliable support for the generation of subsequent adaptive content positioning rules. For example, after collecting the product details page of a Vietnamese e-commerce platform, the complete DOM tree structure of the page is obtained and input into the augmented large language model trained on the Vietnamese e-commerce webpage corpus in a hierarchical node data format. The model can quickly identify node features in the DOM tree related to dynamic product parameters and promotional information, preparing for subsequent accurate node function annotation and the generation of adaptive positioning rules.

[0143] This embodiment significantly improves the semantic understanding ability and node classification accuracy of the large language model for the target minority language webpage structure by constructing a targeted training set of minority language webpage corpus and performing incremental training. This makes the enhanced model more adaptable to the characteristics of minority language webpages, providing solid model support for the accurate identification of dynamic content nodes and the generation of adaptive content positioning rules.

[0144] In summary, compared with existing technologies, the adaptive collection method for minority language web pages based on a large language model provided in this embodiment uses a headless browser to render the target web page to obtain the document object model tree structure, parses the structure through the large language model and generates adaptive content positioning rules, dynamically adjusts the collection frequency by combining semantic similarity calculation, and injects operation sequences that simulate human interaction during the collection process, ultimately achieving complete crawling of dynamic content of minority language web pages, effectively reducing the risk of anti-crawling triggers and ensuring the continuous and stable collection process.

[0145] To facilitate better implementation of the adaptive acquisition method for minority language web pages based on a large language model according to the embodiments of this application, this application also provides an adaptive acquisition device for minority language web pages based on a large language model, which is based on the aforementioned adaptive acquisition method for minority language web pages based on a large language model. The meanings of the terms are the same as in the aforementioned adaptive acquisition method for minority language web pages based on a large language model, and specific implementation details can be found in the description of the method embodiments.

[0146] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a small-language webpage adaptive acquisition device based on a large language model provided in an embodiment of this application. Specifically, this device may include a webpage rendering module 201, a rule generation module 202, a semantic analysis module 203, a strategy generation module 204, and an execution acquisition module 205, as follows:

[0147] The web page rendering module 201 is used to load and render the target language web page through a headless browser and obtain the corresponding document object model tree structure.

[0148] The rule generation module 202 is used to parse and semantically annotate the document object model tree structure based on the large language model, identify dynamic content nodes and generate corresponding adaptive content positioning rules.

[0149] The semantic analysis module 203 is used to perform semantic vectorization on the content of at least two continuously collected web pages based on adaptive content positioning rules, and to calculate the corresponding semantic similarity.

[0150] The strategy generation module 204 is used to dynamically adjust the frequency of subsequent collection requests and generate a collection strategy by querying preset frequency mapping rules based on semantic similarity.

[0151] The collection module 205 is used to execute the collection strategy to access the target webpage and inject operation sequences to simulate human interaction behavior during the collection process.

[0152] Furthermore, in some embodiments, the rule generation module 202 is specifically used for:

[0153] Input the document object model tree structure into a pre-trained multilingual large language model;

[0154] The structured description and semantic annotation information of nodes in the document object model tree are obtained through the large language model. The semantic annotation information includes at least the node function category and content language tag.

[0155] Based on structured descriptions and semantic annotation information, identify target nodes related to dynamic content loading;

[0156] Based on the path information and attribute characteristics of the target node, generate or update adaptive rules for locating the content of the target node.

[0157] Furthermore, in some embodiments, the rule generation module 202 is specifically used for:

[0158] Based on the path information of the target node, construct the initial location path expression;

[0159] Based on the node function categories in the semantic annotation information, the initial localization path expression is generalized or specialized to form adaptive rules;

[0160] Adaptive rules and content language tags are associated and stored in the rule base.

[0161] Furthermore, in some embodiments, the semantic analysis module 203 is specifically used for:

[0162] The continuously collected web page content is input into the cross-language semantic coding model;

[0163] The vector representation of each webpage's content in a unified semantic space is obtained through a cross-language semantic coding model;

[0164] Calculate the cosine distance between at least two vector representations and use the cosine distance as a measure of semantic similarity.

[0165] Furthermore, in some embodiments, the policy generation module 204 is specifically used for:

[0166] At least two semantic similarity threshold intervals are preset, and each threshold interval is associated with a collection frequency level;

[0167] The calculated semantic similarity is matched with the threshold range to determine the collection frequency level to which the semantic similarity belongs;

[0168] Based on the determined collection frequency level, adjust the request time interval for the next batch of collection tasks to generate a collection strategy.

[0169] Furthermore, in some embodiments, the data acquisition module 205 is specifically used for:

[0170] Generate a collection schedule with random delays based on the request time intervals in the collection strategy;

[0171] Construct a set of script instructions that include at least one of mouse movement events, scrolling events, or click events;

[0172] After the headless browser initiates a request and loads the page according to the collection schedule, it executes the script instruction set to extract the corresponding page content.

[0173] Furthermore, in some embodiments, the data acquisition module 205 is specifically used for:

[0174] Determine the jurisdiction of the target webpage based on its domain name or Internet Protocol address, and load the corresponding set of data compliance clauses.

[0175] Parse the data compliance clause set and extract the defined sensitive data patterns;

[0176] The collected raw data and sensitive data patterns are matched in real time, and the sensitive data that is successfully matched is filtered or de-identified.

[0177] Furthermore, in some embodiments, the data acquisition module 205 is specifically used for:

[0178] Use regular expressions or keyword matching algorithms to compare the text fields in the original data with sensitive data patterns;

[0179] When the comparison is successful, the corresponding field is processed according to the processing requirements in the data compliance clause set. The processing operation includes at least one of the following: replacement operation, masking operation, or deletion operation.

[0180] Record the operation type, target field, and timestamp of the disposal operation to generate a compliance audit log.

[0181] Furthermore, in some embodiments, the apparatus further includes a model training module, specifically used for:

[0182] Obtain a training set of web page corpus for the target minority language. The training set of web page corpus includes the web page text and the structural annotations of its corresponding basic document object model.

[0183] The basic multilingual large language model is incrementally trained using a web page corpus training set. The semantic understanding ability and node classification ability of the target minority language web page structure of the basic multilingual large language model are optimized to obtain an enhanced large language model.

[0184] Input the document object model tree structure into the enhanced large language model.

[0185] For specific limitations regarding the adaptive webpage acquisition device for minority languages ​​based on a large language model, please refer to the limitations of the adaptive webpage acquisition method for minority languages ​​based on a large language model mentioned above, which will not be repeated here. Each module in the aforementioned adaptive webpage acquisition device for minority languages ​​based on a large language model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0186] The adaptive webpage acquisition device for minority languages ​​based on a large language model provided in this embodiment uses a headless browser to render the target webpage to obtain the document object model tree structure. The large language model parses the structure and generates adaptive content positioning rules. Combined with semantic similarity calculation, the acquisition frequency is dynamically adjusted. At the same time, operation sequences simulating human interaction are injected during the acquisition process, and finally the complete crawling of dynamic content of minority language webpages is achieved, effectively reducing the risk of anti-crawling triggers and ensuring the continuous and stable acquisition process.

[0187] Furthermore, embodiments of this application also provide an electronic device, such as... Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0188] The electronic device may include components such as a processor 301 with one or more processing cores, a memory 302 with one or more computer-readable storage media, a power supply 303, and an input unit 304. Those skilled in the art will understand that... Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0189] The processor 301 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 302, and by calling data stored in the memory 302, thereby providing overall monitoring of the electronic device. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 301.

[0190] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and a method for adaptive acquisition of minority language web pages based on a large language model by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0191] The electronic device also includes a power supply 303 that supplies power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 303 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0192] The electronic device may also include an input unit 304, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0193] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 301 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 runs the applications stored in the memory 302 to realize various functions, as follows:

[0194] The system loads and renders target language web pages using a headless browser to obtain the corresponding document object model (DOM) tree structure. Based on a large language model, it parses and semantically annotates the DOM tree structure, identifies dynamic content nodes, and generates corresponding adaptive content location rules. Based on these adaptive content location rules, it semantically vectorizes the content of at least two consecutively collected web pages and calculates their semantic similarity. According to the semantic similarity, it queries a preset frequency mapping rule to dynamically adjust the frequency of subsequent collection requests, generating a collection strategy. The collection strategy is then executed to access the target web page, and an operation sequence simulating human interaction is injected during the collection process.

[0195] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0196] This application employs a headless browser to render the target webpage to obtain the document object model tree structure. The structure is then parsed using a large language model to generate adaptive content location rules. Combined with semantic similarity calculation, the collection frequency is dynamically adjusted. Simultaneously, operation sequences simulating human interaction are injected during the collection process, ultimately achieving complete crawling of dynamic content from webpages in less common languages. This effectively reduces the risk of anti-crawling triggers and ensures a continuous and stable collection process.

[0197] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0198] To this end, embodiments of this application provide a storage medium storing multiple instructions that can be loaded by a processor to execute steps in any of the adaptive acquisition methods for minority language web pages based on a large language model provided in embodiments of this application. For example, the instructions can execute the following steps:

[0199] The system loads and renders target language web pages using a headless browser to obtain the corresponding document object model (DOM) tree structure. Based on a large language model, it parses and semantically annotates the DOM tree structure, identifies dynamic content nodes, and generates corresponding adaptive content location rules. Based on these adaptive content location rules, it semantically vectorizes the content of at least two consecutively collected web pages and calculates their semantic similarity. According to the semantic similarity, it queries a preset frequency mapping rule to dynamically adjust the frequency of subsequent collection requests, generating a collection strategy. The collection strategy is then executed to access the target web page, and an operation sequence simulating human interaction is injected during the collection process.

[0200] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0201] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0202] Since the instructions stored in the storage medium can execute the steps in any of the adaptive acquisition methods for minority language web pages based on a large language model provided in the embodiments of this application, the beneficial effects that any of the adaptive acquisition methods for minority language web pages based on a large language model provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0203] The above provides a detailed description of a method and apparatus for adaptive collection of minority language web pages based on a large language model, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for adaptively collecting web pages in minority languages ​​based on a large language model, characterized in that, include: Load and render the target language webpage using a headless browser, and obtain the corresponding document object model tree structure; The document object model tree structure is input into a pre-trained multilingual large language model; the structured description and semantic annotation information of the nodes in the document object model tree are obtained through the large language model, wherein the semantic annotation information includes at least the node function category and content language label; based on the structured description and the semantic annotation information, target nodes related to dynamic content loading are identified; based on the path information and attribute features of the target nodes, adaptive content positioning rules for locating the content of the target nodes are generated or updated. Based on the adaptive content location rule, semantic vectorization is performed on at least two consecutively collected web page contents, and the corresponding semantic similarity is calculated. Based on the preset frequency mapping rules of the semantic similarity query, the frequency of subsequent collection requests is dynamically adjusted to generate a collection strategy; The acquisition strategy is executed to access the target webpage, and an operation sequence for simulating human interaction behavior is injected during the acquisition process.

2. The adaptive acquisition method for minority language web pages based on a large language model according to claim 1, characterized in that, The generation or updating of adaptive content location rules for locating target node content includes: Based on the path information of the target node, construct an initial location path expression; Based on the node function categories in the semantic annotation information, the initial positioning path expression is generalized or specialized to form the adaptive content positioning rule. The adaptive content positioning rules and the content language tags are associated and stored in the rule base.

3. The adaptive acquisition method for minority language web pages based on a large language model according to claim 1, characterized in that, The step of semantically vectorizing at least two consecutively collected webpage contents based on the adaptive content localization rule and calculating the corresponding semantic similarity includes: The continuously collected web page content is input into the cross-language semantic coding model; The cross-language semantic coding model is used to obtain vector representations of the content of each webpage in a unified semantic space; Calculate the cosine distance between at least two of the vector representations, and use the cosine distance as a measure of the semantic similarity.

4. The method for adaptive collection of minority language web pages based on a large language model according to claim 1, characterized in that, The step of dynamically adjusting the frequency of subsequent collection requests and generating a collection strategy based on the preset frequency mapping rules of the semantic similarity query includes: At least two semantic similarity threshold intervals are preset, and each threshold interval is associated with a collection frequency level; The calculated semantic similarity is matched with the threshold range to determine the collection frequency level to which the semantic similarity belongs; Based on the determined collection frequency level, the request time interval for the next batch of collection tasks is adjusted to generate the collection strategy.

5. The adaptive acquisition method for minority language web pages based on a large language model according to claim 1, characterized in that, The execution of the collection strategy to access the target webpage, and the injection of an operation sequence to simulate human interaction behavior during the collection process, includes: Based on the request time interval in the acquisition strategy, a acquisition schedule containing random delays is generated; Construct a set of script instructions that include at least one of mouse movement events, scrolling events, or click events; After the headless browser initiates a request and loads the page according to the collection schedule, it executes the script instruction set to extract the corresponding page content.

6. The method for adaptive collection of minority language web pages based on a large language model according to claim 1, characterized in that, The step of executing the collection strategy to access the target webpage and injecting an operation sequence to simulate human interaction behavior during the collection process further includes: The jurisdiction of the target webpage is determined based on its domain name or Internet Protocol address, and the corresponding set of data compliance clauses is loaded. Parse the set of data compliance clauses and extract the defined sensitive data patterns; The collected raw data is matched with the sensitive data pattern in real time, and the sensitive data that is successfully matched is filtered or de-identified.

7. The method for adaptive collection of minority language web pages based on a large language model according to claim 6, characterized in that, The step of performing real-time matching between the collected raw data and the sensitive data pattern, and filtering or de-identifying the successfully matched sensitive data, includes: Regular expressions or keyword matching algorithms are used to compare the text fields in the original data with the sensitive data patterns. When the comparison is successful, the corresponding field is processed according to the processing requirements in the data compliance clause set. The processing operation includes at least one of the following: replacement operation, masking operation, or deletion operation. Record the operation type, target field, and timestamp of the disposal operation to generate a compliance audit log.

8. The method for adaptive collection of minority language web pages based on a large language model according to claim 1, characterized in that, Before inputting the document object model tree structure into the pre-trained multilingual large language model, the method further includes: Obtain a training set of web page corpus for the target minority language, wherein the training set of web page corpus includes web page text and the structural annotations of the corresponding basic document object model; The basic multilingual large language model is incrementally trained using the web page corpus training set to optimize the semantic understanding ability and node classification ability of the target minority language web page structure of the basic multilingual large language model, thereby obtaining an enhanced large language model. The document object model tree structure is input into the enhanced large language model.

9. A small-language webpage adaptive acquisition device based on a large language model, characterized in that, include: The web page rendering module is used to load and render target language web pages through a headless browser and obtain the corresponding document object model tree structure. The rule generation module is used to input the document object model tree structure into a pre-trained multilingual large language model; obtain the structured description and semantic annotation information of the nodes in the document object model tree through the large language model, wherein the semantic annotation information includes at least the node function category and content language label; identify the target nodes related to dynamic content loading based on the structured description and the semantic annotation information; and generate or update adaptive content positioning rules for locating the content of the target nodes based on the path information and attribute features of the target nodes. The semantic analysis module is used to perform semantic vectorization on at least two consecutively collected web page contents based on the adaptive content positioning rules, and calculate the corresponding semantic similarity. The strategy generation module is used to dynamically adjust the initiation frequency of subsequent collection requests based on the semantic similarity query preset frequency mapping rules, and generate a collection strategy. The collection module is used to execute the collection strategy to access the target webpage and inject operation sequences to simulate human interaction behavior during the collection process.

Images