A method, system, device and storage medium for processing streaming hypertext content

Abstract

The application discloses a processing method, system and device of streaming hypertext content and a storage medium, and relates to the field of data processing. In the method, data segments of the streaming hypertext content are continuously received until complete HTTP header information is acquired, and a hook point is triggered. At the hook point, a target plug-in registered is called to determine whether the streaming hypertext content needs to be customized and modified according to the HTTP header information. When the streaming hypertext content needs to be customized and modified, a customized modification condition is formed according to the target plug-in, and target data segments meeting the customized modification condition in subsequently received data segments are directly modified. The modified data content is transmitted as new data segments. The technical solution provided by the application can realize efficient and accurate customized processing of the streaming hypertext content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a processing method, system and device of streaming hypertext content and a storage medium. BACKGROUND

[0002] Streaming hypertext content processing is one of the key technologies in modern Internet applications, and is widely used in web acceleration, content filtering, and advertisement delivery. With the continuous growth of Internet traffic, efficient and real-time processing of streaming hypertext content has become a key factor in improving user experience and service quality.

[0003] The existing streaming hypertext content processing technology has obvious shortcomings in dealing with complex customization requirements. For example, in the process of dynamically generating and modifying streaming data, it is not flexible to accurately personalize according to different business scenarios and user needs. In addition, the existing technology is prone to performance bottlenecks and data consistency problems when processing a large number of concurrent requests, resulting in a decline in service quality and user experience.

[0004] Therefore, there is an urgent need for a streaming hypertext content processing method that can effectively support complex customization requirements to overcome the limitations of existing technology. SUMMARY

[0005] The present application provides a processing method, system, device and storage medium of streaming hypertext content, which can realize efficient and accurate customization processing of streaming hypertext content and improve the processing capacity and response speed of the system.

[0006] In a first aspect of the present application, a processing method of streaming hypertext content is provided, applied to a streaming hypertext content processing platform, and the method comprises:

[0007] Continuously receiving data segments of streaming hypertext content until the HTTP header information is completely acquired, triggering a hook point;

[0008] At the hook point, calling a registered target plug-in to determine whether customization modification of the streaming hypertext content is needed according to the HTTP header information;

[0009] When customization modification of the streaming hypertext content is needed, forming a customization modification condition according to the target plug-in, and directly modifying the target data segment that meets the customization modification condition in the subsequent received data segment;

[0010] Transmitting the modified data content as a new data segment.

[0011] Optionally, the directly modifying the target data segment that meets the customization modification condition in the subsequent received data segment comprises:

[0012] processing the content of the target data segment by using a keyword matching algorithm to obtain a keyword matched with a preset keyword list;

[0013] starting from a target position where the keyword is located, scanning the HTML code forward and backward until a boundary point conforming to the HTML tag rule is found, and determining a range of HTML tags to be processed according to the boundary point;

[0014] performing customized modification on the content in the range of HTML tags, the customized modification including replacement, deletion or insertion of the content.

[0015] Optionally, the scanning of the HTML code forward and backward until the boundary point conforming to the HTML tag rule is found, and the determination of the range of HTML tags to be processed according to the boundary point include:

[0016] parsing the target data segment, constructing a target tree, determining a page modification expression according to a user-defined content filtering rule, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression;

[0017] obtaining an expression of a nested relationship and attribute characteristics of the HTML tag object;

[0018] verifying whether the HTML tag object satisfies the nested relationship and attribute characteristics described in the expression, and if yes, determining that the HTML tag object and the content contained therein are the range of HTML tags to be processed.

[0019] Optionally, the verification of whether the HTML tag object satisfies the nested relationship and attribute characteristics described in the expression includes:

[0020] embedding a first attribute value into the page modification expression when the content filtering rule uses a regular expression to describe the first attribute value;

[0021] extracting a second attribute value of the HTML tag object, describing the second attribute value by using a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression;

[0022] when the first attribute value and the second attribute value are successfully matched, confirming that the HTML tag object satisfies the attribute characteristic requirement described in the page modification expression.

[0023] Optionally, the direct modification of the target data segment satisfying the customized modification condition in the subsequently received data segment includes:

[0024] When the HTML tag range needing to be processed is located, it is checked whether the replacement content defined in the page modification expression for the HTML tag range is empty;

[0025] If the replacement content defined in the page modification expression for the HTML tag range is empty, the content contained in the HTML tag range is marked as a region to be deleted.

[0026] Optionally, the method further comprises:

[0027] A context state is created and maintained for storing modification information across data shards, and the context state is a memory data structure or a persistent storage;

[0028] When a new data shard is received, the context state is checked to determine whether there is an uncompleted modification task associated with the new data shard;

[0029] When there is an uncompleted modification task associated with the new data shard, a corresponding modification operation is continued to be performed according to information in the context state.

[0030] Optionally, the method further comprises:

[0031] The multiple data shards of the streamable hypertext content are processed in parallel by multiple threads, a lock mechanism is used to share data among the multiple threads, and a message queue is used to ensure the correctness of processing order and data dependency relationship of the multiple threads.

[0032] In a second aspect of the present application, a processing system of streamable hypertext content is provided, comprising a collection module, a judgment module, an execution module and an integration module, wherein:

[0033] The collection module is configured to continuously receive data shards of streamable hypertext content until HTTP header information is completely acquired, and trigger a hook point;

[0034] The judgment module is configured to, at the hook point, call a target plug-in registered according to the HTTP header information to determine whether customized modification of the streamable hypertext content is needed;

[0035] The execution module is configured to, when customized modification of the streamable hypertext content is needed, form a customized modification condition according to the target plug-in, and directly perform corresponding modification on a target data shard meeting the customized modification condition in a data shard received subsequently;

[0036] The integration module is configured to transmit the modified data content as a new data shard.

[0037] In a third aspect of the present application, an electronic device is provided, comprising a processor, a memory, a user interface and a network interface, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, and the processor is configured to execute the instructions stored in the memory to enable the electronic device to perform the method according to any one of the preceding aspects.

[0038] In a fourth aspect of the present application, a computer readable storage medium is provided, which stores instructions that, when executed, perform the method according to any one of the preceding aspects.

[0039] In summary, the one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:

[0040] 1. By continuously receiving data segments of streaming hypertext content and triggering a hook point immediately after obtaining complete HTTP header information, timely judgment and processing of data can be facilitated, unnecessary waiting time can be avoided, and the efficiency of data processing can be improved;

[0041] 2. By calling the registered target plug-in at the hook point, it can be flexibly determined whether the streaming hypertext content needs to be customized and modified according to the HTTP header information. This design enables the processing flow to be dynamically adjusted according to specific needs, thereby enhancing the flexibility and adaptability of the system;

[0042] 3. Only target data segments that meet the customized modification conditions are modified, avoiding comprehensive scanning and processing of the entire streaming data, thereby reducing unnecessary resource consumption. This targeted processing method helps to improve the performance of the system, especially when processing large-scale streaming data;

[0043] 4. The modified data segments are recombined into complete streaming data, which ensures the integrity and consistency of the data. At the same time, since the modification operation is performed during data reception, users can obtain customized content without interrupting the data stream;

[0044] 5. The plug-in mechanism is used to realize the function of customized modification, which makes the system easy to extend and maintain. When new customized requirements are needed, only new plug-ins need to be developed and registered, without modifying the existing processing logic;

[0045] 5. Since the embodiments of the present application perform customized modification during streaming data reception, they can respond to changes in user needs in real time. This is particularly important for real-time application scenarios that require quick response. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1is a flowchart of a processing method of streaming hypertext content disclosed by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of an original access page disclosed by an embodiment of the present application;

[0048] Figure 3 is a schematic diagram of a part of html source code structure of an original access page disclosed by an embodiment of the present application;

[0049] Figure 4 is a schematic diagram of an original access page updated disclosed by an embodiment of the present application;

[0050] Figure 5 is a schematic diagram of a page containing JSON data disclosed by an embodiment of the present application;

[0051] Figure 6 is a schematic diagram of a page containing JSON data updated disclosed by an embodiment of the present application;

[0052] Figure 7 is a schematic diagram of a module of a processing system of streaming hypertext content disclosed by an embodiment of the present application;

[0053] Figure 8 is a schematic diagram of a structure of an electronic device disclosed by an embodiment of the present application.

[0054] Legend of reference signs: 701, acquisition module; 702, judgment module; 703, execution module; 704, integration module; 801, processor; 802, communication bus; 803, user interface; 804, network interface; 805, memory. DETAILED DESCRIPTION

[0055] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in combination with the drawings in the embodiments of the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0056] In the description of the embodiments of the present application, the words such as "for example" or "for instance" are used to represent an example, illustration or description. Any embodiment or design scheme described as "for example" or "for instance" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "for example" or "for instance" are intended to present the relevant concept in a specific way.

[0057] In the description of the embodiments of the present application, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. In addition, the terms "first", "second" are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. The terms "include", "contain", "have" and their variants mean "include but are not limited to", unless otherwise specifically emphasized.

[0058] The embodiment discloses a processing method of streaming hypertext content, applied to a streaming hypertext content processing platform, Figure 1 is a flowchart of the processing method of streaming hypertext content disclosed by the embodiment of the present application, as Figure 1 shown, the method comprises the following steps:

[0059] S110, continuously receiving data segments of streaming hypertext content until the HTTP header information is completely acquired, triggering a hook point;

[0060] The proxy service starts to receive streaming hypertext content (i.e. the content of HTTP request or response) from the client or server. These data contents are transmitted in the form of segments (or data blocks) step by step. The proxy service starts to parse when the first data segment is received, and continues this process until it can completely parse the HTTP header information. The HTTP header information contains important metadata about the request or response, such as target URL, request method, status code, header field, etc. When the HTTP header information is completely parsed, the proxy service will trigger a specific hook point (or callback point, event point). This hook point is a position preset by the proxy service, which is used to execute the operations or plug-ins registered by the developer when a specific event occurs (such as the completion of HTTP header parsing).

[0061] S120, at the hook point, calling the target plug-in registered according to the HTTP header information to determine whether the streaming hypertext content needs to be customized and modified;

[0062] Depending on business needs, one or more functional processing functions can be registered at appropriate hook points. For example, at the hook point after the response header is parsed, developers can obtain various fields of the parsed HTTP header, including content-type, content-length, content-encoding, transfer-encoding, set-cookie, etc., and can then add HTTP fields, modify existing fields, or delete specified fields according to business requirements. Alternatively, they can determine certain fields and, under specific conditions, decide to register plugins for the response content and execute specified functional processing functions. For instance, if the host field contains github.com and the content-type field value is text / html, then certain keywords can be completely replaced in the response text page.

[0063] Once the hook is triggered, the proxy service invokes the target plugins that the developer had previously registered. These plugins are code modules written by the developer to implement specific functions (such as header auditing, header field modification, content customization, etc.). The plugins receive HTTP header information as input and determine whether customized modifications to the streaming hypertext content are needed based on preset logic or rules. For example, a plugin might check whether the target URL matches a specific pattern or whether a header field contains a specific value, and then decide whether to perform customized modifications based on these results.

[0064] S130. When it is necessary to make customized modifications to the streaming hypertext content, customized modification conditions are formed according to the target plugin, and the target data segments that meet the customized modification conditions in the subsequently received data segments are directly modified accordingly.

[0065] If a plugin determines that customized modifications to the streaming hypertext content are needed, it will formulate customized modification conditions based on its own logic or rules. These conditions may include specific text patterns, data ranges, field values, etc., to guide subsequent modification operations. As the proxy service continues to receive subsequent data shards, it checks whether each shard meets the customized modification conditions set by the plugin. For target data shards that meet the conditions, the proxy service will directly modify them according to the modification rules or operations provided by the plugin. This modification may be CRUD operations on data, or it may be re-encoding or formatting of specific fields or data.

[0066] Optionally, directly modifying the target data fragments that meet the customized modification conditions in subsequently received data fragments includes:

[0067] The target data segments are processed using a keyword matching algorithm to obtain keywords that match a preset keyword list;

[0068] starting from the target position where the keyword is located, scanning the HTML code forward and backward until a boundary point that meets the HTML tag rule is found, and determining the range of HTML tags that need to be processed according to the boundary point;

[0069] performing customized modification on the content within the range of HTML tags, which includes replacement, deletion, or insertion of content.

[0070] The proxy service or HTML processing system performs keyword matching on the subsequently received data fragments (i.e., streaming data blocks of HTML content). This typically involves using efficient keyword matching algorithms, such as regular expression matching, string search algorithms, etc., to search for keywords in the data fragments that match a pre-defined keyword list. The pre-defined keyword list is defined by the developer based on business requirements and contains specific words or phrases in the HTML content that need special attention or processing. When a keyword that matches the pre-defined keyword list is found in the data fragment, the system starts scanning the HTML code from the position where the keyword is located (i.e., the target position) and scans forward and backward. The purpose of scanning is to find boundary points that meet the HTML tag rule, which define the range of HTML tags containing the keyword. This typically involves parsing the nested relationship of HTML tags, attribute characteristics, etc., to ensure accurate identification of the range of HTML tags that need to be processed. During the scanning process, the system uses HTML parsing techniques (such as DOM parsing, SAX parsing, etc.) to identify and process HTML tags. These techniques allow the system to understand the structure of the HTML document, including the opening and closing of tags, the setting of attributes, etc. Through scanning and parsing, the system can determine the start and end positions of the HTML tags containing the keyword, thereby determining the range of HTML tags that need to be processed. After determining the range of HTML tags that need to be processed, the system can perform customized modification on the content within the range. Customized modification can include various operations, such as replacement, deletion, or insertion of content. The specific implementation of these operations depends on the developer's requirements and business logic.

[0071] By utilizing a keyword matching algorithm, the system can quickly locate keywords in the target data shard that match a predefined keyword list. This efficient matching mechanism ensures that the system can handle streaming data in real-time and respond promptly to customized modification requirements. Starting from the target location of the keyword, the system can scan the HTML code forward and backward and accurately find the demarcation points that comply with HTML tag rules. This precision ensures that only the content within the HTML tag range directly related to the keyword is processed, avoiding unnecessary global parsing and modification. The system supports various customized modifications to the content within the HTML tag range, including replacement, deletion, or insertion. This flexibility allows developers to make precise adjustments and optimizations to HTML pages based on specific requirements. Since the system only processes target data shards that meet the customized modification conditions and concentrates computing resources on high-precision parsing of HTML objects related to the keyword occurrence location, it can significantly reduce the parsing granularity of the global page and resource consumption. This optimization improves the processing efficiency of the system, especially when handling large-scale streaming HTML data. By customizing the content within the HTML tag range, developers can achieve precise control over the page content. This helps ensure the accuracy and consistency of the page content while improving user experience and page performance. The technical framework supports developers to extend the system's functionality by defining new keyword lists and customized modification rules. This extensibility allows the system to adapt to changing requirements and scenarios. At the same time, since the system adopts a modular design philosophy, it is also convenient for subsequent maintenance and upgrading.

[0072] Optionally, the forward and backward scanning of the HTML code until the demarcation point that complies with the HTML tag rules is found, and determining the HTML tag range that needs to be processed according to the demarcation point includes:

[0073] Parsing the target data shard, constructing a target tree, determining a page modification expression according to a user-defined content filtering rule, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression;

[0074] Obtaining an expression of the nesting relationship and attribute characteristics of the HTML tag object;

[0075] Verifying whether the HTML tag object satisfies the nesting relationship and attribute characteristics described in the expression, and if it does, determining that the HTML tag object and the contained content are the HTML tag range that needs to be processed.

[0076] HTML parsing is performed on the received target data fragment. The purpose of this step is to convert the HTML code into a data structure that is easy to manipulate and query, commonly referred to as a DOM (Document Object Model) tree (i.e., target tree). In the DOM tree, each element of the HTML document is represented as a node, and the nodes are connected through parent-child relationships, sibling relationships, etc., forming a tree-like structure. After the DOM tree is constructed, the system determines the page modification expression based on the user-defined content filtering rules. These rules are usually defined by the user according to specific needs, specifying the HTML elements and their attribute characteristics that need to be filtered or modified. The page modification expression is a specific syntax or format used to describe these rules so that the system can understand and execute them. The page modification expression is used to locate the HTML tag object containing the keyword in the DOM tree. This step usually involves traversing and searching the DOM tree to find nodes that match the page modification expression. Once the matching node is found, the system can determine the HTML tag object where the keyword is located. After determining the HTML tag object where the keyword is located, the system obtains the expression of the nested relationship and attribute characteristics of the tag object. These expressions describe the location of the tag object in the DOM tree, the nested relationship with other tags, and its own attribute characteristics (such as class, id, etc.). This information is crucial for subsequent verification of whether the tag object meets the user-defined rules. The system verifies whether the HTML tag object meets the nested relationship and attribute characteristics described in the user-defined expression. This step usually involves matching and comparing the attributes of the tag object to ensure they meet the user-specified rules. If the tag object meets all conditions, the system can determine that the tag object and its contained content are the HTML tag range that needs to be processed.

[0077] By parsing the target data fragments, the system can construct an accurate target tree (such as a DOM tree) that reflects the structure and hierarchy of the HTML document. This precise parsing is the foundation for subsequent steps, ensuring the accuracy and reliability of subsequent processing. Based on user-defined content filtering rules, the system can determine page modification expressions. These expressions describe the specific locations and characteristics of HTML tag objects that need to be filtered or modified. By matching these expressions with the target tree, the system can efficiently locate the HTML tag objects where the keywords are located. The system can obtain the expressions of the nesting relationship and attribute characteristics of the HTML tag objects and verify them based on these expressions. This flexibility allows the system to handle various complex HTML structures, including nested tags and tags with specific attributes. Through verification, the system can ensure that only HTML tag objects and their contents that meet specific conditions will be processed. When the HTML tag objects that meet the page modification expressions are determined, the system can determine the range of HTML tags that need to be processed based on the nesting relationship and attribute characteristics of these objects. This precise range determination avoids unnecessary global parsing and modification, improving processing efficiency and accuracy. By constructing the target tree and using page modification expressions for positioning, the system can avoid repeated parsing and scanning of the global HTML document. This optimization reduces resource consumption and improves processing speed, especially when dealing with large-scale HTML data. The technical framework supports the extension of user-defined content filtering rules and page modification expressions. This means that developers can add new rules or expressions according to specific needs to adapt to changing requirements and scenarios. At the same time, since the system adopts a modular design philosophy, it is also convenient for subsequent maintenance and upgrading.

[0078] Optionally, the verification of whether the HTML tag object meets the nesting relationship and attribute characteristics described in the expression includes:

[0079] When the content filtering rule uses a regular expression to describe the first attribute value and embeds the first attribute value into the page modification expression;

[0080] Extracting the second attribute value of the HTML tag object and describing the second attribute value through a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression;

[0081] When the first attribute value and the second attribute value match successfully, it is confirmed that the HTML tag object meets the attribute characteristic requirements described in the page modification expression.

[0082] In the content filtering rule, first, a regular expression is used to describe a certain attribute value (i.e., the first attribute value) of the HTML tag object of interest. This attribute value can be a class name (class), an ID, a data attribute (data-*), etc., depending on the characteristics of the tag object that is intended to be located. This first attribute value described using a regular expression is embedded into the page modification expression. The page modification expression is a more extensive rule that describes how to modify or filter a specific part of the HTML document. By embedding the first attribute value, a condition is actually specified, i.e., only when a certain attribute value of the HTML tag object meets this condition, will the tag object be selected for subsequent processing. In the process of parsing the HTML document and building the DOM tree, the system will traverse the tag objects in the document and extract the attribute value of interest (i.e., the second attribute value). This value is obtained from the actual HTML tag and is used to match the first attribute value in the page modification expression. In order to perform the matching, the second attribute value also needs to be converted into a regular expression (i.e., the target regular expression). This usually means treating the second attribute value as a normal string and encapsulating it in the matching pattern of the regular expression. However, in most cases, if the second attribute value itself does not contain any regular expression special characters that need to be escaped, it can be used directly as a fixed string pattern for matching. The target regular expression is matched with the first attribute value in the page modification expression. If the match is successful, it means that the attribute value of the HTML tag object meets the condition specified in the content filtering rule. When the first attribute value matches the second attribute value successfully, it can be confirmed that the HTML tag object meets the attribute characteristic requirements described in the page modification expression. This means that the tag object is one of the targets that is intended to be modified or filtered. When the tag object that meets the condition is confirmed, the system can further determine the range of HTML tags that need to be processed according to other parts of the page modification expression (such as nested relationships, tag names, etc.), and perform the corresponding modification or filtering operation.

[0083] The use of regular expressions in the content filtering rules to describe the first attribute value enables the system to flexibly match various complex attribute characteristics. The powerful matching capability of regular expressions ensures that the system can accurately identify HTML tag objects that meet specific conditions. The system obtains a target regular expression by extracting the second attribute value of the HTML tag object and using a regular expression to describe this value. Matching this target regular expression with the first attribute value in the page modification expression can accurately verify whether the HTML tag object meets the attribute characteristic requirements described in the page modification expression. Due to the efficient matching algorithm of regular expressions, the system can complete a large number of matching operations in a short time. This ensures that the system can still maintain high processing efficiency when processing large-scale HTML data. The use of regular expressions to describe attribute characteristics enables the system to easily add new attribute characteristic matching rules. This scalability allows the system to adapt to changing needs and scenarios, improving the flexibility and practicality of the system. Through precise matching algorithms and strict constraints of regular expressions, the system can reduce false positives and false negatives. This ensures that the system can maintain high accuracy and reliability when processing HTML tag objects. Due to the efficiency and accuracy of regular expressions, the system can avoid unnecessary global scanning and parsing. This reduces resource consumption and improves system performance.

[0084] Optionally, the directly corresponding modification of the target data slice that meets the customized modification condition in the subsequently received data slice comprises:

[0085] When the HTML tag range that needs to be processed is located, it is checked whether the replacement content defined in the page modification expression for the HTML tag range is empty;

[0086] If the replacement content defined in the page modification expression for the HTML tag range is empty, the content contained in the HTML tag range is marked as a region to be deleted.

[0087] The system needs to locate the HTML tag scope that needs to be processed. This is usually done by parsing the HTML document and modifying the expression according to the selector or tag name specified in the page modification expression. The selector can be a CSS selector, an XPath expression, or other methods that can uniquely identify an HTML tag or tag scope. When the system locates the HTML tag scope that needs to be processed, it will then check the replacement content defined in the page modification expression for this scope. The page modification expression is a predefined rule that specifies how to modify a specific part of the HTML document. This expression may contain selectors, attribute conditions, replacement content, and other parts. The system will determine whether the replacement content defined in the page modification expression for the current HTML tag scope is empty according to the definition in the page modification expression. Here, "empty" may mean that no specific replacement text is provided, or the replacement text is an empty string. If the system finds that the replacement content defined in the page modification expression for the current HTML tag scope is empty, it will mark the content contained in this HTML tag scope as a region to be deleted. This means that in the subsequent modification process, the content of this region will be deleted or removed, rather than replaced with other content. The system will modify the HTML document according to the previous judgment result. For the region marked for deletion, the system will delete all content in this region. If specific replacement content is provided in the page modification expression, the system will insert this content into the corresponding position, replacing the original content.

[0088] When the system locates the HTML tag scope that needs to be processed, it will check whether the replacement content defined in the page modification expression for this scope is empty. If it is empty, the system will mark the content contained in this HTML tag scope as a region to be deleted. This precise operation ensures that only HTML content that meets specific conditions will be deleted, avoiding unnecessary global deletion operations. By directly marking the region to be deleted, the system avoids unnecessary parsing and modification operations. This optimization reduces resource consumption and improves system processing efficiency. Especially when dealing with large-scale HTML data, this optimization effect is more significant. The system can flexibly process the content of the HTML tag scope according to different page modification expressions. When the replacement content is empty, the system can automatically perform the deletion operation without additional configuration or rules. This flexibility allows the system to adapt to changing needs and scenarios. Through explicit replacement content checking logic, the system can accurately determine when to perform the deletion operation. This explicit logic reduces the risk of misoperation, ensuring the stability and reliability of the system. Precise deletion operations and optimized resource usage enable the system to process HTML data more quickly, thereby improving user experience. Users can see the modified results more quickly, reducing waiting time.

[0089] Specifically, taking HTML as an example, the structured data tag level processing operation is divided into three steps: filtering to the keyword in the data slice, expanding forward and backward from the position of the keyword to find the label boundary that meets the rule description; processing the content of the label. The page modification expression of the content filtering rule describes the specific position of the object (containing the keyword) in the HTML using a running expression that describes the HTML tag name and the nesting logic between each other. The expression supports the description of HTML tags such as DIV, SPAN, LI, H1-5, A, P, etc. Each tag can describe the attribute characteristics of the tag, such as the characteristics of class="name" in the DIV tag. Each expression requires at least one description of the DIV tag, and the attribute characteristics of the DIV tag must be included in the last DIV tag description. The embodiment of the application can reduce the parsing granularity of the global HTML page, concentrate computing resources, and perform high-precision parsing on the HTML object related to the position of the keyword, while unrelated other HTML text elements can be quickly skipped in the parsing process. An efficient HTML page content parsing and filtering mechanism is implemented.

[0090] The page modification expression of the user-defined content filtering rule is composed of the following two parts:

[0091] 1. An expression describing the nesting relationship of the HTML tag object where the keyword appears and the attribute characteristics of each tag object. The attribute characteristics of the HTML tag object support the format of regular expression description to cope with the characteristics of the flexible and variable position of the HTML tag attribute.

[0092] 2. The target content of the keyword object content replacement, that is, the content of the HTML object that hits the above expression is replaced by the content. It supports string form and can be empty. When the target content of the replacement is empty, the meaning of the current expression is to delete the HTML object, at this time the length of the page content will change, and the text content within the HTML object tag (such as <div ...> .. ) will be deleted together.

[0093] S140, transmit the modified data content as a new data slice.

[0094] In processing data streams, the data is first modified and the modified content is considered as new data pieces. These new data pieces can be generated in different ways, such as direct modification of the original data. To ensure that each new data piece can be uniquely identified, a unique identifier (e.g. timestamp, sequence number, etc.) is usually assigned to each data piece. These identifiers not only help to track the data pieces during transmission, but also can be used for data sorting in subsequent processing. When preparing to transmit these new data pieces, it is important to maintain the integrity and accuracy of the data. This may involve checking the data, such as calculating checksums or hash values, to ensure that the data has not been tampered with or damaged during transmission. At the same time, since the data pieces are new, they usually do not need to be sorted in the order of the original data, but can be organized according to actual needs or requirements of the transmission protocol. During transmission, the real-time and availability of data pieces need to be ensured to meet business needs. This may involve using appropriate transmission protocols and technologies, such as TCP / IP, to ensure that data pieces can reliably reach the target system or application platform. After reaching the target, the receiving system may need to verify the integrity and accuracy of each data piece. This is usually done by recalculating the checksum or hash value and comparing it with the value generated during transmission. If the verification is passed, the data piece is considered valid and can be further processed or stored according to business needs.

[0095] Figure 2 is a schematic diagram of the original access page disclosed by the embodiments of the present application, as shown in Figure 2 , the keyword is "xmind mind map". Figure 3 is a schematic diagram of the original access page disclosed by the embodiments of the present application, as shown in

[0096] Configure "xmind mind map" as a scanning keyword, and configure the label feature of deletion (replacement with empty) as follows:

[0097] cond %{READ_RESPONSE_HDR_HOOK}

[0098] cond %{HTML-BODY:KEYWORD}

[0099] update-response-html DIV(class\=\"md-content\").ARTICLE().DIV(id\=\"post\")"".

[0100] Figure 4 is a schematic diagram of the original access page updated by the embodiments of the present application, as shown in Figure 4As shown, the target paragraph is deleted. Figure 5 is a schematic diagram of a page containing JSON data disclosed by an embodiment of the present application, and the similar effects described above can also be achieved using JSON data. The page containing JSON data is as shown in Figure 5 . Configure "ting2" to scan keywords, and configure the updated label features as follows:

[0101] cond %{READ_RESPONSE_HDR_HOOK}

[0102] cond %{HTML-BODY:KEYWORD}

[0103] update-response-html JSON(qzk).JSON(qzk2).JSON(qzk3).JSON(qzk4).JSON(qzk5).JSON(qzk6).JSON(addr)"flowtest"。

[0104] Figure 6 is a schematic diagram of an updated page containing JSON data disclosed by an embodiment of the present application, as shown in Figure 6 . Thus, the "addr" field containing the keyword in the response content obtained after passing through the proxy program is updated to the configuration value.

[0105] Optionally, the method further comprises:

[0106] creating and maintaining a context state for storing modification information across data shards, the context state being a memory data structure or a persistent storage;

[0107] when a new data shard is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data shard;

[0108] when there is an uncompleted modification task associated with the new data shard, continuing to perform the corresponding modification operation according to the information in the context state.

[0109] The context state is a memory data structure or persistent storage used to store information about modifications across data shards. It can contain various types of information, such as unique identifiers for modification tasks, identifiers for data shards, types and contents of modification operations, statuses of modification tasks (e.g., not completed, completed), etc. The context state needs to persist throughout the data processing flow and be updated continuously as data shards are received and processed. In memory data structures, the context state is usually managed as a global variable or thread-local variable. In persistent storage, databases, file systems, or other storage media can be used to save the context state. When the system receives a new data shard, it first needs to identify and parse it. Before parsing the new data shard, the system needs to check the context state to determine whether there are any outstanding modification tasks associated with this data shard. This is usually done by comparing the identifier of the data shard with the information in the context state. If there are outstanding modification tasks associated with the new data shard, the system needs to determine the specific contents and status of these tasks. Based on the information in the context state, the system can proceed to perform the modification tasks associated with the new data shard. This may involve modifying the contents of the data shard, adding new data elements, deleting unnecessary data, etc. After performing the modification operations, the system needs to update the context state to reflect the status and results of the modification tasks. This may include marking the modification tasks as completed, updating the identifiers or statuses of the data shards, adding new modification tasks, etc.

[0110] By creating and maintaining the context state, the system is able to store modification information across data shards. This means that when the system receives a new data shard, it is able to check the context state to determine if there are any outstanding modification tasks related to that data shard. If there are, the system can continue to perform the corresponding modification operations based on the information in the context state, ensuring the continuity of data processing. The context state not only stores information about modification tasks, but also may store intermediate states or temporary results during the modification process. This allows the system to perform subsequent operations based on the previous modification history and state when processing new data shards, improving the accuracy and consistency of data modification. By maintaining the context state, the system is able to avoid repeatedly parsing and modifying the same data shards, reducing unnecessary resource consumption. In addition, the context state can also help the system more efficiently manage and schedule modification tasks, further improving the processing performance of the system. The context state provides a mechanism for the system to store and process modification information across data shards, allowing the system to support more complex data processing logic. For example, the system can handle complex operations that require aggregation, filtering, or transformation across multiple data shards. The context state can be an in-memory data structure or persistent storage, depending on the specific needs and resource limitations of the system. This flexibility allows the system to choose the most suitable storage method according to actual needs, improving the scalability and flexibility of the system.

[0111] Optionally, the method further comprises:

[0112] By multi-threading parallel processing of multiple streamable hypertext content data shards, a lock mechanism is used to share data among multiple threads, and a message queue is used to ensure the correctness of the processing order and data dependency relationship of the multiple threads.

[0113] Multi-threaded parallel processing is a programming model that allows multiple tasks to be executed simultaneously, which can improve the efficiency of program execution and system throughput. When processing data shards of streaming hypertext content, multi-threaded parallel processing can fully utilize the multi-core CPU resources of a computer, processing multiple data shards simultaneously, thereby speeding up processing. Each thread can independently process a data shard, performing operations such as parsing, filtering, and converting. Because multi-threaded parallel processing can execute multiple tasks simultaneously, it can significantly reduce the total time required to process a large number of data shards. In multi-threaded programming, shared data is a common requirement. However, because multiple threads may access and modify shared data simultaneously, certain synchronization mechanisms are needed to ensure data consistency and integrity. The lock mechanism is a commonly used synchronization mechanism that can be implemented through mutexes. When a thread needs to access shared data, it will first attempt to acquire the mutex. If the lock is already held by another thread, the current thread will be blocked until the lock is released. In this way, it can be ensured that only one thread can access shared data at any time, thereby avoiding data race and deadlock problems. When processing data shards of streaming hypertext content, there may be data dependencies. For example, the processing result of a certain data shard may depend on the processing result of the previous data shard. To ensure the correctness of multi-threaded processing order and data dependencies, a message queue can be used. The message queue is a first-in, first-out data structure that ensures messages are received and processed in the order they are sent. When processing data shards of streaming hypertext content, each data shard can be encapsulated as a message and sent to the message queue. Then, multiple threads can take messages from the message queue in turn for processing. Through the message queue, the following points can be ensured: the processing order of messages is consistent with the sending order, ensuring the correctness of data dependencies; the message queue can act as a buffer, when a thread processes slowly, other threads can continue to process subsequent messages, thereby improving the concurrency performance and throughput of the system; the message queue can also implement message retry and compensation mechanisms, when a message processing fails, it can be put back into the message queue for processing, thereby improving the reliability and stability of the system.

[0114] By multi-threading parallel processing of multiple data shards, the throughput and processing speed of the system can be significantly improved. In processing a large amount of streaming hypertext content, multi-threading parallel processing can complete the task faster, reduce user waiting time, and improve user experience. Lock mechanisms (such as mutexes, read-write locks, etc.) are used to ensure that multiple threads do not conflict when accessing shared data, ensuring data consistency and security. The use of lock mechanisms can also avoid data contention and deadlocks, ensuring the stable operation of multi-threaded programs. Through the use of message queues, the processing order and data dependency relationships of multiple threads can be ensured to be correct, avoiding data errors or inconsistencies caused by improper processing order. In processing streaming hypertext content, there may be complex dependencies between data. Through the combined use of message queues and lock mechanisms, these dependencies can be correctly processed, avoiding data inconsistencies or errors.

[0115] The embodiment also discloses a processing system for streaming hypertext content, Figure 7 is a module schematic diagram of the processing system for streaming hypertext content disclosed by the embodiment of the application, as Figure 7 shown, the system includes a collection module 701, a judgment module 702, an execution module 703, and an integration module 704, wherein:

[0116] The collection module 701 is configured to continuously receive data shards of streaming hypertext content until the HTTP header information is completely acquired, triggering a hook point.

[0117] The judgment module 702 is configured to call a registered target plug-in to determine whether customized modification of the streaming hypertext content is needed according to the HTTP header information at the hook point.

[0118] The execution module 703 is configured to form a customized modification condition according to the target plug-in when customized modification of the streaming hypertext content is needed, and directly modify target data shards that meet the customized modification condition in subsequent received data shards.

[0119] The integration module 704 is configured to transmit the modified data content as new data shards.

[0120] Optionally, the execution module 703 is configured to:

[0121] process the content of the target data shard using a keyword matching algorithm to obtain keywords that match a preset keyword list;

[0122] scan the HTML code forward and backward from the target position where the keyword is located until a demarcation point that meets the HTML tag rule is found, and determine the range of HTML tags that need to be processed according to the demarcation point.

[0123] customized modification on the content within the HTML tag range, the customized modification including replacement, deletion or insertion of the content.

[0124] Optionally, the execution module 703 is configured to:

[0125] parsing the target data fragment, constructing a target tree, determining a page modification expression according to the user-defined content filtering rule, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression;

[0126] obtaining an expression of the nested relationship and attribute characteristics of the HTML tag object;

[0127] verifying whether the HTML tag object satisfies the nested relationship and attribute characteristics described in the expression, and if so, determining that the HTML tag object and the contained content are the HTML tag range to be processed.

[0128] Optionally, the execution module 703 is configured to:

[0129] embedding the first attribute value into the page modification expression when the content filtering rule uses a regular expression to describe the first attribute value;

[0130] extracting a second attribute value of the HTML tag object, describing the second attribute value by a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression;

[0131] when the first attribute value and the second attribute value match successfully, confirming that the HTML tag object satisfies the attribute characteristic requirement described in the page modification expression.

[0132] Optionally, the execution module 703 is configured to:

[0133] checking whether the replacement content defined in the page modification expression for the HTML tag range is empty after locating the HTML tag range to be processed;

[0134] if the replacement content defined in the page modification expression for the HTML tag range is empty, marking the content contained in the HTML tag range as a region to be deleted.

[0135] Optionally, the system further comprises an association module, the association module being configured to:

[0136] creating and maintaining a context state for storing modification information across data shards, the context state being a memory data structure or a persistent storage;

[0137] when a new data shard is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data shard;

[0138] when there is an uncompleted modification task associated with the new data shard, continuing to perform the corresponding modification operation according to the information in the context state.

[0139] Optionally, the system further comprises a parallel module, which is configured to:

[0140] by multi-thread parallel processing of data shards of multiple streaming hypertext contents, using a lock mechanism to share data among the multi-threads, and through a message queue to ensure the correctness of the processing order and data dependency relationship of the multi-threads.

[0141] It should be noted that: the apparatus provided in the above embodiments, when realizing its functions, only takes the above-mentioned division of each functional module as an example for illustration, and in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0142] The embodiment also discloses an electronic device, which refers to Figure 8 The electronic device can include: at least one processor 801, at least one communication bus 802, a user interface 803, a network interface 804, and at least one memory 805.

[0143] The communication bus 802 is used to realize the connection and communication between the components.

[0144] The user interface 803 can include a display screen (Display) and a camera (Camera), and the optional user interface 803 can further include a standard wired interface and a wireless interface.

[0145] The network interface 804 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0146] The processor 801 can include one or more processing cores. The processor 801 connects various parts within the server through various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 805, and calling data stored in the memory 805. Alternatively, the processor 801 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 801 can integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 801, but can be realized by a separate chip.

[0147] The memory 805 can include a random access memory (RAM) and a read-only memory (ROM). Optionally, the memory 805 includes a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 805 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area can store data involved in the above-mentioned various method embodiments, etc. The memory 805 can also be at least one storage device located away from the aforementioned processor 801. As shown, the memory 805 as a computer storage medium can include an operating system, a network communication module, a user interface module, and an application program of the processing method of the streaming hypertext content. Figure 8 As shown, the memory 805 as a computer storage medium can include an operating system, a network communication module, a user interface module, and an application program of the processing method of the streaming hypertext content.

[0148] In Figure 8In the electronic device shown, the user interface 803 is mainly used to provide an interface for user input, and obtain data input by the user; and the processor 801 can be used to invoke an application program of a processing method of the stored streaming hypertext content in the memory 805, which, when executed by one or more processors 801, causes the electronic device to perform the method of one or more of the above-described embodiments.

[0149] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0150] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0151] In several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner for actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical or other forms.

[0152] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0153] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0154] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory 805 and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned memory 805 includes: a U disk, a mobile hard disk, a magnetic or optical disk and various media that can store program codes.

[0155] The above-described are only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the disclosure. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not described in the present disclosure. The scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method of processing streaming hypertext content, characterized by, The method is applied to a streaming hypertext content processing platform, and comprises the following steps: continuously receiving data segments of streaming hypertext content until complete acquisition of HTTP header information, triggering a hook point; at the hook point, calling a registered target plug-in to determine whether customized modification of the streaming hypertext content is required according to the HTTP header information; when customized modification of the streaming hypertext content is required, forming a customized modification condition according to the target plug-in, and directly performing corresponding modification on target data segments of subsequent received data segments that meet the customized modification condition; transmitting the modified data content as new data segments.

2. The method of claim 1, wherein, The direct corresponding modification on target data segments of subsequent received data segments that meet the customized modification condition comprises the following steps: processing the content of the target data segment by using a keyword matching algorithm to obtain keywords that match a preset keyword list; starting from a target position where the keywords are located, scanning the HTML code forward and backward until a demarcation point that meets the HTML tag rule is found, and determining the range of the HTML tag that needs to be processed according to the demarcation point; performing customized modification on the content in the range of the HTML tag, and the customized modification comprises replacement, deletion or insertion of the content.

3. The method of claim 2, wherein, The forward and backward scanning of the HTML code until a demarcation point that meets the HTML tag rule is found, and the determination of the range of the HTML tag that needs to be processed according to the demarcation point comprises the following steps: parsing the target data segment, constructing a target tree, determining a page modification expression according to a user-defined content filtering rule, and locating the HTML tag object where the keywords are located in the target tree according to the page modification expression; obtaining an expression of the nesting relationship and attribute characteristics of the HTML tag object; verifying whether the HTML tag object meets the nesting relationship and attribute characteristics described in the expression, and if so, determining that the HTML tag object and the contained content are the range of the HTML tag that needs to be processed.

4. The method of claim 3, wherein, The verification of whether the HTML tag object meets the nesting relationship and attribute characteristics described in the expression comprises the following steps: using a regular expression to describe a first attribute value in a content filtering rule, and embedding the first attribute value into the page modification expression; extracting a second attribute value of the HTML tag object, describing the second attribute value by using a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression; when the first attribute value and the second attribute value are successfully matched, it is confirmed that the HTML tag object meets the attribute characteristic requirement described in the page modification expression.

5. The method of claim 3, wherein, The direct corresponding modification on target data segments of subsequent received data segments that meet the customized modification condition comprises the following steps: after locating the range of the HTML tag that needs to be processed, checking whether replacement content defined in the page modification expression for the range of the HTML tag is empty; If the replacement content defined in the page modification expression for the HTML tag range is empty, mark the content contained in the HTML tag range as a region to be deleted.

6. The method of claim 1, wherein, The method further comprises: creating and maintaining a context state for storing modification information across data shards, the context state being a memory data structure or a persistent storage; when a new data shard is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data shard; when there is an uncompleted modification task associated with the new data shard, continuing to perform the corresponding modification operation according to the information in the context state.

7. The method of claim 1, wherein, The method further comprises: by multi-thread parallel processing of multiple data shards of streaming hypertext content, using a lock mechanism to share data among multiple threads, and through a message queue to ensure the correctness of the processing order and data dependency relationship of the multiple threads.

8. A system for processing streaming hypertext content, characterized by It comprises a collection module, a judgment module, an execution module and an integration module, wherein: The collection module is configured to continuously receive data shards of streaming hypertext content until the HTTP header information is completely acquired, triggering a hook point. The judgment module is configured to call the registered target plug-in at the hook point to determine whether the streaming hypertext content needs to be customized and modified according to the HTTP header information. The execution module is configured to form a customized modification condition according to the target plug-in when the streaming hypertext content needs to be customized and modified, and directly modify the target data shard that meets the customized modification condition in the subsequent received data shard. The integration module is configured to transmit the modified data content as a new data shard.

9. An electronic device, comprising: The electronic device comprises a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to make the electronic device execute the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions are executed, the method of any one of claims 1-7 is executed.

Images