Streaming hypertext content processing method, system and equipment and storage medium

Abstract

The invention discloses a streaming hypertext content processing method, system and device and a storage medium, and relates to the field of data processing. In the method, data fragments of streaming hypertext content are continuously received until HTTP head information is completely obtained, and a hook point is triggered; on the hook point, calling a registered target plug-in to judge whether the streaming hypertext content needs to be subjected to customized modification or not according to the HTTP header information; when the streaming hypertext content needs to be subjected to customized modification, a customized modification condition is formed according to the target plug-in, and a target data fragment meeting the customized modification condition in the subsequently received data fragments is directly and correspondingly modified; and transmitting the modified data content as a new data fragment. By implementing the technical scheme provided by the invention, efficient and accurate streaming hypertext content customization processing can be realized.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and in particular to a method, system, device and storage medium for processing streaming hypertext content. Background Art

[0002] Streaming hypertext content processing is one of the key technologies in modern Internet applications and is widely used in web page acceleration, content filtering, advertising, etc. With the continuous growth of Internet traffic, efficient and real-time processing of streaming hypertext content has become a key factor in improving user experience and service quality.

[0003] Existing streaming hypertext content processing technologies have obvious deficiencies in dealing with complex customized requirements. For example, in the process of dynamically generating and modifying streaming data, it is not possible to flexibly perform accurate personalized processing according to different business scenarios and user needs. In addition, when processing a large number of concurrent requests, existing technologies are prone to performance bottlenecks and data consistency issues, resulting in reduced service quality and user experience.

[0004] Therefore, there is an urgent need for a streaming hypertext content processing method that can effectively support complex customized requirements to overcome the limitations of existing technologies. Summary of the invention

[0005] The present application provides a method, system, device and storage medium for processing streaming hypertext content, which can realize efficient and accurate customized processing of streaming hypertext content and improve the processing capability and response speed of the system.

[0006] In a first aspect of the present application, a method for processing streaming hypertext content is provided, which is applied to a streaming hypertext content processing platform. The method comprises: Continue to receive data segments of streaming hypertext content until the HTTP header information is completely obtained, triggering the hook point; At the hook point, calling a registered target plug-in to determine whether it is necessary to customize the streaming hypertext content according to the HTTP header information; When it is necessary to make customized modifications to the streaming hypertext content, a customized modification condition is formed according to the target plug-in, and corresponding modifications are directly made to the target data fragments that meet the customized modification condition in the subsequently received data fragments; The modified data content is transmitted as a new data segment.

[0007] Optionally, directly modifying the target data slices satisfying the customized modification condition in the subsequently received data slices accordingly includes: Using a keyword matching algorithm to process the content of the target data segment to obtain keywords that match a preset keyword list; Starting from the target position where the keyword is located, scanning the HTML code forward and backward until a demarcation point that meets the HTML tag rule is found, and determining the range of HTML tags to be processed according to the demarcation point; The content within the HTML tag range is customized and modified, and the customized modification includes replacing, deleting or inserting the content.

[0008] Optionally, scanning the HTML code forward and backward until a demarcation point that complies with HTML tag rules is found, and determining the range of HTML tags to be processed according to the demarcation point includes: Parsing the target data slices, constructing a target tree, determining a page modification expression according to a content filtering rule defined by a user, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression; Obtaining the nesting relationship and attribute feature expressions of the HTML tag object; Verify whether the HTML tag object satisfies the nesting relationship and attribute characteristics described in the expression. If so, determine that the HTML tag object and the included content are the HTML tag range that needs to be processed.

[0009] Optionally, the verifying whether the HTML tag object satisfies the nested relationship and attribute characteristics described in the expression includes: Using a regular expression to describe a first attribute value in a content filtering rule, and embedding the first attribute value into the page modification expression; Extracting a second attribute value of the HTML tag object, describing the second attribute value through a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression; When the first attribute value matches the second attribute value successfully, it is confirmed that the HTML tag object meets the attribute feature requirements described in the page modification expression.

[0010] Optionally, directly modifying the target data slices satisfying the customized modification condition in the subsequently received data slices accordingly includes: After locating the HTML tag range to be processed, checking whether the replacement content defined for the HTML tag range in the page modification expression is empty; If the replacement content defined for the HTML tag range in the page modification expression is empty, the content included in the HTML tag range is marked as a to-be-deleted area.

[0011] Optionally, the method further includes: Create and maintain a context state for storing modification information across data shards, where the context state is an in-memory data structure or persistent storage; When a new data segment is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data segment; When there is an unfinished modification task associated with the new data slice, the corresponding modification operation continues to be performed according to the information in the context state.

[0012] Optionally, the method further includes: The data fragments of multiple streaming hypertext contents are processed in parallel by multi-threading, a lock mechanism is adopted to share data among the multi-threads, and the correctness of the processing sequence and data dependency of the multi-threads is ensured by a message queue.

[0013] In a second aspect of the present application, a system for processing streaming hypertext content is provided, comprising a collection module, a judgment module, an execution module and an integration module, wherein: The acquisition module is configured to continuously receive data segments of streaming hypertext content until the HTTP header information is completely obtained, triggering the hook point; A judgment module configured to call a registered target plug-in at the hook point to judge whether it is necessary to perform customized modification on the streaming hypertext content according to the HTTP header information; An execution module configured to form a customized modification condition according to the target plug-in when a customized modification is required for the streaming hypertext content, and directly perform corresponding modification on the target data fragments satisfying the customized modification condition in the subsequently received data fragments; The integration module is configured to transmit the modified data content as a new data segment.

[0014] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes any one of the methods described above.

[0015] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions, and when the instructions are executed, any of the methods described above is executed.

[0016] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By continuously receiving data segments of streaming hypertext content and triggering the hook point immediately after completely obtaining the HTTP header information, it helps to judge and process data in a timely manner, avoids unnecessary waiting time, and improves the efficiency of data processing; 2. By calling the registered target plug-in at the hook point, it is possible to flexibly determine whether the streaming hypertext content needs to be customized based on the HTTP header information. This design enables the processing flow to be dynamically adjusted according to specific needs, enhancing the flexibility and adaptability of the system; 3. Only the target data shards that meet the customized modification conditions are modified, avoiding the comprehensive scanning and processing of the entire streaming data, thereby reducing unnecessary resource consumption. This targeted processing method helps to improve the performance of the system, especially when processing large-scale streaming data; 4. The modified data shards are reassembled into complete streaming data, which ensures the integrity and consistency of the data. At the same time, since the modification operation is performed during the data reception process, users can obtain customized content without interrupting the data flow; 5. The function of customized modification is realized through the plug-in mechanism, which makes the system easy to expand and maintain. When new customized requirements need to be added, only new plug-ins need to be developed and registered without modifying the existing processing logic; 5. Since the embodiment of the present application performs customized modification during the streaming data reception process, it can respond to changes in user needs in real time, which is particularly important for real-time application scenarios that require rapid response. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for processing streaming hypertext content disclosed in an embodiment of the present application; Figure 2 It is a schematic diagram of the original access page disclosed in the embodiment of the present application; Figure 3 It is a schematic diagram of a partial HTML source code structure of the original access page disclosed in the embodiment of the present application; Figure 4 It is a schematic diagram of an updated original access page disclosed in the embodiment of the present application; Figure 5 is a schematic diagram of a page containing JSON data disclosed in an embodiment of the present application; Figure 6 is a schematic diagram of an updated page containing JSON data disclosed in an embodiment of the present application; Figure 7 It is a module schematic diagram of a system for processing streaming hypertext content disclosed in an embodiment of the present application; Figure 8It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application.

[0018] Explanation of the reference numerals: 701, acquisition module; 702, judgment module; 703, execution module; 704, integration module; 801, processor; 802, communication bus; 803, user interface; 804, network interface; 805, memory. DETAILED DESCRIPTION

[0019] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0020] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0021] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0022] This embodiment discloses a method for processing streaming hypertext content, which is applied to a streaming hypertext content processing platform. Figure 1 is a flow chart of a method for processing streaming hypertext content disclosed in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: S110, continuously receiving data segments of the streaming hypertext content until the HTTP header information is completely obtained, triggering the hook point; The proxy service starts receiving streaming hypertext content (i.e., the content of the HTTP request or response) from the client or server. This data content is transmitted step by step in the form of segments (or data blocks). The proxy service starts parsing when it receives the first data segment and continues this process until it can fully parse the HTTP header information. The HTTP header information contains important metadata about the request or response, such as the target URL, request method, status code, header fields, etc. When the HTTP header information is fully parsed, the proxy service triggers a specific hook point (or callback point, event point). This hook point is a preset location of the proxy service, which is used to execute the developer-registered operations or plug-ins when a specific event occurs (such as the completion of HTTP header parsing).

[0023] S120, at the hook point, calling the registered target plug-in to determine whether it is necessary to customize the streaming hypertext content according to the HTTP header information; According to business needs, one or more function processing functions can be registered at appropriate hook points. For example, at the hook point after the response header is parsed, developers can obtain the various fields of the parsed HTTP header, including content-type, content-length, content-encoding, transfer-encoding, set-cookie, etc., and can add HTTP fields, modify existing fields, and delete specified fields according to business needs; they can also determine certain fields and decide to register plug-ins for the response content when specific conditions are met, and execute specified function processing functions, such as fully replacing certain keywords in the response text page when the host field contains github.com and the content-type field value is text / html.

[0024] After the hook point is triggered, the proxy service will call the target plug-in that the developer has previously registered. These plug-ins are code modules written by developers to implement specific functions (such as header information auditing, header field modification, content customization, etc.). The plug-in receives HTTP header information as input and determines whether the streaming hypertext content needs to be customized based on preset logic or rules. For example, the plug-in may check whether the target URL conforms to a specific pattern, or check whether the header field contains a specific value, and then decide whether to make customized modifications based on these judgment results.

[0025] S130: when it is necessary to make customized modification to the streaming hypertext content, a customized modification condition is formed according to the target plug-in, and corresponding modification is directly made to the target data slices satisfying the customized modification condition in the subsequently received data slices; If the plug-in determines that customized modification of the streaming hypertext content is required, it will form customized modification conditions based on its own logic or rules. These conditions may include specific text patterns, data ranges, field values, etc., which are used to guide subsequent modification operations. As the proxy service continues to receive subsequent data shards, it checks whether each shard meets the customized modification conditions set by the plug-in. For target data shards that meet the conditions, the proxy service will directly modify them according to the modification rules or operations provided by the plug-in. This modification may be the addition, deletion, modification, and query of data, or it may be the recoding or formatting of specific fields or data.

[0026] Optionally, directly modifying the target data slices satisfying the customized modification condition in the subsequently received data slices accordingly includes: Using a keyword matching algorithm to process the content of the target data segment to obtain keywords that match a preset keyword list; Starting from the target position where the keyword is located, scanning the HTML code forward and backward until a demarcation point that meets the HTML tag rule is found, and determining the range of HTML tags to be processed according to the demarcation point; The content within the HTML tag range is customized and modified, and the customized modification includes replacing, deleting or inserting the content.

[0027] The proxy service or HTML processing system will perform keyword matching processing on the subsequently received data segments (i.e., streaming data blocks of HTML content). This usually involves using efficient keyword matching algorithms, such as regular expression matching, string search algorithms, etc., to search for keywords that match the preset keyword list in the data segments. The preset keyword list is defined by the developer based on business needs and contains specific words or phrases in the HTML content that require special attention or processing. When a keyword that matches the preset keyword list is found in the data segment, the system will scan the HTML code forward and backward starting from the location of the keyword (i.e., the target location). The purpose of the scan is to find demarcation points that comply with HTML tag rules, which define the range of HTML tags containing keywords. This usually involves parsing the nesting relationship and attribute characteristics of HTML tags to ensure accurate identification of the range of HTML tags that need to be processed. During the scanning process, the system will use HTML parsing technologies (such as DOM parsing, SAX parsing, etc.) to identify and process HTML tags. These technologies allow the system to understand the structure of HTML documents, including the opening and closing of tags, the setting of attributes, etc. By scanning and parsing, the system can determine the starting and ending positions of HTML tags containing keywords, thereby determining the range of HTML tags that need to be processed. After determining the range of HTML tags that need to be processed, the system can make customized modifications to the content within the range. Customized modifications can include a variety of operations, such as replacing, deleting, or inserting content. The specific implementation of these operations depends on the developer's needs and business logic.

[0028] By using the keyword matching algorithm, the keywords that match the preset keyword list can be quickly located in the target data shard. This efficient matching mechanism ensures that the system can process streaming data in real time and respond to customized modification needs in a timely manner. Starting from the target location where the keyword is located, the system can scan the HTML code forward and backward and accurately find the demarcation point that meets the HTML tag rules. This accuracy ensures that only the content within the HTML tag range that is directly related to the keyword will be processed, thus avoiding unnecessary global parsing and modification. The system supports a variety of customized modifications to the content within the HTML tag range, including replacement, deletion or insertion of content. This flexibility allows developers to accurately adjust and optimize HTML pages according to specific needs. Since the system only processes the target data shards that meet the customized modification conditions and concentrates computing resources to perform high-precision parsing of HTML objects related to the keyword occurrence position, it can significantly reduce the parsing granularity and resource consumption of the global page. This optimization improves the processing efficiency of the system, especially when processing large-scale streaming HTML data. By customizing the content within the HTML tag range, developers can achieve precise control over the page content. This helps ensure the accuracy and consistency of page content while improving user experience and page performance. The technical framework supports developers to expand system functions by defining new keyword lists and customizing modification rules. This scalability enables the system to adapt to changing needs and scenarios. At the same time, since the system adopts a modular design concept, it is also convenient for subsequent maintenance and upgrades.

[0029] Optionally, scanning the HTML code forward and backward until a demarcation point that complies with HTML tag rules is found, and determining the range of HTML tags to be processed according to the demarcation point includes: Parsing the target data slices, constructing a target tree, determining a page modification expression according to a content filtering rule defined by a user, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression; Obtaining the nesting relationship and attribute feature expressions of the HTML tag object; Verify whether the HTML tag object satisfies the nesting relationship and attribute characteristics described in the expression. If so, determine that the HTML tag object and the included content are the HTML tag range that needs to be processed.

[0030] Perform HTML parsing on the received target data fragments. The purpose of this step is to convert the HTML code into a data structure that is easy to operate and query, usually called a DOM (Document Object Model) tree (i.e., the target tree). In the DOM tree, each element of the HTML document is represented as a node, and the nodes are connected through parent-child relationships, sibling relationships, etc., forming a tree structure. After the DOM tree is constructed, the system determines the page modification expression based on the content filtering rules defined by the user. These rules are usually defined by the user based on specific needs to specify the HTML elements and their attribute characteristics that need to be filtered or modified. The page modification expression is a specific syntax or format used to describe these rules so that the system can understand and execute them. Use the page modification expression to locate the HTML tag object containing the keyword in the DOM tree. This step usually involves traversing and searching the DOM tree to find the node that matches the page modification expression. Once the matching node is found, the system can determine the HTML tag object where the keyword is located. After determining the HTML tag object where the keyword is located, the system obtains the expression of the nesting relationship and attribute characteristics of the tag object. These expressions describe the position of the tag object in the DOM tree, its nesting relationship with other tags, and its own attribute characteristics (such as class, id, etc.). This information is crucial for the subsequent verification of whether the tag object meets the user-defined rules. The system verifies whether the HTML tag object meets the nesting relationship and attribute characteristics described in the user-defined expressions. This step usually involves matching and comparing the attributes of the tag object to ensure that they meet the user-specified rules. If the tag object meets all conditions, the system can determine that the tag object and its content are the HTML tag range that needs to be processed.

[0031] By parsing the target data segments, the system can build an accurate target tree (such as a DOM tree), which reflects the structure and hierarchical relationship of the HTML document. This accurate parsing is the basis for subsequent steps and ensures the accuracy and reliability of subsequent processing. Based on the content filtering rules defined by the user, the system can determine the page modification expressions. These expressions describe the specific location and characteristics of the HTML tag objects that need to be filtered or modified. By matching these expressions with the target tree, the system can efficiently locate the HTML tag objects where the keywords are located. The system can obtain the expressions of the nesting relationship and attribute characteristics of the HTML tag objects and verify them based on these expressions. This flexibility enables the system to handle various complex HTML structures, including nested tags and tags with specific attributes. Through verification, the system can ensure that only HTML tag objects and their contents that meet specific conditions will be processed. When the HTML tag objects that meet the page modification expressions are determined, the system can determine the range of HTML tags that need to be processed based on the nesting relationship and attribute characteristics of these objects. This accurate range determination avoids unnecessary global parsing and modification, thereby improving processing efficiency and accuracy. By building the target tree and using page modification expressions for positioning, the system can avoid repeated parsing and scanning of global HTML documents. This optimization reduces resource consumption and improves processing speed, especially when processing large-scale HTML data. The technical framework supports the extension of user-defined content filtering rules and page modification expressions. This means that developers can add new rules or expressions according to specific needs to adapt to changing needs and scenarios. At the same time, since the system adopts a modular design concept, it is also convenient for subsequent maintenance and upgrades.

[0032] Optionally, the verifying whether the HTML tag object satisfies the nested relationship and attribute characteristics described in the expression includes: Using a regular expression to describe a first attribute value in a content filtering rule, and embedding the first attribute value into the page modification expression; Extracting a second attribute value of the HTML tag object, describing the second attribute value through a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression; When the first attribute value matches the second attribute value successfully, it is confirmed that the HTML tag object meets the attribute feature requirements described in the page modification expression.

[0033] In the content filtering rule, a regular expression is first used to describe a certain attribute value of the HTML tag object of interest (i.e., the first attribute value). This attribute value can be a class name (class), ID, data attribute (data-*), etc., depending on the characteristics of the tag object you want to locate. This first attribute value described by the regular expression is embedded in the page modification expression. The page modification expression is a more extensive rule that describes how to modify or filter a specific part of the HTML document. By embedding the first attribute value, you are actually specifying a condition, that is, only when a certain attribute value of the HTML tag object meets this condition, the tag object will be selected for subsequent processing. In the process of parsing the HTML document and building the DOM tree, the system traverses the tag objects in the document and extracts the attribute value of interest (i.e., the second attribute value). This value is obtained from the actual HTML tag and is used to match the first attribute value in the page modification expression. In order to match, the second attribute value also needs to be converted into a regular expression (i.e., the target regular expression). This usually means treating the second attribute value as an ordinary string and encapsulating it in the matching pattern of the regular expression. However, in most cases, if the second attribute value itself does not contain any regular expression special characters that need to be escaped, it can be used directly as a fixed string pattern for matching. Match the target regular expression with the first attribute value in the page modification expression. If the match is successful, it means that the attribute value of the HTML tag object meets the conditions specified in the content filtering rule. When the first attribute value matches the second attribute value successfully, it can be confirmed that the HTML tag object meets the attribute feature requirements described in the page modification expression. This means that the tag object is one of the targets you want to modify or filter. After confirming the tag objects that meet the conditions, the system can further determine the range of HTML tags that need to be processed based on other parts of the page modification expression (such as nested relationships, tag names, etc.), and perform corresponding modification or filtering operations.

[0034] Using regular expressions to describe the first attribute value in content filtering rules enables the system to flexibly match various complex attribute features. The powerful matching ability of regular expressions ensures that the system can accurately identify HTML tag objects that meet specific conditions. The system extracts the second attribute value of the HTML tag object and uses regular expressions to describe this value, thereby obtaining a target regular expression. By matching this target regular expression with the first attribute value in the page modification expression, it can accurately verify whether the HTML tag object meets the attribute feature requirements described in the page modification expression. Due to the efficient matching algorithm of regular expressions, the system can complete a large number of matching operations in a short time. This ensures that the system can still maintain high processing efficiency when processing large-scale HTML data. Using regular expressions to describe attribute features enables the system to easily add new attribute feature matching rules. This scalability enables the system to adapt to changing needs and scenarios, and improves the flexibility and practicality of the system. Through precise matching algorithms and strict constraints of regular expressions, the system can reduce misjudgments and missed judgments. This ensures that the system can maintain high accuracy and reliability when processing HTML tag objects. Due to the efficiency and accuracy of regular expressions, the system can avoid unnecessary global scanning and parsing. This reduces resource consumption and improves system performance.

[0035] Optionally, directly modifying the target data slices satisfying the customized modification condition in the subsequently received data slices accordingly includes: After locating the HTML tag range to be processed, checking whether the replacement content defined for the HTML tag range in the page modification expression is empty; If the replacement content defined for the HTML tag range in the page modification expression is empty, the content included in the HTML tag range is marked as a to-be-deleted area.

[0036] The system needs to locate the HTML tag range that needs to be processed. This is usually achieved by parsing the HTML document and based on the selector or tag name specified in the page modification expression. The selector can be a CSS selector, an XPath expression, or other method that can uniquely identify an HTML tag or tag range. When the HTML tag range that needs to be processed is located, the system will then check the replacement content defined for this range in the page modification expression. A page modification expression is a predefined rule that specifies how to modify a specific part of an HTML document. This expression may contain multiple parts such as selectors, attribute conditions, and replacement content. The system will determine whether the replacement content defined for the current HTML tag range is empty based on the definition in the page modification expression. Here, "empty" may mean that no specific replacement text is provided, or the replacement text is an empty string. If the system finds that the replacement content defined for the current HTML tag range in the page modification expression is empty, it will mark the content contained in this HTML tag range as a to-be-deleted area. This means that in the subsequent modification process, the content of this area will be deleted or removed instead of being replaced with other content. The system will actually modify the HTML document based on the previous judgment results. For areas marked as to-be-deleted, the system will delete all the content in this area. If specific replacement content is provided in the page modification expression, the system will insert this content into the corresponding position to replace the original content.

[0037] When the system locates the HTML tag range that needs to be processed, it checks whether the replacement content defined for the range in the page modification expression is empty. If it is empty, the system marks the content contained in the HTML tag range as a to-be-deleted area. This precise operation ensures that only HTML content that meets specific conditions is deleted, avoiding unnecessary global deletion operations. By directly marking the to-be-deleted area, the system avoids unnecessary parsing and modification operations. This optimization reduces resource consumption and improves the processing efficiency of the system. This optimization effect is particularly significant when processing large-scale HTML data. The system can flexibly process the content of the HTML tag range according to different page modification expressions. When the replacement content is empty, the system can automatically perform the deletion operation without additional configuration or rules. This flexibility enables the system to adapt to changing needs and scenarios. Through clear replacement content checking logic, the system can accurately determine when to perform the deletion operation. This clear logic reduces the risk of misoperation and ensures the stability and reliability of the system. Precise deletion operations and optimized resource usage enable the system to process HTML data faster, thereby improving the user experience. Users can see the modified results faster, reducing waiting time.

[0038] Specifically, taking HTML as an example, the embodiment of the present application divides the structured data tag-level processing operation into three steps: filtering keywords in the data segment, expanding forward and backward from the location of the keywords to find the tag boundary that meets the rule description; processing the content of this tag. The page modification expression of the content filtering rule uses the running expression that describes the HTML tag name and the nested logic between them to describe the specific location of the object (containing keywords) to be filtered in the HTML. The expression supports the description of HTML tags such as DIV, SPAN, LI, H1-5, A, P, etc. Each tag can describe the attribute characteristics of the tag, such as the class="name" feature in the DIV tag. Each expression is required to contain at least one DIV tag description, and the last DIV tag description must contain the attribute characteristics of the DIV tag. The embodiment of the present application can reduce the parsing granularity of the HTML global page, concentrate computing resources, and perform high-precision parsing on HTML objects related to the keyword appearance position, while other irrelevant HTML text elements can be quickly skipped during the parsing process. A highly efficient HTML page content parsing and filtering mechanism is implemented.

[0039] The page modification expression of a user-defined content filtering rule consists of the following two parts: 1. Describe the nested relationship of HTML tag objects where keywords appear and the expression of attribute characteristics of each tag object. The attribute characteristics of HTML tag objects support the format described by regular expressions to cope with the flexible and changeable characteristics of HTML tag attributes in which the appearance position is not fixed.

[0040] 2. The target content of the keyword object content replacement is the content that replaces the content of the HTML object that hits the above expression. It supports string format and can be empty. When the target content of the replacement is empty, the current expression means deleting the HTML object. At this time, the length of the page content will change, and the HTML object tag (such as <div ...> .. ) The text content will be deleted.

[0041] S140: Transmit the modified data content as a new data segment.

[0042] When processing a data stream, the data is first modified and the modified content is treated as a new data segment. These new data segments may be generated in different ways, such as directly modifying the original data. To ensure that each new data segment can be uniquely identified, a unique identifier (such as a timestamp, sequence number, etc.) is usually assigned to each data segment. These identifiers not only help track the data segments during transmission, but can also be used to sort the data in subsequent processing. When preparing these new data segments for transmission, it is important to maintain the integrity and accuracy of the data. This may involve verifying the data, such as calculating a checksum or hash value, to ensure that the data has not been tampered with or damaged during transmission. At the same time, since the data segments are new, they usually do not need to be sorted in the order of the original data, but can be organized according to actual needs or the requirements of the transmission protocol. During the transmission process, it is necessary to ensure the real-time and availability of the data segments to meet business needs. This may involve using appropriate transmission protocols and technologies, such as TCP / IP, to ensure that the data segments can reliably reach the target system or application platform. After arriving at the target, the receiving system may need to verify the integrity and accuracy of each data segment. This is usually done by recalculating a checksum or hash value and comparing it to the value generated during transmission. If the verification passes, the data shard is considered valid and can be further processed or stored based on business needs.

[0043] Figure 2 is a schematic diagram of the original access page disclosed in the embodiment of this application, such as Figure 2 As shown, the keyword is "xmind mind map". Figure 3 This is a schematic diagram of a partial HTML source code structure of the original access page disclosed in the embodiment of the present application, and the box contains text content including HTML object tags.

[0044] Configure "xmind mind map" to scan keywords and configure the tag features of deletion (replace with empty) as follows: cond %{READ_RESPONSE_HDR_HOOK} cond %{HTML-BODY:KEYWORD} update-response-html DIV(class\=\"md-content\").ARTICLE().DIV(id\=\"post\")"".

[0045] Figure 4 is a schematic diagram of the updated original access page disclosed in the embodiment of the present application, such as Figure 4 As shown, the target paragraph is deleted. Figure 5is a schematic diagram of a page containing JSON data disclosed in an embodiment of the present application. Similar effects can also be achieved by using JSON data. A page containing JSON data is as follows Figure 5 Configure "ting2" as a scan keyword and configure the updated tag features as follows: cond %{READ_RESPONSE_HDR_HOOK} cond %{HTML-BODY:KEYWORD} update-response-html JSON(qzk).JSON(qzk2).JSON(qzk3).JSON(qzk4).JSON(qzk5).JSON(qzk6).JSON(addr)"flowtest".

[0046] Figure 6 is a schematic diagram of an updated page containing JSON data disclosed in an embodiment of the present application, such as Figure 6 As shown, the "addr" field containing the keyword in the response content obtained after the proxy program will be updated to the configuration value.

[0047] Optionally, the method further includes: Create and maintain a context state for storing modification information across data shards, where the context state is an in-memory data structure or persistent storage; When a new data segment is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data segment; When there is an unfinished modification task associated with the new data slice, the corresponding modification operation continues to be performed according to the information in the context state.

[0048] The context state is an in-memory data structure or persistent storage used to store cross-data shard modification information. It can contain various types of information, such as a unique identifier for the modification task, an identifier for the data shard, the type and content of the modification operation, the status of the modification task (e.g., unfinished, completed), etc. The context state needs to persist throughout the data processing flow and be continuously updated as data shards are received and processed. In an in-memory data structure, the context state is usually managed as a global variable or a thread-local variable. In a persistent storage, a database, file system, or other storage medium can be used to save the context state. When the system receives a new data shard, it first needs to be identified and parsed. Before parsing the new data shard, the system needs to check the context state to determine if there are any unfinished modification tasks associated with the data shard. This is usually achieved by comparing the identifier of the data shard with the information in the context state. If there are unfinished modification tasks associated with the new data shard, the system needs to determine the specific content and status of these tasks. Based on the information in the context state, the system can continue to execute the modification tasks associated with the new data shard. This may involve modifying the content of the data shard, adding new data elements, deleting unnecessary data, etc. After performing the modification operation, the system needs to update the context state to reflect the status and results of the modification task. This may include marking the modification task as completed, updating the identifier or status of the data shard, adding a new modification task, etc.

[0049] By creating and maintaining context state, the system is able to store modification information across data shards. This means that when the system receives a new data shard, it can check the context state to determine if there are any unfinished modification tasks related to the data shard. If so, the system can continue to perform the corresponding modification operation based on the information in the context state, thereby ensuring the continuity of data processing. The context state not only stores the information of the modification task, but also may store the intermediate state or temporary results during the modification process. This enables the system to perform subsequent operations based on the previous modification history and state when processing new data shards, thereby improving the accuracy and consistency of data modification. By maintaining the context state, the system can avoid repeatedly parsing and modifying the same data shard, thereby reducing unnecessary resource consumption. In addition, the context state can also help the system manage and schedule modification tasks more efficiently, further improving the processing performance of the system. The context state provides the system with a mechanism to store and process modification information across data shards, which enables the system to support more complex data processing logic. For example, the system can handle complex operations that require aggregation, filtering, or transformation across multiple data shards. The context state can be an in-memory data structure or a persistent storage, depending on the specific needs and resource constraints of the system. This flexibility enables the system to select the most suitable storage method according to actual needs, thereby improving the scalability and flexibility of the system.

[0050] Optionally, the method further includes: The data fragments of multiple streaming hypertext contents are processed in parallel by multi-threading, a lock mechanism is adopted to share data among the multi-threads, and the correctness of the processing sequence and data dependency of the multi-threads is ensured by a message queue.

[0051] Multithreaded parallel processing is a programming model that allows multiple tasks to be executed simultaneously, which can improve the execution efficiency of the program and the system throughput. When processing data slices of streaming hypertext content, multithreaded parallel processing can make full use of the computer's multi-core CPU resources to process multiple data slices at the same time, thereby speeding up the processing speed. Each thread can independently process a data slice and perform operations such as parsing, filtering, and conversion. Since multithreaded parallel processing can execute multiple tasks at the same time, it can significantly reduce the total time required to process a large number of data slices. In multithreaded programming, sharing data is a common requirement. However, since multiple threads may access and modify shared data at the same time, certain synchronization mechanisms are required to ensure the consistency and integrity of the data. The lock mechanism is a commonly used synchronization mechanism, which can be implemented through a mutex. When a thread needs to access shared data, it first tries to acquire the mutex. If the lock is already held by another thread, the current thread will be blocked until the lock is released. In this way, it can be ensured that only one thread can access the shared data at any time, thereby avoiding problems such as data competition and deadlock. When processing data slices of streaming hypertext content, data dependencies may exist. For example, the processing result of a data slice may depend on the processing result of the previous data slice. In order to ensure the correctness of the multi-threaded processing order and data dependency, a message queue can be used. A message queue is a first-in-first-out data structure that ensures that messages are received and processed in the order in which they are sent. When processing data slices of streaming hypertext content, each data slice can be encapsulated as a message and sent to a message queue. Then, multiple threads can take out messages from the message queue in turn for processing. Through the message queue, the following points can be ensured: the processing order of messages is consistent with the sending order, thereby ensuring the correctness of data dependencies; the message queue can act as a buffer. When a thread processes slowly, other threads can continue to process subsequent messages, thereby improving the system's concurrency and throughput; the message queue can also implement a message retry and compensation mechanism. When a message fails to be processed, it can be put back into the message queue for processing, thereby improving the reliability and stability of the system.

[0052] By processing multiple data shards in parallel through multithreading, the throughput and processing speed of the system can be significantly improved. When processing a large amount of streaming hypertext content, multithreaded parallel processing can complete tasks faster, reduce user waiting time, and improve user experience. Lock mechanisms (such as mutex locks, read-write locks, etc.) are used to ensure that there is no conflict when multiple threads access shared data, ensuring data consistency and security. The use of lock mechanisms can also avoid problems such as data competition and deadlock, ensuring the stable operation of multithreaded programs. The message queue is used to ensure the correctness of the processing order and data dependencies of multiple threads, avoiding data errors or inconsistencies caused by improper processing order. When processing streaming hypertext content, there may be complex dependencies between data. The combination of message queues and lock mechanisms can ensure that these dependencies are properly processed, avoiding data inconsistencies or errors.

[0053] This embodiment also discloses a system for processing streaming hypertext content. Figure 7 is a module diagram of a processing system for streaming hypertext content disclosed in an embodiment of the present application, such as Figure 7 As shown, the system includes a collection module 701, a judgment module 702, an execution module 703 and an integration module 704, wherein: The acquisition module 701 is configured to continuously receive data segments of the streaming hypertext content until the HTTP header information is completely obtained, thereby triggering the hook point; The judgment module 702 is configured to call the registered target plug-in at the hook point to judge whether it is necessary to customize the streaming hypertext content according to the HTTP header information; The execution module 703 is configured to form a customized modification condition according to the target plug-in when the streaming hypertext content needs to be customized, and directly perform corresponding modification on the target data fragments that meet the customized modification condition in the subsequently received data fragments; The integration module 704 is configured to transmit the modified data content as a new data segment.

[0054] Optionally, the execution module 703 is configured to: Using a keyword matching algorithm to process the content of the target data segment to obtain keywords that match a preset keyword list; Starting from the target position where the keyword is located, scanning the HTML code forward and backward until a demarcation point that meets the HTML tag rule is found, and determining the range of HTML tags to be processed according to the demarcation point; The content within the HTML tag range is customized and modified, and the customized modification includes replacing, deleting or inserting the content.

[0055] Optionally, the execution module 703 is configured to: Parsing the target data slices, constructing a target tree, determining a page modification expression according to a content filtering rule defined by a user, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression; Obtaining the nesting relationship and attribute feature expressions of the HTML tag object; Verify whether the HTML tag object satisfies the nesting relationship and attribute characteristics described in the expression. If so, determine that the HTML tag object and the included content are the HTML tag range that needs to be processed.

[0056] Optionally, the execution module 703 is configured to: Using a regular expression to describe a first attribute value in a content filtering rule, and embedding the first attribute value into the page modification expression; Extracting a second attribute value of the HTML tag object, describing the second attribute value through a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression; When the first attribute value matches the second attribute value successfully, it is confirmed that the HTML tag object meets the attribute feature requirements described in the page modification expression.

[0057] Optionally, the execution module 703 is configured to: After locating the HTML tag range to be processed, checking whether the replacement content defined for the HTML tag range in the page modification expression is empty; If the replacement content defined for the HTML tag range in the page modification expression is empty, the content included in the HTML tag range is marked as a to-be-deleted area.

[0058] Optionally, the system further comprises an association module, wherein the association module is configured to: Create and maintain a context state for storing modification information across data shards, where the context state is an in-memory data structure or persistent storage; When a new data segment is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data segment; When there is an unfinished modification task associated with the new data slice, the corresponding modification operation continues to be performed according to the information in the context state.

[0059] Optionally, the system further comprises a parallel module, wherein the parallel module is configured to: The data fragments of multiple streaming hypertext contents are processed in parallel by multi-threading, a lock mechanism is adopted to share data among the multi-threads, and the correctness of the processing sequence and data dependency of the multi-threads is ensured by a message queue.

[0060] It should be noted that: when the device provided in the above embodiment realizes its function, only the division of the above functional modules is used as an example. In actual application, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0061] This embodiment also discloses an electronic device, referring to Figure 8 The electronic device may include: at least one processor 801 , at least one communication bus 802 , a user interface 803 , a network interface 804 , and at least one memory 805 .

[0062] The communication bus 802 is used to realize the connection and communication between these components.

[0063] The user interface 803 may include a display screen (Display) and a camera (Camera). The optional user interface 803 may also include a standard wired interface and a wireless interface.

[0064] The network interface 804 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0065] Among them, the processor 801 may include one or more processing cores. The processor 801 uses various interfaces and lines to connect various parts in the entire server, and executes various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 805, and calling data stored in the memory 805. Optionally, the processor 801 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 801 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 801, and it can be implemented separately through a chip.

[0066] Among them, the memory 805 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 805 includes a non-transitory computer-readable storage medium. The memory 805 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 805 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 805 may also be optionally at least one storage device located away from the aforementioned processor 801. As Figure 8 As shown, the memory 805 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a method for processing streaming hypertext content.

[0067] exist Figure 8In the electronic device shown, the user interface 803 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 801 can be used to call the application program of the processing method of streaming hypertext content stored in the memory 805. When executed by one or more processors 801, the electronic device executes one or more methods in the above-mentioned embodiments.

[0068] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0069] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0071] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0072] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory 805, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory 805 includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0074] The above is only an exemplary embodiment of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field that are not recorded in the present disclosure. The description and examples are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for processing streaming hypertext content, characterized in that: Applied to a streaming hypertext content processing platform, the method comprises: Continue to receive data segments of streaming hypertext content until the HTTP header information is completely obtained, triggering the hook point; At the hook point, calling a registered target plug-in to determine whether it is necessary to customize the streaming hypertext content according to the HTTP header information; When it is necessary to make customized modifications to the streaming hypertext content, a customized modification condition is formed according to the target plug-in, and corresponding modifications are directly made to the target data fragments that meet the customized modification condition in the subsequently received data fragments; The modified data content is transmitted as a new data segment.

2. The method for processing streaming hypertext content according to claim 1, characterized in that: Directly modifying the target data slices satisfying the customized modification condition in the subsequently received data slices includes: Using a keyword matching algorithm to process the content of the target data segment to obtain keywords that match a preset keyword list; Starting from the target position where the keyword is located, scanning the HTML code forward and backward until a demarcation point that meets the HTML tag rule is found, and determining the range of HTML tags to be processed according to the demarcation point; The content within the HTML tag range is customized and modified, and the customized modification includes replacing, deleting or inserting the content.

3. The method for processing streaming hypertext content according to claim 2, characterized in that: Scanning the HTML code forward and backward until a demarcation point that meets the HTML tag rule is found, and determining the range of HTML tags to be processed according to the demarcation point includes: Parsing the target data slices, constructing a target tree, determining a page modification expression according to a content filtering rule defined by a user, and locating the HTML tag object where the keyword is located in the target tree according to the page modification expression; Obtaining the nesting relationship and attribute feature expressions of the HTML tag object; Verify whether the HTML tag object satisfies the nesting relationship and attribute characteristics described in the expression. If so, determine that the HTML tag object and the included content are the HTML tag range that needs to be processed.

4. The method for processing streaming hypertext content according to claim 3, characterized in that: The verification of whether the HTML tag object satisfies the nesting relationship and attribute characteristics described in the expression includes: Using a regular expression to describe a first attribute value in a content filtering rule, and embedding the first attribute value into the page modification expression; Extracting a second attribute value of the HTML tag object, describing the second attribute value through a regular expression to obtain a target regular expression, and matching the target regular expression with the page modification expression; When the first attribute value matches the second attribute value successfully, it is confirmed that the HTML tag object meets the attribute feature requirements described in the page modification expression.

5. The method for processing streaming hypertext content according to claim 3, characterized in that: Directly modifying the target data slices satisfying the customized modification condition in the subsequently received data slices includes: After locating the HTML tag range to be processed, checking whether the replacement content defined for the HTML tag range in the page modification expression is empty; If the replacement content defined for the HTML tag range in the page modification expression is empty, the content included in the HTML tag range is marked as a to-be-deleted area.

6. The method for processing streaming hypertext content according to claim 1, characterized in that: The method further comprises: Create and maintain a context state for storing modification information across data shards, where the context state is an in-memory data structure or persistent storage; When a new data segment is received, checking the context state to determine whether there is an uncompleted modification task associated with the new data segment; When there is an unfinished modification task associated with the new data slice, the corresponding modification operation continues to be performed according to the information in the context state.

7. The method for processing streaming hypertext content according to claim 1, characterized in that: The method further comprises: The data fragments of multiple streaming hypertext contents are processed in parallel by multi-threading, a lock mechanism is adopted to share data among the multi-threads, and the correctness of the processing sequence and data dependency of the multi-threads is ensured by a message queue.

8. A system for processing streaming hypertext content, characterized in that: It includes acquisition module, judgment module, execution module and integration module, among which: The acquisition module is configured to continuously receive data segments of streaming hypertext content until the HTTP header information is completely obtained, triggering the hook point; A judgment module configured to call a registered target plug-in at the hook point to judge whether it is necessary to perform customized modification on the streaming hypertext content according to the HTTP header information; An execution module configured to form a customized modification condition according to the target plug-in when a customized modification is required for the streaming hypertext content, and directly perform corresponding modification on the target data fragments satisfying the customized modification condition in the subsequently received data fragments; The integration module is configured to transmit the modified data content as a new data segment.

9. An electronic device, characterized in that: It includes a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device executes the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is performed.

Images