Resolution acceleration method, device and equipment based on rule base dynamic adjustment
By dynamically adjusting the parsing rule database, giving priority to the use of high hit rate rules and updating the rule database, the problems of low parsing efficiency and high resource consumption in the existing technology are solved, and the performance and stability of e-commerce product page analysis are improved.
Patent Information
- Application Number
- CN202410920977.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-10
- Publication Date
- 2024-11-08
AI Technical Summary
When analyzing e-commerce product pages, the existing technology needs to try a variety of analysis rules, resulting in poor performance, long time and high resource consumption, and it is impossible to effectively deal with changes in the page structure.
Using a dynamic adjustment method based on the rule base, by obtaining and recording the hits of parsing rules, we give priority to parsing rules with high hit rates, and dynamically update the rule base when the page structure changes to ensure the most effective parsing rules are used.
Improve analysis efficiency and performance, reduce time and resource usage, enhance the system's ability to adapt to page changes, reduce maintenance costs, and ensure the consistency of rules and system stability.
Smart Images

Figure CN118916563A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a parsing speed-up method, device and equipment based on dynamic adjustment of a rule base. Background Art
[0002] At present, in order to provide a reference for sellers’ product selection and operation, it is necessary to obtain the original pages of e-commerce products, keywords, comments, etc., parse out the titles, prices, sales, rankings and other information, and conduct statistics and analysis.
[0003] Due to the large number of products, tens of millions of pages need to be parsed every day, and dozens of fields need to be parsed for each page. Since the pages are often updated, the DOM structure also changes. Some pages need to be parsed with new rules, and the original parsing rules cannot be discarded because some pages still have old structures. In this way, each field must traverse the accumulated parsing rules until the parsing is successful, which has poor performance and consumes a lot of resources. In this way, when parsing data elements from the page, due to the same business, the page has different versions, and parsing a certain data requires trying different parsing rules, which has poor performance.
[0004] In view of this, it is necessary to propose a parsing speed-up method based on dynamic adjustment of the rule base to improve performance and reduce time and resources. Summary of the invention
[0005] The purpose of the present invention is to provide a method, device and equipment for speeding up parsing based on dynamic adjustment of a rule base, so as to overcome the shortcomings of the existing attempts at multiple parsing rules.
[0006] The present invention provides a parsing speed-up method based on dynamic adjustment of a rule base, comprising:
[0007] Obtaining an original HTML page, where the elements on the original HTML page include title, price, comments, sales volume, ranking, and information related to page attributes;
[0008] Based on the rule library, the original HTML page is parsed for the first time by traversing the rule list, converting the HTML string into an operable DOM tree / element list to obtain the parsed page elements;
[0009] When the page structure of the original HTML page changes, the rules that were successfully parsed the last time are obtained and parsed again according to the rules that were successfully parsed the last time;
[0010] When the rules that were successfully parsed in the previous time are not obtained, the rules with high hit rates are prioritized based on the parsing rule hits to perform a second parsing of the page elements until the parsing is successful. When the parsing rule hits change, the rule library is dynamically updated to perform subsequent parsing after the updated rules until the parsing is successful.
[0011] Preferably, the first parsing of the original HTML page by traversing the rule list based on the rule base further comprises:
[0012] Identify the field method that needs to be parsed based on custom annotations;
[0013] Get the value corresponding to the annotation parameter in the path through the get method, and assign the annotation parameter to the string variable parsing rule;
[0014] When the application starts, the annotation is read, and the corresponding parsing rules are read from ZooKeeper according to the annotation information, and the parsing rules are stored in the mapping in the memory;
[0015] The annotation parameters include site, business and field information.
[0016] Preferably, when the page structure of the original HTML page changes, obtaining the rules that were successfully parsed last time and performing a second parsing according to the rules that were successfully parsed last time includes:
[0017] Get the previously parsed successful rule stored in memory and determine whether the rule exists;
[0018] If the rules that were successfully parsed the previous time exist in the memory, the second parsing operation will be performed directly;
[0019] If the rule that was successfully parsed the last time does not exist in the memory, a list of rules sorted by usage from high to low is obtained from the rule center based on the site, business, and field. All rules are traversed in sequence starting from the first rule with the highest usage rate to enter the second parsing operation.
[0020] Preferably, the specific steps of the second analysis include:
[0021] Parse the value of the field from the page according to the specified rules;
[0022] If the parsing is successful and the parsed value is not empty, the data obtained after parsing is counted;
[0023] If the parsing is unsuccessful, return to the previous step and start from the first rule with the highest usage rate, and traverse the next rule for parsing.
[0024] Preferably, the method of preferentially selecting rules with high hit rate to perform a second analysis on page elements based on the hit situation of the analysis rules until the analysis is successful further includes:
[0025] After the second successful analysis, the number of successful analysis of the rule is recorded;
[0026] Determine whether the rule hit ranking has changed based on the current second successful analysis of the rule;
[0027] If the rule hit ranking changes, the rule center is updated, and the application monitors the rule center. When changes occur, the application-side rule cache is updated. The rule hit ranking is to sort the rules by hit rate at a preset time to form a comma-separated string. It is determined whether the string is consistent with that in Zookeeper. If different, the Zookeeper API is called to write data to Zookeeper.
[0028] If the current rule is different from the last successful rule, the last successful rule is updated as the current rule and the value of the current field is saved.
[0029] Preferably, when a change occurs, updating the application-side rule cache further comprises:
[0030] The rule list is stored through the custom concurrent list MyCopyOnWriteArrayList concurrent list;
[0031] When receiving a rule update, the application adds a new rule list to the head at once. It traverses the rule list starting from the head. The old list at the back does not affect the parsing, so the new rules can take effect as soon as possible.
[0032] At the same time, the old list is deleted from the specified position, which is the starting position of the old rules.
[0033] Preferably, the dynamically updated rule base further comprises:
[0034] According to the regular statistics of parsing rule hits, the hit rates are sorted from high to low and written to the Zookeeper cache. When the application starts, the corresponding configuration is read, stored in the memory, and changes are monitored.
[0035] If the hit situation is monitored to change, the application is notified through Zookeeper, and the application reads the corresponding configuration and overwrites the configuration in memory;
[0036] Among them, the currently hit rule is obtained and given priority as the rule for the next analysis; if the analysis fails, the rules in the memory are analyzed in sequence until the analysis succeeds, and the currently hit rule is reset.
[0037] The present invention also provides a parsing speed-up device based on dynamic adjustment of a rule base, comprising:
[0038] An acquisition module is used to acquire an original HTML page, where the elements on the original HTML page include title, price, comments, sales volume, ranking and information related to page attributes;
[0039] The first parsing module is used to parse the original HTML page for the first time based on the rule library traversal rule list, convert the HTML string into an operable DOM tree / element list, and obtain the parsed page elements;
[0040] The secondary parsing module is used to obtain the rules that were successfully parsed the last time and perform a second parsing according to the rules that were successfully parsed the last time when the page structure of the original HTML page changes;
[0041] The update module is used to select the rules with high hit rate to perform a second analysis on the page elements until the analysis is successful when the rules that successfully parsed the previous time are not obtained, based on the parsing rule hit situation, and dynamically update the rule library to perform subsequent analysis after the updated rules when the parsing rule hit situation changes until the analysis is successful.
[0042] The present invention provides an electronic device, comprising:
[0043] A memory, the memory being used to store a processing program;
[0044] A processor, when executing the processing program, implements the parsing speed-up method based on dynamic adjustment of the rule base as described in an embodiment of the present invention.
[0045] The present invention provides a readable storage medium, characterized in that a processing program is stored on the readable storage medium, and when the processing program is executed by a processor, the parsing speed-up method based on dynamic adjustment of a rule base as described in an embodiment of the present invention is implemented.
[0046] With respect to the prior art, the present invention has the following beneficial effects:
[0047] 1. Improve parsing efficiency: By counting the hits of parsing rules, give priority to using rules with high hit rates to parse page elements, improve parsing performance, and reduce time and resource usage; by using a rule library and dynamic adjustment strategies, the system can quickly select the most appropriate parsing rules based on the actual structure and changes of the page, thereby improving the speed and accuracy of parsing, improving parsing performance, and reducing parsing time and resource usage.
[0048] 2. Adaptive page changes: When the HTML page structure changes, the system can automatically obtain the rules that were successfully parsed the previous time and perform a second analysis, or select new rules based on the hit rate, which increases the system's ability to adapt to page changes.
[0049] 3. Optimize rule usage: By counting and monitoring the hits of parsing rules, the system can dynamically update the rule base and give priority to the rules with high hit rates. This ensures that the most effective rules are used first, thereby improving the overall parsing performance.
[0050] 4. Reduce manual intervention: The automated rule adjustment and update process reduces the need for manual intervention, reduces maintenance costs, and improves the system's automation level.
[0051] 5. Improve system stability: Using distributed coordination services such as ZooKeeper to store and update rules can ensure the consistency of rules and the stability of the system, and keep the rules synchronized even in multi-node deployment.
[0052] 6. Flexibility and scalability: Parsing rules are managed through annotations and configuration, making the system more flexible and scalable. New parsing rules can be easily added or existing rules can be modified.
[0053] 7. Real-time feedback and rapid response: The system can collect statistics on analysis status in real time and respond quickly to changes in rules, which ensures that the system can adapt to new analysis requirements and environmental changes in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 This is a diagram showing an example of steps of a parsing speed-up method based on dynamic adjustment of a rule base in an embodiment of the present invention;
[0055] Figure 2 This is a schematic diagram of the first analysis process in an embodiment of the present invention;
[0056] Figure 3 This is an example diagram of the overall process of the parsing speed-up method based on dynamic adjustment of the rule base in an embodiment of the present invention;
[0057] Figure 4 This is an example diagram of a rule list in an embodiment of the present invention;
[0058] Figure 5-6 This is an example diagram of a rule list when the hit situation changes in an embodiment of the present invention. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0060] Embodiment 1
[0061] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0062] It should be noted that the concepts such as "once" and "second" mentioned in the present application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0063] It should be noted that the modifications of "one" and "plurality" mentioned in the disclosure of this application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0064] Currently, we parse titles, price comments, sales, ranking information, etc. from pages. Because pages are often updated, and different rules are needed for the same page, that is, pages are often updated, and the DOM structure also changes. Some pages need to be parsed with new rules, and the old parsing rules cannot be discarded because some pages still have the old structure. In this way, each field must traverse the accumulated parsing rules until the parsing is successful, which has poor performance and consumes a lot of resources.
[0065] In order to solve the above defects, Figure 1-3 As shown, the present invention provides a parsing speed-up method based on dynamic adjustment of a rule base, comprising:
[0066] S1: Obtain the original HTML page, where the elements on the original HTML page include title, price, comments, sales, ranking and information related to page attributes; obtain the original pages of e-commerce products, keywords, comments, etc., parse out the title, price, sales, ranking and other information, perform statistics and analysis, and provide reference for sellers to select products and operate.
[0067] S2: Perform the first parsing of the original HTML page based on the rule library traversing the rule list, converting the HTML string into an operable DOM tree / element list, and obtaining the parsed page elements; the specific implementation steps of the first parsing in this embodiment are as follows: 1. Preparation of the rule library: Define the rule structure: First, it is necessary to define the structure of the rule, for example, the rule can contain the identifier of the element (such as ID, class name, tag name, etc.), attributes, text content, etc. Rule storage: The rules are usually stored in a database, file or distributed storage system, such as ZooKeeper. 2. HTML loading: Get HTML content: Get the original HTML page content through a network request (such as HTTP GET) or other means. Parse HTML: Use an HTML parser (such as JavaScript's DOMParser, Java's Jsoup, Python's BeautifulSoup, etc.) to parse the HTML string into a DOM tree. 3. Rule traversal and matching: Traverse the rule list: Traverse the rule list in the rule library in a predetermined order (such as the frequency of use of the rule, the defined priority, etc.). Matching rules: For each rule, use the API provided by the DOM tree (such as querySelector, getElementById, etc.) to try to match the elements in the page. 4. Data extraction and conversion: Extract data: When the rule matches successfully, extract the text content, attributes or other required data of the relevant elements. Data conversion: Format or convert the extracted data as needed, for example, convert the price string to a numeric type. 5. Parsing result processing: Result storage: Store the parsed data in a suitable data structure, such as a list, map, etc. Error handling: If the match fails or an error occurs in data extraction, record the error message and decide whether to retry or skip. 6. Processing statistics after successful parsing: Record the number of successes of each rule for subsequent rule optimization. Update the rule base: Update the rule base based on the statistical results and preset strategies, such as adjusting the priority of the rules or modifying the content of the rules.
[0068] In order to read configuration rules from a configuration management service such as ZooKeeper, and then parse these rules and store them in a mapping structure so that the corresponding rule list can be quickly found according to the business field name later, the first parsing of the original HTML page based on the rule base traversal rule list further includes: identifying the field method to be parsed according to the custom annotation; obtaining the value corresponding to the annotation parameter in the path through the get method, and assigning the annotation parameter to the string variable parsing rule; reading the annotation when the application starts, and reading the corresponding parsing rule from ZooKeeper according to the annotation information, and storing the parsing rule in the mapping in the memory; wherein the annotation parameter includes site, business and field information. The field parsing method is annotated by the annotation @Parser(station = "us", biz = "product", field = "title"), and the parsing rules are placed in Zookeeper, starting with the original order. This is a custom annotation used to annotate the field method to be parsed. The annotation contains information about the station, business and field. When the application starts, it reads the @Parser annotation and performs corresponding processing: it reads from Zookeeper according to station_biz_field and puts it into memory.
[0069] Stringrules=curatorClient.get(" / parser / station / biz / field");Map <String,List <string>fieldRuleMap=new HashMap<String,MyCopyOnWriteArrayList <string>>();
[0070] List ruleList=rules.split(",");
[0071] fieldRuleMap.put(station_biz_field,ruleList);, see Figure 4 As shown,
[0072] After the parsing is successful, the number of successes increases automatically.
[0073] Map <String,<String,Integer> >fieldRuleSuccessMap=newHashMap <linkedhashmap>();
[0074] station_biz_field=>rule=>number, the number is 0 at the beginning. It can be understood that there is a data structure in the application that is used to track the number of successful parsing of rules. This data structure is a Map, whose key is the string station_biz_field, which represents a specific business field, such as "us_product_title". The mapped value is another Map, whose key is the specific rule and the value is an integer, indicating the number of times the rule is successfully parsed. At the beginning of the program, the number of successes of each rule is initialized to 0. As the program runs, whenever a rule is used to successfully parse data, the corresponding item of this rule in fieldRuleSuccessMap will be found, and its success number will be increased by 1.
[0075] fieldRuleSuccessMap: The key is station_biz_field, and the value is the mapping between the parsing rule and the number of hits. The header has the highest hit rate. station_biz_field includes rule 3:150, rule 2:2333, and rule 5:6699.
[0076] For different businesses, statistics can be set by day, week, or month. If it is set to day, the number of times will be cleared at midnight every day; if it is set to week, the number of times will be cleared at midnight every Monday.
[0077] It can be understood as follows: In this embodiment, the application processes annotations when it starts: the application reads these annotations when it starts, and reads the corresponding parsing rules from ZooKeeper according to the annotation information, and then stores these rules in a map in memory. Track the number of successes of the rules: the application maintains a map fieldRuleSuccessMap to record the number of successes of each rule. Initially, the number is 0. Regularly count and sort rules: according to the configuration such as by day, week, and month, the application will regularly clear the number of successes of the rules. Every hour, the application will sort the rules according to the number of successes of the rules and generate a new rule list string. If the newly generated rule list string is inconsistent with that in ZooKeeper, the application will call the ZooKeeper API to update the data in ZooKeeper. Use a custom concurrent list to listen to and update rules: the application uses a custom concurrent list MyCopyOnWriteArrayList to store the rule list. When a rule update is received, the application will add the new rule list to the head at one time, so that the new rule can take effect as soon as possible. At the same time, the application deletes the old list from the specified location to reduce the number of locks and performance loss.
[0078] S3: When the page structure of the original HTML page changes, the rule that was successfully parsed the last time is obtained and a second parsing is performed according to the rule that was successfully parsed the last time; the specific steps of the second parsing include: parsing the value of the field from the page according to the specified rule; if the parsing is successful and the parsed value is not empty, statistics are performed on the data obtained after the parsing; if the parsing is unsuccessful, return to the previous step and start from the first rule with the highest usage rate, and traverse the next rule for parsing.
[0079] In step S3, when the page structure of the original HTML page changes, obtaining the rules that were successfully parsed the last time and performing a second analysis according to the rules that were successfully parsed the last time includes:
[0080] Get the previously parsed successful rule stored in memory and determine whether the rule exists;
[0081] If the rules that were successfully parsed the previous time exist in the memory, the second parsing operation will be performed directly;
[0082] If the rule that was successfully parsed the last time does not exist in the memory, a list of rules sorted by usage from high to low is obtained from the rule center based on the site, business, and field. All rules are traversed in sequence starting from the first rule with the highest usage rate to enter the second parsing operation.
[0083] S4: When the rules that successfully parsed the previous time are not obtained, the rules with high hit rates are prioritized based on the hit conditions of the parsing rules to perform a second parsing of the page elements until the parsing is successful. When the hit conditions of the parsing rules change, the rule library is dynamically updated to perform subsequent parsing after the updated rules are updated until the parsing is successful. If the parsing is successful here, no further parsing is required this time. Only when the hit rate changes and the rule library is dynamically updated, when subsequent pages such as other product pages and keyword pages need to be parsed, the updated rule library is used to parse from high to low hit rates.
[0084] The steps of dynamically updating the rule base described in this embodiment include: according to the regular statistics of the hit situation of the parsing rules, sorting from high to low according to the hit rate, writing to the Zookeeper cache, reading the corresponding configuration when the application starts, putting it into the memory, and monitoring the changes; if the hit situation is monitored to change, the application end is notified through Zookeeper, the application reads the corresponding configuration, and overwrites the configuration in the memory; wherein, the currently hit rule is obtained and is given priority as the rule for the next parsing; if the parsing fails, the rules are parsed in sequence according to the order of the rules in the memory until the parsing is successful, and the currently hit rule is reset.
[0085] In step S4, based on the hit situation of the parsed rules, the rules with high hit rates are preferentially selected to perform a second analysis on the page elements until the analysis is successful, further including: recording the number of successful analysis of the rule after the second analysis is successful; judging whether the rule hit ranking has changed according to the current rule situation that has been successfully parsed for the second time; if the rule hit ranking has changed, updating the rule center, and the application monitors the rule center. When changes occur, the application-side rule cache is updated. The rule hit ranking is to sort the rules by hit rate within a preset time to form a comma-separated string, and judging whether the string is consistent with that in Zookeeper. If different, calling the Zookeeper API to write data to Zookeeper; if the current rule is different from the last successful rule, updating the last successful rule as the current rule, and saving the value of the current field.
[0086] In this embodiment, when a change occurs, updating the application-side rule cache further includes:
[0087] The rule list is stored through the custom concurrent list MyCopyOnWriteArrayList; the application monitors changes and receives notifications, and uses the MyCopyOnWriteArrayList concurrent list to solve the impact of updates on ongoing parsing.
[0088] When a rule update is received, the application adds a new rule list to the head at once, adds it to the head of the list, and traverses the rule list from the head. The old list at the back does not affect the parsing, so the new rules can take effect as soon as possible; at the same time, the old list is deleted from the specified position, and the specified position is the starting position of the old rule. When an update is received, a new list is added at a time, only one lock is added, and the reference is changed once, reducing method calls and performance loss. The purpose of adding to the head of the list is to take effect as soon as possible (traversing the rule list starts from the head). The old list at the back does not affect the parsing. A new removeAll(index) method is added to delete the old list from the specified position at once, only one lock is added, and the reference is changed once, see Figure 5-6 shown.
[0089] Those skilled in the art can understand that, in this embodiment, step 1, obtaining the last successful rule specifically includes the following: 1.1 Obtain the last successful rule stored in the memory, if there is one, proceed to step 3 1.2 If it is not found, that is, when the application is started, proceed to step 2. Step 2, obtain the rule list from the rule center: 2.1 Obtain the rule list from the rule center according to the site, business and field (arranged from high to low by usage rate 2.2 Start from the first rule with the highest usage rate, traverse in sequence, and proceed to step 3. Step 3, analysis 3.1 According to the specified rule, parse the value of the field from the page, such as the title; 3.2 If the analysis is successful and its value is not empty, proceed to step 4; 3.3 Otherwise, proceed to step 2.2, traverse the next rule for analysis. Step 4, post-analysis processing: 4.1 Record the number of successful analysis of the rule; 4.2 If the rule hit ranking changes, update the rule center, apply the monitoring rule center, and update the application-side rule cache when changes occur; 4.3 If the rule is different from the last successful rule, update the last successful rule to the current rule; 4.4 Save the value of the current field for subsequent processing.
[0090] Embodiment 2
[0091] Based on the same inventive concept, the present invention provides a parsing speed-up device based on dynamic adjustment of a rule base, comprising:
[0092] An acquisition module is used to acquire an original HTML page, where the elements on the original HTML page include title, price, comments, sales volume, ranking and information related to page attributes;
[0093] The first parsing module is used to parse the original HTML page for the first time based on the rule library traversal rule list, convert the HTML string into an operable DOM tree / element list, and obtain the parsed page elements;
[0094] The secondary parsing module is used to obtain the rules that were successfully parsed the last time and perform a second parsing according to the rules that were successfully parsed the last time when the page structure of the original HTML page changes;
[0095] The update module is used to select the rules with high hit rate to perform a second analysis on the page elements until the analysis is successful when the rules that successfully parsed the previous time are not obtained, based on the parsing rule hit situation, and dynamically update the rule library to perform subsequent analysis after the updated rules when the parsing rule hit situation changes until the analysis is successful.
[0096] The principles, contents and implementation methods of the above modules are as described in Example 1 and will not be repeated here.
[0097] Embodiment 3
[0098] In some embodiments of the present application, an electronic device is also provided. The electronic device includes a memory and a processor, wherein the memory is used to store a processing program, and the processor executes the processing program according to instructions. When the processor executes the processing program, the parsing speed-up method based on dynamic adjustment of the rule base in the above embodiment is implemented.
[0099] In some embodiments of the present application, a readable storage medium is also provided, which can be a non-volatile readable storage medium or a volatile readable storage medium. The readable storage medium stores instructions, and when the instructions are executed on a computer, an electronic device including the readable storage medium executes the aforementioned parsing speed-up method based on dynamic adjustment of a rule base.
[0100] It is understandable that, for the aforementioned parsing speed-up method based on dynamic adjustment of the rule base, if it is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0101] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.
[0102] The program code involved in executing the technical solution disclosed in the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also conventional procedural programming languages such as C language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on the remote computing device, or completely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).
[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.< / linkedhashmap> < / string> < / string>
Claims
1. A parsing speed-up method based on dynamic adjustment of rule base, characterized in that: include: Obtaining an original HTML page, where the elements on the original HTML page include title, price, comments, sales volume, ranking, and information related to page attributes; Based on the rule library, the original HTML page is parsed for the first time by traversing the rule list, converting the HTML string into an operable DOM tree / element list to obtain the parsed page elements; When the page structure of the original HTML page changes, the rules that were successfully parsed the last time are obtained and parsed again according to the rules that were successfully parsed the last time; When the rules that were successfully parsed in the previous time are not obtained, the rules with high hit rates are prioritized based on the parsing rule hits to perform a second parsing of the page elements until the parsing is successful. When the parsing rule hits change, the rule library is dynamically updated to perform subsequent parsing after the updated rules until the parsing is successful.
2. The method for speeding up parsing based on dynamic adjustment of rule base according to claim 1, characterized in that: The first parsing of the original HTML page based on the rule base traversal rule list further includes: Identify the field method that needs to be parsed based on custom annotations; Get the value corresponding to the annotation parameter in the path through the get method, and assign the annotation parameter to the string variable parsing rule; When the application starts, the annotation is read, and the corresponding parsing rules are read from ZooKeeper according to the annotation information, and the parsing rules are stored in the mapping in the memory; The annotation parameters include site, business and field information.
3. The method for speeding up parsing based on dynamic adjustment of rule base according to claim 1, characterized in that: When the page structure of the original HTML page changes, obtaining the rules that were successfully parsed the previous time and performing a second parsing according to the rules that were successfully parsed the previous time includes: Get the previously parsed successful rule stored in memory and determine whether the rule exists; If the rules that were successfully parsed the previous time exist in the memory, the second parsing operation will be performed directly; If the rule that was successfully parsed the last time does not exist in the memory, a list of rules sorted by usage from high to low is obtained from the rule center based on the site, business, and field. All rules are traversed in sequence starting from the first rule with the highest usage rate to enter the second parsing operation.
4. The method for speeding up parsing based on dynamic adjustment of rule base according to claim 1, characterized in that: The specific steps of the second analysis include: Parse the value of the field from the page according to the specified rules; If the parsing is successful and the parsed value is not empty, the data obtained after parsing is counted; If the parsing is unsuccessful, return to the previous step and start from the first rule with the highest usage rate, and traverse the next rule for parsing.
5. The method for speeding up parsing based on dynamic adjustment of rule base according to claim 1, characterized in that: The method of preferentially selecting a rule with a high hit rate based on the parsing rule hit situation to perform a second parsing of the page element until the parsing is successful further includes: After the second successful analysis, the number of successful analysis of the rule is recorded; Determine whether the rule hit ranking has changed based on the current second successful analysis of the rule; If the rule hit ranking changes, the rule center is updated, and the application monitors the rule center. When changes occur, the application-side rule cache is updated. The rule hit ranking is to sort the rules by hit rate at a preset time to form a comma-separated string. It is determined whether the string is consistent with that in Zookeeper. If different, the Zookeeper API is called to write data to Zookeeper. If the current rule is different from the last successful rule, the last successful rule is updated as the current rule and the value of the current field is saved.
6. The method for speeding up parsing based on dynamic adjustment of rule base according to claim 5, characterized in that: When a change occurs, updating the application-side rule cache further includes: The rule list is stored through the custom concurrent list MyCopyOnWriteArrayList concurrent list; When receiving a rule update, the application adds a new rule list to the head at once. It traverses the rule list starting from the head. The old list at the back does not affect the parsing, so the new rules can take effect as soon as possible. At the same time, the old list is deleted from the specified position, which is the starting position of the old rules.
7. The method for speeding up parsing based on dynamic adjustment of rule base according to claim 1, characterized in that: The dynamically updated rule base further comprises: According to the regular statistics of parsing rule hits, the hit rates are sorted from high to low and written to the Zookeeper cache. When the application starts, the corresponding configuration is read, stored in the memory, and changes are monitored. If the hit situation is monitored to change, the application is notified through Zookeeper, and the application reads the corresponding configuration and overwrites the configuration in memory; Among them, the currently hit rule is obtained and given priority as the rule for the next analysis; if the analysis fails, the rules in the memory are analyzed in sequence until the analysis succeeds, and the currently hit rule is reset.
8. A parsing speed-up device based on dynamic adjustment of rule base, characterized in that: include: An acquisition module is used to acquire an original HTML page, where the elements on the original HTML page include title, price, comments, sales volume, ranking and information related to page attributes; The first parsing module is used to parse the original HTML page for the first time based on the rule library traversal rule list, convert the HTML string into an operable DOM tree / element list, and obtain the parsed page elements; A secondary parsing module is used to obtain the rules that were successfully parsed the previous time and perform a second parsing according to the rules that were successfully parsed the previous time when the page structure of the original HTML page changes; The update module is used to select the rules with high hit rate to perform a second analysis on the page elements until the analysis is successful when the rules that successfully parsed the previous time are not obtained, based on the parsing rule hit situation, and dynamically update the rule library to perform subsequent analysis after the updated rules when the parsing rule hit situation changes until the analysis is successful.
9. An electronic device, characterized in that: include: A memory, the memory being used to store a processing program; A processor, wherein when executing the processing program, the processor implements the parsing speed-up method based on dynamic adjustment of the rule base as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that: The readable storage medium stores a processing program, and when the processing program is executed by the processor, the parsing speed-up method based on dynamic adjustment of the rule base as described in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Rule matching method and device and rule management method and device
CN107294929A
Distributive focus web crawler webpage crawling method and system
CN107729564A
Method and device for realizing annotation type configuration
CN110795135A
Log analysis method and log analysis device
CN112667672A
Efficient data pattern matching
US20140310225A1
