A webpage content extraction method and a related device thereof

Abstract

The application discloses a webpage content extraction method and a related device thereof. The method comprises the following steps: when a webpage to be used is opened according to a website address to be used, firstly, current page display content of the webpage to be used is acquired; then, loading progress representation data of the webpage to be used is determined according to the current page display content, so that when it is determined that the loading progress representation data does not satisfy a preset progress condition, the step of acquiring the current page display content of the webpage to be used is continued to be executed; until when it is determined that the loading progress representation data satisfies the preset progress condition, the webpage to be used is extracted from the webpage to be used. In this way, the loading progress of the webpage to be used can be acquired in real time, so that the loading completion time of the webpage to be used can be learned in time, and then, the adverse effects caused by webpage content extraction performed too early (or too late) can be effectively avoided, so that the webpage content extraction effect is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method for extracting web page content and related equipment. Background Technology

[0002] With the popularization of information technology, the application scope of web pages is becoming increasingly wide. For example, web pages can be used for news introductions, popular science knowledge dissemination, advertising promotion, and so on.

[0003] In addition, in some application scenarios (such as web page analysis and web page migration), for a web page, it is necessary to first extract the web page content; and then process the extracted web page content accordingly (such as web page content analysis and processing, new web page construction and processing).

[0004] However, due to defects in web page content extraction technology, the extraction effect is relatively poor. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a webpage content extraction method and related equipment, which can improve the webpage content extraction effect.

[0006] To achieve the above objectives, the technical solutions provided in this application are as follows:

[0007] This application provides a method for extracting web page content, the method comprising:

[0008] When a webpage to be used is opened according to the URL to be used, the current page display content of the webpage to be used is obtained; wherein, the URL to be used is used to uniquely identify the webpage to be used;

[0009] Based on the content currently displayed on the page, determine the loading progress representation data of the webpage to be used;

[0010] If it is determined that the loading progress representation data does not meet the preset progress conditions, then the step of obtaining the current page display content of the webpage to be used is returned until it is determined that the loading progress representation data meets the preset progress conditions, at which point the content to be used is extracted from the webpage to be used.

[0011] In one possible implementation, determining the loading progress representation data of the webpage to be used based on the currently displayed page content includes:

[0012] Extract at least one element to be used from the content currently displayed on the page;

[0013] Based on the at least one element to be used, determine the loading progress representation data of the webpage to be used.

[0014] In one possible implementation, determining the loading progress representation data of the webpage to be used based on the at least one element to be used includes:

[0015] The at least one element to be used is classified to obtain at least one set of elements;

[0016] Determine the set score value for each set of elements;

[0017] The loading progress representation data of the webpage to be used is determined based on the sum of the set score values ​​of the at least one set of elements.

[0018] In one possible implementation, classifying the at least one element to be used to obtain at least one set of elements includes:

[0019] The at least one element to be used is classified to obtain the at least one set of elements and the element category identifier of the at least one set of elements;

[0020] The number of element sets is N, and the process of determining the set score value of the nth element set includes:

[0021] The preset value of the element rating corresponding to the element category identifier of the nth element set is found from the first mapping relationship and determined as the individual rating value of the nth element set; wherein, the first mapping relationship includes the correspondence between the element category identifier of the nth element set and the individual rating value of the nth element set; n is a positive integer, n≤N, and N is a positive integer;

[0022] The set score of the nth element set is determined by multiplying the number of elements in the nth element set by the individual score of the nth element set.

[0023] In one possible implementation, the method further includes:

[0024] Determine the webpage architecture description information of the webpage to be used;

[0025] The preset progress conditions are determined based on the webpage architecture description information.

[0026] In one possible implementation, the extraction process of the content to be used includes:

[0027] If the webpage to be used contains lazy-loaded resources that meet the lazy-load conditions, then the address to be completed for the lazy-loaded resources is determined based on the webpage to be used.

[0028] Based on the resource attribute information of the lazy-loaded resources, determine the address completion content;

[0029] The resource address of the lazy-loaded resource is determined based on the address completion content and the address to be completed;

[0030] The content to be used is determined based on the resource address of the lazy-loaded resource.

[0031] In one possible implementation, the resource attribute information includes at least one of the following: resource size, resource display device attributes, and webpage display size list.

[0032] In one possible implementation, the resource attribute information includes resource size, resource display device attributes, and a list of webpage display sizes;

[0033] The step of determining the address completion content based on the resource attribute information of the lazy-loaded resource includes:

[0034] The actual display size of the lazy-loaded resource is determined based on the resource size of the lazy-loaded resource and the resource display device attributes of the lazy-loaded resource;

[0035] The address completion content is determined based on the actual display size and the list of webpage display sizes.

[0036] In one possible implementation, determining the address completion content based on the actual display size and the list of webpage display sizes includes:

[0037] The actual display size is compared with at least one candidate display size in the list of webpage display sizes to obtain a size comparison result;

[0038] Based on the size comparison results, the address completion content is determined.

[0039] In one possible implementation, the extraction process of the content to be used includes:

[0040] If there are elements to be extracted in the webpage to be used that meet the preset element conditions, then at least one candidate element feature and the element extraction rule corresponding to the at least one candidate element feature are determined according to the element type identifier of the element to be extracted.

[0041] The element features of the element to be extracted are matched with the features of at least one candidate element to obtain the feature matching result;

[0042] If the feature matching result indicates that the target element feature matches the element feature of the element to be extracted, then the element to be extracted in the webpage to be used is extracted according to the element extraction rule corresponding to the target element feature to obtain the element extraction content of the element to be extracted; wherein, the at least one candidate element feature includes the target element feature;

[0043] Based on the element extraction content of the element to be extracted, the content to be used is determined.

[0044] This application also provides a web page content extraction device, including:

[0045] The content acquisition unit is used to acquire the current page display content of the webpage to be used when the webpage to be used is opened according to the URL to be used; wherein, the URL to be used is used to uniquely identify the webpage to be used;

[0046] The progress determination unit is used to determine the loading progress representation data of the webpage to be used based on the content currently displayed on the page.

[0047] The judgment loop unit is used to return to the content acquisition unit to perform the step of acquiring the current page display content of the webpage to be used if it is determined that the loading progress representation data does not meet the preset progress conditions;

[0048] The content extraction unit is used to extract the content to be used from the webpage to be used until it is determined that the loading progress representation data meets the preset progress conditions.

[0049] This application also provides a device, the device including a processor and a memory:

[0050] The memory is used to store computer programs;

[0051] The processor is used to execute any implementation of the web page content extraction method provided in the embodiments of this application according to the computer program.

[0052] This application also provides a computer-readable storage medium for storing a computer program that executes any implementation of the web page content extraction method provided in this application.

[0053] This application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any implementation of the web page content extraction method provided in this application.

[0054] Compared with the prior art, the embodiments of this application have at least the following advantages:

[0055] In the technical solution provided in this application embodiment, when opening a webpage to be used according to the URL to be used, the current page display content of the webpage to be used is first obtained; then, based on the current page display content, the loading progress representation data of the webpage to be used is determined, so that if it is determined that the loading progress representation data does not meet the preset progress condition, the above step "obtaining the current page display content of the webpage to be used" is returned to continue; until it is determined that the loading progress representation data meets the preset progress condition, the content to be used is extracted from the webpage to be used. In this way, the loading progress of the webpage to be used can be obtained in real time, so that the loading completion time of the webpage to be used can be known in a timely manner, thereby effectively avoiding the adverse effects caused by extracting webpage content too early (or too late), which is conducive to improving the webpage content extraction effect. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart illustrating a webpage content extraction method provided in this application embodiment;

[0058] Figure 2 A schematic diagram illustrating a webpage loading progress recognition process provided in an embodiment of this application;

[0059] Figure 3 This is a schematic diagram of a web page content extraction device provided in an embodiment of this application. Detailed Implementation

[0060] The inventors discovered in their research on webpage content extraction that, to extract content from a webpage, one must first open the webpage using its URL and then extract the content from it. However, opening a webpage using its URL can be slow due to slow internet speeds, leading to slow page loading. This can result in attempting to extract content from a loading page (i.e., a page that hasn't finished loading), leading to incomplete content extraction (or even no content being extracted at all), thus resulting in poor extraction performance.

[0061] The inventors also discovered that, to overcome the above problems, a fixed network wait time can be added, so that this network wait time can represent the time interval between the webpage opening action and the webpage content retrieval action. However, because different webpages have different page loading times, the webpage content retrieval process based on the network wait time is prone to the following phenomena:

[0062] ① Because some web pages take longer to load than the network waiting time, these web pages are still in the loading state when the web page content extraction process is triggered according to the network waiting time. This causes the web page content extraction process to be performed on these web pages too early, resulting in incomplete web page content extracted from these web pages, and thus poor web page content extraction effect.

[0063] ②Because the page loading time of some web pages is much shorter than the network waiting time, there is a relatively long time interval between the triggering time of the web page content extraction process and the time when the web page is loaded. This results in the web page content extraction process being processed too late, which leads to a high delay in the web page content extraction process and consequently a poor timeliness of web page content extraction.

[0064] Based on the above findings, in order to solve the technical problems shown in the background section, this application provides a webpage content extraction method. The method includes: when opening a webpage to be used according to the URL to be used, first obtaining the current page display content of the webpage to be used; then, based on the current page display content, determining the loading progress representation data of the webpage to be used, so that if it is determined that the loading progress representation data does not meet a preset progress condition, the process returns to continue executing the above step "obtaining the current page display content of the webpage to be used"; until it is determined that the loading progress representation data meets the preset progress condition, the content to be used is extracted from the webpage to be used. This enables real-time acquisition of the loading progress of the webpage to be used, thereby allowing timely knowledge of the completion time of the webpage's loading, effectively avoiding adverse effects caused by extracting webpage content too early (or too late), thus improving the webpage content extraction effect.

[0065] Furthermore, the embodiments of this application do not limit the executing entity of the web page content extraction method. For example, the web page content extraction method provided in the embodiments of this application can be applied to data processing devices such as terminal devices or servers. The terminal device can be a smartphone, computer, personal digital assistant (PDA), or tablet computer, etc. The server can be a standalone server, a cluster server, or a cloud server.

[0066] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0067] Method Example 1

[0068] See Figure 1 The figure is a flowchart of a web page content extraction method provided in an embodiment of this application.

[0069] The webpage content extraction method provided in this application includes S1-S5:

[0070] S1: When the webpage to be used is opened according to the URL to be used, obtain the current page content displayed on the webpage to be used.

[0071] The aforementioned "URL to be used" is used to uniquely identify the webpage to be used, so that the "URL to be used" can be used to access the webpage.

[0072] The aforementioned "web page to be used" refers to a web page that requires web page content extraction processing; moreover, the embodiments of this application do not limit the "web page to be used". For ease of understanding, the following description is based on two application scenarios.

[0073] Scenario 1: For some webpage migration scenarios (e.g., migrating a third-party ad landing page to a first-party ad landing page), the webpages that need to be migrated (e.g., the aforementioned "third-party ad landing page") can all be identified as webpages to be used.

[0074] Scenario 2: For some web page processing scenarios (such as specific web page filtering scenarios, web page hidden word decoding scenarios, etc.), all the web pages that need to be processed can be identified as web pages to be used.

[0075] The aforementioned "current page content" can be used to indicate the page content displayed in real time on the webpage to be used during the loading process of the webpage to be used.

[0076] Furthermore, for web pages that are in the loading process, page content can be fetched from the web page at preset time intervals. Therefore, the aforementioned "current page content" can also be used to represent the page content displayed on the web page at the current fetching time.

[0077] The time interval between the "current capture time" and the "previous capture time" is the aforementioned "preset time interval," and the time interval between the "current capture time" and the "next capture time" is also the aforementioned "preset time interval." The aforementioned "preset time interval" can be preset.

[0078] Based on the above content of S1, if you want to perform webpage content crawling on a webpage to be used, you can first open the webpage to be used by following its URL (i.e., the URL to be used) to start loading the page content. Then, during the loading process of the webpage to be used, you can perform real-time page content crawling on the webpage to obtain the current page content displayed on the webpage. This allows you to determine whether the webpage to be used has finished loading based on the current page content. This helps to promptly identify webpages that have finished loading, thereby effectively avoiding the adverse effects caused by extracting webpage content too early (or too late).

[0079] S2: Determine the loading progress representation data of the webpage to be used based on the content currently displayed on the page.

[0080] The aforementioned "loading progress representation data of the webpage to be used" is used to indicate the loading progress of the webpage to be used.

[0081] Furthermore, the embodiments of this application do not limit the implementation of S2; for example, it may specifically include S21-S22:

[0082] S21: Extract at least one element to be used from the content currently displayed on the page.

[0083] The "elements to be used" mentioned above refer to web page elements (such as text, images, audio, video, animation, tables, etc.) that exist in the content displayed on the current page.

[0084] It should be noted that the implementation of S21 is not limited to the present application embodiments. It can be implemented using any existing or future method that can extract web page elements from web page content.

[0085] S22: Determine the loading progress representation data of the webpage to be used based on at least one element to be used.

[0086] As an example, S22 may specifically include S221-S223:

[0087] S221: Classify at least one element to be used to obtain at least one set of elements.

[0088] The aforementioned "classification process" is used to divide some objects to be classified into different categories; moreover, the embodiments of this application do not limit the implementation of the "classification process". For example, it can be implemented using any existing or future classification method.

[0089] The aforementioned "element set" is used to represent a certain type of web page element (e.g., text, image, audio, etc.).

[0090] Furthermore, the embodiments of this application do not limit the process of determining the "at least one set of elements" as described above. For example, it may specifically include: first classifying at least one element to be used to obtain at least one element cluster; and then determining each element cluster as a separate set of elements.

[0091] In reality, since different element sets represent different types of web page elements, they have different element characteristics, which may lead to different processing procedures (e.g., scoring procedures) for different element sets. Therefore, in order to distinguish between different element sets, this application embodiment also provides another possible implementation of S221, which may specifically include: classifying at least one element to be used to obtain at least one element set and an element category identifier for the at least one element set.

[0092] The "element category identifier of the nth element set" is used to uniquely identify the nth element set; moreover, the embodiments of this application do not limit the "element category identifier of the nth element set," for example, it can be the element category name (e.g., Figure 2 The element category names shown are "text element", "image element", "video element", etc. Here, n is a positive integer, n≤N, where N is a positive integer and represents the number of sets in the above "at least one element set".

[0093] Furthermore, the embodiments of this application do not limit the process of determining the "nth element set" and the "element category identifier of the nth element set". For example, it may specifically include: first, classifying at least one element to be used to obtain N element clusters; then, determining the nth element cluster as the nth element set, and determining the cluster identifier of the nth element cluster as the element category identifier of the nth element set. Wherein, n is a positive integer, n≤N, and N is a positive integer.

[0094] The aforementioned "cluster identifier of the nth element cluster" is used to uniquely identify the nth element cluster; moreover, the embodiments of this application do not limit the "cluster identifier of the nth element cluster". For example, it can be determined based on the element type to which all elements in the nth element cluster belong; and the determination process can specifically include: firstly performing a count of the occurrences of the element types to which all elements in the nth element cluster belong; and then determining the element type with the highest occurrence as the cluster identifier of the nth element cluster, etc.

[0095] Based on the above content of S221, after obtaining at least one element to be used, these elements can be classified to obtain at least one set of elements (and the element category identifier of the at least one set of elements), so that the display degree of the content displayed on the current page (that is, the loading progress of the webpage to be used) can be determined based on these set of elements.

[0096] S222: Determine the set score for each set of elements.

[0097] The "set score of the nth element set" is used to represent the loading progress of the web page elements represented by the nth element set on the web page to be used. Here, n is a positive integer, n≤N, N is a positive integer, and N represents the number of sets in the above "at least one element set".

[0098] Furthermore, the embodiments of this application do not limit the process of determining the "set score value of the nth element set" as described above. For example, it may specifically include steps 11-12:

[0099] Step 11: Find the preset value of the element rating corresponding to the element category identifier of the nth element set from the first mapping relationship, and determine it as the individual rating value of the nth element set.

[0100] The aforementioned "first mapping relationship" is used to record the preset score values ​​of the elements corresponding to each candidate element category identifier. Furthermore, this application embodiment does not limit the "first mapping relationship," which may include, for example, the correspondence between the first candidate element category identifier (e.g., "text element" above) and the first element's preset score value; the correspondence between the second candidate element category identifier (e.g., "image element" above) and the second element's preset score value; ... (and so on); the correspondence between the Qth candidate element category identifier (e.g., "video element" above) and the Qth element's preset score value. Where Q is a positive integer.

[0101] As can be seen, after obtaining the element category identifier of the nth element set, this "element category identifier of the nth element set" can be matched with the Q candidate element category identifiers in the first mapping relationship to obtain the identifier matching result. If the identifier matching result indicates that the "element category identifier of the nth element set" matches the qth candidate element category identifier, then the preset score value of the qth element can be determined as the individual score value of the nth element set. Based on this, it can be known that the above-mentioned "first mapping relationship" includes the correspondence between the element category identifier of the nth element set and the individual score value of the nth element set. Where q∈{1, 2, 3, ..., Q}.

[0102] Step 12: Determine the set score of the nth element set by multiplying the number of elements in the nth element set by the individual score of the nth element set.

[0103] In this embodiment of the application, after obtaining the individual rating values ​​of the nth element set, the set rating value of the nth element set can be determined based on the individual rating values ​​of the nth element set; and the determination process can be: the set rating value of the nth element set is determined by multiplying the number of elements in the nth element set by the individual rating values ​​of the nth element set. Here, "the number of elements in the nth element set" represents the number of web page elements included in the nth element set.

[0104] Based on the above content of S222, after obtaining at least one set of elements, a scoring process can be performed on each set of elements to obtain a set score value for each set of elements. This allows for the determination of the loading progress representation data of the webpage to be used based on these set score values.

[0105] S223: Determine the loading progress representation data of the webpage to be used based on the sum of the set score values ​​of at least one set of elements.

[0106] In this embodiment of the application, after obtaining the set score value of at least one set of elements, the set score values ​​of these set of elements can be added together to obtain the loading progress representation data of the webpage to be used, so that the loading progress representation data can better represent the loading progress of the webpage to be used, so that it can be determined whether the webpage to be used has been loaded based on the loading progress representation data.

[0107] Based on the above content of S2, after obtaining the current page display content of the webpage to be used, the various elements included in the current page display content can be classified and scored to obtain the element scoring status of the current page display content; then, based on the element scoring status, the loading progress representation data of the webpage to be used can be determined so that the loading progress representation data can better represent the loading progress of the webpage to be used, so that it can be determined whether the webpage to be used has been loaded based on the loading progress representation data.

[0108] S3: Determine whether the loading progress representation data meets the preset progress conditions. If yes, execute S5; otherwise, execute S4.

[0109] The aforementioned "preset progress conditions" are used to indicate the conditions reached when the webpage finishes loading.

[0110] Furthermore, the embodiments of this application do not limit the above-mentioned "preset progress conditions". For example, if the above-mentioned "loading progress representation data" is determined based on the set score value of at least one set of elements, then the "preset progress conditions" may specifically be: the above-mentioned "loading progress representation data" reaches a preset page expected score threshold (that is, Figure 2 (as shown in the "predicted value", etc.)

[0111] As can be seen, after obtaining the loading progress representation data of the webpage to be used, it can be determined whether the loading progress representation data has reached the preset expected page score threshold. If it has, it can be determined that the loading progress representation data meets the preset progress condition, thus determining that the page loading of the webpage to be used has been completed, and the webpage content extraction process can be directly performed on the webpage to be used. If it has not reached the preset progress condition, it can be determined that the loading progress representation data does not meet the preset progress condition, thus determining that the webpage to be used is still in the page loading state. Therefore, in order to avoid incomplete extraction of webpage content, it can return to continue to execute the above steps "obtain the current page display content of the webpage to be used" and its subsequent steps to realize the next round of identification process for the loading state of the webpage to be used.

[0112] Furthermore, the embodiments of this application do not limit the method of obtaining the aforementioned "preset progress conditions," for example, they can be preset. As another example, to improve the flexibility of webpage content extraction, the following method can be used. Method Example 2 The process of determining the "preset schedule conditions" shown is implemented.

[0113] S4: Return to step S1, "Get the current page content of the webpage to be used".

[0114] It should be noted that the implementation of S4 is not limited in this application. For ease of understanding, the following description combines two scenarios.

[0115] In scenario 1, to improve the real-time performance of identifying webpage loading status, S4 can specifically be as follows: after determining that the loading progress representation data does not meet the preset progress conditions, immediately return to execute the above steps "obtain the current page display content of the webpage to be used" and its subsequent steps, so as to realize the next round of identification process for the loading status of the webpage to be used. This can effectively shorten the time interval between two adjacent webpage loading status identification processes, thereby facilitating the more timely identification of the webpage to be used that has been loaded, and thus effectively reducing the latency of extracting the webpage content for the webpage to be used.

[0116] Scenario 2, to avoid excessively increasing computational resource consumption, such as Figure 2 As shown, S4 can specifically be: after determining that the loading progress representation data does not meet the preset progress conditions, wait for a preset time; then return to execute the above steps "obtain the current page display content of the webpage to be used" and its subsequent steps, so as to realize the next round of identification process for the loading status of the webpage to be used. This can effectively reduce the number of times the webpage loading status identification process is executed, thereby effectively reducing the computational resource consumption of the multi-round webpage loading status identification process, and thus effectively avoiding excessive increase in computational resource consumption.

[0117] Based on the relevant content of S4 above, it can be seen that after determining that the loading progress representation data of the webpage to be used does not meet the preset progress conditions, it can be determined that the page loading process of the webpage to be used is still not complete. Therefore, the above step "obtain the current page display content of the webpage to be used" and its subsequent steps can be returned to execute, so as to realize the next round of identification process for the loading status of the webpage to be used. This is conducive to timely discovery of the webpage to be used that has been loaded, thereby effectively avoiding the adverse effects caused by extracting webpage content too early (or too late).

[0118] S5: Extract content to be used from the webpage to be used.

[0119] The aforementioned "content to be used" refers to the page content (e.g., structured content of the webpage) in the webpage to be used; moreover, the embodiments of this application do not limit the extraction process of the "content to be used." For example, any existing or future webpage content extraction method (e.g., a method for extracting general structured content from a webpage) can be used for implementation. Furthermore, to improve the webpage content extraction effect, the following methods can be employed... Method Example 3 and / or Method Example 4 The extraction process of the "content to be used" shown is implemented.

[0120] As can be seen, in this embodiment of the application, after determining that the loading progress representation data of the webpage to be used meets the preset progress conditions, it can be determined that the page loading process of the webpage to be used has been completed. Therefore, the webpage content can be directly extracted from the webpage to be used to obtain the content to be used, so that the content to be used can better represent the webpage content of the webpage to be used.

[0121] Based on the above-mentioned content of S1 to S5, it can be seen that for the web page content extraction method provided in this application embodiment, when opening the web page to be used according to the URL to be used, the current page display content of the web page to be used is first obtained; then, based on the current page display content, the loading progress representation data of the web page to be used is determined, so that if it is determined that the loading progress representation data does not meet the preset progress condition, the above step "obtaining the current page display content of the web page to be used" is returned to continue; until it is determined that the loading progress representation data meets the preset progress condition, the content to be used is extracted from the web page to be used. In this way, the loading progress of the web page to be used can be obtained in real time, so that the loading completion time of the web page to be used can be known in a timely manner, thereby effectively avoiding the adverse effects caused by extracting web page content too early (or too late), which is conducive to improving the web page content extraction effect.

[0122] Method Example 2

[0123] In reality, different web pages have different content, resulting in different preset progress conditions for each web page. Therefore, to further improve the web page content extraction effect, this application embodiment also provides a possible implementation for determining the aforementioned "preset progress conditions," which specifically may include steps 21-22:

[0124] Step 21: Determine the webpage architecture description information of the webpage to be used.

[0125] The above "web page architecture description information" is used to represent the web page architecture of the web page to be used.

[0126] Furthermore, the embodiments of this application do not limit the process of determining the "web page architecture description information" described above. For example, it can specifically be steps 211-212:

[0127] Step 211: Determine the webpage template to be used.

[0128] This application does not limit the process of determining the "web page template of the web page to be used" as described above. For example, it can be determined based on the template information provided by the user for the web page to be used. Alternatively, it can specifically involve matching the web page to be used with at least one candidate web page template to obtain a first template matching result. When the first template matching result indicates the existence of a candidate web page template that matches the web page to be used, that "candidate web page template that matches the web page to be used" is determined as the web page template of the web page to be used.

[0129] Step 212: Determine the webpage structure description information of the webpage to be used based on the webpage template.

[0130] It should be noted that the implementation of step 212 is not limited in this application embodiment. For example, it can specifically be: determining the webpage template of the webpage to be used as the webpage architecture description information of the webpage to be used. Or, it can specifically be: first searching for the template identifier corresponding to the webpage template of the webpage to be used in the second mapping relationship, and determining it as the template identifier to be used; then determining the template identifier to be used as the webpage architecture description information of the webpage to be used.

[0131] The aforementioned "second mapping relationship" is used to record the template identifiers corresponding to different candidate webpage templates; moreover, this application embodiment does not limit the "second mapping relationship," for example, it may specifically include: the correspondence between the first candidate webpage template and the first template identifier, the correspondence between the second candidate webpage template and the second template identifier, ..., and the correspondence between the Yth candidate webpage template and the Yth template identifier. Wherein, Y is a positive integer.

[0132] As can be seen, after obtaining the webpage template of the webpage to be used, this "webpage template of the webpage to be used" can be matched with Y candidate webpage templates in the second mapping relationship to obtain the second template matching result; if the second template matching result indicates that the "webpage template of the webpage to be used" matches the y-th candidate webpage template, then the y-th template can be identified and determined as the webpage architecture description information of the webpage to be used. Where y∈{1,2,3,...,Y}.

[0133] Based on the relevant content of step 21 above, after obtaining the webpage to be used, the webpage architecture description information of the webpage to be used can be determined so that the preset progress conditions corresponding to the webpage to be used can be determined based on the webpage architecture description information.

[0134] Step 22: Determine the preset progress conditions based on the webpage architecture description information.

[0135] It should be noted that the implementation of step 22 is not limited in this application embodiment. For example, when the above-mentioned "preset progress condition" is to reach a pre-set page expected score threshold, step 22 may specifically include: firstly, searching for the page score expected value corresponding to the web page architecture description information in the third mapping relationship, and determining it as the page expected score threshold; then, determining the preset progress condition based on the page expected score threshold.

[0136] The aforementioned "third mapping relationship" is used to record the expected page rating value corresponding to each candidate webpage architecture information; moreover, the embodiments of this application do not limit this "third mapping relationship," for example, it may specifically include: the correspondence between the first candidate webpage architecture information and the first expected page rating value; the correspondence between the second candidate webpage architecture information and the second expected page rating value; ... (and so on); the correspondence between the Tth candidate webpage architecture information and the Tth expected page rating value. Wherein, T is a positive integer.

[0137] As can be seen, after obtaining the webpage architecture description information of the webpage to be used, this webpage architecture description information can be matched with the T candidate webpage architecture information in the third mapping relationship to obtain the architecture matching result. If the architecture matching result indicates that the t-th candidate webpage architecture information matches the webpage architecture description information, then the expected score of the t-th page can be determined as the expected score threshold for the webpage to be used, so that the preset progress conditions corresponding to the webpage to be used can be determined based on the expected score threshold. Where t∈{1, 2, 3, ..., T}.

[0138] Based on the relevant content of steps 21 to 22 above, after obtaining the webpage to be used, the preset progress conditions corresponding to the webpage to be used can be determined according to the webpage template of the webpage to be used, so that the preset progress conditions can better represent the conditions reached when the webpage to be used is loaded. This is conducive to improving the recognition accuracy of the loaded webpage, thereby improving the webpage content extraction effect.

[0139] Method Example 3

[0140] In reality, some web pages may contain lazy-loaded resources (e.g., lazy-loaded image resources). Therefore, to further improve the web page content extraction effect, this application embodiment also provides a possible implementation method for extracting the above-mentioned "content to be used", which may specifically include steps 31-34:

[0141] Step 31: If the webpage to be used contains a lazy-loaded resource that meets the lazy-load conditions, then determine the address to be completed for that lazy-loaded resource based on the webpage to be used.

[0142] The aforementioned "lazy loading condition" is used to indicate the conditions that must be met for lazy loading of resources (e.g., lazy loading of image resources); and the "lazy loading condition" can be preset.

[0143] The aforementioned "lazy-loaded resources" refer to resources that are loaded late in the webpage to be used (e.g., image resources).

[0144] The "to be completed URL for lazy-loaded resources" mentioned above refers to the incomplete URL of the resource recorded in the webpage to be used for that lazy-loaded resource. For example, the "to be completed URL for lazy-loaded resources" could be https: / / www.xxxx.xx?imgs / ${width}.png.

[0145] It should be noted that the embodiments of this application do not limit the process of obtaining the "address to be completed for lazy-loaded resources" as described above. For example, any existing or future method of obtaining the resource address of lazy-loaded resources from a webpage can be used for implementation.

[0146] Step 32: Determine the address completion content based on the resource attribute information of the lazy-loaded resources.

[0147] The aforementioned "resource attribute information of lazy-loaded resources" is used to describe the lazy-loaded resource; moreover, the embodiments of this application do not limit the "resource attribute information of lazy-loaded resources". For example, it may include at least one of the following: the resource size of the lazy-loaded resource, the resource display device attribute of the lazy-loaded resource, and the webpage display size list of the lazy-loaded resource.

[0148] The "resource size of the lazy-loaded resource" mentioned above describes the size of the lazy-loaded resource. For example, if the "lazy-loaded resource" is an image, then the "resource size of the lazy-loaded resource" can include the image height and image width.

[0149] Furthermore, the embodiments of this application do not limit the method of obtaining the "resource size of lazy-loaded resources" as described above. For example, any existing or future method of obtaining the resource size of lazy-loaded resources can be used for implementation.

[0150] The aforementioned "resource display device attributes of lazy-loaded resources" is used to describe the attribute information of the display device of the lazy-loaded resource (e.g., device pixel ratio, etc.); and the embodiments of this application do not limit the "resource display device attributes of lazy-loaded resources", for example, it may include the device pixel ratio (i.e., devicePixelRatio) of the display device of the lazy-loaded resource.

[0151] Furthermore, the embodiments of this application do not limit the method of obtaining the above-mentioned "resource display device attributes of lazy-loaded resources". For example, it can be implemented by any existing or future method of obtaining resource display device attributes of lazy-loaded resources (for example, it can be obtained by using the interface provided by the browser that displays the webpage to be used).

[0152] The aforementioned "list of webpage display sizes for lazy-loaded resources" describes a list of available resource sizes provided by the webpage to be used for the lazy-loaded resource; and the "list of webpage display sizes for lazy-loaded resources" includes at least one candidate display size. For example, if the aforementioned "lazy-loaded resource" is an image, then the "resource size of the lazy-loaded resource" can be a list of available image widths [200, 400, 600, 800], and the list of available image widths includes four candidate display sizes (that is, 200, 400, 600, and 800).

[0153] Furthermore, the embodiments of this application do not limit the method of obtaining the "list of webpage display sizes for lazy-loaded resources" described above. For example, it can be implemented using any existing or future method of obtaining the list of webpage display sizes for lazy-loaded resources (for example, it can be obtained directly from the page to be used).

[0154] The "address completion content" mentioned above refers to the content that needs to be filled in the position to be completed in the "address to be completed" above. For example, when the "address to be completed" above is https: / / www.xxxx.xx?imgs / ${width}.png, then the "address completion content" can be 600, so that the "address completion content" can complete the "width" in the "address to be completed" above.

[0155] Furthermore, the embodiments of this application do not limit the process of determining the "address completion content" described above. For example, when the "resource attribute information" includes resource size, resource display device attributes, and a list of webpage display sizes, the process of determining the "address completion content" may specifically include steps 321-322:

[0156] Step 321: Determine the actual display size of the lazy-loaded resource based on its resource size and the device attributes of its display.

[0157] As an example, when the above-mentioned "lazy-loaded resource" is an image, the above-mentioned "resource size of the lazy-loaded resource" includes the image width of the lazy-loaded resource, and the above-mentioned "resource display device attribute of the lazy-loaded resource" includes the pixel ratio of the resource display device of the lazy-loaded resource, step 321 can specifically be: determining the actual display size of the lazy-loaded resource (e.g., 600) by multiplying the image width of the lazy-loaded resource (e.g., 300px) by the pixel ratio of the resource display device of the lazy-loaded resource (e.g., 2).

[0158] Step 322: Determine the address completion content based on the actual display size of the lazy-loaded resource and the list of webpage display sizes for that lazy-loaded resource.

[0159] As an example, step 322 may specifically include steps 3221-3222:

[0160] Step 3221: Compare the actual display size of the lazy-loaded resource with at least one candidate display size in the list of webpage display sizes for the lazy-loaded resource to obtain the size comparison result.

[0161] The above "size comparison result" is used to represent the relative size between the above "actual display size of lazy-loaded resources" and at least one candidate display size in the list of webpage display sizes.

[0162] Furthermore, the embodiments of this application do not limit the above-mentioned "size comparison result". For example, if the above-mentioned "list of webpage display sizes of lazy-loaded resources" includes a target display size (e.g., 600), then the "size comparison result" may include: the above-mentioned "actual display size of lazy-loaded resources" (e.g., 600) is the same as the target display size.

[0163] For example, if the list of webpage display sizes includes D candidate display sizes, then the "size comparison result" can include: the difference between the "actual display size of the lazy-loaded resource" and the first candidate display size, the difference between the "actual display size of the lazy-loaded resource" and the second candidate display size, ... (and so on), and the difference between the "actual display size of the lazy-loaded resource" and the Dth candidate display size. Here, D is a positive integer.

[0164] Step 3222: Determine the address completion content based on the size comparison results.

[0165] It should be noted that the implementation of step 3222 is not limited in this application embodiment. For example, if the above "size comparison result" indicates that the above "actual display size of lazy-loaded resources" is the same as the target display size in the webpage display size list, then the target display size can be determined as the address completion content.

[0166] For example, when the above "size comparison result" includes the difference between the above "actual display size of lazy-loaded resources" and the d-th candidate display size, and d is a positive integer, d≤D, step 3222 may specifically include: first performing a minimum value analysis on the "size comparison result" to obtain the minimum size difference; then determining the candidate display size with the minimum size difference as the address padding content.

[0167] Based on the relevant content of step 32 above, for the "actual display size of the lazy-loaded resource", if there is a candidate display size in the webpage display size list of the lazy-loaded resource that is equal to the "actual display size of the lazy-loaded resource", then the candidate display size can be directly determined as the address completion content; however, if there is no candidate display size in the webpage display size list of the lazy-loaded resource that is equal to the "actual display size of the lazy-loaded resource", then the candidate display size that is closest to the "actual display size of the lazy-loaded resource" can be determined as the address completion content.

[0168] Step 33: Determine the resource address of the lazy-loaded resource based on the address completion content and the address to be completed.

[0169] In this embodiment, after obtaining the address completion content and the address to be completed, the address completion content can be added to the position that needs to be completed in the address to be completed, thus obtaining the resource address of the lazy-loaded resource. For example, when the above "address to be completed" is https: / / www.xxxx.xx?imgs / ${width}.png, and the above "address completion content" is 600, the resource address of the lazy-loaded resource can be https: / / www.xxxx.xx?imgs / 600.png.

[0170] Step 34: Determine the content to be used based on the resource address of the lazy-loaded resource.

[0171] This application embodiment does not limit the implementation of step 34. For example, it may specifically include: directly combining the resource address of the lazy-loaded resource with other extracted web page content (e.g., text, images, and other general structured content in web pages) to obtain the content to be used. Alternatively, step 34 may specifically include steps 341-342:

[0172] Step 341: Obtain the actual resource data (i.e., the loaded resource data) of the lazy-loaded resource according to its resource address.

[0173] In this embodiment of the application, after obtaining the resource address of the lazy-loaded resource, the actual resource data of the lazy-loaded resource can be obtained according to the resource address of the lazy-loaded resource, so that the actual resource data can represent the loading and display effect of the lazy-loaded resource on the webpage to be used.

[0174] Step 342: Determine the content to be used based on the actual resource data of the lazy-loaded resources.

[0175] This application does not limit the implementation of step 342. For example, it can specifically be: combining the actual resource data of the lazy-loaded resource with other extracted web page content (e.g., text, images, and other general structured content in web pages) to obtain the content to be used. Alternatively, step 342 can specifically be: first, according to the web page display position of the lazy-loaded resource, filling the web page to be used with the actual resource data of the lazy-loaded resource to obtain the filled web page; then, extracting the content to be used from the filled web page. Here, "web page display position of the lazy-loaded resource" describes the position of the lazy-loaded resource in the web page to be used.

[0176] Based on the relevant content of steps 31 to 34 above, for each lazy-loaded resource in the webpage to be used, the resource address of each lazy-loaded resource can be determined first by using steps 31 to 33; then, based on the resource addresses of these lazy-loaded resources, the content to be used can be determined, so that the content to be used includes not only the general structured content of the webpage (e.g., elements such as images, videos, text, and links and their size coordinates), but also the loading and display content of the lazy-loaded resources. This can overcome the defect that lazy-loaded resources in the webpage cannot be crawled, thereby improving the webpage content extraction effect.

[0177] Method Example 4

[0178] In reality, some web pages not only include general elements (e.g., images, videos, text, links, etc.), but may also include special elements (e.g., carousels, form components, etc.). Therefore, in order to further improve the web page content extraction effect, this application embodiment also provides a possible implementation method for extracting the above-mentioned "content to be used", which may specifically include steps 41-44:

[0179] Step 41: If there is an element to be extracted in the webpage that meets the preset element conditions, then determine at least one candidate element feature and the element extraction rule corresponding to the at least one candidate element feature based on the element type identifier of the element to be extracted.

[0180] The aforementioned "preset element conditions" are used to represent the conditions achieved by a certain type of non-general element; moreover, this application does not limit the "preset element conditions", for example, they can be preset.

[0181] The term "elements to be extracted" refers to any non-generic element (e.g., a carousel) that appears in the webpage to be used.

[0182] The aforementioned "element type identifier of the element to be extracted" is used to uniquely identify the element category to which the element to be extracted belongs; moreover, the embodiments of this application do not limit the "element type identifier of the element to be extracted", for example, it can be the element category name of the element to be extracted (e.g., carousel element).

[0183] The aforementioned “candidate element features” refer to the element features that need to be referenced when determining the extraction rules of the aforementioned “elements to be extracted”; moreover, the “candidate element features” are used to describe a non-general element (in particular, a commonly used non-general element) under the element category to which the aforementioned “elements to be extracted” belong.

[0184] The embodiments of this application are not limited to the above-described process of obtaining "at least one candidate element feature". For example, if the non-general element represented by the above-described "element to be extracted" is a carousel, the process of obtaining "at least one candidate element feature" may specifically be: firstly, searching for some commonly used carousel components in the carousel component library; and then determining the element features of each commonly used carousel component as each candidate element feature.

[0185] The "element extraction rule corresponding to the k-th candidate element feature" is used to represent the extraction rule required when extracting content from web page elements that have the k-th candidate element feature. Furthermore, this application embodiment does not limit the "element extraction rule corresponding to the k-th candidate element feature," for example, it can be preset. Here, k is a positive integer, k≤K, K is a positive integer, and K represents the number of the aforementioned "at least one candidate element feature."

[0186] This application embodiment does not limit the implementation of step 41. For example, it can specifically be: first, searching for at least one reference element feature corresponding to the element type identifier of the element to be extracted from the fourth mapping relationship, and determining it as at least one candidate element feature corresponding to the element to be extracted; then, searching for the element extraction rules corresponding to each candidate element feature from the fifth mapping relationship. The "fourth mapping relationship" is used to record at least one reference element feature corresponding to each candidate element type identifier. The "fifth mapping relationship" is used to record the element extraction rules corresponding to each reference element feature.

[0187] Based on the relevant content of step 41 above, after determining that there are elements to be extracted in the webpage that meet the preset element conditions, at least one candidate element feature and the element extraction rule corresponding to the at least one candidate element feature can be determined according to the element type identifier of the element to be extracted, so that these element extraction rules can be referenced in the future to perform content extraction processing on the element to be extracted in the webpage.

[0188] Step 42: Match the element features of the element to be extracted with at least one candidate element feature to obtain the feature matching result.

[0189] The above "feature matching result" is used to indicate whether the element features of the element to be extracted match the features of each candidate element.

[0190] Step 43: If the feature matching result indicates that the target element feature matches the element feature of the element to be extracted, then the element to be extracted from the webpage to be used is extracted according to the element extraction rule corresponding to the target element feature, thereby obtaining the element extraction content of the element to be extracted. Here, "at least one candidate element feature" includes the target element feature.

[0191] The “target element features” mentioned above are used to represent candidate element features that match the element features of the element to be extracted.

[0192] Step 44: Determine the content to be used based on the element extraction content of the element to be extracted.

[0193] It should be noted that the implementation of step 44 is not limited in this application embodiment. For example, it may specifically include: combining the element extraction content of the element to be extracted with other web page content that has been extracted (e.g., general structured content in web pages such as text and images, and actual resource data of lazy-loaded resources, etc.) to obtain the content to be used.

[0194] Based on the relevant content of steps 41 to 44 above, for each non-general element in the webpage to be used, the element extraction content of each non-general element can be determined first by using steps 41 to 43; then, based on the element extraction content of these non-general elements, the content to be used can be determined, so that the content to be used includes not only the general structured content of the webpage (e.g., elements such as page images, videos, text, links, etc. and their size coordinate information), but also the structured content of non-general elements in the webpage. This can overcome the defect of not being able to extract non-general elements in the webpage, thereby improving the webpage content extraction effect.

[0195] Based on the web page content extraction method provided in the above embodiments, this application also provides a web page content extraction device, which will be explained and described below with reference to the accompanying drawings.

[0196] Device Examples

[0197] For technical details of the webpage content extraction device provided in the device embodiment, please refer to the above method embodiment.

[0198] See Figure 3 The figure is a schematic diagram of the structure of a web page content extraction device provided in an embodiment of this application.

[0199] The webpage content extraction device 300 provided in this application embodiment includes:

[0200] The content acquisition unit 301 is used to acquire the current page display content of the webpage to be used when the webpage to be used is opened according to the URL to be used; wherein, the URL to be used is used to uniquely identify the webpage to be used;

[0201] The progress determination unit 302 is used to determine the loading progress representation data of the webpage to be used based on the content displayed on the current page.

[0202] The judgment loop unit 303 is used to return to the content acquisition unit to perform the step of acquiring the current page display content of the webpage to be used if it is determined that the loading progress characterization data does not meet the preset progress conditions;

[0203] The content extraction unit 304 is used to extract the content to be used from the webpage to be used until it is determined that the loading progress representation data meets the preset progress conditions.

[0204] In one possible implementation, the progress determination unit 302 includes:

[0205] An element extraction subunit is used to extract at least one element to be used from the content displayed on the current page.

[0206] The progress determination subunit is used to determine the loading progress representation data of the webpage to be used based on the at least one element to be used.

[0207] In one possible implementation, the progress determination subunit includes:

[0208] An element classification subunit is used to classify the at least one element to be used to obtain at least one set of elements;

[0209] The element scoring subunit is used to determine the set score value for each set of elements;

[0210] The rating summary subunit is used to determine the loading progress representation data of the webpage to be used based on the sum of the set rating values ​​of the at least one set of elements.

[0211] In one possible implementation, the element classification subunit is specifically used to: classify the at least one element to be used to obtain the at least one set of elements and the element category identifier of the at least one set of elements;

[0212] The number of elements in the set is N;

[0213] The element scoring subunit is specifically used for: finding the preset element score value corresponding to the element category identifier of the nth element set from the first mapping relationship, and determining it as the individual score value of the nth element set; determining the set score value of the nth element set based on the product between the number of elements in the nth element set and the individual score value of the nth element set; wherein, the first mapping relationship includes the correspondence between the element category identifier of the nth element set and the individual score value of the nth element set; n is a positive integer, n≤N, and N is a positive integer.

[0214] In one possible implementation, the webpage content extraction device 300 further includes:

[0215] The condition determination unit is used to determine the webpage architecture description information of the webpage to be used; and to determine the preset progress conditions based on the webpage architecture description information.

[0216] In one possible implementation, the content extraction unit 304 includes:

[0217] The first determining subunit is used to determine the address to be completed for the lazy-loaded resource based on the webpage to be used if there is a lazy-loaded resource in the webpage to be used that meets the lazy-load conditions.

[0218] The second determining subunit is used to determine the address completion content based on the resource attribute information of the lazy-loaded resource;

[0219] The third determining subunit is used to determine the resource address of the lazy-loaded resource based on the address completion content and the address to be completed;

[0220] The fourth determining subunit is used to determine the content to be used based on the resource address of the lazy-loaded resource.

[0221] In one possible implementation, the resource attribute information includes at least one of the following: resource size, resource display device attributes, and webpage display size list.

[0222] In one possible implementation, the resource attribute information includes resource size, resource display device attributes, and a list of webpage display sizes;

[0223] The second determining subunit includes:

[0224] The fifth determining subunit is used to determine the actual display size of the lazy-loaded resource based on the resource size of the lazy-loaded resource and the resource display device attributes of the lazy-loaded resource;

[0225] The sixth determining subunit is used to determine the address completion content based on the actual display size and the list of webpage display sizes.

[0226] In one possible implementation, the sixth determining subunit is specifically used to: compare the actual display size with at least one candidate display size in the webpage display size list to obtain a size comparison result; and determine the address completion content based on the size comparison result.

[0227] In one possible implementation, the content extraction unit 304 includes:

[0228] The seventh determining subunit is used to determine at least one candidate element feature and the element extraction rule corresponding to the at least one candidate element feature if there is an element to be extracted in the webpage to be used that meets the preset element conditions.

[0229] The eighth determining subunit is used to match the element features of the element to be extracted with the features of at least one candidate element to obtain a feature matching result;

[0230] An element extraction subunit is used to extract the element to be extracted from the webpage according to the element extraction rules corresponding to the target element feature if the feature matching result indicates that the target element feature matches the element feature of the element to be extracted; wherein, the at least one candidate element feature includes the target element feature.

[0231] The ninth determining subunit is used to determine the content to be used based on the element extraction content of the element to be extracted.

[0232] Based on the aforementioned content of the webpage content extraction device 300, it can be understood that when the webpage to be used is opened according to the URL to be used, the device first obtains the current page display content of the webpage to be used; then, based on the current page display content, it determines the loading progress representation data of the webpage to be used, so that if it is determined that the loading progress representation data does not meet the preset progress conditions, it returns to continue executing the above step "obtain the current page display content of the webpage to be used"; until it is determined that the loading progress representation data meets the preset progress conditions, the content to be used is extracted from the webpage to be used. In this way, the loading progress of the webpage to be used can be obtained in real time, so that the loading completion time of the webpage to be used can be known in a timely manner, thereby effectively avoiding the adverse effects caused by extracting webpage content too early (or too late), which is conducive to improving the webpage content extraction effect.

[0233] Furthermore, embodiments of this application also provide a device, the device including a processor and a memory:

[0234] The memory is used to store computer programs;

[0235] The processor is used to execute any implementation of the web page content extraction method provided in the embodiments of this application according to the computer program.

[0236] Furthermore, embodiments of this application also provide a computer-readable storage medium for storing a computer program for executing any implementation of the web page content extraction method provided in embodiments of this application.

[0237] Furthermore, this application also provides a computer program product, which, when run on a terminal device, causes the terminal device to execute any implementation of the web page content extraction method provided in this application.

[0238] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0239] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the technical solutions of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall still fall within the protection scope of the technical solutions of the present invention.

Claims

1. A method for extracting webpage content, characterized in that, The method includes: When a webpage is opened according to the URL to be used, the current page display content of the webpage to be used is obtained, and the current page display content indicates the page content displayed on the webpage to be used; wherein, the URL to be used is used to uniquely identify the webpage to be used; Based on the content currently displayed on the page, the loading progress representation data of the webpage to be used is determined, and the loading progress representation data indicates the loading progress of the webpage to be used; If it is determined that the loading progress representation data does not meet the preset progress conditions, then return to the step of obtaining the current page display content of the webpage to be used, until it is determined that the loading progress representation data meets the preset progress conditions, then extract the content to be used from the webpage to be used, and the content to be used is the webpage content of the webpage to be used; The extraction process of the content to be used includes: if there is an element to be extracted in the webpage to be used that meets preset element conditions, then, based on the element type identifier of the element to be extracted, at least one candidate element feature and an element extraction rule corresponding to the at least one candidate element feature are determined; the element feature of the element to be extracted is matched with the at least one candidate element feature to obtain a feature matching result; if the feature matching result indicates that the target element feature matches the element feature of the element to be extracted, then, according to the element extraction rule corresponding to the target element feature, the element to be extracted in the webpage to be used is extracted to obtain the element extraction content of the element to be extracted; wherein, the at least one candidate element feature includes the target element feature; and the content to be used is determined based on the element extraction content of the element to be extracted.

2. The method according to claim 1, characterized in that, The step of determining the loading progress representation data of the webpage to be used based on the content currently displayed on the page includes: Extract at least one element to be used from the content currently displayed on the page; Based on the at least one element to be used, determine the loading progress representation data of the webpage to be used.

3. The method according to claim 2, characterized in that, The step of determining the loading progress representation data of the webpage to be used based on the at least one element to be used includes: The at least one element to be used is classified to obtain at least one set of elements; Determine the set score value for each set of elements; The loading progress representation data of the webpage to be used is determined based on the sum of the set score values ​​of the at least one set of elements.

4. The method according to claim 3, characterized in that, The process of classifying the at least one element to be used to obtain at least one set of elements includes: The at least one element to be used is classified to obtain the at least one set of elements and the element category identifier of the at least one set of elements; The number of element sets is N, and the process of determining the set score value of the nth element set includes: The preset value of the element rating corresponding to the element category identifier of the nth element set is found from the first mapping relationship and determined as the individual rating value of the nth element set; wherein, the first mapping relationship includes the correspondence between the element category identifier of the nth element set and the individual rating value of the nth element set; n is a positive integer, n≤N, and N is a positive integer; The set score of the nth element set is determined by multiplying the number of elements in the nth element set by the individual score of the nth element set.

5. The method according to claim 1, characterized in that, The method further includes: Determine the webpage architecture description information of the webpage to be used; The preset progress conditions are determined based on the webpage architecture description information.

6. The method according to claim 1, characterized in that, The extraction process of the content to be used includes: If the webpage to be used contains lazy-loaded resources that meet the lazy-load conditions, then the address to be completed for the lazy-loaded resources is determined based on the webpage to be used. Based on the resource attribute information of the lazy-loaded resources, determine the address completion content; The resource address of the lazy-loaded resource is determined based on the address completion content and the address to be completed; The content to be used is determined based on the resource address of the lazy-loaded resource.

7. The method according to claim 6, characterized in that, The resource attribute information includes at least one of the following: resource size, resource display device attributes, and webpage display size list.

8. The method according to claim 6, characterized in that, The resource attribute information includes resource size, resource display device attributes, and a list of webpage display sizes; The step of determining the address completion content based on the resource attribute information of the lazy-loaded resource includes: The actual display size of the lazy-loaded resource is determined based on the resource size of the lazy-loaded resource and the resource display device attributes of the lazy-loaded resource; The address completion content is determined based on the actual display size and the list of webpage display sizes.

9. The method according to claim 8, characterized in that, The step of determining the address completion content based on the actual display size and the list of webpage display sizes includes: The actual display size is compared with at least one candidate display size in the list of webpage display sizes to obtain a size comparison result; Based on the size comparison results, the address completion content is determined.

10. A webpage content extraction device, characterized in that, include: The content acquisition unit is used to acquire the current page display content of the webpage to be used when the webpage to be used is opened according to the URL to be used, wherein the current page display content indicates the page content displayed on the webpage to be used; wherein the URL to be used is used to uniquely identify the webpage to be used; The progress determination unit is used to determine the loading progress representation data of the webpage to be used based on the content displayed on the current page, wherein the loading progress representation data indicates the loading progress of the webpage to be used. The judgment loop unit is used to return to the content acquisition unit to perform the step of acquiring the current page display content of the webpage to be used if it is determined that the loading progress representation data does not meet the preset progress conditions; A content extraction unit is used to extract content to be used from the webpage to be used until the loading progress representation data is determined to meet a preset progress condition. The content to be used is the webpage content of the webpage to be used. The extraction process of the content to be used includes: if there is an element to be extracted in the webpage to be used that meets a preset element condition, then, based on the element type identifier of the element to be extracted, at least one candidate element feature and an element extraction rule corresponding to the at least one candidate element feature are determined; the element feature of the element to be extracted is matched with the at least one candidate element feature to obtain a feature matching result; if the feature matching result indicates that the target element feature matches the element feature of the element to be extracted, then, according to the element extraction rule corresponding to the target element feature, the element to be extracted in the webpage to be used is extracted to obtain the element extraction content of the element to be extracted; wherein, the at least one candidate element feature includes the target element feature; and the content to be used is determined based on the element extraction content of the element to be extracted.

11. A webpage content extraction device, characterized in that, The device includes a processor and a memory: The memory is used to store computer programs; The processor is configured to perform the method according to any one of claims 1-9 according to the computer program.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-9.

13. A computer program product, characterized in that, When the computer program product is run on a terminal device, it causes the terminal device to perform the method according to any one of claims 1-9.

Images