A webpage deduplication method and device, a storage medium and an electronic device
By calculating the duplication rate of webpage summary text and adjusting the order of the webpage list, the problem of duplicate webpages in search engines was solved, achieving efficient and accurate deduplication and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING QIHOOD TECHNOLOGY CO LTD
- Filing Date
- 2022-08-22
- Publication Date
- 2026-07-03
AI Technical Summary
Existing technologies often result in duplicate web pages in the search engine results, and current deduplication methods cannot simultaneously achieve both accuracy and efficiency.
By obtaining the summary text of web pages, calculating the duplication rate based on the string set, adjusting the order of the web page list using the average duplication rate, and deleting duplicate web pages.
It improves the accuracy and efficiency of webpage lists, reduces webpage duplication, and enhances the user experience.
Smart Images
Figure CN117668347B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, storage medium and electronic device for deduplicating web pages. Background Technology
[0002] With the development and widespread application of the internet, online information has exploded, and the internet has become a crucial source of information for people. Search engines help people quickly find the information they need online, facilitating their work and daily lives. Search engines have gradually become one of the most frequently used online services.
[0003] However, many duplicate web pages appear in the search engine results list, and the problem of duplicate web pages has always been a major issue encountered by users when using search engines. Summary of the Invention
[0004] This application provides a method, apparatus, storage medium, and electronic device for deduplicating web pages, which can deduplicat web pages in a search engine's returned list, improving the problem of duplicate web pages in the list. The technical solution is as follows:
[0005] In a first aspect, the present application provides a webpage deduplication method, characterized in that the method includes:
[0006] Obtain the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list;
[0007] Obtain a first set of strings from the first summary text, and determine a first repetition rate of the first summary text in the second summary text based on the first set of strings;
[0008] Obtain a second set of strings from the second summary text, and determine a second repetition rate of the second summary text in the first summary text based on the second set of strings;
[0009] Based on the first repetition rate and the second repetition rate, the deduplication results of the first webpage and the second webpage are determined, and the order of the webpages in the webpage list is adjusted based on the deduplication results.
[0010] Secondly, this application provides a webpage deduplication device, the device comprising:
[0011] The summary text acquisition module is used to acquire the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list;
[0012] The first repetition rate calculation module is used to obtain a first set of strings in the first summary text and determine the first repetition rate of the first summary text in the second summary text based on the first set of strings.
[0013] The second repetition rate calculation module is used to obtain a second string set in the second summary text and determine the second repetition rate of the second summary text in the first summary text based on the second string set.
[0014] The webpage list adjustment module is used to determine the duplication results of the first webpage and the second webpage based on the first duplication rate and the second duplication rate, and adjust the arrangement order of the webpages in the webpage list based on the duplication results.
[0015] Thirdly, embodiments of this application provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.
[0016] Fourthly, embodiments of this application provide an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.
[0017] In one or more embodiments of this application, firstly, a first summary text corresponding to a first webpage and a second summary text corresponding to a second webpage are obtained. Then, based on the first string set in the first summary text, a first repetition rate of the first summary text in the second summary text is determined. Based on the second string set in the second summary text, a second repetition rate of the second summary text in the first summary text is determined. Finally, based on the first and second repetition rates, a deduplication result is determined. Based on the deduplication result, the order of webpages in the webpage list is adjusted to eliminate deduplication in the webpage list, improve the problem of duplicate webpages in the webpage list, and enhance the user experience. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 A flowchart illustrating a webpage deduplication method provided in this application embodiment;
[0020] Figure 2A flowchart illustrating a webpage deduplication method provided in this application embodiment;
[0021] Figure 3 This is a schematic diagram of the structure of a webpage deduplication device provided in an embodiment of this application;
[0022] Figure 4 This is a schematic diagram of the structure of a webpage deduplication device provided in an embodiment of this application;
[0023] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.
[0026] In related technologies, deduplication for duplicate web pages typically employs two methods: URL-based deduplication and content-based deduplication. URL-based deduplication is less effective for duplicate pages with different URLs but identical content. Content-based deduplication requires extracting content features from different web pages for similarity comparison; however, extracting too few features results in low accuracy, while extracting too many features leads to lengthy processing times, making it impossible to simultaneously achieve both accuracy and efficiency.
[0027] Based on this, this application proposes a webpage deduplication method. This method involves obtaining a first summary text corresponding to a first webpage and a second summary text corresponding to a second webpage. Then, based on the first string set in the first summary text, a first repetition rate of the first summary text in the second summary text is determined. Based on the second string set in the second summary text, a second repetition rate of the second summary text in the first summary text is determined. Finally, based on the combined first and second repetition rates, a deduplication result is determined. Based on the deduplication result, the order of webpages in the webpage list is adjusted to achieve deduplication of the webpage list, improve the problem of duplicate webpages in the webpage list, and enhance the user experience.
[0028] The following detailed description is provided with reference to specific embodiments. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. The flowcharts shown in the accompanying drawings are merely illustrative and are not necessarily to be performed in accordance with the steps shown. For example, some steps are parallel and do not have a strict logical order; therefore, the actual execution order is variable.
[0029] Please see Figure 1 This is a flowchart illustrating a webpage deduplication method provided in an embodiment of this application. In a specific embodiment, the webpage deduplication method is applied to a webpage deduplication device 1 or an electronic device equipped with a webpage deduplication device 1. The following will focus on... Figure 1 The process shown will be explained in detail. The webpage deduplication method may specifically include the following steps:
[0030] S101, obtain the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list;
[0031] Specifically, the first and second web pages are identified from the list of web pages returned by the search engine, and the first summary text of the first web page and the second summary text of the second web page are obtained.
[0032] The first webpage and the second webpage are different webpages in the same list of webpages returned by the search engine.
[0033] The summary text is a summary of the content on a webpage that is similar to the user's search query. Understandably, after a search engine generates a list of webpages based on the user's search input, for performance reasons, the list typically only displays a portion of the summary text for each webpage, rather than the full text. In this case, the summary text is directly used as the content representation of the webpage.
[0034] It should be noted that in the embodiments of this application, the use of the first and second description methods is only to distinguish the event elements obtained under different circumstances. For example, the first webpage and the second webpage are only to distinguish two different webpages in the webpage list. The first summary text is the summary text in the first webpage, and the second summary text is the summary text in the second webpage.
[0035] In one embodiment, the web pages in the web page list are sorted from highest to lowest priority. The priority of each web page is related to the search terms entered by the user.
[0036] S102, Obtain the first string set in the first summary text, and determine the first repetition rate of the first summary text in the second summary text based on the first string set;
[0037] Specifically, the characters in the first summary text are segmented to obtain a first string set consisting of multiple first strings corresponding to the first summary text. Based on the repetition of each first string in the first string set in the second summary text, the first repetition rate of the first summary text in the second summary text is determined.
[0038] Optionally, obtaining the first string set in the first summary text can be achieved by performing character segmentation on the first summary text based on a preset step size and a preset string length to obtain the first string set corresponding to the first summary text.
[0039] Optionally, determining the first repetition rate of the first summary text in the second summary text based on the first string set can be achieved by: sequentially searching for each first string in the first string set in the second summary text, determining the first number of first strings successfully found in the second summary text, and calculating the first repetition rate based on the first number and the first total number of first strings in the first string set. Wherein, the first repetition rate = the first number / the first total number.
[0040] S103, Obtain the second string set in the second summary text, and determine the second repetition rate of the second summary text in the first summary text based on the second string set;
[0041] Specifically, the characters in the second summary text are segmented to obtain a set of second strings consisting of multiple second strings corresponding to the second summary text. Based on the repetition of each second string in the second string set in the first summary text, the second repetition rate of the second summary text in the first summary text is determined.
[0042] Optionally, obtaining the second string set in the second summary text can be achieved by performing character segmentation on the second summary text based on a preset step size and a preset string length to obtain the second string set corresponding to the second summary text.
[0043] Optionally, determining the second repetition rate of the second summary text in the first summary text based on the second string set can be achieved by: sequentially searching for each second string in the second string set in the first summary text, determining the second number of second strings successfully found in the first summary text, and calculating the second repetition rate based on the second number and the second total number of second strings in the second string set. Wherein, the second repetition rate = the second number / the second total number.
[0044] S104. Based on the first repetition rate and the second repetition rate, determine the plagiarism detection results of the first webpage and the second webpage, and adjust the order of the webpages in the webpage list based on the plagiarism detection results.
[0045] Specifically, the average repetition rate is calculated based on the first and second repetition rates. The plagiarism detection results for the first and second webpages are then determined based on the average repetition rate and a preset repetition rate threshold. If the average repetition rate is greater than the preset repetition rate threshold, the first and second webpages are determined to be duplicates; if the average repetition rate is less than or equal to the preset repetition rate threshold, the first and second webpages are determined not to be duplicates. Furthermore, when the first and second webpages are determined to be duplicates, either the first or second webpage is deleted.
[0046] In one or more embodiments of this application, firstly, a first summary text corresponding to a first webpage and a second summary text corresponding to a second webpage are obtained. Then, based on the first string set in the first summary text, a first repetition rate of the first summary text in the second summary text is determined. Based on the second string set in the second summary text, a second repetition rate of the second summary text in the first summary text is determined. Finally, based on the first and second repetition rates, a deduplication result is determined. Based on the deduplication result, the order of webpages in the webpage list is adjusted to eliminate deduplication in the webpage list, improve the problem of duplicate webpages in the webpage list, and enhance the user experience.
[0047] Please see Figure 2 This is a schematic flowchart illustrating a webpage deduplication method provided in an embodiment of this application. The following will focus on... Figure 2 The process shown is explained in detail. The webpage deduplication method may include the following steps:
[0048] S201, Obtain the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list;
[0049] Specifically, the first and second web pages are identified from the list of web pages returned by the search engine, and the first summary text of the first web page and the second summary text of the second web page are obtained.
[0050] The first webpage and the second webpage are different webpages in the same list of webpages returned by the search engine.
[0051] The summary text is a summary of the content on a webpage that is similar to the user's search query. Understandably, after a search engine generates a list of webpages based on the user's search input, for performance reasons, the list typically only displays a portion of the summary text for each webpage, rather than the full text. In this case, the summary text is directly used as the content representation of the webpage.
[0052] It should be noted that in the embodiments of this application, the use of the first and second description methods is only to distinguish the event elements obtained under different circumstances. For example, the first webpage and the second webpage are only to distinguish two different webpages in the webpage list. The first summary text is the summary text in the first webpage, and the second summary text is the summary text in the second webpage.
[0053] In one embodiment, the web pages in the web page list are sorted from highest to lowest priority. The priority of each web page is related to the search terms entered by the user.
[0054] S202, perform character stringing on the first summary text based on the preset step size and preset string length to obtain the first string set corresponding to the first summary text;
[0055] Specifically, after obtaining the first summary text corresponding to the first webpage, the first summary text is split into characters based on a preset step size and a preset string length to obtain a set of first strings composed of multiple first strings.
[0056] The preset string length is the length of the first string, and the preset step size is the distance between the first characters of two adjacent first strings.
[0057] For example, if the first summary text is "The weather in Beijing is very good today", and the preset step size is 2 and the preset string length is 3, then the first summary text is processed by character segmentation based on the preset step size and the preset string length, resulting in the following first string: "Today in Beijing", "Beijing", "The weather", "The weather is very good".
[0058] S203, sequentially search for each first string in the first string set in the second summary text, and determine the first number of first strings that were successfully found;
[0059] Specifically, after performing character string splitting on the first summary text to obtain the first string set corresponding to the first summary text, the second summary text corresponding to the second webpage performs deduplication on each of the first strings in the first string set to determine the first number of duplicate first strings contained in the second summary text.
[0060] The process of performing duplicate processing on each first string in the first string set in the second summary text corresponding to the second webpage involves sequentially searching for each first string in the second summary text, identifying the first string that is successfully found as a duplicate first string, and recording the first number of duplicate first strings.
[0061] For example, if the second summary text is "Traffic in Beijing is very congested today", and the first string set contains the following first strings: "Today North", "Beijing", "The weather", and "The air is very good". If each first string in the first string set is searched in the second summary text, and "Today North" and "Beijing" are found, then the first count of the repeated first strings in the second summary text is 2.
[0062] S204, calculate the first repetition rate based on the first quantity and the first total quantity of the first string in the first string set;
[0063] Specifically, the first quantity is the number of times the first string appears repeatedly in the second summary text in the first string set, and the ratio of the first quantity to the first total quantity of the first string in the first string set is used as the first repetition rate of the first summary text in the second summary text.
[0064] For example, if the first string set contains the following six first strings: "Today North", "Beijing", "The weather", and "The weather is very good", and the second summary text finds two first strings: "Today North" and "Beijing", then the first number is 2, the total number of first strings is 6, and the first repetition rate is 2 / 6 ≈ 33.33%.
[0065] S205, perform character stringing on the second summary text based on the preset step size and preset string length to obtain the second string set corresponding to the second summary text;
[0066] Specifically, after obtaining the second summary text corresponding to the second webpage, the second summary text is split into characters based on a preset step size and a preset string length to obtain a set of second strings composed of multiple second strings.
[0067] The preset string length is the length of the second string, and the preset step size is the distance between the first characters of two adjacent second strings.
[0068] S206, sequentially search for each second string in the second string set in the first summary text, and determine the second number of second strings that were successfully found;
[0069] Specifically, after performing character segmentation on the second summary text to obtain the second string set corresponding to the second summary text, the first summary text corresponding to the first webpage performs deduplication on each second string in the second string set to determine the second number of duplicate second strings contained in the first summary text.
[0070] The process of performing duplicate processing on each second string in the second string set in the first summary text corresponding to the first webpage involves sequentially searching for each second string in the first summary text, identifying the successfully found second strings as duplicate second strings, and recording the second number of duplicate second strings.
[0071] S207, the second repetition rate is calculated based on the second quantity and the second total quantity of the second string in the second string set;
[0072] Specifically, the second quantity is the number of second strings that appear repeatedly in the first summary text within the second string set, and the ratio of the second quantity to the second total quantity of second strings in the second string set is used as the second repetition rate of the second summary text in the first summary text.
[0073] S208, the average repetition rate of the first webpage and the second webpage is calculated based on the first repetition rate and the second repetition rate;
[0074] Specifically, the average repetition rate of the first webpage and the second webpage is obtained by averaging the first repetition rate and the second repetition rate.
[0075] It should be noted that the first repetition rate is the repetition rate of the first summary text of the first webpage in the second summary text of the second webpage, and the second repetition rate is the repetition rate of the second summary text of the second webpage in the first summary text of the first webpage. The average repetition rate is the average of the first and second repetition rates.
[0076] In one scenario, if the second summary text is longer than the first summary text, and the second summary text completely encompasses the first summary text, the calculated first repetition rate will be very high, while the second repetition rate will be very low. In this case, the first and second repetition rates exhibit strong one-sidedness, making it difficult to independently assess whether the first and second webpages are duplicated. However, calculating the average repetition rate based on the first and second repetition rates avoids the inaccuracy of evaluating whether the first and second webpages are duplicated solely based on either the first or second repetition rate, effectively improving the accuracy of webpage deduplication.
[0077] S209: When the average repetition rate is greater than the preset repetition rate threshold, it is determined that the first webpage and the second webpage are repetitive, and the first webpage or the second webpage is deleted from the webpage list.
[0078] Specifically, a preset repetition rate threshold is set. When the average repetition rate is greater than the preset repetition rate threshold, it is determined that the first webpage and the second webpage are duplicates, and the first webpage or the second webpage is deleted from the webpage list.
[0079] It is understandable that a webpage list contains multiple webpages. When two webpages are identified as duplicates, one of them is deleted from the webpage list to ensure that the webpages in the list are no longer duplicated, making it more concise and easier for users to read.
[0080] In one embodiment, the web pages in the web page list are sorted according to priority, with the first and second web pages determined based on their priority. If the first and second web pages are determined to be duplicates, the web page with the lower priority is deleted. If the first web page has a higher priority than the second web page, the second web page is deleted when the first and second web pages are determined to be duplicates. For example, if the first web page is in the first position of the web page list and the second web page is in the third position, and the first web page has a higher priority, then when the first and second web pages are determined to be duplicates, the second web page is deleted from the web page list, and all web pages following the second web page are moved up one position.
[0081] S210, when the average repetition rate is less than or equal to the preset repetition rate threshold, determine that the first webpage and the second webpage are not repetitive.
[0082] Specifically, when the average repetition rate is less than or equal to the preset repetition rate threshold, it is determined that the first webpage and the second webpage are not repetitive, and no processing is performed on the first webpage and the second webpage.
[0083] In this embodiment, the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage are first obtained. Then, the first and second summary texts are processed by character segmentation according to a preset step size and a preset string length to obtain a first string set and a second string set. Based on the first string set in the first summary text, a first repetition rate of the first summary text in the second summary text is determined. Based on the second string set in the second summary text, a second repetition rate of the second summary text in the first summary text is determined. Finally, the average repetition rate is calculated based on the first and second repetition rates. When the average repetition rate is greater than a preset repetition rate threshold, the first and second webpages are determined to be duplicates, and either the first or second webpage is deleted from the webpage list. This achieves deduplication of the webpage list, improves the problem of duplicate webpages in the webpage list, and enhances the user experience. Furthermore, this embodiment performs character segmentation of the summary text based on a preset step size and a preset string length, and calculates the repetition rate based on the strings in the summary text. There is no complicated calculation process, resulting in fast and efficient webpage deduplication. By calculating the average repetition rate and judging whether a webpage is duplicated based on the average repetition rate, the accuracy of the judgment is ensured.
[0084] The following will be combined with the appendix Figure 3 This paper provides a detailed description of the webpage deduplication device provided in the embodiments of this application. It should be noted that the appendix... Figure 3 The webpage deduplication device 1 is used to perform the functions described in this application. Figure 1 , Figure 2 The methods shown in the embodiments are for illustrative purposes only, illustrating the parts relevant to the embodiments of this application. For specific technical details not disclosed, please refer to this application. Figure 1 , Figure 2 The example shown.
[0085] Please see Figure 3 This is a schematic diagram of a webpage deduplication device provided in an embodiment of this application. Figure 3 As shown, the webpage deduplication device 1 can be implemented as all or part of an electronic device through software, hardware, or a combination of both. According to some embodiments, the webpage deduplication device 1 includes a summary text acquisition module 11, a first duplication rate calculation module 12, a second duplication rate calculation module 13, and a webpage list adjustment module 14, specifically including:
[0086] Summary text acquisition module 11 is used to acquire the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list;
[0087] The first repetition rate calculation module 12 is used to obtain a first string set in the first summary text and determine the first repetition rate of the first summary text in the second summary text based on the first string set.
[0088] The second repetition rate calculation module 13 is used to obtain a second string set in the second summary text and determine the second repetition rate of the second summary text in the first summary text based on the second string set.
[0089] The webpage list adjustment module 14 is used to determine the duplication results of the first webpage and the second webpage based on the first duplication rate and the second duplication rate, and adjust the arrangement order of the webpages in the webpage list based on the duplication results.
[0090] Optionally, when the first repetition rate calculation module 12 performs the step of obtaining the first string set in the first summary text, it is specifically used for:
[0091] The first summary text is processed by character segmentation based on a preset step size and a preset string length to obtain the first string set corresponding to the first summary text.
[0092] Optionally, the first repetition rate calculation module 12, when performing the step of determining the first repetition rate of the first summary text in the second summary text based on the first string set, is specifically used for:
[0093] In the second summary text, search sequentially for each first string in the first string set, and determine the first number of first strings that were successfully found;
[0094] The first repetition rate is calculated based on the first quantity and the first total quantity of the first string in the first string set.
[0095] Optionally, when the second repetition rate calculation module 13 performs the step of obtaining the second string set in the second summary text, it is specifically used for:
[0096] The second summary text is processed by character segmentation based on a preset step size and a preset string length to obtain the second string set corresponding to the second summary text.
[0097] Optionally, when the second repetition rate calculation module 13 performs the step of determining the second repetition rate of the second summary text in the first summary text based on the second string set, it is specifically used for:
[0098] Search sequentially for each second string in the second string set in the first summary text, and determine the second number of second strings that were successfully found.
[0099] The second repetition rate is calculated based on the second quantity and the second total quantity of the second string in the second string set.
[0100] Optionally, the webpage list adjustment module 14 is specifically used for:
[0101] The average repetition rate of the first webpage and the second webpage is calculated based on the first repetition rate and the second repetition rate.
[0102] When the average repetition rate is greater than a preset repetition rate threshold, it is determined that the first webpage and the second webpage are duplicates, and the first webpage or the second webpage is deleted from the webpage list.
[0103] Optionally, the webpage list adjustment module 14 is further configured to:
[0104] When the average repetition rate is less than or equal to a preset repetition rate threshold, it is determined that the first webpage and the second webpage do not repetite.
[0105] Optional, please see Figure 4 This is a schematic diagram illustrating an example of a webpage deduplication device provided in an embodiment of this application. Figure 4 As shown, the device further includes a webpage list acquisition module 15, which is specifically used for:
[0106] Obtain a list of web pages, wherein the list includes at least two web pages, and the at least two web pages are sorted according to priority;
[0107] Based on the priority sorting, the first webpage and the second webpage are determined in the webpage list, respectively.
[0108] In this embodiment, the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage are first obtained. Then, the first and second summary texts are processed by character segmentation according to a preset step size and a preset string length to obtain a first string set and a second string set. Based on the first string set in the first summary text, a first repetition rate of the first summary text in the second summary text is determined. Based on the second string set in the second summary text, a second repetition rate of the second summary text in the first summary text is determined. Finally, the average repetition rate is calculated based on the first and second repetition rates. When the average repetition rate is greater than a preset repetition rate threshold, the first and second webpages are determined to be duplicates, and either the first or second webpage is deleted from the webpage list. This achieves deduplication of the webpage list, improves the problem of duplicate webpages in the webpage list, and enhances the user experience. Furthermore, this embodiment performs character segmentation of the summary text based on a preset step size and a preset string length, and calculates the repetition rate based on the strings in the summary text. There is no complicated calculation process, resulting in fast and efficient webpage deduplication. By calculating the average repetition rate and judging whether a webpage is duplicated based on the average repetition rate, the accuracy of the judgment is ensured.
[0109] It should be noted that the webpage deduplication device provided in the above embodiments is only illustrated by the division of the above functional modules when performing the webpage deduplication method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the webpage deduplication device and the webpage deduplication method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.
[0110] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0111] This application also provides a computer storage medium that can store multiple instructions, which are adapted to be loaded and executed by a processor as described above. Figures 1-2 The webpage deduplication method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1-2 The specific details of the illustrated embodiments will not be elaborated here.
[0112] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the processor as described above. Figures 1-2 The webpage deduplication method described in the illustrated embodiment can be found in the following documentation for its specific execution process. Figures 1-2 The specific details of the illustrated embodiments will not be elaborated here.
[0113] Please refer to Figure 5 This diagram illustrates a structural block diagram of an electronic device provided in an exemplary embodiment of this application. The electronic device in this application may include one or more components such as a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, memory 120, input device 130, and output device 140 may be connected via the bus 150.
[0114] Processor 110 may include one or more processing cores. Processor 110 connects to various parts of the electronic device using various interfaces and lines, and executes various functions of terminal 100 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 120, and by calling data stored in memory 120. Optionally, processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). Processor 110 may integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user page, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into processor 110 and may be implemented separately using a communication chip.
[0115] The memory 120 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 120 may include non-transitory computer-readable storage medium. The memory 120 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the various method embodiments described above, etc. The operating system may be the Android system, including systems deeply developed based on the Android system, the iOS system developed by Apple Inc., including systems deeply developed based on the iOS system, or other systems.
[0116] The memory 120 can be divided into operating system space and user space. The operating system runs in the operating system space, while native and third-party applications run in user space. To ensure that different third-party applications can achieve good running performance, the operating system allocates corresponding system resources for each application. However, different application scenarios within the same third-party application have different requirements for system resources. For example, in local resource loading scenarios, third-party applications have high requirements for disk read speed; in animation rendering scenarios, third-party applications have high requirements for GPU performance. Since the operating system and third-party applications are independent of each other, the operating system often cannot promptly perceive the current application scenario of a third-party application, resulting in the operating system's inability to adapt system resources accordingly.
[0117] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to establish data communication between the third-party applications and the operating system. This would allow the operating system to obtain the current scenario information of the third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.
[0118] The input device 130 is used to receive input instructions or data, and includes, but is not limited to, a keyboard, mouse, camera, microphone, or touch device. The output device 140 is used to output instructions or data, and includes, but is not limited to, a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 can be a touch display screen.
[0119] The touch display screen can be designed as a full-screen, curved screen, or irregularly shaped screen. It can also be designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen; however, this application does not limit the specific design in this regard.
[0120] In addition, those skilled in the art will understand that the structure of the electronic device shown in the above figures does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the electronic device may also include radio frequency circuits, input units, sensors, audio circuits, Wireless Fidelity (WiFi) modules, power supplies, Bluetooth modules, etc., which will not be described in detail here.
[0121] exist Figure 5 In the illustrated electronic device, the processor 110 can be used to call the webpage deduplication program stored in the memory 120 and execute it to implement the webpage deduplication method as described in the various method embodiments of this application.
[0122] In this embodiment, the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage are first obtained. Then, the first and second summary texts are processed by character segmentation according to a preset step size and a preset string length to obtain a first string set and a second string set. Based on the first string set in the first summary text, a first repetition rate of the first summary text in the second summary text is determined. Based on the second string set in the second summary text, a second repetition rate of the second summary text in the first summary text is determined. Finally, the average repetition rate is calculated based on the first and second repetition rates. When the average repetition rate is greater than a preset repetition rate threshold, the first and second webpages are determined to be duplicates, and either the first or second webpage is deleted from the webpage list. This achieves deduplication of the webpage list, improves the problem of duplicate webpages in the webpage list, and enhances the user experience. Furthermore, this embodiment performs character segmentation of the summary text based on a preset step size and a preset string length, and calculates the repetition rate based on the strings in the summary text. There is no complicated calculation process, resulting in fast and efficient webpage deduplication. By calculating the average repetition rate and judging whether a webpage is duplicated based on the average repetition rate, the accuracy of the judgment is ensured.
[0123] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.
[0124] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.
Claims
1. A method for deduplicating web pages, characterized in that, The method includes: Obtain the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list; Obtain a first set of strings from the first summary text, and determine a first repetition rate of the first summary text in the second summary text based on the first set of strings; wherein, the first set of strings includes multiple first strings corresponding to the first summary text, and the first repetition rate is determined based on the repetition of each first string in the second summary text; Obtain a second string set from the second summary text, and determine a second repetition rate of the second summary text in the first summary text based on the second string set; wherein, the second string set includes multiple second strings corresponding to the second summary text, and the second repetition rate is determined based on the repetition of each second string in the first summary text; The average repetition rate of the first webpage and the second webpage is calculated based on the first repetition rate and the second repetition rate; when the average repetition rate is greater than a preset repetition rate threshold, it is determined that the first webpage and the second webpage are repetitive, and the first webpage or the second webpage is deleted from the webpage list; when the average repetition rate is less than or equal to the preset repetition rate threshold, it is determined that the first webpage and the second webpage are not repetitive.
2. The method according to claim 1, characterized in that, The step of obtaining the first set of strings in the first summary text includes: The first summary text is processed by character segmentation based on a preset step size and a preset string length to obtain the first string set corresponding to the first summary text.
3. The method according to claim 2, characterized in that, Determining the first repetition rate of the first summary text in the second summary text based on the first string set includes: In the second summary text, search sequentially for each first string in the first string set, and determine the first number of first strings that were successfully found; The first repetition rate is calculated based on the first quantity and the first total quantity of the first string in the first string set.
4. The method according to claim 1, characterized in that, The step of obtaining the second set of strings in the second summary text includes: The second summary text is processed by character segmentation based on a preset step size and a preset string length to obtain the second string set corresponding to the second summary text.
5. The method according to claim 4, characterized in that, Determining the second repetition rate of the second summary text in the first summary text based on the second string set includes: Search sequentially for each second string in the second string set in the first summary text, and determine the second number of second strings that were successfully found. The second repetition rate is calculated based on the second quantity and the second total quantity of the second string in the second string set.
6. The method according to claim 1, characterized in that, Before obtaining the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, the method further includes: Obtain a list of web pages, wherein the list includes at least two web pages, and the at least two web pages are sorted according to priority; Based on the priority sorting, the first webpage and the second webpage are determined in the webpage list, respectively.
7. A webpage deduplication device, characterized in that, The device includes: The summary text acquisition module is used to acquire the first summary text corresponding to the first webpage and the second summary text corresponding to the second webpage, wherein the first webpage and the second webpage are different webpages located in the same webpage list; The first repetition rate calculation module is used to obtain a first string set in the first summary text and determine a first repetition rate of the first summary text in the second summary text based on the first string set; wherein, the first string set includes multiple first strings corresponding to the first summary text, and the first repetition rate is determined based on the repetition of each first string in the second summary text; The second repetition rate calculation module is used to obtain a second string set in the second summary text and determine a second repetition rate of the second summary text in the first summary text based on the second string set; wherein, the second string set includes multiple second strings corresponding to the second summary text, and the second repetition rate is determined based on the repetition status of each second string in the first summary text; The webpage list adjustment module is used to calculate the average repetition rate of a first webpage and a second webpage based on the first repetition rate and the second repetition rate; when the average repetition rate is greater than a preset repetition rate threshold, it is determined that the first webpage and the second webpage are repetitive, and the first webpage or the second webpage is deleted from the webpage list; when the average repetition rate is less than or equal to the preset repetition rate threshold, it is determined that the first webpage and the second webpage are not repetitive.
8. The apparatus according to claim 7, characterized in that, The first repetition rate calculation module is also used to perform character segmentation processing on the first summary text based on a preset step size and a preset string length to obtain a first string set corresponding to the first summary text.
9. The apparatus according to claim 8, characterized in that, The first repetition rate calculation module is also used to sequentially search for each first string in the first string set in the second summary text, and determine the first number of first strings that were successfully found; The first repetition rate is calculated based on the first quantity and the first total quantity of the first string in the first string set.
10. The apparatus according to claim 7, characterized in that, The second repetition rate calculation module is also used to perform character segmentation on the second summary text based on a preset step size and a preset string length to obtain a second string set corresponding to the second summary text.
11. The apparatus according to claim 10, characterized in that, The second repetition rate calculation module is also used to sequentially search for each second string in the second string set in the first summary text, determine the second number of successfully found second strings, and calculate the second repetition rate based on the second number and the second total number of second strings in the second string set.
12. The apparatus according to claim 7, characterized in that, The device further includes: A webpage list acquisition module is used to acquire a webpage list, which includes at least two webpages, and the at least two webpages are sorted according to priority; based on the priority sorting, a first webpage and a second webpage are determined from the webpage list respectively.
13. A storage medium storing a plurality of instructions, characterized in that, When the instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 6.
14. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the steps of the method as claimed in any one of claims 1 to 6.