Webpage text processing method and device, electronic equipment and storage medium
By obtaining and analyzing text in the source web page and jumping web page, calculating membership and updating the weighted text model, the problem of insufficient web browsing intention analysis is solved, and the quality and user experience of recommended content are improved.
Patent Information
- Application Number
- CN202410990265.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art only relies on text analysis for network links, resulting in insufficient analysis of web browsing intentions, which in turn leads to the limitations of the inability to meet user needs.
By obtaining the text in the source and jumping web pages, compute the membership between the two, and updating the target text weight in the weighted text model based on the membership, ensuring the accuracy of the recommended content and the satisfaction of user needs.
It realizes more accurate web browsing intention analysis, improves the quality of recommended content, and ensures that recommended content can better meet user needs.
Smart Images

Figure CN118939876A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a method and device for processing web page text, an electronic device, and a storage medium. Background Art
[0002] There are a lot of texts on web pages, some of which have network links. By jumping to the network links of the texts, you can jump to another web page. By collecting the text corresponding to the network links of the jump operation and analyzing the collected text, you can get the web browsing intention. According to the web browsing intention, you can recommend content related to the web browsing intention to achieve customized recommended content services. In practical applications, only relying on analyzing the text corresponding to the network links may have the limitation of insufficient web browsing intention analysis, which in turn leads to the limitation that the recommended content cannot meet user needs. Summary of the invention
[0003] The present disclosure provides a webpage text processing method and device, electronic device and storage medium. Its main purpose is to solve the problem that only relying on analyzing the text corresponding to the network link may have the limitation of insufficient analysis of webpage browsing intention, which leads to the limitation that the recommended content cannot meet the user's needs.
[0004] According to a first aspect of the present disclosure, a method for processing webpage text is provided, which includes:
[0005] Acquire a first text of a network link in a source web page, and acquire a second text in a redirected web page; the redirected web page is a web page to which the network link in the source web page is triggered;
[0006] Calculating the membership degree between the second text and the first text to obtain the membership degree between the first text and the second text;
[0007] If the membership degree is greater than a preset threshold, determining whether there is a first target text identical to the second text in the first weighted text model; the first weighted text model is a model trained based on the historical second text of the jump web page;
[0008] When it is determined that the first target text exists in the first weighted text model, the first weight of the first target text in the first weighted text model is updated to obtain an updated first weight, so as to recommend the first target text based on the updated first weight.
[0009] Optionally, after obtaining the first text of the network link in the source web page, the method further includes:
[0010] Determine whether there is a second target text identical to the first text in a second weighted text model; the second weighted text model is a model trained based on the historical first text of the source web page;
[0011] When it is determined that the second target text exists in the second weighted text model, the second weight of the second target text in the second weighted text model is updated to obtain an updated second weight, so as to recommend the second target text based on the updated second weight.
[0012] Optionally, after calculating the membership degree between the second text and the first text to obtain the membership degree between the first text and the second text, the method further includes:
[0013] When it is determined that the second target text exists in the second weighted text model and the degree of membership is less than or equal to the preset threshold, the second target text is deleted from the second weighted text model.
[0014] Optionally, after determining whether there is a first target text identical to the second text in the first weighted text model, the method further includes:
[0015] When it is determined that the first target text does not exist in the first weighted text model, the second text is stored in the first weighted text model.
[0016] Optionally, after determining whether there is a second target text identical to the first text in the second weighted text model, the method further includes:
[0017] When it is determined that the second target text does not exist in the second weighted text model, the first text is stored in the first weighted text model.
[0018] Optionally, updating the first weight of the first target text in the first weighted text model to obtain the updated first weight includes:
[0019] assigning a first preset weight increment to the first target text;
[0020] Adding the first weight and the first preset weight increment to obtain the updated first weight;
[0021] The updating of the second weight of the second target text in the second weighted text model to obtain the updated second weight includes:
[0022] assigning a second preset weight increment to the second target text;
[0023] The second weight and the second preset weight increment are added together to obtain the updated second weight.
[0024] According to a second aspect of the present disclosure, a webpage text processing device is provided, comprising:
[0025] An acquisition unit is used to acquire a first text of a network link in a source web page, and acquire a second text in a jump web page; the jump web page is a web page to which the network link in the source web page is triggered;
[0026] a calculation unit, configured to calculate the degree of membership between the second text and the first text, so as to obtain the degree of membership between the first text and the second text;
[0027] A determination unit, configured to determine whether there is a first target text identical to the second text in the first weighted text model when the membership degree is greater than a preset threshold; the first weighted text model is a model trained based on the historical second text of the jump web page;
[0028] An updating unit is used to update the first weight of the first target text in the first weighted text model when it is determined that the first target text exists in the first weighted text model, so as to obtain an updated first weight, so as to recommend the first target text based on the updated first weight.
[0029] Optionally, the device further comprises:
[0030] The determination unit is further used to, after obtaining the first text of the network link in the source web page, determine whether there is a second target text identical to the first text in the second weighted text model; the second weighted text model is a model trained based on the historical first text of the source web page;
[0031] The updating unit is also used to, when it is determined that the second target text exists in the second weighted text model, update the second weight of the second target text in the second weighted text model to obtain an updated second weight, so as to recommend the second target text based on the updated second weight.
[0032] Optionally, the device further comprises:
[0033] A deleting unit is used to delete the second target text in the second weighted text model after calculating the membership between the second text and the first text to obtain the membership between the first text and the second text, and when it is determined that the second target text exists in the second weighted text model and the membership is less than or equal to the preset threshold.
[0034] Optionally, the device further comprises:
[0035] A storage unit is used to store the second text in the first weighted text model after determining whether there is a first target text identical to the second text in the first weighted text model and if it is determined that the first target text does not exist in the first weighted text model.
[0036] Optionally, the device further comprises:
[0037] The storage unit is also used for, after determining whether there is a second target text identical to the first text in the second weighted text model, storing the first text in the first weighted text model if it is determined that the second target text does not exist in the second weighted text model.
[0038] Optionally, the updating unit includes:
[0039] An allocating module, configured to allocate a first preset weight increment to the first target text;
[0040] The calculation module is used for adding the first weight and the first preset weight increment to obtain the updated first weight;
[0041] The updating unit further comprises:
[0042] The allocation module is further used to allocate a second preset weight increment to the second target text;
[0043] The calculation module is further used to add the second weight and the second preset weight increment to obtain the updated second weight.
[0044] According to a third aspect of the present disclosure, there is provided an electronic device, including:
[0045] at least one processor; and
[0046] a memory communicatively connected to the at least one processor; wherein,
[0047] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect.
[0048] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the first aspect.
[0049] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method as described in the first aspect above.
[0050] The web page text processing method and device, electronic device and storage medium provided by the present disclosure obtain the first text of the network link in the source web page, and obtain the second text in the jump web page; the jump web page is the web page that is jumped by the network link in the source web page; the membership degree of the second text and the first text is calculated to obtain the membership degree between the first text and the second text; if the membership degree is greater than a preset threshold, it is determined whether there is a first target text that is the same as the second text in the first weighted text model; the first weighted text model is a model obtained by training based on the historical second text of the jump web page; when it is determined that the first target text exists in the first weighted text model, the first weight of the first target text in the first weighted text model is updated to obtain the updated first weight, so as to recommend the target text based on the updated first weight. Compared with the related art, the embodiment of the present disclosure calculates the membership of the first text of the network link in the source web page and the second text in the jump web page to obtain the membership of the second text, and when the membership is greater than a preset threshold, if there is a first target text identical to the second text in the first weighted text model, the first weight of the first target text in the first weighted text model is updated, so that the first target text is recommended based on the updated first weight, thereby realizing the recommendation of the second text in the jump web page by using the first text of the network link in the source web page, so that the recommended content can meet user needs.
[0051] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0053] Figure 1 A flowchart of a method for processing webpage text provided by an embodiment of the present disclosure;
[0054] Figure 2 A schematic flow chart of a method for updating a second weighted text model of a source web page provided by an embodiment of the present disclosure;
[0055] Figure 3 A schematic diagram of an updating process of a weighted text model provided by an embodiment of the present disclosure;
[0056] Figure 4 A schematic diagram of the structure of a document object model provided by an embodiment of the present disclosure;
[0057] Figure 5 A schematic diagram of the structure of a webpage text processing device provided by an embodiment of the present disclosure;
[0058] Figure 6 A schematic diagram of the structure of another device for processing web page text provided by an embodiment of the present disclosure;
[0059] Figure 7 A schematic block diagram of an exemplary electronic device provided for an embodiment of the present disclosure. DETAILED DESCRIPTION
[0060] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0061] The following describes a method and device for processing web page text, an electronic device, and a storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.
[0062] Figure 1 A flowchart of a method for processing web page text provided by an embodiment of the present disclosure.
[0063] like Figure 1 As shown, the method is applied in a server, and the method comprises the following steps:
[0064] Step 101, obtaining a first text of a network link in a source web page, and obtaining a second text in a jump web page; the jump web page is a web page to which the network link in the source web page is triggered.
[0065] The source web page is the web page where the web link is located; the web link is the link in the source web page that points to other web pages or resources. The web link is usually displayed as clickable text or image, which will jump to another web page after clicking; the first text is the visible text part of the web link in the source web page, that is, the clickable text content seen in the source web page; the jump web page refers to the web page that jumps to by clicking the web link in the source web page, and the jump web page usually contains content that is related to the first text; the second text is the text content contained in the jump web page.
[0066] The code of the web link is extracted from the source web page by a web crawler or a related web crawling tool, and the code of the web link includes the title tag of the web link, that is, the first text; the method for obtaining the second text is similar to the method for obtaining the first text, but it should be clear that this statement is not intended to limit the method for obtaining the first text and the second text to only be implemented by the above method, and the first text and the second text can also be obtained by other methods.
[0067] By obtaining the first text of the network link in the source web page and the second text in the jump web page, the correlation between the first text and the second text can be understood. The correlation between the first text and the second text can be used to construct a knowledge graph, perform text classification, recommendation system or advertising positioning, etc.
[0068] Step 102: Calculate the membership degree between the second text and the first text to obtain the membership degree between the first text and the second text.
[0069] The membership degree is the similarity or correlation between the first text and the second text. Various text similarity calculation methods (such as cosine similarity, edit distance, etc.) can be used to quantify the membership degree between the two texts. The membership degree is usually between 0 and 1, indicating the degree of association between the second text and the first text. The closer the membership degree is to 1, the more similar or related the two texts are, and the closer the membership degree is to 0, the less similar or related the two texts are.
[0070] Membership can help determine the degree of association between two texts, thereby better understanding the text content, matching related information, or making recommendations.
[0071] Step 103, if the membership degree is greater than a preset threshold, determine whether there is a first target text identical to the second text in the first weighted text model; the first weighted text model is a model trained based on the historical second text of the jump web page.
[0072] The weighted text model is a model built based on the historical texts on web pages. The weighted text model assigns different weights to different historical texts on web pages. Each historical text in the weighted text model has a corresponding weight, and the weight reflects the importance or influence of the historical text in the weighted text model. The higher the weight of the historical text, the more important the historical text is and the easier it is to be recommended to users. Each web page has a corresponding weighted text model.
[0073] Since there are a large number of second texts in the jump web page, some of the second texts have a high correlation with the first text, and some of the second texts have a low correlation with the first text. The historical second texts in the first weighted text model corresponding to the jump web page are all texts with a high correlation with the first text. If the degree of membership is greater than the preset threshold, it means that the correlation between the first text and the second text is high, and it is necessary to determine whether there is a first target text that is the same as the second text in the first weighted text model, where the preset threshold can be an arbitrary value and is not limited in the embodiments of the present disclosure.
[0074] Determining whether the same first target text exists can help avoid repeated processing and storage of identical or highly similar texts, which can not only save computing resources but also improve processing efficiency.
[0075] Step 104, when it is determined that the first target text exists in the first weighted text model, the first weight of the first target text in the first weighted text model is updated to obtain an updated first weight, so as to recommend the first target text based on the updated first weight.
[0076] When the first target text exists in the first weighted text model, it means that the first target text is used frequently and the first weight of the first target text in the first weighted text model needs to be updated. The updating method can use the first target text once and increase the preset weight increment for the first weight once. The specific value of the preset weight increment is not limited in the embodiment of the present disclosure.
[0077] According to the existence of the first target text that is the same as the second text in the first weighted text model, the first weight of the first target text in the first weighted text model is dynamically adjusted to make the recommendation results more in line with the user's actual needs and interests. By continuously updating the weights, the recommendation performance can be continuously optimized to improve user experience and satisfaction.
[0078] To facilitate understanding, an example is provided. Suppose that when the source web page jumps to the jump web page for the first time, there are three texts on the jump web page, namely a, b, and c. Since it is the first time to open the jump web page, there is no historical text on the jump web page. It is necessary to build a weighted text model for the jump web page, store a, b, and c in the weighted text model, and assign the same weight A to a, b, and c respectively. When the jump web page is opened for the second time, it is assumed that there are a, d, and e on the jump web page, among which the frequency of occurrence of a increases once, and the weight corresponding to a in the weighted text model needs to be updated. Assuming the preset weight increment is B, the updated weight of a is A+B.
[0079] The present disclosure provides a method for processing web page text, which includes obtaining a first text of a network link in a source web page, and obtaining a second text in a jump web page; the jump web page is a web page to which the network link in the source web page is triggered; calculating the membership between the second text and the first text to obtain the membership between the first text and the second text; if the membership is greater than a preset threshold, determining whether there is a first target text identical to the second text in a first weighted text model; the first weighted text model is a model obtained by training based on the historical second text of the jump web page; when it is determined that the first target text exists in the first weighted text model, updating the first weight of the first target text in the first weighted text model to obtain an updated first weight, so as to recommend the target text based on the updated first weight. Compared with the related art, the embodiment of the present disclosure calculates the membership of the first text of the network link in the source web page and the second text in the jump web page to obtain the membership of the second text, and when the membership is greater than a preset threshold, if there is a first target text identical to the second text in the first weighted text model, the first weight of the first target text in the first weighted text model is updated, so that the first target text is recommended based on the updated first weight, thereby realizing the recommendation of the second text in the jump web page by using the first text of the network link in the source web page, so that the recommended content can meet user needs.
[0080] In practical applications, each web page has a corresponding weighted text model. After obtaining the first text of the network link in the source web page, in order to improve the accuracy of the recommendation degree of the first text of the source web page, it is necessary to update the second weighted text model of the source web page, which can be achieved by but not limited to the following methods: Figure 2 As shown, Figure 2 A flow chart of a method for updating a second weighted text model of a source web page provided by an embodiment of the present disclosure includes:
[0081] Step 201, determining whether there is a second target text identical to the first text in a second weighted text model; the second weighted text model is a model trained based on the historical first text of the source web page.
[0082] The second weighted text model is a model trained based on the historical first text of the source web page. The first text may be a newly appeared text in the source web page, or it may be a historical text in the source web page. By determining whether there is a second target text that is the same as the first text in the second weighted text model, it can be determined whether the first text is a newly appeared text or a historical text.
[0083] By determining whether there is a second target text identical to the first text in the second weighted text model, repeated processing and calculation of the same or similar texts can be avoided, computing resources can be saved, and processing efficiency and performance can be improved, enabling it to respond to user requests more quickly and provide accurate recommendations or search results.
[0084] Step 202, when it is determined that the second target text exists in the second weighted text model, the second weight of the second target text in the second weighted text model is updated to obtain an updated second weight, so as to recommend the second target text based on the updated second weight.
[0085] When the second target text exists in the second weighted text model, it means that the second target text is used frequently and the second weight of the second target text in the second weighted text model needs to be updated. The updating method can use the second target text once and increase the preset weight increment for the second weight.
[0086] According to the existence of a second target text that is the same as the first text in the second weighted text model, the second weight of the second target text in the second weighted text model is dynamically adjusted to make the recommendation results more in line with the user's actual needs and interests. By continuously updating the weights, the recommendation performance can be continuously optimized to improve user experience and satisfaction.
[0087] In practical applications, after calculating the membership degree between the second text and the first text and obtaining the membership degree between the first text and the second text, the content in the jump web page jumped by the network link corresponding to the second target text in the second weighted text model may have a low correlation with the second target text, and the second target text needs to be deleted from the second weighted text model. This can be achieved by but is not limited to the following method: when it is determined that the second target text exists in the second weighted text model and the membership degree is less than or equal to the preset threshold, the second target text is deleted from the second weighted text model; deleting the second target text can prevent the second weighted text model from being interfered by low-quality or irrelevant information, thereby improving the accuracy and reliability of the content recommended by the second weighted text model.
[0088] To facilitate understanding, an example is provided. Suppose that the first text is f, and there is g that is the same as a in the second weighted text model. The second text of the jump web page corresponding to the network link of f is h. The membership degree between f and h is 0.4, and the preset threshold is 0.5. Since 0.4 is less than 0.5, g needs to be deleted in the second weighted text model.
[0089] In actual applications, after determining whether there is a first target text identical to the second text in the first weighted text model, the second text may be a newly added text in the jump web page. The first weighted text model does not have a first target text identical to the second text, and the second text needs to be stored in the first weighted text model. This can be achieved by, but is not limited to, the following method: when it is determined that the first target text does not exist in the first weighted text model, the second text is stored in the first weighted text model, and a preset weight is assigned to the second text, and the preset weight can be an arbitrary value; the first weighted text model is updated by using the newly added text in the jump web page, thereby enriching the content of the text of the first weighted text model.
[0090] In actual applications, after determining whether there is a second target text identical to the first text in the second weighted text model, the first text may be a newly added text in the source web page, and the second weighted text model does not have a second target text identical to the first text. It is necessary to store the first text in the second weighted text model, which can be achieved by but not limited to the following methods: when it is determined that the second target text does not exist in the second weighted text model, the first text is stored in the first weighted text model, and a preset weight is assigned to the first text, and the preset weight can be an arbitrary value; the second weighted text model is updated by using the newly added text in the source web page, thereby enriching the text content of the second weighted text model.
[0091] As a refinement of step 104, when executing the updating of the first weight of the first target text in the first weighted text model to obtain the updated first weight, it can be implemented in but not limited to the following manner: assigning a first preset weight increment to the first target text; adding the first weight and the first preset weight increment to obtain the updated first weight; the first preset weight increment can be any value, which is not limited by the embodiment of the present disclosure; the purpose of assigning the first preset weight increment to the first target text and adding the first weight and the first preset weight increment is to personalize the adjustment, dynamically update the weight, optimize the model performance and improve the user experience, which helps to improve the effect of content recommendation and user satisfaction.
[0092] For ease of understanding, an example is provided. Assuming that the first weight of the first target text is 5 and the first preset weight increment is 1, the updated first weight is 6.
[0093] As a refinement of the above embodiment, when executing the updating of the second weight of the second target text in the second weighted text model to obtain the updated second weight, it can be implemented in but not limited to the following manner: assigning a second preset weight increment to the second target text; adding the second weight and the second preset weight increment to obtain the updated second weight; the second preset weight increment can be any value, which is not limited to the embodiment of the present disclosure, and the first preset weight increment and the second preset weight increment can be the same or different; the purpose of assigning the second preset weight increment to the second target text and adding the second weight and the second preset weight increment is to personalize the adjustment, dynamically update the weight, optimize the model performance and improve the user experience, which helps to improve the effect of content recommendation and user satisfaction.
[0094] In one possible implementation of the embodiment of the present disclosure, in order to better understand the updating process of the weighted text model, as follows: Figure 3 As shown, Figure 3 The following is a schematic diagram of an update process of a weighted text model provided in an embodiment of the present disclosure. The Document Object Model (DOM) is a standard programming interface for processing extensible markup language recommended by the World Wide Web Consortium (W3C). It provides access to data and structures in a document. The basic nodes of the DOM tree include document nodes, element nodes, attribute nodes, text nodes, etc. In order to better understand the DOM tree, Figure 4 As shown, Figure 4A schematic diagram of the structure of a document object model provided by an embodiment of the present disclosure, by converting a web page document into a DOM tree structure, selecting each element node in the DOM tree and its child nodes, filter the selected nodes, keep only the child nodes of text type, and finally get the element node All the text content in the A page is extracted to form independent and complete paragraphs, and then the paragraphs are combined into segmented web page text content. 1, p2……p n )
[0095] The segmented text content acquisition algorithm can effectively obtain the text content in the element node, avoid the interference of the element type or attribute type subnode in the element node, and finally obtain a relatively complete paragraph text content with high versatility and accuracy. Of course, in addition to the main text, the obtained web page text may also contain some irrelevant content. This requires further optimization of these text contents.
[0096] The Fuzzy C-Means (FCM) algorithm is used to cluster the first text corresponding to the source web page's network link as the cluster center p. x , extract the second text (ξ1ξ2……ξ n ) is clustered, and a second weighted text model for jumping to the web page is constructed according to the clustering results.
[0097] FCM is a fuzzy clustering method based on the objective function, which provides more flexible clustering results. Compared with the original FCM fuzzy clustering algorithm, it is necessary to gradually determine the cluster center through iterative optimization. x As the cluster center, for the known cluster center p x Calculate the second text (ξ1ξ2……ξ n )'s membership degree (μ1μ2……μ n ), and set the preset threshold T, set Ω x For p x The cluster centered on Sometimes, there is i ∈Ω x , indicating the content i For p x The clustering of content ξ i After weighted judgment, it can be classified into the first weighted text model.
[0098] When users browse web pages, they trigger the jump of network links through operations, which indicates that there is a connection between the web pages before and after the jump. More specifically, the second text of the jump web page should be consistent with the first text corresponding to the network link of the source web page to meet the user's browsing intention. Therefore, after converting web page documents into DOM tree structures by using Web information extraction technology based on DOM tree structures, it is not only convenient to extract the page content, but also can simultaneously extract the content in the DOM structure. Tags to monitor.
[0099] Obtain the first text p corresponding to the network link of the source web page by monitoring x , when p x If the first text p corresponding to the network link of the source web page does not exist in the second weighted text model of the source web page, the first text p corresponding to the network link of the source web page is added to the second weighted text model. x When p x When it already exists in the second weighted text model, x The corresponding weight is increased. Thus, a second weighted text model is constructed, which reflects the usage frequency of the first text through weighting, making the constructed model more practical and convenient for subsequent application in search recommendation.
[0100] The first text corresponding to the network link of the source web page is the cluster center p x , cluster the second text in the jump web page, and determine the corresponding content ξ by setting the membership threshold i For That is, all the second texts in the jump web page, the first text p corresponding to the network link of the source web page x They do not belong to the same category. This indicates that there is a certain degree of uncertainty in the operation content of the previous page, and the correlation between it and the corresponding jump page is weak, which does not rule out the possibility of certain risks. Therefore, if the clustering results of the jump web pages are inconsistent with the first text corresponding to the network link of the source web page, they are removed from the second weighted text model of the source web page, so that the constructed second weighted text model is more stable and secure.
[0101] In summary, the embodiments of the present disclosure can achieve the following effects:
[0102] 1. The disclosed embodiment calculates the membership of the first text of the network link in the source web page and the second text in the jump web page to obtain the membership of the second text, and when the membership is greater than a preset threshold, if there is a first target text identical to the second text in the first weighted text model, the first weight of the first target text in the first weighted text model is updated so that the first target text is recommended based on the updated first weight, thereby realizing the recommendation of the second text in the jump web page by using the first text of the network link in the source web page, so that the recommended content can meet the needs of users.
[0103] 2. The disclosed embodiment can utilize the correlation between the previous and next pages browsed by the user during the page browsing process to construct a second weighted text model based on the source web page. Compared with the existing web page content extraction technology, the method of dividing the main content of the page through structural or semantic analysis of the large amount of extracted page content, the present application clusters the second text in the jump web page with the first text corresponding to the network link in the source web page as the clustering center, realizes the effective extraction of the key content of the web page, and can provide more accurate search push and content recommendation for the entire platform.
[0104] 3. The weighted text model constructed in the embodiment of the present disclosure is mainly similar data obtained by clustering analysis of the first text corresponding to the network link in the source web page and the second text in the jump web page. Compared with the existing web content extraction technology that mainly focuses on the single-page model, this solution is more in line with the actual user browsing, and the extracted text has more application value. At the same time, compared with the single key content extraction of the prior art, this solution also introduces a data statistics link, that is, obtaining the first text corresponding to the network link in the source web page and weighting it in the weighted text model according to the operation frequency, so that while obtaining the weighted text model, it also realizes the attention statistics of page operations.
[0105] 4. In the process of constructing a weighted text model based on the first text clustering of network links in the source web page, the embodiment of the present disclosure takes into account the correlation between the previous and next pages during the user's browsing process, and introduces a reverse verification link for the operation content of the previous page, that is, according to the clustering analysis results of the current page, reversely eliminate the invalid or irrelevant operation content in the previous page, which can further avoid risky operations in the page. For existing page monitoring operations, the embodiment of the present disclosure adds a risk judgment link for the accuracy and compliance of page operations on the basis of monitoring and collecting statistical data.
[0106] Corresponding to the above-mentioned webpage text processing method, the present invention also provides a webpage text processing device. Since the device embodiment of the present invention corresponds to the above-mentioned method embodiment, the details not disclosed in the device embodiment can be referred to the above-mentioned method embodiment, and will not be repeated in the present invention.
[0107] Figure 5 A schematic diagram of a web page text processing device provided in an embodiment of the present disclosure, wherein the device is applied to a server, such as Figure 5 As shown, including:
[0108] The acquisition unit 31 is used to acquire the first text of the network link in the source web page, and acquire the second text in the jump web page; the jump web page is the web page that is jumped to by triggering the network link in the source web page;
[0109] A calculation unit 32, configured to calculate the degree of membership between the second text and the first text, to obtain the degree of membership between the first text and the second text;
[0110] A determination unit 33, configured to determine whether there is a first target text identical to the second text in the first weighted text model when the membership degree is greater than a preset threshold; the first weighted text model is a model trained based on the historical second text of the jump web page;
[0111] The updating unit 34 is used to update the first weight of the first target text in the first weighted text model when it is determined that the first target text exists in the first weighted text model, so as to obtain an updated first weight, so as to recommend the first target text based on the updated first weight.
[0112] The web page text processing device provided by the present disclosure obtains the first text of the network link in the source web page, and obtains the second text in the jump web page; the jump web page is the web page that is jumped by the network link in the source web page; the membership degree of the second text and the first text is calculated to obtain the membership degree between the first text and the second text; if the membership degree is greater than a preset threshold, it is determined whether there is a first target text that is the same as the second text in the first weighted text model; the first weighted text model is a model obtained by training based on the historical second text of the jump web page; when it is determined that the first target text exists in the first weighted text model, the first weight of the first target text in the first weighted text model is updated to obtain the updated first weight, so as to recommend the target text based on the updated first weight. Compared with the related art, the embodiment of the present disclosure calculates the membership of the first text of the network link in the source web page and the second text in the jump web page to obtain the membership of the second text, and when the membership is greater than a preset threshold, if there is a first target text identical to the second text in the first weighted text model, the first weight of the first target text in the first weighted text model is updated, so that the first target text is recommended based on the updated first weight, thereby realizing the recommendation of the second text in the jump web page by using the first text of the network link in the source web page, so that the recommended content can meet user needs.
[0113] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 6 As shown, the device also includes:
[0114] The determination unit 33 is also used to, after obtaining the first text of the network link in the source web page, determine whether there is a second target text identical to the first text in the second weighted text model; the second weighted text model is a model trained based on the historical first text of the source web page;
[0115] The updating unit 34 is also used to, when it is determined that the second target text exists in the second weighted text model, update the second weight of the second target text in the second weighted text model to obtain an updated second weight, so as to recommend the second target text based on the updated second weight.
[0116] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 6 As shown, the device also includes:
[0117] The deleting unit 35 is used to delete the second target text in the second weighted text model after calculating the membership between the second text and the first text to obtain the membership between the first text and the second text, and when it is determined that the second target text exists in the second weighted text model and the membership is less than or equal to the preset threshold.
[0118] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 6 As shown, the device also includes:
[0119] The storage unit 36 is used to store the second text in the first weighted text model after determining whether there is a first target text identical to the second text in the first weighted text model and if it is determined that the first target text does not exist in the first weighted text model.
[0120] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 6 As shown, the device also includes:
[0121] The storage unit 36 is further configured to, after determining whether there is a second target text identical to the first text in the second weighted text model, store the first text in the first weighted text model if it is determined that the second target text does not exist in the second weighted text model.
[0122] Furthermore, in a possible implementation of the embodiment of the present disclosure, as Figure 6 As shown, the updating unit 34 includes:
[0123] An allocating module 341 is used to allocate a first preset weight increment to the first target text;
[0124] The calculation module 342 is used to add the first weight and the first preset weight increment to obtain the updated first weight;
[0125] The updating unit 34 further includes:
[0126] The allocation module 341 is further used to allocate a second preset weight increment to the second target text;
[0127] The calculation module 342 is further configured to add the second weight and the second preset weight increment to obtain the updated second weight.
[0128] It should be noted that the above explanation of the method embodiment is also applicable to the device of the embodiment of the present disclosure, and the principle is the same, which is no longer limited in the embodiment of the present disclosure.
[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0130] Figure 7 A schematic block diagram of an example electronic device 400 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0131] like Figure 7 As shown, the device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 402 or a computer program loaded from a storage unit 408 to a RAM (Random Access Memory) 403. In the RAM 403, various programs and data required for the operation of the device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An I / O (Input / Output) interface 405 is also connected to the bus 404.
[0132] A number of components in the device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, a mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, an optical disk, etc.; and a communication unit 409, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 409 allows the device 400 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0133] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as a method for processing web page text. For example, in some embodiments, the method for processing web page text may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 400 via the ROM 402 and / or the communication unit 409. When the computer program is loaded into the RAM 403 and executed by the computing unit 401, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 401 may be configured to execute the aforementioned web page text processing method in any other appropriate manner (for example, by means of firmware).
[0134] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0135] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0136] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an EPROM (Electrically Programmable Read-Only-Memory) or a flash memory, an optical fiber, a CD-ROM (Compact Dis sc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0138] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.
[0139] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.
[0140] It should be noted that artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0141] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0142] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for processing webpage text, characterized in that: include: Obtaining the first text of the network link in the source web page, and obtaining the second text in the redirected web page; The jump webpage is a webpage to which the network link in the source webpage is triggered; Calculating the membership degree between the second text and the first text to obtain the membership degree between the first text and the second text; If the membership degree is greater than a preset threshold, determining whether there is a first target text identical to the second text in the first weighted text model; The first weighted text model is a model trained based on the historical second text of the redirected web page; When it is determined that the first target text exists in the first weighted text model, the first weight of the first target text in the first weighted text model is updated to obtain an updated first weight, so as to recommend the first target text based on the updated first weight.
2. The method according to claim 1, characterized in that After obtaining the first text of the network link in the source web page, the method further includes: Determine whether there is a second target text identical to the first text in a second weighted text model; the second weighted text model is a model trained based on the historical first text of the source web page; When it is determined that the second target text exists in the second weighted text model, the second weight of the second target text in the second weighted text model is updated to obtain an updated second weight, so as to recommend the second target text based on the updated second weight.
3. The method according to claim 2, characterized in that After calculating the membership degree between the second text and the first text to obtain the membership degree between the first text and the second text, the method further includes: When it is determined that the second target text exists in the second weighted text model and the degree of membership is less than or equal to the preset threshold, the second target text is deleted from the second weighted text model.
4. The method according to claim 1, characterized in that: After determining whether there is a first target text identical to the second text in the first weighted text model, the method further includes: When it is determined that the first target text does not exist in the first weighted text model, the second text is stored in the first weighted text model.
5. The method according to claim 2, characterized in that: After determining whether there is a second target text identical to the first text in the second weighted text model, the method further includes: When it is determined that the second target text does not exist in the second weighted text model, the first text is stored in the first weighted text model.
6. The method according to claim 2, characterized in that The updating of the first weight of the first target text in the first weighted text model to obtain the updated first weight includes: assigning a first preset weight increment to the first target text; Adding the first weight and the first preset weight increment to obtain the updated first weight; The updating of the second weight of the second target text in the second weighted text model to obtain the updated second weight includes: assigning a second preset weight increment to the second target text; The second weight and the second preset weight increment are added together to obtain the updated second weight.
7. A web page text processing device, characterized in that: include: An acquisition unit, used for acquiring a first text of a network link in a source web page, and acquiring a second text in a redirected web page; The jump webpage is a webpage to which the network link in the source webpage is triggered; a calculation unit, configured to calculate the degree of membership between the second text and the first text, so as to obtain the degree of membership between the first text and the second text; A determination unit, configured to determine whether there is a first target text identical to the second text in the first weighted text model when the degree of membership is greater than a preset threshold; The first weighted text model is a model trained based on the historical second text of the redirected web page; An updating unit is used to update the first weight of the first target text in the first weighted text model when it is determined that the first target text exists in the first weighted text model, so as to obtain an updated first weight, so as to recommend the first target text based on the updated first weight.
8. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.
10. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Citation Information
Patent Citations
Method for identifying re-loading relation between internet news texts
CN102651012A
User interest model incremental update method of personalized recommendation system
CN103488705A
Article recommendation method and device, computer equipment and storage medium
CN110955831A
Content recommendation method and device, equipment, medium and computer program product
CN114297470A
Webpage similarity model training method and device, electronic equipment and medium
CN114742168A
