Web page similarity model training method, device, electronic device and medium

Abstract

This disclosure provides a method, apparatus, electronic device, and medium for training a webpage similarity model, relating to the fields of computer technology, particularly natural language processing and deep learning. A specific implementation scheme comprises: obtaining webpage data for a sample webpage; extracting body sentences from the webpage data using a no-playback sampling strategy to obtain sentence data; obtaining sample data based on the sentence data and webpage data; and training a pre-trained language model based on the sample data to obtain the webpage similarity model, which is used to predict the similarity of a target webpage.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly to technical fields such as natural language processing and deep learning. Background Art

[0002] With the development of Internet technology, how to find the required similar pages from massive web pages and judge the similarity of the content described between web pages has become an important research direction.

[0003] At present, web page similarity analysis technology generally extracts features from web page texts and interacts with representations, and then learns the content of web pages to analyze web pages with similar content. Summary of the Invention

[0004] The present disclosure provides a method, device, electronic device, and readable storage medium for training a web page similarity model.

[0005] According to one aspect of the present disclosure, a method for training a web page similarity model is provided, comprising:

[0006] Obtain web page data of a sample web page;

[0007] Extracting text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data;

[0008] Obtaining sample data based on the sentence data and web page data;

[0009] The pre-trained language model is trained according to the sample data to obtain the web page similarity model, which is used to predict the similarity of the web page to be tested.

[0010] According to another aspect of the present disclosure, another method for predicting web page similarity is provided, comprising:

[0011] Get the web page data of any two web pages to be tested;

[0012] Extracting text sentences from the webpage data of any two webpages to be tested according to a no-playback sampling strategy to obtain sentence data;

[0013] A prediction result of the web page similarity is obtained based on the web page data of the arbitrary two web pages to be tested and the sentence data.

[0014] According to another aspect of the present disclosure, a device for training a web page similarity model is provided, comprising:

[0015] An acquisition unit, configured to acquire webpage data of a sample webpage;

[0016] an extraction unit, configured to extract text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data;

[0017] an obtaining unit, configured to obtain sample data based on the sentence data and the web page data;

[0018] The training unit is used to train the pre-trained language model according to the sample data to obtain the web page similarity model for predicting the similarity of the web page to be tested.

[0019] According to another aspect of the present disclosure, another apparatus for predicting web page similarity is provided, comprising:

[0020] An acquisition unit, used to acquire web page data of any two web pages to be tested;

[0021] an extraction unit, configured to extract text sentences from the webpage data of the arbitrary two webpages to be tested respectively according to a no-playback sampling strategy to obtain sentence data;

[0022] The obtaining unit is configured to obtain a prediction result of the web page similarity based on the web page data of the arbitrary two web pages to be tested and the sentence data.

[0023] According to another aspect of the present disclosure, there is provided an electronic device, including:

[0024] at least one processor; and

[0025] a memory communicatively connected to the at least one processor; wherein,

[0026] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any possible implementation manner and the aspects described above.

[0027] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method of the above-mentioned aspect and any possible implementation manner.

[0028] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method of the aspect and any possible implementation manner as described above.

[0029] It can be seen from the above technical solution that, on the one hand, the embodiment of the present disclosure obtains web page data of a sample web page, and then can extract text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data, and then obtain sample data based on the sentence data and the web page data, and then can train the pre-trained language model according to the sample data to obtain the web page similarity model, which is used to predict the similarity of the web page to be tested. Since the pre-trained language model is trained according to the training sample data obtained based on the no-playback sampling strategy to obtain the web page similarity model, the model training can be performed according to sample data with more comprehensive and rich web page text information, which can make it easier for the model to learn the global text information of the web page, reduce the information loss of learning, and thus enhance the robustness of the model prediction.

[0030] It can be seen from the above technical solution that, on the other hand, the embodiment of the present disclosure obtains the web page data of any two web pages to be tested, and then can extract the text sentences from the web page data of the any two web pages to be tested according to the no-playback sampling strategy to obtain sentence data, so that the prediction results of web page similarity can be obtained based on the web page data of the any two web pages to be tested and the sentence data. Since the relevant data of the web pages used for web page similarity prediction is enriched, the completeness of the web page information used for prediction is enhanced, thereby improving the reliability of the prediction results of web page similarity.

[0031] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0033] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0034] Figure 2 is a schematic diagram of a process of obtaining sample data according to the first embodiment of the present disclosure;

[0035] Figure 3 is a schematic diagram of unsupervised model training according to the first embodiment of the present disclosure;

[0036] Figure 4 is a schematic diagram of supervised model training according to the first embodiment of the present disclosure;

[0037] Figure 5 is a schematic diagram according to a second embodiment of the present disclosure;

[0038] Figure 6is a schematic diagram of similarity prediction of news web pages according to the second embodiment of the present disclosure;

[0039] Figure 7 is a schematic diagram according to a third embodiment of the present disclosure;

[0040] Figure 8 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0041] Figure 9 The block diagram is a block diagram of an electronic device for implementing the web page similarity model training method or web page similarity prediction method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0043] Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0044] It should be noted that the terminal devices involved in the embodiments of the present disclosure may include but are not limited to mobile phones, personal digital assistants (PDAs), wireless handheld devices, tablet computers and other smart devices; display devices may include but are not limited to personal computers, televisions and other devices with display functions.

[0045] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0046] At present, most of the related web page similarity analysis technologies match short texts with short texts or short texts with long texts. A general pre-trained encoder is trained based on contrastive learning, and then the pre-trained encoder is used to encode the texts separately to calculate the web page similarity.

[0047] However, in the solutions of the related art, because the sentences in the chapters (articles) of the web pages are too long, the web page related data for prediction is generally obtained by truncating the sentences or selecting the main sentences. Therefore, the solutions of the related art are difficult to effectively migrate to the similarity prediction tasks directly targeting the chapter-level web pages.

[0048] For example, when training a webpage similarity model based on a bidirectional encoder representation from transformers (BERT), the most similar paragraphs are selected from the webpages and used as input text for model training. However, because the model still needs to use the most similar paragraphs obtained based on similarity ranking for model training, it is difficult to effectively predict the similarity of the entire text of two webpages under test.

[0049] For example, when training a webpage similarity model based on SimCSE, the length limit of the SimCSE pre-trained model makes it difficult to train the model on long texts and passages. Therefore, the text is typically truncated to the maximum length to obtain input text for model training. However, this method easily loses learning information, resulting in the trained model being unable to effectively learn the global information of the webpage.

[0050] Therefore, there is an urgent need to provide a training method for a web page similarity model that can enable the model to learn the global text information of the web page, reduce the information loss of learning, and thus enhance the robustness of the model prediction.

[0051] Figure 1 is a schematic diagram according to the first embodiment of the present disclosure, as shown in Figure 1 shown.

[0052] 101. Obtain web page data of a sample web page.

[0053] 102. Extract text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data.

[0054] 103. Obtain sample data based on the sentence data and web page data.

[0055] 104. Train the pre-trained language model based on the sample data to obtain the web page similarity model for predicting the similarity of the web page to be tested.

[0056] It should be noted that the web page data of the sample web page may include but is not limited to a title, keywords, and a segmented article.

[0057] Optionally, the sample web page may be pre-processed to obtain web page data of the sample web page.

[0058] Specifically, first, the title, keywords, and article of the sample web page can be extracted. Then, the article is segmented into sentences, and the order of each sentence in the article can be obtained according to the order of the sentences in the article, that is, S1, S2, S3, ..., S n And other sentences.

[0059] It should be noted that the sample web pages may include chapter-level web pages such as news web pages.

[0060] It should be noted that the no-replay sampling strategy means that when sample individuals are drawn one by one from the sample population, the sample individuals drawn each time are not put back into the sample population to participate in the next drawing.

[0061] It should be noted that part or all of the execution entities of 101 to 104 may be applications located in the local terminal, or may also be functional units such as plug-ins or software development kits (SDKs) set in the applications located in the local terminal, or may also be processing engines located in the network-side server, or may also be distributed systems located on the network side, for example, a processing engine or distributed system in a web page analysis platform on the network side, etc. This embodiment does not specifically limit this.

[0062] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0063] In this way, by obtaining the web page data of the sample web page, the body sentences can be extracted from the web page data according to the no-playback sampling strategy to obtain sentence data, and then the sample data can be obtained according to the sentence data and the web page data, and then the pre-trained language model can be trained according to the sample data to obtain the web page similarity model, which is used to predict the similarity of the web page to be tested. Since the pre-trained language model is trained according to the training sample data obtained based on the no-playback sampling strategy to obtain the web page similarity model, the model training can be carried out according to the sample data with more comprehensive and rich web page body information, which can make it easier for the model to learn the global text information of the web page, reduce the information loss of learning, and thus enhance the robustness of the model prediction.

[0064] Optionally, in a possible implementation of this embodiment, in 102, sentence extraction processing can be performed on the text in the web page data based on the no-playback sampling strategy and the input threshold of the pre-trained language model to obtain the text sentences, and then the text sentences can be sorted according to the sequence identification of the text sentences, so that the sentence data can be obtained based on the result of the sorting processing.

[0065] In this implementation, the no-playback sampling strategy may include but is not limited to no-playback sampling based on uniform distribution and no-playback sampling based on sentence weights, wherein the sentence weights may be determined according to the length of the sentence.

[0066] In this implementation, the pre-trained language model can be a model pre-trained based on a large corpus. In natural language processing tasks, using a pre-trained language model can avoid training a new model from scratch.

[0067] Specifically, the pre-trained language model may include a pre-trained multilingual model, such as ERNIE-M. In this way, multilingual synchronous learning is performed on web pages in different languages, enhancing the learning ability of web pages in languages ​​with less training resources (such as Icelandic).

[0068] In a specific implementation process of this implementation method, sentence extraction processing can be performed on the text in the web page data according to the no-playback sampling strategy to obtain the text sentences, and then if the total length of the extracted text sentences reaches the input threshold of the pre-trained language model, the sentence extraction processing is terminated.

[0069] In this specific implementation process, the input threshold of the pre-trained language model may include a length threshold of the text sequence input to the model, that is, the input length. The input length of the pre-trained language model may be a fixed length, such as 512 text characters (Tokens).

[0070] In one case of this specific implementation process, for any sample web page, multiple text sentences of the body can be extracted from the web page data. When the total length of the multiple text sentences reaches the input threshold of the pre-trained language model, the sentence extraction process is terminated.

[0071] In this way, by extracting main text sentences from web page data based on the no-replacement sampling strategy and the input threshold of the pre-trained language model, direct truncation of the main text can be avoided, that is, sentence data that is more comprehensive for characterizing the global information of the web page main text can be obtained, so that the model can be trained using sample data determined based on the sentence data in the subsequent process, so that the model can learn more comprehensive and complete main text feature information of the web page.

[0072] In another case of this specific implementation process, for any sample web page, after terminating the sentence extraction process, that is, terminating the extraction of the sentences in the text, the sentences can be sorted according to the sequence identifiers of the sentences in the text, so that the sentence data can be obtained according to the sorting results.

[0073] It is understandable that for any sample web page, the operations of extracting text sentences and sorting the extracted text sentences may be performed multiple times to obtain multiple groups of sentence data.

[0074] Thus, in this implementation, sentences can be extracted from the main text of the web page data using a no-replay sampling strategy and the input threshold of the pre-trained language model to obtain the main text sentences. The sentences are then sorted according to the order identifiers of the main text sentences to obtain sentence data. This allows for more comprehensive sentence data that represents the global information of the web page main text, making it easier to subsequently use this sentence data for model training, thereby enabling the model to learn more comprehensive feature information about the main text of the web page.

[0075] Furthermore, the extracted sentences are sorted according to their order of appearance in the text, generating sorted sentence data. This ensures that the obtained sentence data reflects the original meaning of the text as closely as possible, improving the validity of the sentence data and, in turn, the validity of subsequent sample data. This allows the model to learn more comprehensive text feature information about the webpage when using this sample data for model training, further enhancing the robustness of the model's predictions.

[0076] Optionally, in a possible implementation of this embodiment, in 103, the title data and keyword data in the web page data may be obtained, and then sample data may be obtained based on the sentence data, title and keyword.

[0077] In a specific implementation process of this implementation method, for any sample web page, the sentence data obtained by step 102 and the title and keywords obtained by pre-processing can be used as sample data.

[0078] Figure 2 is a schematic diagram of the process of obtaining sample data according to the first embodiment of the present disclosure, such as Figure 2 As shown, specifically, for any sample web page, the title, keywords, and sentence data can be separated using a separator [SEP] to serve as the sample data of the web page.

[0079] Specifically, for any sample web page, the body sentences can be extracted from the web page data according to the no-playback sampling strategy, that is, from S1, S2, S3, ..., Sn Randomly extract sentences from the 10 ,S2,…,S5, then, according to the order of the sentences in the text, 10 ,S2,…,S5 are sorted to obtain S2,S5,…,S 10 S2,S5,…,S 10 It can be used as sentence data for the sample web page.

[0080] In another specific implementation of this method, for any sample web page, multiple operations of extracting body sentences and sorting the extracted sentences can be performed to obtain multiple sets of sentence data. Based on the title, keywords, and multiple sets of sentence data of the web page, multiple sets of sample data of the web page can be obtained.

[0081] In this way, in this implementation method, sample data can be obtained by obtaining sentence data based on the title, keywords and sampling of the web page, which can enhance the sample data and enrich the sample data for model training. It can facilitate the model to learn the global text information of the web page, thereby further enhancing the robustness of the model prediction.

[0082] It should be noted that the specific implementation process for obtaining sample data provided in this implementation can be combined with the various specific implementation processes for obtaining sentence data provided in the aforementioned implementations to implement the webpage similarity model training method of this embodiment. A detailed description can be found in the relevant content of the aforementioned implementations and will not be repeated here.

[0083] Optionally, in a possible implementation of this embodiment, there may be multiple sample web pages, and the operations 102 - 103 may be performed multiple times on each of the multiple sample web pages to obtain multiple batches of sample data.

[0084] For example, the number of sample web pages may be 10, and operations 102-103 are performed three times on each sample web page to obtain three batches of sample data, each batch of sample data including sample data of 10 web pages.

[0085] Optionally, in a possible implementation of this embodiment, in 104, the pre-trained language model may be subjected to unsupervised training and / or supervised training based on the sample data.

[0086] In a specific implementation process of this implementation manner, the unsupervised training may include unsupervised training of contrastive learning.

[0087] Figure 3 is a schematic diagram of unsupervised model training according to the first embodiment of the present disclosure, such as Figure 3Specifically, the pre-trained language model can be a model based on a twin neural network, that is, a structure that can include a pre-trained language model 1 and a second pre-trained language model 2, and the pre-trained language model 1 and the second pre-trained language model 2 can share parameters. The pre-trained language model can be used to extract feature vectors of sample data.

[0088] For example, first, multiple groups of two batch sample data can be input into the pre-trained language model 1 and the second pre-trained language model 2 respectively to obtain the positive sample feature vector pair (h, h + ). Then, the loss function (loss) of contrastive learning is calculated based on the following formula (1): i Finally, when the training termination condition is met, the web page similarity model can be obtained.

[0089]

[0090] Among them, sim can represent the cosine similarity, It can represent the feature vector pair obtained by inputting the i-th sample into the pre-trained language model, and τ can represent the model's hyperparameters. The numerator of the formula can represent the positive sample, and the denominator of the formula can represent the positive and negative samples.

[0091] It is understood that each batch of sample data can be sample data from multiple sample web pages. For any one of the multiple sample web pages, the sample data from that sample web page can be used as a positive sample, and the sample data from the other sample web pages can be used as negative samples. Therefore, each batch of sample data can include both positive and negative samples.

[0092] In a specific implementation process of this implementation method, if the sample data obtained through 103 has a label, the sample data can be input into a pre-trained language model for supervised training.

[0093] Figure 4 is a schematic diagram of supervised model training according to the first embodiment of the present disclosure, such as Figure 4 Specifically, the pre-trained language model may be a model based on a twin neural network, that is, it may include a pre-trained language model 1 and a pre-trained language model 2, and the two pre-trained language models may share parameters.

[0094] For example, first, the sample data of each two sample web pages can be input into pre-trained language model 1 and second pre-trained language model 2, respectively, to obtain the feature vectors u and v corresponding to each sample web page. Then, based on u and v, a classifier is used to calculate the web page similarity. Finally, when the training termination condition is met, a web page similarity model can be obtained.

[0095] In this way, in this implementation, a web page similarity model can be obtained by performing unsupervised training and / or supervised training on the pre-trained language model based on the obtained sample data. Since the sample data after data enhancement can be applied to model training in different training modes, the universality of the web page sample data is guaranteed.

[0096] It should be noted that the specific implementation process of model training provided in this implementation can be combined with the various specific implementation processes for obtaining sample data provided in the aforementioned implementations to implement the webpage similarity model training method of this embodiment. A detailed description can be found in the relevant content of the aforementioned implementations and will not be repeated here.

[0097] In this embodiment, by obtaining web page data of a sample web page, text sentences can be extracted from the web page data according to a no-playback sampling strategy to obtain sentence data, and then sample data can be obtained based on the sentence data and the web page data. Then, a pre-trained language model can be trained based on the sample data to obtain the web page similarity model, which is used to predict the similarity of the web page to be tested. Since the pre-trained language model is trained based on the training sample data obtained based on the no-playback sampling strategy to obtain the web page similarity model, model training can be performed based on sample data with more comprehensive and rich web page text information, which can make it easier for the model to learn the global text information of the web page, reduce the information loss of learning, and thus enhance the robustness of the model prediction.

[0098] In addition, by adopting the technical solution provided in this embodiment, the main text sentences can be extracted from the web page data according to the no-replacement sampling strategy and the input threshold of the pre-trained language model, which can avoid direct truncation of the main text, that is, sentence data that is more comprehensive for characterizing the global information of the web page text can be obtained, so that the sample data determined based on the sentence data can be used for subsequent model training, so that the model can learn more comprehensive and complete text feature information of the web page.

[0099] Furthermore, the technical solution provided by this embodiment allows for sentence extraction from the main text of web page data based on a no-playback sampling strategy and the input threshold of a pre-trained language model to obtain the main text sentences. The sentences are then sorted based on the sequence identifiers of the main text sentences to obtain sentence data. This allows for more comprehensive sentence data representing the global information of the web page main text, allowing for subsequent model training using this sentence data, thereby enabling the model to learn more comprehensive feature information about the main text of the web page.

[0100] Furthermore, the extracted sentences are sorted according to their order of appearance in the text, generating sorted sentence data. This ensures that the obtained sentence data reflects the original meaning of the text as closely as possible, improving the validity of the sentence data and, in turn, the validity of subsequent sample data. This allows the model to learn more comprehensive text feature information about the webpage when using this sample data for model training, further enhancing the robustness of the model's predictions.

[0101] In addition, by adopting the technical solution provided in this embodiment, sample data can be obtained by obtaining sentence data based on the title, keywords and sampling of the web page, which can enhance the sample data and enrich the sample data for model training. It can facilitate the model to learn the global text information of the web page, thereby enhancing the robustness of the model prediction.

[0102] Figure 5 is a schematic diagram according to the second embodiment of the present disclosure, as shown in Figure 5 shown.

[0103] 501. Obtain web page data of any two web pages to be tested.

[0104] 502. According to the no-playback sampling strategy, extract text sentences from the webpage data of the arbitrary two webpages to be tested to obtain sentence data.

[0105] 503. Obtain a prediction result of web page similarity based on the web page data of the arbitrary two web pages to be tested and the sentence data.

[0106] It should be noted that the web pages to be tested may include news web pages and other chapter-level web pages.

[0107] Specifically, if Figure 6 As shown, the similarities of the news texts of the news web pages can be predicted, for example, 0.9, 4 and 1. Based on the similarities, it can be determined which web pages have similar news contents.

[0108] It should be noted that part or all of the execution entities of 501 to 503 may be applications located in the local terminal, or may also be functional units such as plug-ins or software development kits (SDKs) set in the applications located in the local terminal, or may also be processing engines located in the network-side server, or may also be distributed systems located on the network side, for example, processing engines or distributed systems in the web page analysis platform on the network side, etc. This embodiment does not specifically limit this.

[0109] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0110] In this way, by obtaining the web page data of any two web pages to be tested, the main text sentences can be extracted from the web page data of the any two web pages to be tested respectively according to the no-playback sampling strategy to obtain sentence data, so that the prediction results of web page similarity can be obtained based on the web page data of the any two web pages to be tested and the sentence data. Since the relevant data of the web pages used for web page similarity prediction is enriched, the completeness of the web page information used for prediction is enhanced, thereby improving the reliability of the prediction results of web page similarity.

[0111] Optionally, in a possible implementation of this embodiment, in 503 , the webpage data of the arbitrary two webpages to be tested and the sentence data may be input into a webpage similarity model to obtain a prediction result of the webpage similarity.

[0112] In this implementation, the webpage similarity model may be a model determined by training a pre-trained language model using sample data, wherein the sample data is obtained based on webpage data of a sample webpage and sentence data obtained by extracting body sentences from the webpage data of the sample webpage according to a no-playback sampling strategy.

[0113] It is understandable that the web page similarity model can be obtained by using the web page similarity model training method in the aforementioned embodiment.

[0114] In this implementation, the web page similarity model can support multi-language web page similarity prediction capabilities.

[0115] In this implementation, the web page data may also include titles and keywords.

[0116] In this way, by using the web page similarity model to predict the similarity of any two web pages under test, more accurate prediction results can be obtained, more effectively identifying web pages with similar text content, facilitating the recommendation of related web pages, thereby improving web page information retrieval and semantic computing capabilities. Furthermore, the web page similarity model can support multilingual web page similarity prediction capabilities, thereby improving similarity prediction analysis for both single-language and multilingual web pages.

[0117] In a specific implementation process of this implementation method, sentence extraction processing can be performed on the text in the web page data of any two web pages to be tested, respectively, based on the no-playback sampling strategy and the input threshold of the web page similarity model, so as to obtain the text sentences of the any two web pages to be tested, and then the text sentences can be sorted according to the sequence identifiers of the text sentences of the any two web pages to be tested, so that the sentence data can be obtained according to the results of the sorting processing.

[0118] In this specific implementation, first, for any two web pages to be tested, sentence extraction processing can be performed on the text of the web page data of the two web pages to be tested based on the no-replay sampling strategy and the input threshold of the web page similarity model repeatedly multiple times to obtain the text sentences of the two web pages to be tested. The text sentences of the two web pages to be tested are then sorted based on the sequence identifiers of the text sentences of the two web pages to be tested to obtain multiple groups of sentence data. Then, multiple groups of data to be tested can be obtained based on the web page data and the multiple groups of sentence data.

[0119] One scenario of this specific implementation process is that when using a web page similarity model to predict any two web pages to be tested, multiple groups of data to be tested corresponding to any two web pages to be tested can be input into the web page similarity model, and the output similarities can be integrated and processed, and the prediction result of the web page similarity is obtained based on the similarities after the integrated processing.

[0120] Specifically, the integration process may include a voting process and a weighted average process.

[0121] In this way, by extracting and sorting sentences from the text of any two test web pages based on the no-replay sampling strategy and the input threshold of the web page similarity model, sentence data for prediction can be obtained. This can provide more abundant and accurate sentence data for prediction, thereby improving the reliability and accuracy of the prediction results.

[0122] In this embodiment, the web page data of any two web pages to be tested can be obtained, and then the main text sentences can be extracted from the web page data of the any two web pages to be tested according to the no-playback sampling strategy to obtain sentence data, so that the prediction results of web page similarity can be obtained based on the web page data of the any two web pages to be tested and the sentence data. Since the relevant data of the web pages used for web page similarity prediction is enriched, the completeness of the web page information used for prediction is enhanced, thereby improving the reliability of the prediction results of web page similarity.

[0123] Furthermore, the technical solution provided by this embodiment allows similarity prediction to be performed on any two web pages tested using the web page similarity model. This allows for more accurate prediction results and more effective identification of web pages with similar text content, facilitating the recommendation of related web pages. This improves web page information retrieval and semantic computing capabilities. Furthermore, the web page similarity model supports multilingual web page similarity prediction, thereby enhancing similarity prediction analysis for both single-language and multilingual web pages.

[0124] Furthermore, the technical solution provided by this embodiment can extract and sort sentences from the text of any two test web pages based on a no-playback sampling strategy and the input threshold of a webpage similarity model to obtain sentence data for prediction. This allows for more comprehensive and accurate sentence data for prediction, thereby improving the reliability and accuracy of prediction results.

[0125] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0126] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] Figure 7 is a schematic diagram according to the third embodiment of the present disclosure, as shown in Figure 7 As shown. The web page similarity model training device 700 of this embodiment may include an acquisition unit 701, an extraction unit 702, an acquisition unit 703 and a training unit 704. The acquisition unit is used to acquire web page data of a sample web page; the extraction unit is used to extract text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data; the acquisition unit is used to obtain sample data based on the sentence data and the web page data; the training unit is used to train a pre-trained language model based on the sample data to obtain the web page similarity model for predicting the similarity of the web page to be tested.

[0128] It should be noted that part or all of the training device for the web page similarity model of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal. This embodiment does not specifically limit this.

[0129] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0130] Optionally, in a possible implementation of this embodiment, the extraction unit 702 can be specifically used to perform sentence extraction processing on the text in the web page data according to the no-playback sampling strategy and the input threshold of the pre-trained language model to obtain text sentences, sort the text sentences according to the sequence identification of the text sentences, and obtain the sentence data based on the result of the sorting processing.

[0131] Optionally, in a possible implementation of this embodiment, the extraction unit 702 can also be used to perform sentence extraction processing on the text in the web page data according to the no-playback sampling strategy to obtain the text sentences. If the total length of the extracted text sentences reaches the input threshold of the pre-trained language model, the sentence extraction processing is terminated.

[0132] Optionally, in a possible implementation of this embodiment, the obtaining unit 703 may be specifically configured to obtain titles and keywords in the web page data, and obtain the sample data according to the sentence data, the title, and the keywords.

[0133] In this embodiment, by obtaining web page data of a sample web page, text sentences can be extracted from the web page data according to a no-playback sampling strategy to obtain sentence data, and then sample data can be obtained based on the sentence data and the web page data. Then, a pre-trained language model can be trained based on the sample data to obtain the web page similarity model, which is used to predict the similarity of the web page to be tested. Since the pre-trained language model is trained based on the training sample data obtained based on the no-playback sampling strategy to obtain the web page similarity model, model training can be performed based on sample data with more comprehensive and rich web page text information, which can make it easier for the model to learn the global text information of the web page, reduce the information loss of learning, and thus enhance the robustness of the model prediction.

[0134] In addition, by adopting the technical solution provided in this embodiment, the main text sentences can be extracted from the web page data according to the no-replacement sampling strategy and the input threshold of the pre-trained language model, which can avoid direct truncation of the main text, that is, sentence data that is more comprehensive for characterizing the global information of the web page text can be obtained, so that the sample data determined based on the sentence data can be used for subsequent model training, so that the model can learn more comprehensive and complete text feature information of the web page.

[0135] Furthermore, the technical solution provided by this embodiment allows for sentence extraction from the main text of web page data based on a no-playback sampling strategy and the input threshold of a pre-trained language model to obtain the main text sentences. The sentences are then sorted based on the sequence identifiers of the main text sentences to obtain sentence data. This allows for more comprehensive sentence data representing the global information of the web page main text, allowing for subsequent model training using this sentence data, thereby enabling the model to learn more comprehensive feature information about the main text of the web page.

[0136] Furthermore, the extracted sentences are sorted according to their order of appearance in the text, generating sorted sentence data. This ensures that the obtained sentence data reflects the original meaning of the text as closely as possible, improving the validity of the sentence data and, in turn, the validity of subsequent sample data. This allows the model to learn more comprehensive text feature information about the webpage when using this sample data for model training, further enhancing the robustness of the model's predictions.

[0137] In addition, by adopting the technical solution provided in this embodiment, sample data can be obtained by obtaining sentence data based on the title, keywords and sampling of the web page, which can enhance the sample data and enrich the sample data for model training. It can facilitate the model to learn the global text information of the web page, thereby enhancing the robustness of the model prediction.

[0138] Figure 8 is a schematic diagram according to a fourth embodiment of the present disclosure, as shown in Figure 8 As shown. The web page similarity prediction device 800 of this embodiment may include an acquisition unit 801, an extraction unit 802, and an acquisition unit 803. The acquisition unit 801 is used to acquire web page data of any two web pages to be tested; the extraction unit 802 is used to extract text sentences from the web page data of the any two web pages to be tested according to the no-playback sampling strategy to obtain sentence data; the acquisition unit 803 is used to obtain the prediction result of the web page similarity based on the web page data of the any two web pages to be tested and the sentence data.

[0139] It should be noted that part or all of the web page similarity prediction device of this embodiment may be an application located in the local terminal, or may also be a functional unit such as a plug-in or software development kit (SDK) set in the application located in the local terminal. This embodiment does not specifically limit this.

[0140] It is understandable that the application may be a native program (nativeApp) installed on the local terminal, or may be a webpage program (webApp) of a browser on the local terminal, which is not limited in this embodiment.

[0141] Optionally, in a possible implementation of this embodiment, the obtaining unit 803 may be specifically configured to input the webpage data of the arbitrary two webpages to be tested and the sentence data into a webpage similarity model to obtain a prediction result of the webpage similarity.

[0142] Optionally, in a possible implementation of this embodiment, the extraction unit 802 can be specifically used to perform sentence extraction processing on the text in the web page data of any two web pages to be tested according to the no-playback sampling strategy and the input threshold of the web page similarity model to obtain the text sentences of the any two web pages to be tested, sort the text sentences according to the sequence identifiers of the text sentences of the any two web pages to be tested, and obtain the sentence data based on the results of the sorting processing.

[0143] In this embodiment, the web page data of any two web pages to be tested can be obtained by the acquisition unit, and then the extraction unit can extract the text sentences from the web page data of the any two web pages to be tested respectively according to the no-playback sampling strategy to obtain sentence data, so that the acquisition unit can obtain the prediction result of the web page similarity based on the web page data of the any two web pages to be tested and the sentence data. Since the relevant data of the web pages used for web page similarity prediction is enriched, the completeness of the web page information used for prediction is enhanced, thereby improving the reliability of the prediction result of the web page similarity.

[0144] Furthermore, the technical solution provided by this embodiment allows similarity prediction to be performed on any two web pages tested using the web page similarity model. This allows for more accurate prediction results and more effective identification of web pages with similar text content, facilitating the recommendation of related web pages. This improves web page information retrieval and semantic computing capabilities. Furthermore, the web page similarity model supports multilingual web page similarity prediction, thereby enhancing similarity prediction analysis for both single-language and multilingual web pages.

[0145] Furthermore, the technical solution provided by this embodiment can extract and sort sentences from the text of any two test web pages based on a no-playback sampling strategy and the input threshold of a webpage similarity model to obtain sentence data for prediction. This allows for more comprehensive and accurate sentence data for prediction, thereby improving the reliability and accuracy of prediction results.

[0146] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0148] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Figure 9 As shown, the electronic device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0150] Multiple components in the electronic device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the electronic device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 901 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the training method of the web page similarity model and the prediction method of web page similarity. For example, in some embodiments, the training method of the web page similarity model and the prediction method of web page similarity can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the training method of the web page similarity model and the prediction method of web page similarity described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the web page similarity model training method and the web page similarity prediction method in any other appropriate manner (eg, by means of firmware).

[0152] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

Claims

1. A method for training a web page similarity model, comprising: Obtain web page data of a sample web page; Extracting text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data; Obtaining sample data according to the sentence data and the web page data; Training the pre-trained language model based on the sample data to obtain a webpage similarity model for predicting the similarity between any two webpages to be tested; The extracting of text sentences from the web page data according to the no-playback sampling strategy to obtain sentence data includes: Perform sentence extraction processing on the text in the webpage data according to the no-playback sampling strategy and the input threshold of the pre-trained language model to obtain text sentences; Sorting the text sentences according to the sequence identifiers of the text sentences; Obtaining the sentence data according to the result of the sorting process; Wherein, the pre-trained language model includes a pre-trained multi-language model; For any sample web page, the sample data includes multiple sets of sample data obtained through multiple extractions; The method of training a pre-trained language model based on the sample data to obtain a webpage similarity model includes: The sample data of every two sample web pages are respectively input into two pre-trained language models with shared parameters to obtain a feature vector corresponding to each sample web page, and supervised training is performed based on the feature vector to obtain the web page similarity model.

2. The method according to claim 1, wherein The method of performing sentence extraction processing on the text in the webpage data according to the no-replay sampling strategy and the input threshold of the pre-trained language model to obtain text sentences includes: Performing sentence extraction processing on the text in the web page data according to the no-playback sampling strategy to obtain the text sentences; If the total length of the extracted text sentences reaches the input threshold of the pre-trained language model, the sentence extraction process is terminated.

3. The method according to any one of claims 1 to 2, wherein The obtaining of sample data according to the sentence data and the webpage data includes: Obtaining titles and keywords from the web page data; The sample data is obtained based on the sentence data, the title, and the keyword.

4. A method for predicting web page similarity, comprising: Get the web page data of any two web pages to be tested; Extracting text sentences from the webpage data of any two webpages to be tested according to a no-playback sampling strategy to obtain sentence data; A web page similarity model trained by the method according to any one of claims 1 to 3 is used to obtain a prediction result of the web page similarity based on the web page data of the arbitrary two web pages to be tested and the sentence data.

5. The method according to claim 4, wherein The step of obtaining a prediction result of the webpage similarity based on the webpage data of the arbitrary two webpages to be tested and the sentence data includes: The web page data of the arbitrary two web pages to be tested and the sentence data are input into a web page similarity model to obtain a prediction result of the web page similarity.

6. The method according to claim 5, wherein: The method of extracting text sentences from the webpage data of any two webpages to be tested according to the no-playback sampling strategy to obtain sentence data includes: performing sentence extraction processing on the texts in the webpage data of the arbitrary two webpages to be tested according to the no-replay sampling strategy and the input threshold of the webpage similarity model to obtain the text sentences of the arbitrary two webpages to be tested; sorting the text sentences of the two web pages to be tested according to their sequence identifiers; The sentence data is obtained according to the result of the sorting process.

7. A training device for a web page similarity model, comprising: An acquisition unit, configured to acquire webpage data of a sample webpage; an extraction unit, configured to extract text sentences from the web page data according to a no-playback sampling strategy to obtain sentence data; an obtaining unit, configured to obtain sample data based on the sentence data and the web page data; A training unit, configured to train a pre-trained language model based on the sample data to obtain the web page similarity model for predicting the similarity of the web page to be tested; The extraction unit is specifically used for: Perform sentence extraction processing on the text in the webpage data according to the no-playback sampling strategy and the input threshold of the pre-trained language model to obtain text sentences; Sorting the text sentences according to the sequence identifiers of the text sentences; as well as Obtaining the sentence data according to the result of the sorting process; Wherein, the pre-trained language model includes a pre-trained multi-language model; For any sample web page, the sample data includes multiple sets of sample data obtained through multiple extractions; The training unit is specifically used to: The sample data of every two sample web pages are respectively input into two pre-trained language models with shared parameters to obtain a feature vector corresponding to each sample web page, and supervised training is performed based on the feature vector to obtain the web page similarity model.

8. The device according to claim 7, wherein The extraction unit is further configured to perform sentence extraction processing on the text in the web page data according to the no-playback sampling strategy to obtain the text sentences; If the total length of the extracted text sentences reaches the input threshold of the pre-trained language model, the sentence extraction process is terminated.

9. The device according to any one of claims 7 to 8, wherein the obtaining unit is specifically configured to Obtaining titles and keywords from the web page data; The sample data is obtained based on the sentence data, the title, and the keyword.

10. A device for predicting web page similarity, comprising: An acquisition unit, used to acquire web page data of any two web pages to be tested; an extraction unit, configured to extract text sentences from the webpage data of the arbitrary two webpages to be tested respectively according to a no-playback sampling strategy to obtain sentence data; The obtaining unit is configured to obtain a prediction result of the web page similarity based on the web page data of the arbitrary two web pages to be tested and the sentence data, using the web page similarity model trained by the method according to any one of claims 1 to 3.

11. The device according to claim 10, wherein The obtaining unit is specifically used for The web page data of the arbitrary two web pages to be tested and the sentence data are input into a web page similarity model to obtain a prediction result of the web page similarity.

12. The device according to claim 11, wherein The extraction unit is specifically used for performing sentence extraction processing on the texts in the webpage data of the arbitrary two webpages to be tested according to the no-replay sampling strategy and the input threshold of the webpage similarity model to obtain the text sentences of the arbitrary two webpages to be tested; sorting the text sentences of the two web pages to be tested according to their sequence identifiers; as well as The sentence data is obtained according to the result of the sorting process.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images