A search term ranking model training, search term ranking method and device

Abstract

Embodiments of the present invention provide a search term ranking model training, search term ranking method, and device for use in the field of intelligent analysis technology. The method includes: obtaining sample features of N samples in a first sample set; constructing M sample feature groups based on the obtained sample features, and determining first labeling information for each sample feature group; training a pre-trained model using each sample feature group and the first labeling information of each sample feature group, and returning to the step of obtaining sample features of N samples in the first sample set until the loss function value of the pre-trained model converges; training the search term ranking model to be trained using sample features in a second sample set until the loss function value of the search term ranking model to be trained converges, thereby obtaining a trained search term ranking model. Through this solution, the search term ranking model can be made to more accurately perform search term ranking.

Description

Technical Field

[0001] The present invention relates to the field of intelligent analysis technology, and in particular to a search term ranking model training, a search term ranking method and a device. Background Art

[0002] The search recommendation system can greatly improve the efficiency of users in finding resources. For example, when a user searches in the search interface, it can provide the user with the complete search content that the user may need to search based on the content the user has entered, and then search based on the search content selected by the user from the provided complete search content, and display the final resource search results. The user can then select the resources he needs from the displayed resource search results.

[0003] Generally speaking, the content entered by the user is called a token, and the complete search content used in the final search is called a query. For example, in a search, if a user enters the name of an artist "X" and the system recommends and selects "X's movie," then "X" is the token, and "X's movie" is the search term. To improve search quality, related technologies often use tokens and search terms to train neural network models to obtain a search term ranking model.

[0004] However, since unpopular resources have a low search volume, there are fewer token words or search words for unpopular resources, making it impossible for the trained search word ranking model to fully learn the token words or search words for unpopular resources, resulting in a low accuracy rate for search word ranking using the search word ranking model. Summary of the Invention

[0005] The purpose of the embodiments of the present invention is to provide a search term ranking model training, search term ranking method and device, so that the search term ranking model can more accurately perform search term ranking. The specific technical solution is as follows:

[0006] In a first aspect of the present invention, a method for training a search term ranking model is provided, the method comprising:

[0007] Obtaining sample features of N samples in a first sample set; wherein N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first category of samples and a second category of samples, the token words and / or search words included in the first category of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second category of samples have a search quantity greater than a second quantity threshold within the specified time period, and the sample features of each sample include a token feature determined based on the token words in the sample and a search feature determined based on the search word in the sample;

[0008] Based on the acquired sample features, M sample feature groups are constructed, and the first annotation information of each sample feature group is determined; where M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first annotation information is a positive sample, otherwise it is a negative sample;

[0009] Training a pre-trained model using each sample feature group and the first labeling information of each sample feature group, and returning to the step of obtaining sample features of N samples in the first sample set until a loss function value of the pre-trained model converges, wherein the loss function value of the pre-trained model is determined based on positive and negative sample prediction results output by the pre-trained model for the sample feature group and the first labeling information, and the pre-trained model includes a feature extraction layer that extracts features from token features and search features;

[0010] The search term ranking model to be trained is trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges, thereby obtaining a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is determined based on the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-trained model that has been trained.

[0011] Optionally, constructing M sample feature groups based on the acquired sample features includes:

[0012] Determine a sample feature from the N sample features as a first feature to be combined;

[0013] Determine at least one sample feature from the N sample features as a second feature to be combined;

[0014] combining the first feature to be combined with at least one second feature to be combined to obtain a sample feature group;

[0015] Return to the step of determining a sample feature from the N sample features as the first feature to be combined, until the number of obtained sample feature groups reaches M.

[0016] Optionally, the using each sample feature group and the first annotation information of each sample feature group to train the pre-trained model includes:

[0017] Selecting a sample feature group from each sample feature group as the sample feature group to be used;

[0018] Inputting each sample feature included in the sample feature group to be utilized into a pre-trained model to obtain a classification result predicted by the pre-trained model;

[0019] Determining a first loss function value of the pre-trained model based on the obtained classification result and the first labeling information of the sample feature group to be utilized;

[0020] Adjusting model parameters in the pre-trained model based on the first loss function value;

[0021] Return to the step of selecting a sample feature group from each sample feature group until all the sample feature groups are selected.

[0022] Optionally, the step of training the search term ranking model to be trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges to obtain a trained search term ranking model includes:

[0023] Acquire at least one sample feature from the second sample set as a sample feature to be used;

[0024] Inputting at least one sample feature to be utilized into the search term ranking model to be trained, respectively, to obtain a click-through rate prediction result output by the search term ranking model for each sample feature to be utilized;

[0025] Determining a loss function value of the search term ranking model to be trained based on the second annotation information pre-annotated for the at least one sample feature to be utilized and the obtained click-through rate prediction result as the second loss function value; wherein the second annotation result of any sample feature is determined based on the click-through rate of the search term corresponding to the sample feature;

[0026] Adjusting model parameters in the search term ranking model to be trained based on the second loss function value;

[0027] Return to the step of obtaining the sample features to be used from the second sample set until the second loss function value meets convergence, thereby obtaining a trained search term ranking model.

[0028] Optionally, determining the loss function value of the search term ranking model to be trained based on the second annotation information pre-annotated for the at least one sample feature to be utilized and the obtained click-through rate prediction result includes:

[0029] The loss function value of the search term ranking model to be trained is calculated using the following formula:

[0030]

[0031] L i =-[y i *log(p i )+(1-y i )*log(1-p i )

[0032] Wherein, L is the second loss function value of the search term ranking model to be trained, K is the number of the at least one sample feature to be used, and L i is the loss value of the i-th sample feature to be used in the at least one sample feature to be used, y i is the second annotation information of the i-th sample feature to be used, p i The click-through rate prediction result output by the search term ranking model to be trained for the i-th sample feature to be utilized.

[0033] In a second aspect of the present invention, a search term ranking method is provided, comprising:

[0034] Obtain at least one search term containing a target token term;

[0035] Determining feature information of the at least one search term; wherein the feature information of each search term includes a search feature of the search term and a token feature of the target token term;

[0036] The feature information of the at least one search term is input into a pre-trained search term ranking model to obtain a ranking result of the at least one search term; wherein the ranking result of the at least one search term is determined based on the click-through rate prediction result of each search term based on the search term ranking model, and the search term ranking model is a search term ranking model obtained by training using the method described in the first aspect.

[0037] In a third aspect of the present invention, a search term ranking model training device is provided, comprising:

[0038] a feature acquisition module, configured to acquire sample features of N samples in a first sample set; wherein N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first category of samples and a second category of samples, the token words and / or search words included in the first category of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second category of samples have a search quantity greater than a second quantity threshold within the specified time period, and the sample features of each sample include a token feature determined based on the token words in the sample and a search feature determined based on the search word in the sample;

[0039] A feature group construction module is used to construct M sample feature groups based on the acquired sample features, and determine the first labeling information of each sample feature group; wherein M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first labeling information is a positive sample, otherwise it is a negative sample;

[0040] a first training module, configured to train a pre-trained model using each sample feature group and first labeling information of each sample feature group, and return to executing the step of obtaining sample features of N samples in the first sample set until a loss function value of the pre-trained model converges, wherein the loss function value of the pre-trained model is determined based on positive and negative sample prediction results output by the pre-trained model for the sample feature group and the first labeling information, and the pre-trained model includes a feature extraction layer for extracting features from token features and search features;

[0041] The second training module is used to train the search term ranking model to be trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges to obtain a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is: determined based on the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-training model that has been trained.

[0042] In a fourth aspect of the present invention, a search term ranking device is provided, comprising:

[0043] A search term acquisition module, configured to acquire at least one search term containing a target token word;

[0044] An information determination module, configured to determine feature information of the at least one search term; wherein the feature information of each search term includes a search feature of the search term and a token feature of the target token term;

[0045] An information input module is used to input the feature information of the at least one search term into a pre-trained search term ranking model to obtain a ranking result of the at least one search term; wherein the ranking result of the at least one search term is determined based on the click-through rate prediction result of each search term based on the search term ranking model, and the search term ranking model is a search term ranking model obtained by training using the device described in the third aspect.

[0046] In the fifth aspect of the implementation of the present invention, an electronic device is also provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; the processor is used to implement any of the above-mentioned search term ranking model training methods or search term ranking method steps when executing the programs stored in the memory.

[0047] In the sixth aspect of the implementation of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned search term ranking model training methods or search term ranking methods.

[0048] In the seventh aspect of the implementation of the present invention, a computer program product comprising instructions is also provided, which, when run on a computer, enables the computer to execute any of the above-mentioned search term ranking model training methods or search term ranking methods.

[0049] In a search term ranking model training method provided by an embodiment of the present invention, sample features of N samples in a first sample set can be obtained, and then M sample feature groups are constructed based on the obtained sample features, and the first annotation information of each sample feature group is determined; wherein M is a positive integer greater than N, and each sample feature group and the first annotation information of each sample feature group are used to train the pre-trained model, and the step of obtaining sample features of N samples in the first sample set is returned to execute until the loss function value of the pre-trained model converges; and the search term ranking model to be trained is trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges to obtain a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is determined based on the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the annotated click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the trained pre-trained model. Since the M sample feature groups are constructed by the sample features of N samples in the first sample set, the number of feature extraction layers included in the pre-training model that are trained can be increased, and because the first sample set contains first and second samples, and the first sample contains a smaller number of token words and / or search words, therefore, by increasing the number, the feature extraction layer can better extract features from token features and search features with a smaller number of retrievals, and thus the search term ranking model constructed based on the feature extraction layer has a better ability to extract features from token features and search features with a smaller number of retrievals, so that the search term ranking model can more accurately rank search terms.

[0050] On the other hand, an embodiment of the present invention also provides a search term ranking method, which can obtain at least one search term containing a target token term, and then determine the feature information of at least one search term; wherein, the feature information of each search term includes the search feature of the search term and the token feature of the target token term, and the feature information of at least one search term is input into a pre-trained search term ranking model to obtain a ranking result of at least one search term; wherein, the ranking result of at least one search term is determined based on the click-through rate prediction result of each search term based on the search term ranking model, and the search term ranking model is a search term ranking model obtained by training using the search term ranking model training method provided by the embodiment of the present invention. Since the search term ranking model obtained by training using the search term ranking model training method provided by the embodiment of the present invention has a better ability to extract features for token features and search features with a smaller number of retrievals, the search term ranking is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for describing the embodiments or the prior art.

[0052] Figure 1 A flowchart of a search term ranking model training method provided by an embodiment of the present invention;

[0053] Figure 2 A schematic diagram of the structure of a pre-training model provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the structure of a search term ranking model to be trained provided by an embodiment of the present invention;

[0055] Figure 4 A flowchart of a search term ranking method provided by an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the structure of a search term ranking model training device provided by an embodiment of the present invention;

[0057] Figure 6 A schematic diagram of the structure of a search term ranking device provided by an embodiment of the present invention;

[0058] Figure 7 This is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0060] In order to improve the accuracy of search word ranking using a search word ranking model, embodiments of the present invention provide search word ranking model training, a search word ranking method, and an apparatus.

[0061] It should be noted that, in specific applications, the embodiments of the present invention can be applied to various electronic devices, such as personal computers, servers, mobile phones, and other devices with data processing capabilities. Furthermore, the search term ranking model training method or search term ranking method provided by the embodiments of the present invention can be implemented using software, hardware, or a combination of software and hardware.

[0062] In one implementation, the search term ranking model training method provided by an embodiment of the present invention can be applied to a model training device, and the search term ranking method provided by an embodiment of the present invention can be applied to a client or a server deployed with a search term ranking model.

[0063] The search term ranking model training method provided by the embodiment of the present invention may include:

[0064] Obtaining sample features of N samples in a first sample set; wherein N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first category of samples and a second category of samples, the token words and / or search words included in the first category of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second category of samples have a search quantity greater than a second quantity threshold within a specified time period, and the sample features of each sample include a token feature determined based on the token words in the sample and a search feature determined based on the search word in the sample;

[0065] Based on the acquired sample features, M sample feature groups are constructed, and the first annotation information of each sample feature group is determined; where M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first annotation information is a positive sample, otherwise it is a negative sample;

[0066] Training the pre-trained model using each sample feature group and the first labeling information of each sample feature group, and returning to the step of obtaining sample features of N samples in the first sample set until a loss function value of the pre-trained model converges, wherein the loss function value of the pre-trained model is determined based on positive and negative sample prediction results output by the pre-trained model for the sample feature group and the first labeling information, and the pre-trained model includes a feature extraction layer that extracts features from token features and search features;

[0067] The search term ranking model to be trained is trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges, thereby obtaining a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is determined by: the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-trained model that has been trained.

[0068] In the above scheme of the embodiment of the present invention, since the M sample feature groups are constructed by the sample features of N samples in the first sample set, the number of feature extraction layers included in the pre-training model that are trained can be increased, and because the first sample set contains first-category samples and second-category samples, and the first-category samples contain a smaller number of token words and / or search words, therefore, by increasing the number, the feature extraction layer can better extract features from token features and search features with a smaller number of retrievals, and thus the search term ranking model constructed based on the feature extraction layer has a better ability to extract features from token features and search features with a smaller number of retrievals, so that the search term ranking model can more accurately rank search terms.

[0069] The following will describe in detail the search term ranking model training method provided by the embodiment of the present invention in conjunction with the accompanying drawings.

[0070] like Figure 1 As shown, an embodiment of the present invention provides a search term ranking model training method, including steps S101-S104, wherein:

[0071] S101, obtaining sample features of N samples in a first sample set;

[0072] Wherein, N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first type of samples and a second type of samples, the token words and / or search words included in the first type of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second type of samples have a search quantity greater than a second quantity threshold within a specified time period, and the sample feature of each sample includes a token feature determined based on the token words in the sample and a search feature determined based on the search words in the sample;

[0073] The first sample set may be a pre-collected dataset containing multiple samples, wherein each sample in the first sample set may include at least a token word and / or a search word, and each search word includes the token word in the sample in which it is located. A token word refers to a word or sentence input by a user, while a search word is obtained by token word expansion, i.e., the search word includes at least one token word. For example, if the token word input by the user is the name of an artist "X", then "X's movie", "X's music", and "X's variety show" obtained by token word expansion are all search words containing token words. It should be noted that the search word corresponding to each token word is not unique. For example, the token word "X" mentioned above corresponds to at least three search words: "X's movie", "X's music", and "X's variety show". Similarly, the token word corresponding to each search word is not unique. For example, the search word "X's movie" mentioned above is both the search word corresponding to the token word "X" and the search word corresponding to the token word "movie".

[0074] The first sample set includes a first category of samples and a second category of samples, wherein the token words and / or search words included in the first category of samples have a search count less than a first quantity threshold within a specified time period, and the token words and / or search words included in the second category of samples have a search count greater than a second quantity threshold within a specified time period. The specified time period may be any preset time period, and its corresponding duration may be one year, one month, one week, or one day, and of course it may be longer or shorter. The first quantity threshold and the second quantity threshold may be quantity thresholds set according to requirements. Optionally, the first quantity threshold may be the same as or different from the second quantity threshold. Optionally, the first quantity threshold may be less than the second quantity threshold. Since the token words and / or search words included in the first category of samples have a search count less than the first quantity threshold within a specified time period, it means that the first category of samples may be samples collected for unpopular resources, which refer to resources with a small number of views and / or searches. Similarly, the second category of samples may be samples collected for popular resources, which refer to resources with a large number of views and / or searches.

[0075] The sample features of each of the above samples include token features determined based on the token words in the sample, and search features determined based on the search words in the sample. The above token features can be embedding features extracted for token words to obtain embedding features corresponding to the token words, and the above search features can be embedding features extracted for search words to obtain embedding features corresponding to the search words. Optionally, in addition to token word features and search word features, the sample features of each sample can also include statistical features for token word features and search word features, wherein the statistical features for token word features and search word features include: after searching and sorting using the token words to which the token word features belong and the search words to which the search word features belong, the number of clicks on the search sorting results, the proportion of each data source, and the ratio of the number of searches to the number of clicks, etc.

[0076] Optionally, the above embedding features can be represented by feature vectors of specified dimensions. In one implementation, each sample feature can be represented as:

[0077] f i =[meta,embd token ,embd query ]

[0078] Among them, f i is the i-th sample feature, meta is the statistical feature, embd token is the token word feature of the token word, embd quiery The search term feature of the search term.

[0079] in, in is the statistical feature set corresponding to each sample, is the token feature set corresponding to each sample, is the search feature set corresponding to each sample.

[0080] The above-mentioned N can be a positive integer set according to requirements, which can be the number of samples required to be included in each training batch. Optionally, N samples can be randomly selected from the first sample set to determine the sample characteristics of the N samples. Alternatively, samples can be selected from the first sample set according to a specified selection rule. The selection rule can be determined according to requirements, for example, selecting n1 first-class samples and n2 second-class samples, where n1+n2=N.

[0081] S102, constructing M sample feature groups based on the acquired sample features, and determining first labeling information for each sample feature group;

[0082] In this step, after obtaining N sample features, M sample feature groups can be constructed based on the obtained sample features, and the first annotation information for each sample feature group can be determined. Where M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first annotation information is a positive sample, otherwise it is a negative sample.

[0083] In one implementation, a sample feature can be determined from N sample features as the first feature to be combined, and then at least one sample feature can be determined from the N sample features as the second feature to be combined, and the first feature to be combined and at least one second feature to be combined can be combined to obtain a sample feature group, and the step of determining a sample feature from the N sample features as the first feature to be combined can be returned to execute until the number of sample feature groups obtained reaches M.

[0084] Exemplarily, N sample features are: (token1, query1), (token2, query2) and (token3, query3), then (token1, query1) is randomly selected as the first feature to be combined, and then at least one sample feature is selected from the above three sample features as the second feature to be combined. The at least one sample feature can be (token2, query2), (token1, query1), or (token1, query1) and (token3, query3), which are all possible, and then the sample feature groups (token1, query1, token2, query2), (token1, query1, token1, query2), or (token1, query1, token1, query1, token3, query3) are obtained by combination, which are all possible.

[0085] Optionally, in order to facilitate input into the neural network model, the number of sample features contained in the above sample feature group can be fixed, for example, containing 2, 3 or 4 sample features. Exemplarily, the combination of sample features is illustrated below by taking an example containing 2 sample features. Exemplarily, the N sample features are: (token1, query1), represented by (t1, q1), (token2, query2), represented by (t2, q2), (token3, query3), represented by (t3, q3), and (token4, query4), represented by (t4, q4). By combining the above 4 sample features, a maximum of 16 sample feature groups can be obtained, as shown in Table 1:

[0086] Table 1

[0087] (t1,q1,t1,q1,) (t1,q1,t2,q2,) (t1,q1,t3,q3,) (t1,q1,t4,q4,) (t2,q2,t1,q1,) (t2,q2,t2,q2,) (t2,q2,t3,q3,) (t2,q2,t4,q4,) (t3,q3,t1,q1,) (t3,q3,t2,q2,) (t3,q3,t3,q3,) (t3,q3,t4,q4,) (t4,q4,t1,q1,) (t4,q4,t2,q2,) (t4,q4,t3,q3,) (t4,q4,t4,q4,)

[0088] After obtaining M sample feature groups, it is necessary to determine the first label information of each sample feature group. Optionally, if the token feature and the search feature in each sample feature in the sample feature group are the same, the first label information is a positive sample, otherwise it is a negative sample.

[0089] For example, the token features and search features in the sample feature groups (t1,q1,t1,q1,), (t2,q2,t2,q2,), (t3,q3,t3,q3,) and (t4,q4,t4,q4,) in Table 1 above are the same. Therefore, their first label information is a positive sample, and the remaining sample feature groups are negative samples, so Table 1 contains 4 positive samples and 12 negative samples.

[0090] In order to conveniently represent positive samples and negative samples, the label of the positive sample can be set to 1, and the label of the negative sample can be set to -1.

[0091] S103, training the pre-trained model using each sample feature group and the first annotation information of each sample feature group, and returning to the step of obtaining sample features of N samples in the first sample set until the loss function value of the pre-trained model converges;

[0092] Among them, the loss function value of the pre-training model is: determined based on the positive and negative sample prediction results output by the pre-training model for the sample feature group and the first annotation information, and the pre-training model includes a feature extraction layer for feature extraction of token features and search features.

[0093] Optionally, the pre-trained model may be pre-built and include a feature extraction layer for extracting token features and search features. The pre-trained model may be a Siamese network comprising at least one branch. Optionally, the number of sample features included in the sample feature group may be consistent with the number of branches in the pre-trained model.

[0094] For example, Figure 2 As shown, an embodiment of the present invention provides a structural diagram of a pre-training model. Figure 2The pre-training model includes two feature extraction layers and classification layers, namely feature extraction layer a1 and feature extraction layer a2. Taking the sample feature group containing sample feature 1 and sample feature 2 as an example, in the training stage, sample feature 1 is input into feature extraction layer a1 to obtain feature 1 extracted by feature extraction layer a1 for sample feature 1. At the same time, sample feature 2 is input into feature extraction layer a2 to obtain feature 2 extracted by feature extraction layer a1 for sample feature 1. Feature 1 and feature 2 are then input into the classification layer to obtain the final positive and negative sample prediction results, and then the loss function value is determined based on the positive and negative sample prediction results and the first labeling information of the sample feature group. In the above example, feature extraction layer a1 or feature extraction layer a2 is a feature extraction layer that extracts features for token features and search features. It should be emphasized that the above pre-training model may also include a single feature extraction layer, such as Figure 2 At this time, in the process of training with the sample feature group, the sample features in the sample feature group can be input into the feature extraction layer in sequence to obtain the features extracted by each sample feature, and then the features extracted by each sample feature can be input into the classification layer to obtain the final positive and negative sample prediction results. This is also possible.

[0095] In one implementation, the above-mentioned training of the pre-trained model using each sample feature group and the first annotation information of each sample feature group may include steps A1 to A4:

[0096] Step A1, selecting a sample feature group from each sample feature group as the sample feature group to be used;

[0097] A sample feature group is selected from each sample feature group as the sample feature group to be used. A sample feature group can be randomly selected from each sample feature group as the sample feature group to be used. Alternatively, the sample feature group to be used can be selected from each sample feature group according to a predetermined selection rule as needed, which is not specifically limited in the present invention.

[0098] Step A2: inputting each sample feature included in the sample feature group to be used into the pre-trained model to obtain a classification result predicted by the pre-trained model;

[0099] After determining the target sample feature set, each feature contained in the target sample feature set can be input into the pre-trained model. The feature extraction network in the pre-trained model can then extract features from the target sample feature set. The classification layer then predicts the extracted features to obtain a classification result. The classification result can be 1 or -1, where 1 indicates a positive sample and -1 indicates a negative sample.

[0100] In one implementation, the method of inputting each sample feature contained in the sample feature group to be used into the pre-training model may be different depending on the structure of the pre-training model. In one implementation, when the pre-training model includes multiple feature input layers, for example, Figure 2 In the case of two feature input layers shown, each sample feature can be input simultaneously. In another implementation, each sample feature can also be input sequentially. Both are acceptable.

[0101] After inputting each sample feature into the pre-training model, the classification result predicted by the training model can be obtained, that is, the positive and negative sample prediction results of the sample feature group to which each input sample feature belongs.

[0102] Step A3: determining a first loss function value of the pre-trained model based on the obtained classification result and the first labeling information of the sample feature group to be used;

[0103] After obtaining the classification result, a first loss function value of the pre-trained model can be calculated. Optionally, the classification result and the first annotation information of the sample feature group to be used can be calculated according to a specified loss function to obtain the first loss function value of the pre-trained model. Optionally, the specified loss function can be a pairwise loss.

[0104] Step A4: Adjust the model parameters in the pre-trained model based on the first loss function value, and return to step A1 until all sample feature groups are selected;

[0105] After obtaining the first loss function value, the model parameters in the pre-trained model can be adjusted using the first loss function value. For example, an optimization algorithm such as gradient descent or Newton's method can be used to optimize the model parameters in the pre-trained model using the first loss function value.

[0106] After adjusting the model parameters in the pre-trained model, the process may return to step A1 until all sample feature groups are selected.

[0107] S104, training the search term ranking model to be trained using sample features in the second sample set until the loss function value of the search term ranking model to be trained converges, thereby obtaining a trained search term ranking model;

[0108] Among them, the loss function value of the search term ranking model to be trained is: determined by the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-trained model that has been trained.

[0109] Optionally, the search term ranking model to be trained can be a search term ranking model constructed based on the feature extraction layer in the pre-trained model that has been trained. The search term ranking model can be a multi-layer neural network, which includes at least one feature extraction layer, and the at least one feature extraction layer includes the feature extraction layer in the pre-trained model that has been trained.

[0110] For example, Figure 3 As shown, an embodiment of the present invention provides a structural diagram of a search term ranking model to be trained. Figure 3 The search term ranking model to be trained includes u feature extraction layers and click-through rate prediction layers, where u is a positive integer. The u feature extraction layers are feature extraction layers b1-bu, which include the feature extraction layers in the pre-trained model trained above. For example, Figure 3 The feature extraction layer bi shown in FIG. 1 and the click-through rate prediction layer are used to predict the click-through rate of the search term based on the features extracted by the u feature extraction layers, and then output the click-through rate prediction result.

[0111] Optionally, during the training process of the search term ranking model to be trained, the model parameters of the feature extraction layer in the pre-trained model that has been trained can be frozen, so that in the subsequent process of adjusting the model parameters, the model parameters of the feature extraction layer in the pre-trained model that has been trained remain unchanged.

[0112] Optionally, after the pre-trained model is trained, the model parameters in the feature extraction layer can be frozen using a gradient fixation method. Developers can then use the feature extraction layer in the pre-trained model to extract token features and search features to build a search term ranking model to be trained.

[0113] Since unpopular resources have fewer sample features, increasing the number of training samples can significantly improve the feature extraction of unpopular resource sample features by the feature extraction layer in the pre-trained model. Although the feature extraction layer in the search term ranking model has been trained, the other network layers in the search term ranking model have not yet been trained. In this case, the sample features in the second sample set can be used to train the search term ranking model to be trained, resulting in a trained search term ranking model.

[0114] In one implementation, the training of the search term ranking model to be trained using the sample features in the second sample set may include steps B1 to B4:

[0115] Step B1, obtaining at least one sample feature from the second sample set as a sample feature to be used;

[0116] The second sample set may be the same as the first sample set, or may be different from the first sample set. Optionally, one or more sample features may be read from the second sample set as the sample features to be used.

[0117] In one implementation, random reading may be performed, or reading may be performed from the second sample set according to a pre-set reading rule, both of which are possible.

[0118] Step B2: inputting at least one sample feature to be utilized into the search term ranking model to be trained, and obtaining a click-through rate prediction result output by the search term ranking model for each sample feature to be utilized;

[0119] After reading the sample features to be used, at least one sample feature to be used can be input into the search term ranking model to be trained, so that each feature extraction layer in the search term ranking model can extract the features of the input sample features to be used, and then predict the click-through rate of the search term corresponding to the input sample features to be used through the extracted features, and obtain the click-through rate prediction result of each sample feature to be used.

[0120] Step B3: determining a loss function value of the search term ranking model to be trained based on the second annotation information pre-annotated for at least one sample feature to be utilized and the obtained click-through rate prediction result, as a second loss function value;

[0121] The second annotation result of any sample feature is determined based on the click-through rate of the search term corresponding to the sample feature. After obtaining the click-through rate prediction result of each sample feature to be used, the second loss function value of the search term ranking model to be trained can be determined based on the second annotation information pre-annotated for at least one sample feature to be used and the obtained click-through rate prediction result.

[0122] Optionally, in one implementation, the following formula may be used to calculate the second loss function value of the search term ranking model to be trained:

[0123]

[0124] L i =-[y i *log(p i )+(1-y i )*log(1-p i )

[0125] Where L is the second loss function value of the search term ranking model to be trained, K is the number of at least one sample feature to be used, and L i is the loss value of the i-th sample feature to be used in at least one sample feature to be used, y iis the second annotation information of the i-th sample feature to be used, p i The click-through rate prediction result output by the search term ranking model to be trained for the i-th sample feature to be utilized.

[0126] In step B4, based on the second loss function value, the model parameters in the search term ranking model to be trained are adjusted, and the process returns to step B1 until the second loss function value meets the convergence requirement, thereby obtaining a trained search term ranking model.

[0127] After obtaining the second loss function value, the model parameters of the search term ranking model to be trained can be adjusted based on the second loss function value. Since the feature extraction layer in the pre-trained model of the search term ranking model to be trained has been frozen, it is only necessary to adjust the model parameters of other feature extraction layers or network layers in the search term ranking model to be trained. Optionally, an optimization algorithm such as gradient descent or Newton's method can be used to optimize the model parameters of the search term ranking model to be trained using the second loss function value.

[0128] After the adjustment is completed, the process may return to step B1 until the second loss function value meets convergence, thereby obtaining a trained search term ranking model.

[0129] In the above scheme of the embodiment of the present invention, since the M sample feature groups are constructed by the sample features of N samples in the first sample set, the number of feature extraction layers included in the pre-training model that are trained can be increased, and because the first sample set contains first-category samples and second-category samples, and the first-category samples contain a smaller number of token words and / or search words, therefore, by increasing the number, the feature extraction layer can better extract features from token features and search features with a smaller number of retrievals, and thus the search term ranking model constructed based on the feature extraction layer has a better ability to extract features from token features and search features with a smaller number of retrievals, so that the search term ranking model can more accurately rank search terms.

[0130] like Figure 4 As shown, an embodiment of the present invention provides a search term ranking method, including steps S401-S403, wherein:

[0131] S401, obtaining at least one search term containing a target token word;

[0132] Among them, the embodiment of the present invention can be applied to a client or a server, and the client or the server is deployed with a search term ranking model obtained by training using the search term ranking model training method provided by the embodiment of the present invention.

[0133] The target token word may be a word input by the user, and the at least one search word containing the target token word is a word or sentence containing the target token word determined based on the target token word. For example, if the target token word is "movie," then the at least one search word containing movie may be "hot movie," "X's movie," "comedy movie," etc.

[0134] S402, determining feature information of at least one search term; wherein the feature information of each search term includes a search feature of the search term and a token feature of a target token term;

[0135] After obtaining at least one search term, feature information of each search term may be extracted. For a specific method, refer to the description related to step S101 , which will not be described in detail in the present invention.

[0136] S403: Input feature information of at least one search term into a pre-trained search term ranking model to obtain a ranking result of at least one search term.

[0137] After obtaining the feature information for each search term, the feature information for at least one search term can be input into a pre-trained search term ranking model to obtain a ranking result for the at least one search term. The ranking result for the at least one search term is determined based on the click-through rate prediction results for each search term using the search term ranking model. Simply put, the click-through rate prediction results for each search term are determined using the search term ranking model, and then the search terms are ranked according to their click-through rate prediction results to obtain a ranking result for the at least one search term.

[0138] In the above scheme of the embodiment of the present invention, the search term ranking model obtained by training using the search term ranking model training method provided by the embodiment of the present invention has a better ability to extract features from token features and search features with a smaller number of retrievals, making the search term ranking more accurate.

[0139] Corresponding to the search term ranking model training method provided by the above embodiment of the present invention, Figure 5 As shown, an embodiment of the present invention further provides a search term ranking model training device, the device comprising:

[0140] A feature acquisition module 501 is configured to acquire sample features of N samples in a first sample set; wherein N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first category of samples and a second category of samples, the token words and / or search words included in the first category of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second category of samples have a search quantity greater than a second quantity threshold within the specified time period, and the sample features of each sample include a token feature determined based on the token words in the sample and a search feature determined based on the search word in the sample;

[0141] A feature group construction module 502 is configured to construct M sample feature groups based on the acquired sample features, and determine first labeling information for each sample feature group; wherein M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first labeling information is a positive sample, otherwise it is a negative sample;

[0142] A first training module 503 is configured to train a pre-trained model using each sample feature group and the first labeling information of each sample feature group, and return to executing the step of obtaining sample features of N samples in the first sample set until a loss function value of the pre-trained model converges, wherein the loss function value of the pre-trained model is determined based on the positive and negative sample prediction results output by the pre-trained model for the sample feature group and the first labeling information, and the pre-trained model includes a feature extraction layer for extracting token features and search features;

[0143] The second training module 504 is used to train the search term ranking model to be trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges to obtain a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is: determined based on the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-trained model that has been trained.

[0144] Optionally, the feature group construction module is specifically used to determine a sample feature from the N sample features as a first feature to be combined; determine at least one sample feature from the N sample features as a second feature to be combined; combine the first feature to be combined and at least one second feature to be combined to obtain a sample feature group; return to execute the step of determining a sample feature from the N sample features as the first feature to be combined until the number of obtained sample feature groups reaches M.

[0145] Optionally, the first training module is specifically used to select a sample feature group from each sample feature group as the sample feature group to be used; input each sample feature contained in the sample feature group to be used into the pre-training model to obtain a classification result predicted by the pre-training model; determine the first loss function value of the pre-training model based on the obtained classification result and the first labeling information of the sample feature group to be used; adjust the model parameters in the pre-training model based on the first loss function value; return to execute the step of selecting a sample feature group from each sample feature group until all the sample feature groups are selected.

[0146] Optionally, the second training module is specifically used to obtain at least one sample feature from the second sample set as a sample feature to be used; input at least one sample feature to be used into the search term ranking model to be trained, and obtain the click-through rate prediction result of each sample feature to be used output by the search term ranking model; based on the second annotation information pre-annotated for the at least one sample feature to be used and the obtained click-through rate prediction result, determine the loss function value of the search term ranking model to be trained as the second loss function value; wherein, the second annotation result of any sample feature is determined based on the click-through rate of the search term corresponding to the sample feature; based on the second loss function value, adjust the model parameters in the search term ranking model to be trained; return to execute the step of obtaining the sample feature to be used from the second sample set until the second loss function value meets convergence, and obtain the trained search term ranking model.

[0147] Optionally, the second training module is specifically configured to calculate the loss function value of the search term ranking model to be trained using the following formula:

[0148]

[0149] L i =-[y i *log(p i )+(1-y i )*log(1-p i )

[0150] Wherein, L is the second loss function value of the search term ranking model to be trained, K is the number of the at least one sample feature to be used, and L i is the loss value of the i-th sample feature to be used in the at least one sample feature to be used, y i is the second annotation information of the i-th sample feature to be used, p iThe click-through rate prediction result output by the search term ranking model to be trained for the i-th sample feature to be utilized.

[0151] In the above scheme of the embodiment of the present invention, since the M sample feature groups are constructed by the sample features of N samples in the first sample set, the number of feature extraction layers included in the pre-training model that are trained can be increased, and because the first sample set contains first-category samples and second-category samples, and the first-category samples contain a smaller number of token words and / or search words, therefore, by increasing the number, the feature extraction layer can better extract features from token features and search features with a smaller number of retrievals, and thus the search term ranking model constructed based on the feature extraction layer has a better ability to extract features from token features and search features with a smaller number of retrievals, so that the search term ranking model can more accurately rank search terms.

[0152] Corresponding to the search word ranking method provided by the above embodiment of the present invention, as Figure 6 As shown, an embodiment of the present invention further provides a search term ranking device, the device comprising:

[0153] A search term acquisition module 601 is used to acquire at least one search term containing a target token word;

[0154] An information determination module 602 is configured to determine feature information of the at least one search term, wherein the feature information of each search term includes a search feature of the search term and a token feature of the target token term;

[0155] The information input module 603 is used to input the feature information of the at least one search term into a pre-trained search term ranking model to obtain the ranking result of the at least one search term; wherein the ranking result of the at least one search term is determined based on the click-through rate prediction result of each search term based on the search term ranking model, and the search term ranking model is a search term ranking model obtained by training using the search term ranking model training device provided by an embodiment of the present invention.

[0156] In the above scheme of the embodiment of the present invention, the search term ranking model obtained by training using the search term ranking model training method provided by the embodiment of the present invention has a better ability to extract features from token features and search features with a smaller number of retrievals, making the search term ranking more accurate.

[0157] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0158] Memory 703, for storing computer programs;

[0159] The processor 701 is configured to implement the steps of the search term ranking model training method or the search term ranking method provided in the above-mentioned embodiment of the present invention when executing the program stored in the memory 703 .

[0160] The communication bus mentioned in the terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0161] The communication interface is used for communication between the above terminal and other devices.

[0162] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0163] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0164] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, it implements the search term ranking model training method or the search term ranking method described in any of the above embodiments.

[0165] In another embodiment provided by the present invention, a computer program product containing instructions is also provided, which, when executed on a computer, enables the computer to execute the search term ranking model training method or the search term ranking method described in any one of the above embodiments.

[0166] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0167] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0168] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between other embodiments. In particular, since the apparatus, electronic device, computer-readable storage medium, and computer program product embodiments are generally similar to the method embodiments, their descriptions are relatively simplified. For related portions, reference can be made to the descriptions of the method embodiments.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A search term ranking model training method, characterized in that: The method comprises: Obtaining sample features of N samples in a first sample set; wherein N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first category of samples and a second category of samples, the token words and / or search words included in the first category of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second category of samples have a search quantity greater than a second quantity threshold within the specified time period, and the sample features of each sample include a token feature determined based on the token words in the sample and a search feature determined based on the search word in the sample; Based on the acquired sample features, M sample feature groups are constructed, and the first annotation information of each sample feature group is determined; where M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first annotation information is a positive sample, otherwise it is a negative sample; Training a pre-trained model using each sample feature group and the first labeling information of each sample feature group, and returning to the step of obtaining sample features of N samples in the first sample set until a loss function value of the pre-trained model converges, wherein the loss function value of the pre-trained model is determined based on positive and negative sample prediction results output by the pre-trained model for the sample feature group and the first labeling information, and the pre-trained model includes a feature extraction layer that extracts features from token features and search features; The search term ranking model to be trained is trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges, thereby obtaining a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is determined based on the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-trained model that has been trained.

2. The method according to claim 1, characterized in that The step of constructing M sample feature groups based on the acquired sample features includes: Determine a sample feature from the N sample features as a first feature to be combined; Determine at least one sample feature from the N sample features as a second feature to be combined; combining the first feature to be combined with at least one second feature to be combined to obtain a sample feature group; Return to the step of determining a sample feature from the N sample features as the first feature to be combined, until the number of obtained sample feature groups reaches M.

3. The method according to claim 1 or 2, characterized in that The method of training the pre-trained model by using each sample feature group and the first annotation information of each sample feature group includes: Selecting a sample feature group from each sample feature group as the sample feature group to be used; Inputting each sample feature included in the sample feature group to be utilized into a pre-trained model to obtain a classification result predicted by the pre-trained model; Determining a first loss function value of the pre-trained model based on the obtained classification result and the first labeling information of the sample feature group to be utilized; Adjusting model parameters in the pre-trained model based on the first loss function value; Return to the step of selecting a sample feature group from each sample feature group until all the sample feature groups are selected.

4. The method according to claim 1 or 2, characterized in that The method of training the search term ranking model to be trained by using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges to obtain a trained search term ranking model includes: Acquire at least one sample feature from the second sample set as a sample feature to be used; Inputting at least one sample feature to be utilized into the search term ranking model to be trained, respectively, to obtain a click-through rate prediction result output by the search term ranking model for each sample feature to be utilized; Determining a loss function value of the search term ranking model to be trained based on the second annotation information pre-annotated for the at least one sample feature to be utilized and the obtained click-through rate prediction result as the second loss function value; wherein the second annotation result of any sample feature is determined based on the click-through rate of the search term corresponding to the sample feature; Adjusting model parameters in the search term ranking model to be trained based on the second loss function value; Return to the step of obtaining the sample features to be used from the second sample set until the second loss function value meets convergence, thereby obtaining a trained search term ranking model.

5. The method according to claim 4, characterized in that The determining of the loss function value of the search term ranking model to be trained based on the second annotation information pre-annotated for the at least one sample feature to be utilized and the obtained click-through rate prediction result includes: The loss function value of the search term ranking model to be trained is calculated using the following formula: L i =-[y i *log(p i )+(1-y i )*log(1-p i ) Wherein, L is the second loss function value of the search term ranking model to be trained, K is the number of the at least one sample feature to be used, and L i is the loss value of the i-th sample feature to be used in the at least one sample feature to be used, y i is the second annotation information of the i-th sample feature to be used, p i The click-through rate prediction result output by the search term ranking model to be trained for the i-th sample feature to be utilized.

6. A search term ranking method, characterized in that: The method comprises: Obtain at least one search term containing a target token term; Determining feature information of the at least one search term; wherein the feature information of each search term includes a search feature of the search term and a token feature of the target token term; Input the feature information of the at least one search term into a pre-trained search term ranking model to obtain a ranking result of the at least one search term; wherein the ranking result of the at least one search term is determined based on the click-through rate prediction result of each search term based on the search term ranking model, and the search term ranking model is a search term ranking model obtained by training using any one of the methods described in claims 1-5.

7. A search term ranking model training device, characterized in that: The device comprises: a feature acquisition module, configured to acquire sample features of N samples in a first sample set; wherein N is a positive integer, each sample includes a token word and a search word, each search word includes the token word in the sample in which it is located, the first sample set includes a first category of samples and a second category of samples, the token words and / or search words included in the first category of samples have a search quantity less than a first quantity threshold within a specified time period, the token words and / or search words included in the second category of samples have a search quantity greater than a second quantity threshold within the specified time period, and the sample features of each sample include a token feature determined based on the token words in the sample and a search feature determined based on the search word in the sample; A feature group construction module is used to construct M sample feature groups based on the acquired sample features, and determine the first labeling information of each sample feature group; wherein M is a positive integer greater than N; if the token feature and the search feature in each sample feature in the sample feature group are the same, then the first labeling information is a positive sample, otherwise it is a negative sample; a first training module, configured to train a pre-trained model using each sample feature group and first labeling information of each sample feature group, and return to executing the step of obtaining sample features of N samples in the first sample set until a loss function value of the pre-trained model converges, wherein the loss function value of the pre-trained model is determined based on positive and negative sample prediction results output by the pre-trained model for the sample feature group and the first labeling information, and the pre-trained model includes a feature extraction layer for extracting features from token features and search features; The second training module is used to train the search term ranking model to be trained using the sample features in the second sample set until the loss function value of the search term ranking model to be trained converges to obtain a trained search term ranking model; wherein the loss function value of the search term ranking model to be trained is: determined based on the click-through rate prediction result output by the search term ranking model to be trained for the sample features and the marked click-through rate, and the search term ranking model to be trained is constructed based on the feature extraction layer in the pre-training model that has been trained.

8. A search term ranking device, characterized in that: The device comprises: A search term acquisition module, configured to acquire at least one search term containing a target token word; An information determination module, configured to determine feature information of the at least one search term; wherein the feature information of each search term includes a search feature of the search term and a token feature of the target token term; An information input module is used to input the feature information of the at least one search term into a pre-trained search term ranking model to obtain a ranking result of the at least one search term; wherein the ranking result of the at least one search term is determined based on the click-through rate prediction result of each search term based on the search term ranking model, and the search term ranking model is a search term ranking model obtained by training using the device described in claim 7.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 5 or 6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps of any one of claims 1 to 5 or 6 are implemented.

Images