Search term processing method and device, equipment, storage medium and program product

Abstract

This application provides a method, apparatus, device, storage medium, and program product for processing search terms, applicable to video search, which can improve the deduplication coverage of search terms. In this application, multiple candidate search terms corresponding to the input text are determined; based on the semantic representations of any two candidate search terms and the input text, the semantic repetition between any two candidate search terms is determined; the repetition between the search results of any two candidate search terms is calculated to obtain the repetition between any two candidate search terms based on the search results; the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms is calculated to obtain the repetition between any two candidate search terms based on object click behavior; and deduplication of candidate search terms is performed based on the semantic repetition, the search result repetition, and the object click behavior repetition.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, computer device, storage medium, and computer program product for processing search terms. Background Technology

[0002] With the rapid development of internet technology and smart devices, users can search for information on various platforms, such as searching for videos, products, articles, answers, and so on. Typically, to facilitate user searches, after a user enters text, the platform provides a list of more informative and targeted candidate search terms, allowing the user to select a term before triggering the search.

[0003] However, the number of candidate search terms currently provided by the platform is limited. Even with this limited number, there is still a high degree of duplication among the candidate search terms. For example, there may be two candidate search terms: "films and television works starring Alice" and "movies starring Alice". This reduces the number of candidate search terms that users can actually choose from, resulting in decreased search efficiency and a negative impact on the search experience. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer device, computer-readable storage medium, and computer program product for accurately deduplicating candidate search terms to improve search efficiency, thereby addressing the aforementioned technical problems.

[0005] This application provides a method for processing search terms, the method comprising:

[0006] Identify multiple candidate search terms corresponding to the input text;

[0007] Based on the semantic representations of any two candidate search terms among the plurality of candidate search terms and the input text, the degree of repetition between any two candidate search terms based on text semantics is determined;

[0008] The degree of repetition between the search results of any two candidate search terms is calculated to obtain the degree of repetition between any two candidate search terms based on the search results.

[0009] The redundancy between the tag features corresponding to the historical click object groups of any two candidate search terms is calculated to obtain the redundancy between any two candidate search terms based on object click behavior; the tag features represent the interest distribution of the historical click object groups on the content of various tags;

[0010] The multiple candidate search terms are deduplicated based on the degree of repetition between any two candidate search terms based on textual semantics, the degree of repetition based on search results, and the degree of repetition based on object click behavior.

[0011] This application provides a search term processing apparatus, the apparatus comprising:

[0012] The candidate search term determination module is used to determine multiple candidate search terms corresponding to the input text;

[0013] The first repetition acquisition module is used to determine the repetition degree between any two candidate search terms based on the semantic representation of the input text, based on the semantic representation of any two candidate search terms among the plurality of candidate search terms.

[0014] The second repetition acquisition module is used to calculate the repetition between the search results of any two candidate search terms, and obtain the repetition between any two candidate search terms based on the search results.

[0015] The third repetition acquisition module is used to calculate the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms, and obtain the repetition between any two candidate search terms based on object click behavior; the tag features represent the interest distribution of the historical click object group on the content of various tags;

[0016] The deduplication module is used to deduplicatize the multiple candidate search terms based on the degree of repetition between any two candidate search terms based on textual semantics, the degree of repetition based on search results, and the degree of repetition based on object click behavior.

[0017] In one embodiment, the candidate search term determination module is further configured to: acquire input text; query a candidate search term database to obtain candidate search terms that match the input text; and sort the candidate search terms that match the input text according to the relevance of the input text to the candidate search terms, the popularity of the candidate search terms, and the historical click rate, thereby obtaining multiple candidate search terms corresponding to the input text.

[0018] In one embodiment, the first repetition acquisition module is further configured to input the arbitrary two candidate search terms and the input text into a semantic repetition prediction model; obtain the deep semantic representations of the arbitrary two candidate search terms and the input text respectively through the semantic representation network of the semantic repetition prediction model, and then perform concatenation processing to obtain the concatenated deep semantic representation; predict the text semantic repetition between the arbitrary two candidate search terms based on the concatenated deep semantic representation through the fully connected network connected to the semantic representation network in the semantic repetition prediction model.

[0019] In one embodiment, the apparatus includes a first training module for acquiring training samples of the semantic repetition prediction model. The training samples of the semantic repetition prediction model include positive samples and negative samples. The positive samples consist of the input text and candidate search terms marked as duplicates in the candidate search terms of the input text. The negative samples consist of the input text and candidate search terms not marked as duplicates in the candidate search terms of the input text. Based on the positive and negative samples included in the training samples of the semantic repetition prediction model, the network parameters of the semantic representation network and the fully connected network in the semantic repetition prediction model are optimized until training stops.

[0020] In one embodiment, the any two candidate search terms include a first candidate search term and a second candidate search term; the second repetition acquisition module is further configured to acquire a first search result list corresponding to the first candidate search term; acquire a second search result list corresponding to the second candidate search term; and calculate the repetition between multiple search results ranked at the top of the first search result list and multiple search results ranked at the top of the second search result list to obtain the repetition between the first candidate search term and the second candidate search term based on search results, wherein the higher the ranking of the search result, the higher the relevance between it and the corresponding candidate search term.

[0021] In one embodiment, the second repetition acquisition module is further configured to determine the video intersection and video union formed by the top-ranked videos in the first search results list and the top-ranked videos in the second search results list; for each video in the video union, the duration, number of plays within a first preset time period, and average playback completeness of each video are statistically analyzed to obtain the statistical value of each video; for each video in the video intersection, the duration, number of plays within a first preset time period, and average playback completeness of each video are statistically analyzed to obtain the statistical value of each video; based on the ratio of the sum of the statistical values ​​of each video in the video intersection to the sum of the statistical values ​​of each video in the video union, the repetition degree between the first candidate search term and the second candidate search term based on the search results is obtained.

[0022] In one embodiment, the device further includes a completeness acquisition module, configured to acquire the playback duration and playback count of the video within a second preset time period; and to use the ratio between the playback duration and playback count of the video within the second preset time period as the average playback completeness of the video.

[0023] In one embodiment, the third repetition acquisition module is further configured to determine the historical click object group of the candidate search term; for each historical click object in the historical click object group, obtain the interest distribution of each historical click object in the content of various tags according to the viewing records of the historical click object; calculate the mean of the interest distribution of each historical click object in the content of various tags in the historical click object group of the candidate search term to obtain the tag features corresponding to the historical click object group of the candidate search term; and for any two candidate search terms, obtain the repetition between any two candidate search terms based on object click behavior according to the repetition between the tag features corresponding to the historical click object groups of the two candidate search terms.

[0024] In one embodiment, the third repetition acquisition module is further configured to calculate the cosine distance between the tag features corresponding to the historical click object groups of any two candidate search terms, and based on the cosine distance, obtain the repetition degree between any two candidate search terms based on object click behavior, wherein the repetition degree based on object click behavior is inversely correlated with the cosine distance.

[0025] In one embodiment, the search results obtained based on the search terms are videos. The third repetition acquisition module is further used to determine various tags pre-set for the videos; obtain the tags of the videos watched by the historical click object according to the viewing records of the historical click object; for each tag, determine the historical click object's interest in videos of each tag according to the video duration, playback completion rate, and playback time of the corresponding videos watched by the historical click object; normalize the historical click object's interest in videos of each tag to obtain the distribution of the historical click object's interest in videos of each tag.

[0026] In one embodiment, the deduplication module is further configured to perform a weighted summation of the repetition based on text semantics, the repetition based on search results, and the repetition based on object click behavior between any two candidate search terms to obtain the repetition between any two candidate search terms; based on the repetition between any two candidate search terms, identify multiple repetition groups from the plurality of candidate search terms, wherein the repetition between candidate search terms belonging to the same repetition group is higher than a set threshold, and the repetition between candidate search terms belonging to different repetition groups is lower than a set threshold; and perform deduplication on candidate search terms belonging to the same repetition group.

[0027] In one embodiment, the deduplication module is further configured to construct a candidate search term distance graph based on the repetition between any two candidate search terms; the nodes in the candidate search term distance graph represent candidate search terms, and the distance between nodes is inversely correlated with the repetition between candidate search terms; the candidate search term distance graph is mined to obtain multiple repetition groups.

[0028] In one embodiment, the deduplication module is further configured to obtain the object tag sequence corresponding to the object into which the input text is input; for a repeating group with a number of candidate search terms greater than 1, input any candidate search term in the repeating group, the object tag sequence, and the input text into the click probability prediction model, and output the click probability of the object for any candidate search term in the repeating group; within the repeating group, remove candidate search terms with click probabilities lower than a threshold to obtain deduplicated candidate search terms.

[0029] In one embodiment, the deduplication module is further configured to input any candidate search term within the repeating group, the object tag sequence, and the input text into a click probability prediction model; obtain the deep semantic representations of the candidate search term, the object tag sequence, and the input text respectively through the semantic representation network of the click probability prediction model, and then concatenate them to obtain the concatenated deep semantic representation; predict the click probability of the object for the candidate search term based on the concatenated deep semantic representation through the fully connected network connected to the semantic representation network in the click probability prediction model.

[0030] In one embodiment, the apparatus further includes a second training module for acquiring training samples of the click probability prediction model. The training samples of the click probability prediction model include positive samples and negative samples. The positive samples consist of sample input text, candidate search terms for objects that have been clicked by the input of the sample input text, and object label sequences for the objects that have been clicked by the input of the sample input text. The negative samples consist of the sample input text, candidate search terms for objects that have not been clicked by the input of the sample input text, and object label sequences for the objects that have been clicked by the input of the sample input text. Based on the positive and negative samples included in the training samples of the click probability prediction model, the network parameters of the semantic representation network and the fully connected network in the click prediction probability model are optimized until training stops.

[0031] This application provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor executes the above-described method for processing search terms.

[0032] This application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor using the above-described method for processing search terms.

[0033] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-mentioned method for processing search terms.

[0034] The aforementioned search term processing method, apparatus, computer equipment, storage medium, and computer program product, after obtaining multiple candidate search terms corresponding to the input text, determine the degree of repetition between any two candidate search terms from multiple dimensions. Specifically: based on the semantic representation of the input text, the semantic representations of any two candidate search terms are mined to determine the degree of repetition between any two candidate search terms based on text semantics; based on the search results of any two candidate search terms, the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the search results; furthermore, the historical click object groups of any two candidate search terms are determined, and the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the interest distribution of the historical click object groups on the content of various tags. This is not limited to starting from the candidate search terms themselves, but expands the information dimensions that deduplication can rely on, which helps to improve the deduplication effect; finally, based on the multi-dimensional repetition of any two search terms, candidate search terms are deduplicated, improving the deduplication effect of candidate search terms, and ultimately improving the user's search efficiency and search experience. Attached Figure Description

[0035] Figure 1 This is an application environment diagram of a method for processing search terms in one embodiment;

[0036] Figure 2 This is a flowchart illustrating a method for processing search terms in one embodiment;

[0037] Figure 3 This is a schematic diagram of a video search interface in one embodiment;

[0038] Figure 4 This is a flowchart illustrating the process of calculating text semantic-based repetition in one embodiment;

[0039] Figure 5 This is a distance map of candidate search terms in one embodiment;

[0040] Figure 6 This is a flowchart illustrating the process of predicting the click probability of a first candidate search term in one embodiment;

[0041] Figure 7 This is a framework diagram of a method for processing search terms in one embodiment;

[0042] Figure 8 This is a flowchart illustrating the method for processing search terms in another embodiment;

[0043] Figure 9 This is a structural block diagram of a word search processing device in one embodiment;

[0044] Figure 10This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. Those skilled in the art will understand, explicitly and implicitly, that the embodiments described in this application can be combined with other embodiments. It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0047] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0048] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0049] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0050] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0051] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, smart customer service, vehicle networking, and intelligent transportation. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0052] Figure 1 This diagram illustrates the application environment of a search term processing method in one embodiment. Terminal 102 and server 104 communicate via a communication network. Terminal 102 can interact with server 104 through the communication network; terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server, a server cluster consisting of multiple servers, or a cloud server. Server 104 can utilize a data storage system to store candidate search terms; this data storage system can be integrated into server 104 or set up separately from server 104.

[0053] In one embodiment, after obtaining the input text, the terminal 102 can send the input text to the server 104. The server 104 determines multiple candidate search terms corresponding to the input text. Based on the semantic representations of any two candidate search terms and the input text, the server 104 can determine the semantic repetition between any two candidate search terms. The server 104 can calculate the repetition between the search results of any two candidate search terms to obtain the repetition between any two candidate search terms based on the search results. The server 104 can calculate the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms to obtain the repetition between any two candidate search terms based on object click behavior. Tag features represent the distribution of interest of the historical click object group to the content of various tags. Based on the semantic repetition, the search result repetition, and the object click behavior repetition between any two candidate search terms, the server 104 can deduplicate the multiple candidate search terms and feed back the deduplicated candidate search terms to the terminal 102.

[0054] Figure 2 This is a flowchart illustrating a method for processing search terms in one embodiment. This method can be executed by a server or a terminal, or jointly by both. In this embodiment, the method is described using server execution as an example. The method includes the following steps:

[0055] Step S202: Determine multiple candidate search terms corresponding to the input text.

[0056] When an individual wants to find relevant videos, products, or other content on platforms such as video platforms or shopping platforms, they can enter relevant text, which can be called input text (input text belongs to the search query), so that the platform can search for the relevant content.

[0057] Search terms can be search suggestions built based on the input text, allowing an object to select the search terms it wants from the search terms without having to enter the complete text, or only a part of the text, so that the platform can perform content search based on the candidate search terms selected by the object.

[0058] Let's take video search as an example:

[0059] Figure 3 This is a schematic diagram of a video search interface in one embodiment. (Refer to...) Figure 3The video search interface 301 can be displayed on the terminal. When an object wants to find movies or TV shows starring Alice, it can enter "Alice" in the video search box 3011 of the video search interface 301. The terminal can send "Alice" as input text to the server. After receiving the input text, the server can construct multiple search terms based on the input text and return them to the terminal. The search terms returned by the server to the terminal can include: "Alice 2021 New Year's Eve", "Alice's participation in variety shows", "Works co-starring Alice and Bob", "Film and television works starring Alice", and "Movies starring Alice". After receiving multiple search terms from the server, the terminal can display them on the video search interface 301. Figure 3 The video search interface 301 can display various search terms in the search term bars 3012 to 3016 respectively. When a user selects the search term "films starring Alice" and clicks the search control 3018, the terminal can send the selected search term "films starring Alice" to the server. The server performs a video search based on the search term "films starring Alice" and sends the search results back to the terminal. Additionally, if a user wants to delete text from the video search box 3011, they can click the delete control 3017.

[0060] The search terms "films and television works starring Alice" and "movies starring Alice" are duplicates. Displaying both simultaneously to the user might impair search efficiency. Therefore, after initially identifying multiple search terms based on the input text, the server performs deduplication on these terms and then sends the deduplicated search terms back to the user. These initially identified search terms can be referred to as candidate search terms.

[0061] After the server obtains multiple candidate search terms based on the input text, it can perform a multi-dimensional repetition calculation on every two candidate search terms. The multi-dimensional repetition calculation is described in steps S204 to S208.

[0062] In one embodiment, when determining multiple candidate search terms corresponding to input text, the server may perform the following steps: obtain the input text; query the candidate search term database to obtain candidate search terms that match the input text; sort the candidate search terms that match the input text according to the relevance between the input text and the candidate search terms, the popularity of the candidate search terms, and the historical click rate, to obtain multiple candidate search terms corresponding to the input text.

[0063] The candidate search term database includes a large number of candidate search terms. These terms are typically determined based on the search content on the server; for example, the database can be built based on the description information of the search content and historical input text. In a video search scenario, the description information of the search results might include the video title, the names of the actors, the character names, and the music. In a product search scenario, the description information of the search results might include the product name, product size, and product usage.

[0064] The above content is illustrated using video search as an example:

[0065] After receiving the input text from the object, the server queries the candidate search term database based on the input text. Using descriptive information such as video names, actor names, character names, and music included in the database, as well as historical input text from other objects, the server constructs candidate search terms that match the input text. Next, the server sorts the constructed candidate search terms according to their relevance to the input text, their popularity, and historical click-through rates. Based on the sorting results, the server then selects multiple candidate search terms corresponding to the input text from the constructed candidate search terms that match the input text for deduplication.

[0066] In the above embodiments, after obtaining the input text, candidate search terms that match the input text can be obtained based on the candidate search term database. By combining the relevance of the candidate search terms to the input text, the popularity of the candidate search terms, and the historical click rate, multiple candidate search terms for deduplication can be determined, which can improve the deduplication efficiency of search terms.

[0067] Step S204: Based on the semantic representations of any two candidate search terms and the input text, determine the semantic repetition between any two candidate search terms.

[0068] The repetition between candidate search terms is reflected not only in their literal meaning but also in their semantic meaning. In this application, semantic meaning is used as one dimension to analyze the repetition between candidate search terms. To further improve the accuracy of the semantic repetition between candidate search terms, the input text mentioned in step S202 is analyzed, serving as the context for calculating the semantic repetition between candidate search terms.

[0069] In the following description, among the two candidate search terms for calculating the repetition of text semantics, one candidate search term is referred to as the first candidate search term, and the other candidate search term is referred to as the second candidate search term.

[0070] After obtaining the input text mentioned in step S202 and the first and second candidate search terms corresponding to the input text, the server obtains the semantic representation of the input text, the semantic representation of the first candidate search term, and the semantic representation of the second candidate search term. With the input text as the context, the server combines the semantic representation of the first and second candidate search terms with the semantic representation of the input text to determine the degree of repetition between the first and second candidate search terms based on text semantics.

[0071] In one embodiment, when calculating text-based semantic repetition, the server can input any two candidate search terms and the input text into the semantic repetition prediction model; through the semantic representation network of the semantic repetition prediction model, obtain the deep semantic representations of each of the two candidate search terms and the input text respectively, and then concatenate them to obtain the concatenated deep semantic representation; through the fully connected network connected to the semantic representation network in the semantic repetition prediction model, predict the text-based semantic repetition between any two candidate search terms based on the concatenated deep semantic representation.

[0072] The semantic repetition prediction model is used to predict the repetition between candidate search terms based on textual semantics. This semantic repetition prediction model may include a semantic representation network and a fully connected network connected to the semantic representation network. The semantic representation network may be a bidirectional encoder representation network (BERT).

[0073] The semantic repetition prediction model can include multiple independent semantic representation networks, up to three in number. The network parameters of these independent semantic representation networks are shared, and they are used to extract the deep semantic representation of the input text, the deep semantic representation of the first candidate search term, and the deep semantic representation of the second candidate search term, respectively.

[0074] By extracting the deep semantic representations of the input text, the first candidate search term, and the second candidate search term from mutually independent semantic representation networks with shared network parameters, the computational efficiency of text semantic repetition can be improved.

[0075] Figure 4 This is a flowchart illustrating the calculation of text semantic-based repetition in one embodiment; see reference. Figure 4After receiving the input text, the first candidate search term, and the second candidate search term, the server inputs these terms into semantic representation networks that share network parameters, respectively, to obtain deep semantic representations of the input text, the first candidate search term, and the second candidate search term. Next, the server can concatenate these deep semantic representations to obtain a concatenated deep semantic representation. Then, the server can input this concatenated deep semantic representation into the fully connected network of a semantic repetition prediction model. The output of this model is used as the semantic repetition between the first and second candidate search terms, given the input text as context.

[0076] In the above embodiments, the accuracy of the repetition can be improved by using a pre-built semantic repetition prediction model to obtain the repetition between candidate search terms based on text semantics. Furthermore, the accuracy of the repetition can be further improved by combining the input text with text semantics-based repetition calculation.

[0077] In one embodiment, the training steps of the semantic repetition prediction model include: obtaining training samples for the semantic repetition prediction model, which include positive samples and negative samples. Positive samples consist of the input text and candidate search terms marked as duplicates in the input text, while negative samples consist of the input text and candidate search terms not marked as duplicates in the input text. Based on the positive and negative samples included in the training samples of the semantic repetition prediction model, the network parameters of the semantic representation network and the fully connected network in the semantic repetition prediction model are optimized until training stops.

[0078] The semantic repetition prediction model predicts the repetition between candidate search terms based on textual semantics within the context of the input text. Therefore, the training samples for the semantic repetition prediction model need to take the context into account.

[0079] Taking video search as an example: The server can acquire log data, including historical input text and corresponding candidate search terms. Heuristic mining is performed on the log data, using the historical input text as sample input text. Then, in the candidate search terms corresponding to this sample input text's context, duplicate candidate search terms are manually labeled. This sample input text and the labeled duplicate candidate search terms are used as positive samples in the training samples. The server can also randomly select non-duplicate candidate search terms from the candidate search terms corresponding to this sample input text's context, and label these randomly selected non-duplicate candidate search terms as non-duplicate. This sample input text and the unlabeled duplicate candidate search terms are used as negative samples in the training samples. The server can then input the positive and negative samples into the semantic repetition prediction model to optimize the network parameters of the semantic representation network and the fully connected network of the semantic repetition prediction model until training stops.

[0080] In the above embodiments, positive and negative samples are obtained with the sample input text as context. The model is trained with these positive and negative samples, so that the semantic repetition prediction model has the corresponding ability to calculate the repetition of candidate search terms based on text semantics with the input text as context.

[0081] Step S206: Calculate the degree of repetition between the search results of any two candidate search terms to obtain the degree of repetition between any two candidate search terms based on the search results.

[0082] Search results are the content obtained based on search terms. The search results will vary depending on the search scenario. For example, in a video search scenario, the search results will be videos; in a product search scenario, the search results will be products; and in an image search scenario, the search results will be images.

[0083] If the search results for two candidate search terms are basically the same, then these two candidate search terms are likely to be duplicates. In other words, the higher the consistency of the search results for two candidate search terms, the higher the degree of duplication between the two candidate search terms based on the search results.

[0084] Let's take the first and second candidate search terms as examples:

[0085] After obtaining the first candidate search term and the second candidate search term, the server performs a search using the first candidate search term to obtain the search results corresponding to the first candidate search term, and then performs a search using the second candidate search term to obtain the search results corresponding to the second candidate search term. Next, the server can compare the consistency between the search results corresponding to the first candidate search term and the search results corresponding to the second candidate search term. If the consistency is higher, it is determined that the first candidate search term and the second candidate search term have a higher degree of repetition based on the search results.

[0086] In product search scenarios, when comparing the consistency between search results for the first and second candidate search terms, the server can perform consistency comparisons based on aspects such as product links, product names, and product uses. In image search scenarios, when comparing the consistency between search results for the first and second candidate search terms, the server can perform consistency comparisons based on aspects such as image semantics and image pixel distribution. In video search scenarios, consistency comparisons can be performed based on aspects such as video duration, number of video views, and average video completeness.

[0087] Step S208: Calculate the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms to obtain the repetition between any two candidate search terms based on object click behavior; tag features represent the distribution of interest of the historical click object group to the content of various tags.

[0088] The historical click object group of the candidate search term can include historical objects that have clicked the candidate search term. These historical objects that have clicked the candidate search term can be referred to as historical click objects.

[0089] The tag features corresponding to the historical click object group represent the distribution of the historical click object group's interest in the content of various tags. The tag features corresponding to the historical click object group can be denoted as: [(tag_1,prob_1), (tag_2,prob_2), (tag_3,prob_3), ..., (tag_n,prob_n)], where tag_n represents tag n, and prob_n represents the interest of the historical click object group in the content of tag n. This interest can be represented by probability.

[0090] In the same context as the input text mentioned in step S202, if two candidate search terms are clicked by similar historical click groups, it indicates that the two candidate search terms have a relatively consistent degree of satisfaction with the group in the same context, suggesting that the two candidate search terms are likely to represent the same meaning. In other words, the two candidate search terms may be duplicate candidate search terms. Correspondingly, the lower the probability that the object clicks both candidate search terms, the lower the degree of duplication between the two candidate search terms based on the object's click behavior.

[0091] Let's take the first and second candidate search terms as examples:

[0092] After obtaining the first and second candidate search terms, the server can retrieve the tag features corresponding to the historical click target groups of the first candidate search term and record them as [(tag_1,prob_11), (tag_2,prob_21), (tag_3,prob_31), ..., (tag_n,prob_n1)], and retrieve the tag features corresponding to the historical click target groups of the second candidate search term and record them as [(tag_1,prob_12), (tag_2,prob_22), (tag_3,prob_32), ..., [(tag_n,prob_n2)]; Then, for the tag features corresponding to the historical click object group of the first candidate search term and the tag features corresponding to the historical click object group of the second candidate search term, the server calculates the consistency of the interest distribution of the content of various tags corresponding to these two tag features. The higher the consistency, the higher the degree of repetition between the tag features corresponding to the historical click object group of the first candidate search term and the tag features corresponding to the historical click object group of the second candidate search term, and the lower the degree of repetition between the first candidate search term and the second candidate search term based on object click behavior.

[0093] Step S210: Based on the textual semantic repetition, search result repetition, and object click behavior repetition between any two candidate search terms, deduplicatize multiple candidate search terms.

[0094] Let's take the first and second candidate search terms as examples:

[0095] After obtaining the textual semantic repetition, search result-based repetition, and object click behavior-based repetition between the first and second candidate search terms through steps S204 to S208, the server can combine these three dimensions to obtain the repetition between the first and second candidate search terms. The higher the repetition between the first and second candidate search terms, the more similar the meanings represented by the first and second candidate search terms are, and the greater the probability that they are duplicates. Therefore, the server can eliminate one candidate search term and retain the other candidate search term to feed back the retained candidate search term to the terminal.

[0096] In the above-mentioned method for processing search terms, after obtaining multiple candidate search terms corresponding to the input text, the degree of repetition between any two candidate search terms is determined from multiple dimensions. Specifically: based on the semantic representation of the input text, the semantic representations of any two candidate search terms are mined to determine the degree of repetition between any two candidate search terms based on text semantics; based on the search results of any two candidate search terms, the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the search results; in addition, the historical click target groups of any two candidate search terms are determined, and the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the interest distribution of the historical click target groups on the content of various tags. This method is not limited to starting from the candidate search terms themselves, but expands the information dimensions that deduplication can rely on, which helps to improve the deduplication effect; finally, based on the multi-dimensional repetition of any two search terms, candidate search terms are deduplicated to improve the deduplication effect, ultimately improving the user's search efficiency and search experience.

[0097] In one embodiment, when calculating the repetition between any two candidate search terms based on search results, the server may also obtain a first search result list corresponding to the first candidate search term; obtain a second search result list corresponding to the second candidate search term; and calculate the repetition between the top-ranked search results in the first search result list and the top-ranked search results in the second search result list to obtain the repetition between the first candidate search term and the second candidate search term based on search results, wherein the higher the ranking of the search results, the higher the relevance between the multiple search results and the corresponding candidate search term.

[0098] Taking video search, the first candidate search term, and the second candidate search term as an example, the search result is a video:

[0099] After obtaining the first and second candidate search terms, the server uses the first candidate search term as an index to search for the corresponding video in the video search index, forming a search result list for the first candidate search term, which can be called the first search result list. Then, the server can use the second candidate search term as an index to search for the corresponding video in the video search index, forming a search result list for the second candidate search term, which can be called the second search result list.

[0100] Since videos and candidate search terms are semantically related, after obtaining the first and second search result lists, the videos in each search result list can be sorted. The higher the relevance between the video and the candidate search terms, the higher the video will be ranked.

[0101] After calculating the semantic relevance between the first candidate search term and each video in the first search results list, the server sorts the videos in the first search results list, with videos having higher relevance ranking higher. Similarly, after calculating the semantic relevance between the second candidate search term and each video in the second search results list, the server sorts the videos in the second search results list, with videos having higher relevance ranking higher.

[0102] Next, the server can determine the number of videos ranked at the top of the first search results list (e.g., the top 200) and the number of videos ranked at the top of the second search results list (e.g., the top 200), and determine the degree of repetition between the number of videos ranked at the top of the first search results list and the number of videos ranked at the top of the second search results list, thereby obtaining the degree of repetition between the first candidate search term and the second candidate search term based on the search results.

[0103] In the above embodiments, based on the relevance between candidate search terms and search results, a portion of the search results are selected from the search results list for duplicate calculation, which improves the efficiency of duplicate calculation while ensuring the accuracy of duplicate calculation.

[0104] In one embodiment, in a video search scenario, the search results obtained based on search terms are videos. The server can determine the video intersection and video union formed by multiple videos ranked at the top of the first search results list and multiple videos ranked at the top of the second search results list. For each video in the video union, the server calculates its duration, number of plays within a first preset time period, and average playback completeness to obtain its statistical value. For each video in the video intersection, the server calculates its duration, number of plays within a first preset time period, and average playback completeness to obtain its statistical value. The server obtains the degree of repetition between the first and second candidate search terms based on the search results by comparing the sum of the statistical values ​​of the videos in the video intersection with the sum of the statistical values ​​of the videos in the video union.

[0105] Here, video length refers to the inherent duration of the video, such as a video with a length of 4 minutes. Video playback duration refers to the actual playback time of the video, such as a 4-minute video that is actually played for 2 minutes. Average playback completeness represents the percentage of times the video is played in its entirety; the higher the average playback completeness, the more likely the video is played in its entirety.

[0106] The video union includes multiple videos ranked high in the first search results list and multiple videos ranked high in the second search results list. Taking the first 200 as an example, the video union can include the first 200 videos in the first search results list and the first 200 videos in the second search results list.

[0107] The video intersection includes videos that appear repeatedly in both the first and second search results lists; for example, if a video appears in both the first and second search results lists, then that video can be considered part of the video intersection.

[0108] Let's take the example of a video union including video a, video b, video c, and video d, and a video intersection including video a and video d:

[0109] After determining the union and intersection of the top-ranked videos in the first search results list and the top-ranked videos in the second search results list, the server performs statistical analysis on each video in the union to obtain a corresponding statistical value. Taking video 'a' as an example: the server can obtain the duration of video 'a', the number of times video 'a' was played within a first preset time period T1, and the average playback completeness of video 'a', and then calculate the statistical value of video 'a' based on these factors. Specifically, the method for calculating the duration, number of plays, and average playback completeness of video 'a' can be by multiplying these factors together; the statistical value of video 'a' is the product result.

[0110] Following the same method used to calculate the statistical value of video a, the server calculates the statistical values ​​of all other videos in the video set, obtaining the statistical values ​​of video b, video c, and video d. Next, the server can calculate the sum of the statistical values ​​of video a, video b, video c, and video d within the video set, and denoted as CA.

[0111] For each video in the video intersection, such as the statistical values ​​of video a and video d, the above calculation results can be directly reused to obtain the statistical values ​​of video a and video d. Then, the server can calculate the sum of the statistical values ​​of all videos in the video intersection, that is, the sum of the statistical values ​​of video a and video d, and denoted as CI.

[0112] Next, the server can obtain the overlap (denoted as P_search) between the top-ranked search results in the first search results list and the top-ranked search results in the second search results list based on CA and CI. CA and P_search are inversely correlated, while CI and P_search are positively correlated. Specifically, the server can use the ratio of CI to CA as P_search, i.e., P_search = CI / CA, and use P_search as the overlap between the first and second candidate search terms based on the search results.

[0113] In the above embodiments, the repetition rate between candidate search terms based on search results is obtained from three aspects: video duration, number of times the video is played within a first preset time period, and average video playback completeness, which can improve the accuracy of the repetition rate.

[0114] In one embodiment, regarding the average playback completeness of a video, the server can obtain the playback duration and number of plays of the video within a second preset time period; and use the ratio between the playback duration and the number of plays of the video within the second preset time period as the average playback completeness of the video.

[0115] The second preset time period can be the same as or different from the first preset time period. When the second preset time period is different from the first preset time period, the second preset time period can be longer than the first preset time period.

[0116] Taking video a as an example: the playback duration of video a within the second preset time period, and the number of times video a is played within the second preset time period. Then, the server can use the ratio between the playback duration and the number of times video a is played within the second preset time period as the average playback completeness of video a.

[0117] In the above embodiments, the average playback completeness of the video is obtained based on the ratio between the playback duration and the number of playbacks within the second preset time period, which can more comprehensively and accurately reflect the situation of the video being played in its entirety.

[0118] In one embodiment, when calculating the repetition between any two candidate search terms based on object click behavior, the server can determine the historical click object group of the candidate search terms. For each historical click object in the historical click object group, based on the viewing history of the historical click object, the server obtains the interest distribution of each historical click object in the content of various tags. The server calculates the mean of the interest distribution of each historical click object in the content of various tags in the historical click object group of the candidate search terms to obtain the tag features corresponding to the historical click object group of the candidate search terms. For any two candidate search terms, the server obtains the repetition between any two candidate search terms based on the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms.

[0119] The historical click targets of a candidate search term are: objects that have previously clicked on that candidate search term. To improve the repetition rate based on object click behavior, the context of the input text can be incorporated; under the context of the input text, the historical click targets of a candidate search term are: objects that have previously clicked on that candidate search term when the input text is the same as the input text.

[0120] Suppose a video platform has n tags: tag_1, tag_2, ..., tag_n, and the videos published on this video platform are labeled with the corresponding tags.

[0121] Taking the calculation of the tag features corresponding to the historical click group of the first candidate search term as an example:

[0122] If the input text is the same as the input text mentioned in step S202, objects that have clicked on the first candidate search term are considered as historical click objects of the first candidate search term and form a historical click object group for the first candidate search term. The interest distribution of each historical click object for the first candidate search term across various tags is then obtained. If the historical click object group for the first candidate search term includes user_1 and user_2, then the calculation method for the interest distribution of user_1 across various tags can be:

[0123] Based on user_1's viewing history on the video platform, we can determine user_1's interest in the content of various tags and obtain the distribution of user_1's interest in the content of various tags, denoted as [(tag_1,prob_user_11), (tag_2,prob_user_12), (tag_3,prob_user_13), ..., (tag_n,prob_user_1n)].

[0124] Similarly, the interest distribution of user_1 in the content of various tags can be obtained in the same way as above, denoted as [(tag_1,prob_user_21), (tag_2,prob_user_22), (tag_3,prob_user_23), ..., (tag_n,prob_user_2n)].

[0125] Then, for the same type of tag, the mean interest of each historical click object of the first candidate search term under that type of tag is calculated. For example, for tag_1, the mean of prob_user_11 and prob_user_21 is calculated and recorded as prob_11. The server can obtain the mean interest of the historical click object group of the first candidate search term under various types of tags, resulting in [(tag_1,prob_11), (tag_2,prob_21), (tag_3,prob_31), ..., (tag_n,prob_n1)], which can be called the tag features corresponding to the historical click object group of the first candidate search term.

[0126] The server can calculate the tag features corresponding to the historical click object group of the second candidate search term using the same method as described above. The calculated features are: [(tag_1,prob_12), (tag_2,prob_22), (tag_3,prob_32), ..., (tag_n,prob_n2)]. Based on the repetition between the tag features of the historical click object group of the first and second candidate search terms, the server can determine the repetition degree between them based on object click behavior. A higher repetition degree between the tag features of the historical click object group of the first and second candidate search terms indicates a lower repetition degree based on object click behavior.

[0127] In the above embodiments, based on the viewing records of historical click objects of candidate search terms, the interest distribution of historical click objects in the content of various tags is obtained. By means of the average, the tag features corresponding to the historical click object group of candidate search terms are obtained, which more accurately reflects the interest of the historical click object group of candidate search terms in the content of various tags, and improves the accuracy of the repetition of candidate search terms based on object click behavior.

[0128] In one embodiment, the server can calculate the cosine distance between the tag features corresponding to the historical click object groups of any two candidate search terms. Based on the cosine distance, the repetition degree between any two candidate search terms based on object click behavior is obtained. The repetition degree based on object click behavior is inversely correlated with the cosine distance.

[0129] Let's take the first and second candidate search terms as examples:

[0130] After the server obtains the tag features [(tag_1,prob_11), (tag_2,prob_21), (tag_3,prob_31), ..., (tag_n,prob_n1)] corresponding to the historical click object group of the first candidate search term, and the tag features [(tag_1,prob_12), (tag_2,prob_22), (tag_3,prob_32), ..., (tag_n,prob_n2)] corresponding to the historical click object group of the second candidate search term, it can calculate the cosine distance between the two. Based on the cosine distance, the repetition degree (denoted as P_user) based on object click behavior between the first and second candidate search terms is obtained. This repetition degree based on object click behavior is inversely correlated with the cosine distance; the specific calculation method can be: P_user = 1 - cosine distance.

[0131] In the above embodiments, based on the cosine distance between the tag features corresponding to the historical click object groups of candidate search terms, and given that the cosine distance and the repetition based on object click behavior are inversely correlated, the repetition based on object click behavior among candidate search terms is obtained, thereby improving the accuracy of the repetition based on object click behavior.

[0132] In one embodiment, regarding the distribution of interest in content across various tags for each historical click object, the server can determine the various tags pre-defined for the video; based on the viewing history of the historical click object, the tags of the videos watched by the historical click object are obtained; for each tag, the interest of the historical click object in the videos of each tag is determined based on the video duration, completion rate, and playback time of the corresponding videos watched by the historical click object; the interest of the historical click object in the videos of each tag is normalized to obtain the distribution of the historical click object's interest in the videos of each tag.

[0133] The playback completion rate of a viewed video represents the ratio between the video content watched by the object and the complete video content. A higher completion rate indicates that the watched video content is closer to the complete video content. Specifically, it can be determined by the ratio between the video's playback duration and its total length. The playback time point of the viewed video is the moment the object watches the video.

[0134] Let's take calculating the interest of user_1, the historical click target of the first candidate search term, in the video of tag_1 as an example:

[0135] On video platforms, various tags can be pre-defined for videos. Based on user_1's viewing history, it can be determined which videos user_1 watched and the tags assigned to these videos. When calculating user_1's interest in tag_1, the video duration of the videos belonging to tag_1 watched by user_1, the completion rate of user_1 for the videos belonging to tag_1, and the playback time of the videos belonging to tag_1 watched by user_1 can be obtained to obtain user_1's interest in the videos belonging to tag_1. The specific calculation method can be: user_1's interest in the videos belonging to tag_1 = (video duration of the videos belonging to tag_1 watched by user_1 × completion rate of user_1 for the videos belonging to tag_1) / log(e + number of weeks since the playback time of the videos belonging to tag_1 watched by user_1).

[0136] Following the method described above for calculating user_1's interest in the video of tag_1, the server can obtain user_1's interest in tag_2, tag_3, ..., tag_n, and normalize each interest level to form the distribution of user_1's interest in various tags [(tag_1,prob_user_11), (tag_2,prob_user_12), (tag_3,prob_user_13), ..., (tag_n,prob_user_1n)].

[0137] In the above embodiments, when calculating the interest distribution of historical click objects in videos of various tags, the video duration, playback completion rate, and playback time of the videos of various tags watched by the historical click objects are combined. This allows the interest distribution to more accurately reflect the degree of interest of historical click objects in videos of various tags. Furthermore, normalizing the interest of historical click objects in videos of various tags allows for comparison and calculation on the same dimension, improving the accuracy of the repetition of candidate search terms based on object click behavior.

[0138] In one embodiment, the server can perform a weighted summation of the repetition between any two candidate search terms based on textual semantics, search results, and object click behavior to obtain the repetition between any two candidate search terms; based on the repetition between any two candidate search terms, multiple repetition groups are identified from multiple candidate search terms; among the multiple repetition groups, the repetition between candidate search terms belonging to the same repetition group is higher than a set threshold, and the repetition between candidate search terms belonging to different repetition groups is lower than a set threshold; and the candidate search terms belonging to the same repetition group are deduplicated.

[0139] Among them, the repetition based on text semantics is denoted as P_text, the repetition based on search results is denoted as P_search, and the repetition based on object click behavior is denoted as P_user. Taking the first candidate search term and the second candidate search term as examples:

[0140] After obtaining the text-based semantic repetition P_text, the search result-based repetition P_search, and the object click behavior-based repetition P_user between the first and second candidate search terms, the server can perform a weighted sum of these repetitions according to preset weights to obtain the repetition degree between the first and second candidate search terms. Specifically, the calculation method can be: Repetition degree between the first and second candidate search terms = w1 × P_text + w2 × P_search + w3 × P_user; where w1, w2, and w3 can be floating-point weights, and w1 + w2 + w3 = 1.0. The server can perform a grid search on the dataset to determine reasonable values ​​for w1, w2, and w3, such as w1 = 0.5, w2 = 0.15, and w3 = 0.35.

[0141] For the multiple candidate search terms of the input text mentioned in step S202, the server can obtain the repetition between every two candidate search terms in the above manner, and based on the repetition between every two candidate search terms, identify multiple repetition groups from the multiple candidate search terms. Among them, the repetition between candidate search terms in the same repetition group is higher than a set threshold, and the repetition between candidate search terms in different repetition groups is lower than the set threshold. After obtaining multiple repetition groups, the server can determine whether to deduplicate the candidate search terms in the same repetition group based on the number of candidate search terms included in the repetition group. For example, if a repetition group includes only one candidate search term, then deduplication of the candidate search terms in the repetition group is not required. Or, for example, if a repetition group includes two or more candidate search terms, then deduplication of the candidate search terms in the repetition group can be performed.

[0142] The deduplication process for candidate search terms within the same repeating group can be performed by: randomly selecting one candidate search term to retain, or retaining one candidate search term based on the interests of the object of the input text mentioned in step S202.

[0143] In the above embodiments, after obtaining the repetition degree between candidate search terms based on text semantics, the repetition degree based on search results, and the repetition degree based on object click behavior, a weighted sum can be performed according to a preset weight. Based on the repetition degree obtained by the weighted sum, grouping and deduplication can be performed, which can improve the accuracy of deduplication of candidate search terms.

[0144] In one embodiment, the server can construct a candidate search term distance graph based on the repetition between any two candidate search terms; the nodes in the candidate search term distance graph represent candidate search terms, and the distance between nodes is inversely correlated with the repetition between candidate search terms; the candidate search term distance graph is mined to obtain multiple repetition groups.

[0145] For the multiple candidate search terms of the input text mentioned in step S202, the server can construct a distance map of candidate search terms by using the candidate search terms as nodes and determining the distance between nodes based on the repetition between the candidate search terms; wherein, the higher the repetition between candidate search terms, the farther the distance between the corresponding nodes.

[0146] Figure 5 As shown in one embodiment, a distance map of candidate search terms is provided. Figure 5 Using the candidate search terms of the input text mentioned in step S202 as nodes, nodes 1 to 11 are formed. The greater the distance between nodes, the lower the repetition between the corresponding candidate search terms. Next, the server can mine the candidate search term distance graph to obtain multiple repetition groups. When mining the candidate search term distance graph, the server can use a community detection algorithm (such as a graph algorithm based on modularity) to iteratively mine the candidate search term distance graph to ensure that the distance between candidate search terms within the same repetition group is less than a threshold, and the distance between repetition groups is greater than the threshold.

[0147] In the above embodiments, after obtaining the repetition between candidate search terms by comprehensively considering the repetition across multiple dimensions, the candidate search terms can be grouped using graph mining to improve the deduplication efficiency of candidate search terms.

[0148] In one embodiment, the server can obtain the object tag sequence corresponding to the object in the input text; for repeated groups with more than 1 candidate search terms, the server inputs any candidate search term, object tag sequence, and input text into the click probability prediction model, and outputs the click probability of the object for any candidate search term in the repeated group; within the repeated group, the server removes candidate search terms with click probabilities lower than a threshold to obtain deduplicated candidate search terms.

[0149] The object tag sequence is used to represent the tags to which the content the object is interested in belongs. For example, if the object is interested in videos with tags_1 and tag_3, the object tag sequence for that object would be [tag_1, tag_3]. The click probability prediction model can be used to predict the probability that each candidate search term corresponding to the input text mentioned in step S202 will be clicked by the object that input the input text mentioned in step S202, given the context of the input text mentioned in step S202.

[0150] For duplicate groups containing more than one candidate search term, the server can deduplicate the candidate search terms within the duplicate group and retain one of the candidate search terms.

[0151] Let's take a specific repeating group that includes the first candidate search term and the second candidate search term as an example:

[0152] The server can obtain the object tag sequence of the object inputting the text mentioned in step S202, and input this object tag sequence, along with the first candidate search term and the input text mentioned in step S202, into a click probability prediction model. The output of the click probability prediction model is taken as the probability that the first candidate search term is clicked by the object inputting the text mentioned in step S202 within the context of the input text mentioned in step S202, i.e., the click probability of the object for the first candidate search term. Similarly, the server can obtain the click probability of the object for the second candidate search term and remove candidate search terms with click probabilities below a threshold to obtain deduplicated candidate search terms. When removing the first and second candidate search terms, the server can remove candidate search terms with low click probabilities and retain candidate search terms with high click probabilities.

[0153] In the above embodiments, the input text is used as context, and the click probability of the object to the candidate search terms is obtained by combining the object's object tag sequence and the click probability prediction model to remove duplicates. Taking object tags into account, personalized deduplication is achieved.

[0154] In one embodiment, the server can input any candidate search term, object tag sequence, and input text from the repeating group into the click probability prediction model; through the semantic representation network of the click probability prediction model, the deep semantic representations of the candidate search term, the object tag sequence, and the input text are obtained respectively, and then concatenated to obtain the concatenated deep semantic representation; through the fully connected network in the click probability prediction model connected to the semantic representation network, the click probability of the object for the candidate search term is predicted based on the concatenated deep semantic representation.

[0155] Click probability prediction models can include a semantic representation network, a fully connected network connected to the semantic representation network, and a classification network connected to the fully connected network. The semantic representation network can be a bidirectional encoder representation network.

[0156] The click probability prediction model can include multiple independent semantic representation networks, up to three in number. The network parameters of these independent semantic representation networks are shared, and they are used to extract deep semantic representations of the input text, deep semantic representations of candidate search terms, and deep semantic representations of object label sequences, respectively.

[0157] By extracting deep semantic representations of the input text, candidate search terms, and object label sequences from mutually independent semantic representation networks with shared network parameters, the prediction efficiency of click probability can be improved.

[0158] Let's take calculating the click probability of an object for the first candidate search term as an example:

[0159] Figure 6 This is a flowchart illustrating the process of predicting the click probability for a first candidate search term in one embodiment; see reference. Figure 6 After obtaining the input text, the first candidate search term, and the corresponding object label sequence mentioned in step S202, the server inputs the input text, the first candidate search term, and the corresponding object label sequence into the semantic representation networks that share network parameters, respectively, to obtain the deep semantic representation of the input text, the first candidate search term, and the object label sequence mentioned in step S202. These are then concatenated to obtain the concatenated deep semantic representation. Next, the server inputs the concatenated deep semantic representation into the fully connected network connected to the semantic representation network in the click probability prediction model, and inputs the output of the fully connected network into the classification network. The output of the classification network is used as the click probability of the object for the first candidate search term.

[0160] In the above embodiments, the pre-built click probability prediction model is used to obtain the click probability of an object on candidate search terms under the context of input text, thereby improving the accuracy of click probability and realizing the deduplication of personalized search terms.

[0161] In one embodiment, the training steps of the click probability prediction model include: the server can obtain training samples of the click probability prediction model, which include positive samples and negative samples. Positive samples consist of the input text, candidate search terms for objects that have been clicked in the input text, and a sequence of object labels for the objects in the input text. Negative samples consist of the input text, candidate search terms for objects that have not been clicked in the input text, and a sequence of object labels for the objects in the input text. Based on the positive and negative samples included in the training samples of the click probability prediction model, the network parameters of the semantic representation network and the fully connected network in the click probability prediction model are optimized until training stops.

[0162] The server can acquire log data, including historical input text and corresponding candidate search terms. The server can use historical input text as sample input text, and then, from the candidate search terms corresponding to that sample input text's context, determine the candidate search terms that the object that input the sample input text has clicked. The sample input text, the candidate search terms clicked by the object that input the sample input text, and the object label sequence of the object that input the sample input text are considered positive examples. The server can randomly determine candidate search terms that the object that input the sample input text has not clicked, and the sample input text, the candidate search terms that the object that input the sample input text has not clicked, and the object label sequence of the object that input the sample input text are considered negative examples. The server can then input the positive and negative examples into the click probability prediction model to optimize the network parameters of the semantic representation network, fully connected network, and classification network in the click probability prediction model until training stops.

[0163] In the above embodiments, with the sample input text as the context, the corresponding positive and negative samples are obtained by combining the object label sequence. The model is trained using these positive and negative samples, so that the click probability prediction model has the corresponding ability to predict the probability of the object clicking the corresponding candidate search term when the input text is used as the context and the object label sequence of the object corresponding to the input text is combined, thereby achieving personalized deduplication.

[0164] To better understand the above method, an application embodiment of the search term processing method of this application is described in detail below. This application embodiment corresponds to a video search scenario, and accordingly, the search results are videos.

[0165] Figure 7 Here is a framework diagram of a method for processing search terms in one embodiment, referencing... Figure 7The server can generate multiple candidate search terms corresponding to the input text of the object. Then, it determines the repetition between the candidate search terms from three aspects: repetition based on text semantics, repetition based on search results, and repetition based on object click behavior, forming repetition groups. Then, for the same repetition group, the candidate search terms are deduplicated based on the object's object tag sequence.

[0166] Figure 8 This is a flowchart illustrating the method for processing search terms in another embodiment, see reference. Figure 8 This method can be executed by a server or a terminal, or by both a server and a terminal. In this embodiment, the method is described using the server as an example. The method mainly includes the following steps:

[0167] Step S802: Obtain the input text.

[0168] Step S804: Query the candidate search term database to obtain candidate search terms that match the input text.

[0169] Step S806: Based on the relevance of the input text to the candidate search terms, the popularity of the candidate search terms, and the historical click rate, sort the candidate search terms that match the input text to obtain multiple candidate search terms corresponding to the input text.

[0170] Step S808: Based on the semantic representations of any two candidate search terms and the input text, determine the semantic repetition between any two candidate search terms.

[0171] Specifically, step S808 may include the following steps:

[0172] Input any two candidate search terms and the input text into the semantic redundancy prediction model;

[0173] The semantic representation network of the semantic repetition prediction model obtains the deep semantic representations of each of the two candidate search terms and the input text, and then concatenates them to obtain the concatenated deep semantic representation.

[0174] By using a fully connected network connected to the semantic representation network in the semantic repetition prediction model, the textual semantic repetition between any two candidate search terms is predicted based on the concatenated deep semantic representation.

[0175] Step S810: Calculate the degree of repetition between the search results of any two candidate search terms to obtain the degree of repetition between any two candidate search terms based on the search results.

[0176] Taking any two candidate search terms, including the first candidate search term and the second candidate search term, as an example, step S810 specifically includes the following steps:

[0177] Retrieve the list of first search results corresponding to the first candidate search term;

[0178] Retrieve the list of second search results corresponding to the second candidate search term;

[0179] Determine the union and intersection of the top-ranked videos in the first search result list and the top-ranked videos in the second search result list; wherein, the higher the ranking of the search results, the higher the relevance between the multiple search results and the corresponding candidate search terms.

[0180] For each video in the video set, the video's duration, number of plays within a first preset time period, and average playback completeness are counted to obtain the video's statistical values.

[0181] For each video in the video intersection, the video's duration, number of plays within the first preset time period, and average playback completeness are counted to obtain the video's statistical values.

[0182] The degree of overlap between the first and second candidate search terms based on search results is obtained by comparing the sum of the statistical values ​​of the videos in the video intersection with the sum of the statistical values ​​of the videos in the video union.

[0183] Step S812: Calculate the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms to obtain the repetition between any two candidate search terms based on object click behavior.

[0184] Among them, tag features represent the distribution of interest in the content of various tags by the historical click object group;

[0185] Step S812 may specifically include the following steps:

[0186] Determine the historical click object group of the candidate search term. For each historical click object in the historical click object group, obtain the interest distribution of each historical click object in the content of various tags based on the viewing history of the historical click object. Calculate the mean of the interest distribution of each historical click object in the content of various tags in the historical click object group of the candidate search term to obtain the tag features corresponding to the historical click object group of the candidate search term.

[0187] Calculate the cosine distance between the tag features corresponding to the historical click object groups of any two candidate search terms. Based on the cosine distance, obtain the repetition degree between any two candidate search terms based on object click behavior. The repetition degree based on object click behavior is inversely correlated with the cosine distance.

[0188] Step S814: The repetition between any two candidate search terms is obtained by weighted summation of the repetition based on text semantics, the repetition based on search results, and the repetition based on object click behavior.

[0189] Step S816: Based on the degree of repetition between any two candidate search terms, identify multiple repeating groups from multiple candidate search terms.

[0190] Among them, the repetition between candidate search terms belonging to the same repeating group is higher than the set threshold, while the repetition between candidate search terms belonging to different repeating groups is lower than the set threshold.

[0191] Step S816 may specifically include the following steps:

[0192] Based on the repetition between any two candidate search terms, a distance graph of candidate search terms is constructed; the nodes in the distance graph represent candidate search terms, and the distance between nodes is inversely correlated with the repetition between candidate search terms;

[0193] By mining the distance graph of candidate search terms, multiple repeating groups were obtained.

[0194] Step S818: For repeated groups with more than 1 candidate search terms, input any candidate search term, object tag sequence, and input text into the click probability prediction model, and output the click probability of the object for any candidate search term in the repeated group.

[0195] Step S818 may specifically include the following steps:

[0196] Input any candidate search term, object tag sequence, and input text from the repeating group into the click probability prediction model;

[0197] The semantic representation network of the click probability prediction model is used to obtain the deep semantic representation of candidate search terms, the deep semantic representation of object tag sequence and the deep semantic representation of input text, and then they are concatenated to obtain the concatenated deep semantic representation.

[0198] By using a fully connected network connected to the semantic representation network in the click probability prediction model, the click probability of an object for candidate search terms is predicted based on the concatenated deep semantic representation.

[0199] Step S820: For duplicate groups with more than 1 candidate search terms, remove candidate search terms with a click probability lower than the threshold to obtain deduplicated candidate search terms.

[0200] In this application embodiment, after obtaining multiple candidate search terms corresponding to the input text, the degree of repetition between any two candidate search terms is determined from multiple dimensions. Specifically: based on the semantic representation of the input text, the semantic representations of any two candidate search terms are mined to determine the degree of repetition between any two candidate search terms based on text semantics; based on the search results of any two candidate search terms, the degree of repetition between any two candidate search terms is obtained according to the degree of repetition between the search results; furthermore, the historical click target groups of any two candidate search terms are determined, and the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the historical click target groups and the distribution of their interest in the content of various tags. The repetition of click behavior is not limited to the candidate search terms themselves, but expands the information dimensions that deduplication can rely on, which helps to improve the deduplication effect. Finally, based on the repetition of any two search terms in multiple dimensions, candidate search terms are deduplicated, which improves the deduplication effect of candidate search terms and ultimately improves the user's search efficiency and search experience. In addition, this application embodiment can select candidate search terms that are more friendly to the user from the same repetitive group of candidate search terms based on the user's personalized interest in videos of various tags, thereby optimizing the deduplication of search terms, improving the richness of search terms, making it easier for users to conduct selective video searches based on search terms, and improving the user's video search efficiency.

[0201] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0202] Based on the same inventive concept, this application also provides a search term processing apparatus for implementing the search term processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations and technical effects of one or more search term processing apparatus embodiments provided below can be found in the limitations and technical effects of the search term processing method described above, and will not be repeated here.

[0203] Figure 9 This is a structural block diagram of a term-searching processing device in one embodiment. (Refer to...) Figure 9 The device includes:

[0204] The candidate search term determination module 902 is used to determine multiple candidate search terms corresponding to the input text;

[0205] The first repetition acquisition module 904 is used to determine the repetition degree between any two candidate search terms based on the semantic representation of the input text, based on the semantic representation of any two candidate search terms among multiple candidate search terms.

[0206] The second repetition acquisition module 906 is used to calculate the repetition between the search results of any two candidate search terms, and obtain the repetition between any two candidate search terms based on the search results.

[0207] The third repetition acquisition module 908 is used to calculate the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms, and obtain the repetition between any two candidate search terms based on object click behavior; the tag features represent the distribution of interest of the historical click object group to the content of various tags;

[0208] The deduplication module 910 is used to deduplicat multiple candidate search terms based on the degree of repetition between any two candidate search terms based on textual semantics, the degree of repetition based on search results, and the degree of repetition based on object click behavior.

[0209] In one embodiment, the candidate search term determination module 902 is further configured to: acquire input text; query the candidate search term database to obtain candidate search terms that match the input text; and sort the candidate search terms that match the input text according to the relevance between the input text and the candidate search terms, the popularity of the candidate search terms, and the historical click rate, thereby obtaining multiple candidate search terms corresponding to the input text.

[0210] In one embodiment, the first repetition acquisition module 904 is further configured to input any two candidate search terms and the input text into the semantic repetition prediction model; obtain the deep semantic representations of each of the two candidate search terms and the input text respectively through the semantic representation network of the semantic repetition prediction model, and then perform concatenation processing to obtain the concatenated deep semantic representation; predict the text semantic repetition between any two candidate search terms based on the concatenated deep semantic representation through the fully connected network connected to the semantic representation network in the semantic repetition prediction model.

[0211] In one embodiment, the apparatus includes a first training module for acquiring training samples of a semantic repetition prediction model. The training samples of the semantic repetition prediction model include positive samples and negative samples. Positive samples consist of the sample input text and candidate search terms in the sample input text that are marked as duplicates. Negative samples consist of the sample input text and candidate search terms in the sample input text that are not marked as duplicates. Based on the positive and negative samples included in the training samples of the semantic repetition prediction model, the network parameters of the semantic representation network and the fully connected network in the semantic repetition prediction model are optimized until training stops.

[0212] In one embodiment, any two candidate search terms include a first candidate search term and a second candidate search term; the second repetition acquisition module 906 is further configured to acquire a first search result list corresponding to the first candidate search term; acquire a second search result list corresponding to the second candidate search term; and calculate the repetition between the top-ranked search results in the first search result list and the top-ranked search results in the second search result list to obtain the repetition between the first candidate search term and the second candidate search term based on search results, wherein the higher the ranking of the search result, the higher the relevance between it and the corresponding candidate search term.

[0213] In one embodiment, the second repetition acquisition module 906 is further configured to determine the video intersection and video union formed by multiple videos ranked at the top of the first search results list and multiple videos ranked at the top of the second search results list; for each video in the video union, the duration, number of plays within a first preset time period, and average playback completeness of each video are counted to obtain the statistical value of each video; for each video in the video intersection, the duration, number of plays within a first preset time period, and average playback completeness of each video are counted to obtain the statistical value of each video; based on the ratio of the sum of the statistical values ​​of each video in the video intersection to the sum of the statistical values ​​of each video in the video union, the repetition degree between the first candidate search term and the second candidate search term based on the search results is obtained.

[0214] In one embodiment, the device further includes a completeness acquisition module, used to acquire the playback duration and number of times the video is played within a second preset time period; and to use the ratio between the playback duration and the number of times the video is played within the second preset time period as the average playback completeness of the video.

[0215] In one embodiment, the third repetition acquisition module 908 is further configured to determine the historical click object group of the candidate search term; for each historical click object in the historical click object group, based on the viewing records of the historical click object, obtain the interest distribution of each historical click object in the content of various tags; calculate the mean of the interest distribution of each historical click object in the content of various tags in the historical click object group of the candidate search term, and obtain the tag features corresponding to the historical click object group of the candidate search term; for any two candidate search terms, obtain the repetition between any two candidate search terms based on object click behavior based on the repetition between the tag features corresponding to the historical click object groups of any two candidate search terms.

[0216] In one embodiment, the third repetition acquisition module 908 is further configured to calculate the cosine distance between the tag features corresponding to the historical click object groups of any two candidate search terms, and based on the cosine distance, obtain the repetition degree between any two candidate search terms based on object click behavior, wherein the repetition degree based on object click behavior is inversely correlated with the cosine distance.

[0217] In one embodiment, the search results obtained based on the search terms are videos. The third repetition acquisition module 908 is further used to determine various tags pre-set for the videos; obtain the tags of the videos watched by the historical clicked object based on the viewing records of the historical clicked object; for each tag, determine the historical clicked object's interest in the videos of each tag based on the video duration, playback completion rate and playback time of the corresponding videos watched by the historical clicked object; normalize the historical clicked object's interest in the videos of each tag to obtain the distribution of the historical clicked object's interest in the videos of each tag.

[0218] In one embodiment, the deduplication module 910 is further configured to perform a weighted summation of the repetition based on text semantics, the repetition based on search results, and the repetition based on object click behavior between any two candidate search terms to obtain the repetition between any two candidate search terms; based on the repetition between any two candidate search terms, identify multiple repetition groups from multiple candidate search terms, wherein the repetition between candidate search terms belonging to the same repetition group is higher than a set threshold, and the repetition between candidate search terms belonging to different repetition groups is lower than a set threshold; and perform deduplication on candidate search terms belonging to the same repetition group.

[0219] In one embodiment, the deduplication module 910 is further configured to construct a candidate search term distance graph based on the repetition between any two candidate search terms; the nodes in the candidate search term distance graph represent candidate search terms, and the distance between nodes is inversely correlated with the repetition between candidate search terms; the candidate search term distance graph is mined to obtain multiple repetition groups.

[0220] In one embodiment, the deduplication module 910 is further configured to obtain the object tag sequence corresponding to the object in the input text; for a repeating group containing more than one candidate search term, the module inputs any candidate search term in the repeating group, the object tag sequence, and the input text into a click probability prediction model, and outputs the click probability of the object for any candidate search term in the repeating group; within the repeating group, candidate search terms with click probabilities lower than a threshold are removed to obtain deduplicated candidate search terms.

[0221] In one embodiment, the deduplication module 910 is further configured to input any candidate search term, object tag sequence, and input text from the duplicate group into the click probability prediction model; obtain the deep semantic representations of the candidate search term, the object tag sequence, and the input text respectively through the semantic representation network of the click probability prediction model, and then concatenate them to obtain the concatenated deep semantic representation; predict the click probability of the object for the candidate search term based on the concatenated deep semantic representation through the fully connected network connected to the semantic representation network in the click probability prediction model.

[0222] In one embodiment, the apparatus further includes a second training module for acquiring training samples of the click probability prediction model. The training samples of the click probability prediction model include positive samples and negative samples. Positive samples consist of sample input text, candidate search terms for objects that have been clicked in the input sample input text, and object label sequences for objects in the input sample input text. Negative samples consist of sample input text, candidate search terms for objects that have not been clicked in the input sample input text, and object label sequences for objects in the input sample input text. Based on the positive and negative samples included in the training samples of the click probability prediction model, the network parameters of the semantic representation network and the fully connected network in the click prediction probability model are optimized until training stops.

[0223] In the aforementioned search term processing device, after obtaining multiple candidate search terms corresponding to the input text, the degree of repetition between any two candidate search terms is determined from multiple dimensions. Specifically: based on the semantic representation of the input text, the semantic representations of any two candidate search terms are mined to determine the degree of repetition between any two candidate search terms based on text semantics; based on the search results of any two candidate search terms, the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the search results; furthermore, the historical click object groups of any two candidate search terms are determined, and the degree of repetition between any two candidate search terms is obtained based on the degree of repetition between the interest distribution of the historical click object groups on the content of various tags. This is not limited to starting from the candidate search terms themselves, but expands the information dimensions that deduplication can rely on, which helps to improve the deduplication effect; finally, based on the multi-dimensional repetition of any two search terms, candidate search terms are deduplicated, improving the deduplication effect of candidate search terms, and ultimately improving the user's search efficiency and search experience.

[0224] Each module in the aforementioned search term processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0225] In one embodiment, a computer device is provided, which may be a terminal or a server, and its internal structure diagram may be as follows: Figure 10 As shown, this computer device includes a processor, memory, input / output interfaces (I / O interfaces), and a communication interface connected via a system bus. The processor, memory, and I / O interfaces are connected via the system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores processing data for search terms. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for processing search terms.

[0226] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0227] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the various method embodiments described above.

[0228] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the various method embodiments described above.

[0229] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the various method embodiments described above.

[0230] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0231] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0232] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0233] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for processing search terms, characterized in that, The method includes: Identify multiple candidate search terms corresponding to the input text; Based on the semantic representations of any two candidate search terms among the plurality of candidate search terms and the input text, the degree of repetition between any two candidate search terms based on text semantics is determined; The degree of repetition between the search results of any two candidate search terms is calculated to obtain the degree of repetition between any two candidate search terms based on the search results. A historical click object group is determined for the candidate search terms. For each historical click object in the historical click object group, the interest distribution of each historical click object in the content of various tags is obtained based on the viewing history of the historical click object. The mean of the interest distribution of each historical click object in the content of various tags in the historical click object group of the candidate search terms is calculated to obtain the tag features corresponding to the historical click object group of the candidate search terms. The tag features characterize the interest distribution of the historical click object group in the content of various tags. For any two candidate search terms, the degree of repetition between the two candidate search terms based on the repetition of the tag features corresponding to the historical click object groups of the two candidate search terms is obtained. The multiple candidate search terms are deduplicated based on the degree of repetition between any two candidate search terms based on textual semantics, the degree of repetition based on search results, and the degree of repetition based on object click behavior.

2. The method according to claim 1, characterized in that, The determination of multiple candidate search terms corresponding to the input text includes: Get the input text; The candidate search term database is queried to obtain candidate search terms that match the input text; Based on the relevance of the input text to the candidate search terms, the popularity of the candidate search terms, and the historical click rate, the candidate search terms that match the input text are sorted to obtain multiple candidate search terms corresponding to the input text.

3. The method according to claim 1, characterized in that, The step of determining the semantic repetition between any two candidate search terms based on the semantic representations of the input text and any two candidate search terms from the plurality of candidate search terms includes: The two candidate search terms and the input text are input into the semantic redundancy prediction model; After obtaining the deep semantic representations of each of the two candidate search terms and the input text through the semantic representation network of the semantic repetition prediction model, the two terms are then concatenated to obtain the concatenated deep semantic representation. The semantic repetition prediction model uses a fully connected network connected to the semantic representation network to predict the textual semantic repetition between any two candidate search terms based on the concatenated deep semantic representation.

4. The method according to claim 3, characterized in that, The training steps for the semantic redundancy prediction model include: Obtain training samples for the semantic repetition prediction model. The training samples for the semantic repetition prediction model include positive samples and negative samples. The positive samples consist of the input text and the candidate search terms in the input text that are marked as duplicates. The negative samples consist of the input text and the candidate search terms in the input text that are not marked as duplicates. Based on the positive and negative examples included in the training samples of the semantic repetition prediction model, the network parameters of the semantic representation network and the fully connected network in the semantic repetition prediction model are optimized until training stops.

5. The method according to claim 1, characterized in that, The two candidate search terms include a first candidate search term and a second candidate search term. The step of calculating the overlap between the search results of any two candidate search terms, to obtain the overlap between the search results of any two candidate search terms, includes: Obtain the first search results list corresponding to the first candidate search term; Obtain the second search results list corresponding to the second candidate search term; The repetition between the top-ranked search results in the first search results list and the top-ranked search results in the second search results list is calculated to obtain the repetition between the first candidate search term and the second candidate search term based on the search results. The higher the ranking of the search result, the higher the relevance between it and the corresponding candidate search term.

6. The method according to claim 5, characterized in that, The search results obtained based on the search terms are videos; the step of calculating the repetition between the top-ranked search results in the first search results list and the top-ranked search results in the second search results list to obtain the repetition between the first candidate search terms and the second candidate search terms based on search results includes: Determine the video intersection and video union formed by the top-ranked videos in the first search results list and the top-ranked videos in the second search results list; For each video in the video set, the video's duration, the number of times it was played within a first preset time period, and the average playback completeness are counted to obtain the video's statistical values. For each video in the video intersection, the video's duration, number of times it was played within a first preset time period, and average playback completeness are statistically analyzed to obtain the video's statistical value. The degree of repetition between the first candidate search term and the second candidate search term based on search results is obtained by comparing the sum of the statistical values ​​of the videos in the video intersection with the sum of the statistical values ​​of the videos in the video union.

7. The method according to claim 6, characterized in that, The method further includes: Obtain the playback duration and number of times the video is played within a second preset time period; The ratio between the playback duration and the number of times the video is played within a second preset time period is used as the average playback completeness of the video.

8. The method according to claim 1, characterized in that, For any two candidate search terms, the degree of repetition between any two candidate search terms based on the repetition of tag features corresponding to the historical click object groups of the two candidate search terms is obtained, including: Calculate the cosine distance between the tag features corresponding to the historical click object groups of any two candidate search terms. Based on the cosine distance, obtain the repetition degree between any two candidate search terms based on object click behavior. The repetition degree based on object click behavior is inversely correlated with the cosine distance.

9. The method according to claim 1, characterized in that, The search results obtained based on the search terms are videos. The step of obtaining the interest distribution of each historically clicked object across various tags based on its viewing history includes: Identify the various tags pre-set for the video; Based on the viewing history of the historical clicked object, the tags of the videos watched by the historical clicked object are obtained. For each type of tag, the interest of the historical clicked object in the videos of each type of tag is determined based on the video duration, playback completion rate and playback time of the corresponding videos watched by the historical clicked object. The interest level of the historical clicked object in videos of various tags is normalized to obtain the interest distribution of the historical clicked object in videos of various tags.

10. The method according to claim 1, characterized in that, The process of deduplicating the multiple candidate search terms based on the semantic redundancy between any two candidate search terms, the redundancy based on search results, and the redundancy based on object click behavior includes: The repetition between any two candidate search terms is obtained by weighted summing of the repetition based on text semantics, the repetition based on search results, and the repetition based on object click behavior. Based on the repetition between any two candidate search terms, multiple repetition groups are identified from the multiple candidate search terms. Among the multiple repetition groups, the repetition between candidate search terms belonging to the same repetition group is higher than a set threshold, and the repetition between candidate search terms belonging to different repetition groups is lower than a set threshold. Remove duplicates from candidate search terms that belong to the same repeating group.

11. The method according to claim 10, characterized in that, The method of identifying multiple repeating groups from the multiple candidate search terms based on the degree of repetition between any two candidate search terms includes: Based on the repetition between any two candidate search terms, a candidate search term distance graph is constructed; the nodes in the candidate search term distance graph represent candidate search terms, and the distance between nodes is inversely correlated with the repetition between candidate search terms. The distance graph of the candidate search terms is mined to obtain multiple repeating groups.

12. The method according to claim 10, characterized in that, The process of deduplicating candidate search terms belonging to the same repeating group includes: Obtain the sequence of object labels corresponding to the object into which the input text is input; For a repeating group containing more than 1 candidate search terms, each candidate search term in the repeating group, the object tag sequence, and the input text are input into the click probability prediction model, and the click probability of the object for the candidate search terms in the repeating group is output. Within the duplicate group, candidate search terms with a click probability lower than a threshold are removed to obtain deduplicated candidate search terms.

13. The method according to claim 12, characterized in that, The step of inputting any candidate search term within the repeating group, the object tag sequence, and the input text into the click probability prediction model, and outputting the click probability of the object for the candidate search term within the repeating group, includes: Input any candidate search term in the repeating group, the object tag sequence, and the input text into the click probability prediction model; The deep semantic representations of the candidate search terms, the object tag sequence, and the input text are obtained through the semantic representation network of the click probability prediction model. Then, they are concatenated to obtain the concatenated deep semantic representation. The click probability prediction model uses a fully connected network connected to the semantic representation network to predict the click probability of the object for the candidate search term based on the concatenated deep semantic representation.

14. The method according to claim 12, characterized in that, The training steps for the click probability prediction model include: Obtain training samples for the click probability prediction model. The training samples for the click probability prediction model include positive samples and negative samples. The positive samples consist of the sample input text, candidate search terms for objects that have been clicked by the input of the sample input text, and object label sequences for the objects that have been clicked by the input of the sample input text. The negative samples consist of the sample input text, candidate search terms for objects that have not been clicked by the input of the sample input text, and object label sequences for the objects that have been clicked by the input of the sample input text. Based on the positive and negative examples included in the training samples of the click probability prediction model, the network parameters of the semantic representation network and the fully connected network in the click probability prediction model are optimized until training stops.

15. A device for processing search terms, characterized in that, The device includes: The candidate search term determination module is used to determine multiple candidate search terms corresponding to the input text; The first repetition acquisition module is used to determine the repetition degree between any two candidate search terms based on the semantic representation of the input text, based on the semantic representation of any two candidate search terms among the plurality of candidate search terms. The second repetition acquisition module is used to calculate the repetition between the search results of any two candidate search terms, and obtain the repetition between any two candidate search terms based on the search results. The third repetition acquisition module is used to determine the historical click object group of the candidate search terms. For each historical click object in the historical click object group, based on the viewing history of the historical click object, the interest distribution of each historical click object in the content of various tags is obtained. The mean of the interest distribution of each historical click object in the content of various tags in the historical click object group of the candidate search terms is calculated to obtain the tag features corresponding to the historical click object group of the candidate search terms. For any two candidate search terms, the repetition between the two candidate search terms based on the object click behavior is obtained according to the repetition between the tag features corresponding to the historical click object groups of the two candidate search terms. The tag features characterize the interest distribution of the historical click object group in the content of various tags. The deduplication module is used to deduplicatize the multiple candidate search terms based on the degree of repetition between any two candidate search terms based on textual semantics, the degree of repetition based on search results, and the degree of repetition based on object click behavior.

16. The search term processing apparatus according to claim 15, characterized in that, The candidate search term determination module is also used to acquire input text; query the candidate search term database to obtain candidate search terms that match the input text; and sort the candidate search terms that match the input text according to the relevance between the input text and the candidate search terms, the popularity of the candidate search terms, and the historical click rate to obtain multiple candidate search terms corresponding to the input text.

17. The search term processing apparatus according to claim 15, characterized in that, The first repetition acquisition module is further configured to input the arbitrary two candidate search terms and the input text into the semantic repetition prediction model; obtain the deep semantic representations of the arbitrary two candidate search terms and the input text respectively through the semantic representation network of the semantic repetition prediction model, and then perform concatenation processing to obtain the concatenated deep semantic representation; predict the text semantic repetition between the arbitrary two candidate search terms based on the concatenated deep semantic representation through the fully connected network connected to the semantic representation network in the semantic repetition prediction model.

18. The search term processing apparatus according to claim 17, characterized in that, The device further includes a first training module, which is used to acquire training samples of the semantic repetition prediction model. The training samples of the semantic repetition prediction model include positive samples and negative samples. The positive samples consist of the sample input text and candidate search terms marked as duplicates in the candidate search terms of the sample input text. The negative samples consist of the sample input text and candidate search terms not marked as duplicates in the candidate search terms of the sample input text. Based on the positive and negative samples included in the training samples of the semantic repetition prediction model, the network parameters of the semantic representation network and the fully connected network in the semantic repetition prediction model are optimized until training stops.

19. The search term processing apparatus according to claim 15, characterized in that, The two candidate search terms include a first candidate search term and a second candidate search term; the second repetition rate acquisition module is further configured to acquire a first search result list corresponding to the first candidate search term; and acquire a second search result list corresponding to the second candidate search term; The repetition between the top-ranked search results in the first search results list and the top-ranked search results in the second search results list is calculated to obtain the repetition between the first candidate search term and the second candidate search term based on the search results. The higher the ranking of the search result, the higher the relevance between it and the corresponding candidate search term.

20. The search term processing apparatus according to claim 19, characterized in that, The search results obtained based on the search terms are videos; the second repetition acquisition module is also used to determine the video intersection and video union formed by the top-ranked videos in the first search results list and the top-ranked videos in the second search results list; For each video in the union of the videos, the duration, number of times it is played within a first preset time period, and average playback completeness are counted to obtain the statistical value of each video; for each video in the intersection of the videos, the duration, number of times it is played within a first preset time period, and average playback completeness are counted to obtain the statistical value of each video. The degree of repetition between the first candidate search term and the second candidate search term based on search results is obtained by comparing the sum of the statistical values ​​of the videos in the video intersection with the sum of the statistical values ​​of the videos in the video union.

21. The search term processing apparatus according to claim 20, characterized in that, The device further includes a completeness acquisition module, which is used to acquire the playback duration and number of times the video is played within a second preset time period; and to use the ratio between the playback duration and the number of times the video is played within the second preset time period as the average playback completeness of the video.

22. The search term processing apparatus according to claim 15, characterized in that, The third repetition acquisition module is also used to calculate the cosine distance between the tag features corresponding to the historical click object groups of any two candidate search terms, and based on the cosine distance, obtain the repetition degree between any two candidate search terms based on object click behavior, wherein the repetition degree based on object click behavior is inversely correlated with the cosine distance.

23. The search term processing apparatus according to claim 15, characterized in that, The search results obtained based on the search terms are videos, and the third repetition acquisition module is also used to determine various tags that are pre-set for the videos; Based on the viewing history of the historical clicked object, the tags of the videos watched by the historical clicked object are obtained. For each type of tag, the interest of the historical clicked object in the videos of each type of tag is determined based on the video duration, playback completion rate and playback time of the corresponding videos watched by the historical clicked object. The interest level of the historical clicked object in videos of various tags is normalized to obtain the interest distribution of the historical clicked object in videos of various tags.

24. The search term processing apparatus according to claim 15, characterized in that, The deduplication module is also used to perform a weighted summation of the repetition degree between any two candidate search terms based on text semantics, the repetition degree based on search results, and the repetition degree based on object click behavior, to obtain the repetition degree between any two candidate search terms. Based on the repetition between any two candidate search terms, multiple repetition groups are identified from the multiple candidate search terms. Among the multiple repetition groups, the repetition between candidate search terms belonging to the same repetition group is higher than a set threshold, and the repetition between candidate search terms belonging to different repetition groups is lower than a set threshold. The candidate search terms belonging to the same repetition group are deduplicated.

25. The search term processing apparatus according to claim 24, characterized in that, The deduplication module is also used to construct a candidate search term distance map based on the repetition between any two candidate search terms; the nodes in the candidate search term distance map represent candidate search terms, and the distance between nodes is inversely correlated with the repetition between candidate search terms; the candidate search term distance map is mined to obtain multiple repetition groups.

26. The search term processing apparatus according to claim 24, characterized in that, The deduplication module is further used to obtain the object tag sequence corresponding to the object into which the input text is input; for a repeating group with more than 1 candidate search terms, each candidate search term in the repeating group, the object tag sequence, and the input text are input into the click probability prediction model, and the click probability of the object for the candidate search terms in the repeating group is output; within the repeating group, candidate search terms with click probabilities lower than a threshold are removed to obtain deduplicated candidate search terms.

27. The search term processing apparatus according to claim 26, characterized in that, The deduplication module is further configured to input any candidate search term within the repeating group, the object tag sequence, and the input text into the click probability prediction model; through the semantic representation network of the click probability prediction model, obtain the deep semantic representation of the candidate search term, the deep semantic representation of the object tag sequence, and the deep semantic representation of the input text respectively, and then concatenate them to obtain the concatenated deep semantic representation; through the fully connected network in the click probability prediction model connected to the semantic representation network, predict the click probability of the object for the candidate search term based on the concatenated deep semantic representation.

28. The search term processing apparatus according to claim 26, characterized in that, The device further includes a second training module, which is used to acquire training samples of the click probability prediction model. The training samples of the click probability prediction model include positive samples and negative samples. The positive samples consist of the sample input text, candidate search terms for objects that have been clicked by the input of the sample input text, and object label sequences for the objects that have been clicked by the input of the sample input text. The negative samples consist of the sample input text, candidate search terms for objects that have not been clicked by the input of the sample input text, and object label sequences for the objects that have been clicked by the input of the sample input text. Based on the positive and negative samples included in the training samples of the click probability prediction model, the network parameters of the semantic representation network and the fully connected network in the click probability prediction model are optimized until training stops.

29. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 14.

30. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 14.

31. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 14.

Images