Data search method and device

Abstract

The invention provides a data search method and device. The method comprises the steps of obtaining a search instruction; the search instruction comprises at least one target keyword; obtaining at least one information source database, and performing semantic similarity search on the at least one target keyword in each information source database to obtain an initial search result under each information source; according to the semantic similarity between the initial search results and the target keyword, performing first sorting operation on each initial search result to obtain a first search result set; and obtaining a second sorting model, and screening from the first search result set to obtain a target search result set. In this way, the accuracy and comprehensiveness of the search result can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data search method and device. BACKGROUND

[0002] Existing data search is usually based on static keyword matching to achieve fast search of massive data, but this scheme can only directly perform accurate matching based on the keywords given by the user, for example, only when the webpage text contains content completely consistent with the keyword, the webpage is taken as a search result.

[0003] However, the search results obtained by relying on keyword accurate matching in the existing search scheme may miss information that is semantically the same or similar but cannot be accurately matched with the keyword, and may miss information with reference value, and the search results may not really contain the information required by the user, so as to fail to provide valuable search results, thus resulting in the problems of low overall accuracy and poor comprehensiveness of the existing search method. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data search method and device to solve the problem of low overall accuracy and poor comprehensiveness of the search results of the existing search method.

[0005] The present application provides a data search method, which comprises: obtaining a search instruction; the search instruction comprises at least one target keyword; obtaining at least one source database, performing semantic similarity search on the at least one target keyword in each source database to obtain initial search results under each source; performing first sorting operation on each initial search result according to the semantic similarity between the initial search result and the target keyword to obtain a first search result set; obtaining a second sorting model to filter a target search result set from the first search result set.

[0006] The present application also provides a data search device, which comprises: an acquisition module for acquiring a search instruction; the search instruction comprises at least one target keyword; a search module for obtaining at least one source database, performing semantic similarity search on the at least one target keyword in each source database to obtain initial search results under each source; a first sorting module for performing first sorting operation on the initial search result according to the semantic similarity between the initial search result and the target keyword to obtain a first search result set; The second sorting module is configured to obtain a second sorting model, and filter a target search result set from the first search result set.

[0007] The data search method and device provided by the embodiments of the present application can improve the semantic understanding ability of the search instruction and improve the accuracy and comprehensiveness of the search result by performing semantic similarity search on the target keyword in the search instruction in each source database; the combination of the first sorting operation based on the semantic similarity and the second sorting operation based on the second sorting model can ensure the recall rate and precision rate of the target search result filtered finally and the search instruction, improve the accuracy of the search result, and improve the search quality.

[0008] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and easy to understand, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for a detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0010] Figure 1 A flow chart of a data search method provided by the embodiments of the present application is shown; Figure 2 A structural schematic diagram of a data search device provided by the embodiments of the present application is shown; Figure 3 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0011] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more apparent, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by those skilled in the art without creative labor belongs to the scope of protection of the present application.

[0012] It is found through research that existing data search is usually based on static keyword matching to achieve fast search of massive data, but this scheme can only directly perform accurate matching based on the keywords given by the user, for example, only when the webpage text contains content completely consistent with the keywords, the webpage is taken as a search result.

[0013] However, the search results obtained by relying on keyword accurate matching in the existing search scheme may miss information that is semantically the same or similar but cannot be accurately matched with the keywords, may miss information with reference value, and the search results may not really contain the information required by the user, so as to fail to provide valuable search results, thus resulting in the problems of low overall accuracy and poor comprehensiveness of the existing search method.

[0014] Based on this, the embodiments of the present application provide a data search method and device to achieve the purpose of improving the overall accuracy and comprehensiveness of the search results.

[0015] Please refer to Figure 1 , Figure 1 The flowchart of a data search method provided by the embodiments of the present application. The data search method provided by the embodiments of the present application can be implemented based on software and / or hardware. As shown in Figure 1 The data search method provided by the embodiments of the present application includes the following steps: S101, obtaining a search instruction.

[0016] The search instruction includes at least one target keyword; the target keyword is a keyword that can indicate a search intent and is related to the expected search content.

[0017] In one example, the search instruction can be input by the user, in which case the search instruction input by the user can be obtained through a human-computer interaction device and / or a human-computer interaction interface. The user input method can include voice input, text input, etc., which is not limited herein.

[0018] In another example, the search instruction can also be triggered automatically by a machine. For example, the embodiments of the present application can be applied to a large language model (LLM) to provide search content support for the question answering and writing functions of the large language model; in this case, the search instruction can be automatically generated by the large language model, that is, the search instruction is obtained through data interaction with the large language model.

[0019] S102, obtaining at least one source database, and performing semantic similarity search on the at least one target keyword in each source database to obtain an initial search result under each source.

[0020] Here, the source refers to an entity or source that generates public information, for example, including public social media, public forums, public search engines, etc. In the embodiments of the present application, each source has a corresponding source database, and the source database includes at least one piece of data.

[0021] In this step, semantic similarity search can be performed on the target keyword in the source database corresponding to each source, that is, the search results similar in semantics to the target keyword are found from each source database, and the initial search results under each source are obtained.

[0022] In specific implementation, each piece of data in the source database should be pre-processed and structured data with index. When constructing the source database, first, duplicate data or web pages are deleted, and irrelevant data is removed. Irrelevant data is the part of each piece of data that is irrelevant to searchable information, such as advertisements, formatting, headers, footers, etc. After deduplication and removal of irrelevant information, each piece of data is converted into a high-dimensional vector xi, and the source database can be represented as D={x1,x2,……,xn}. The conversion method of the high-dimensional vector is not limited in the embodiments of the present application, for example, existing technologies such as BERT can be used. An index is extracted for each piece of data, and each index can store the keywords of each data document and record the positions and frequencies of each keyword appearing in the data document, which is used to realize accurate keyword matching and fast document retrieval subsequently; here, preferably, each different page data in the sources is regarded as a piece of data. In this way, when searching for a target keyword, the index can be matched to quickly return all data documents containing the target keyword and similar in semantics to the target keyword in the source database.

[0023] In this way, standardization processing is performed when data is warehoused, rather than after searching. On the one hand, because searching is the key link that needs to process the most data, consumes the most time and directly affects the quality of search results in the whole process, it is necessary to process data when warehousing data, which can reduce the amount of calculation when searching, and the standardized format can also avoid the error rate and search understanding cost caused by non-uniform formats, and ensure higher search efficiency.

[0024] Further, the embodiments of the present application also consider that the data modalities in the current source have diversity, such as a static web page usually includes text and pictures at the same time, and some web pages also include videos, moving pictures and other modalities. Among these contents, there may only be text, or there may be no text at all, or there may be both non-text elements such as landscapes, figures, graphics, and text. The prior art usually uses OCR to process pictures or video frames, but OCR can only recognize text in pictures and cannot recognize other elements, nor can it be used to process moving pictures or videos, resulting in the inability to match text and other modal data during search, and the search results are prone to omissions. Therefore, in the embodiments of the present application, a multi-modal processing method is used to construct a source database, so as to process pictures, videos, moving pictures and other contents. Regardless of whether the web page data includes one or more of pictures, videos, moving pictures and text, the multi-modal processing capability can map the entire web page data into a feature vector of the same dimension.

[0025] Correspondingly, the method for constructing the source database can include: Step a1, for each source, pre-processing each data in the source, and decomposing each data into modal data corresponding to each modal type according to the modal type.

[0026] In this step, each data in the source is parsed and pre-processed, and then each data result is decomposed into modal data corresponding to each modal type according to the modal type, such as text modal data, picture modal data, video modal data, etc. If the original data is a video containing sound, text and pictures, the decomposed data includes sound modal data, text modal data and video modal data containing only pictures.

[0027] Here, the decomposed modal data also includes timestamp data. If an original data is decomposed into more than one modal data, each modal data obtained by decomposition needs to be aligned in time through a timestamp. The timestamp is used to record the relative time or absolute time. Assuming that the original data is a 2-minute video, which includes 2-minute picture images and occasional sound data, and the image is provided with subtitles when the sound data is triggered, the relative time or absolute time when each sound data and text data of the subtitles exist is represented by a timestamp to ensure the synchronization of sound, text and pictures. The relative time refers to taking the start time of the original data as the timing origin, and taking the time length between the occurrence time of the corresponding modal content and the timing origin as the timestamp data. The absolute time refers to taking the actual time as the timestamp data.

[0028] Step a2, extracting semantic vectors of the modal data corresponding to each modal type, and mapping each modal data to the same semantic vector space.

[0029] In this step, multimodal processing is first performed, converting all data of each modality type into the corresponding modal vector. For example, all text in each webpage is concatenated and converted into a single text vector, and all image vectors are concatenated and converted into a single image vector. Similarly, a video can be seen as being obtained by concatenating a large number of image frames in chronological order, so it can also be converted into a large number of image vectors. Here, in order to maintain the alignment of multiple modal data in a single dataset, the concatenation points can be connected using timestamps.

[0030] The purpose of multimodal processing here is to extract semantic data from non-textual modalities, ultimately mapping vectors of different modalities and dimensions into the same semantic vector space. Within this semantic vector space, the concept of "modality" and the data analysis impact of different modalities can be eliminated, ensuring that semantically similar data from different modalities are also close in distance. For example, the textual form of "a cat" and a photograph of a cat are close in distance within this semantic vector space. Thus, in subsequent data searches, only the distances between vectors within this semantic vector space need to be calculated to achieve cross-modal, semantic search and matching. In specific implementations, multimodal processing methods can refer to existing technologies, such as multimodal embedding models (e.g., Contrastive Language–Image Pre-training, CLIP, etc.), and this application does not impose any limitations on these methods.

[0031] Step a3: Construct the source database corresponding to the source based on each semantic vector in the semantic vector space.

[0032] In this step, a data index is constructed in the source database based on each semantic vector in the semantic vector space. The data index is used to match each piece of data in the source by keywords. The source database is constructed based on each piece of data in each source and the data index.

[0033] In this way, text-based modal data and non-text-based modal data describing the same semantic concept are mapped to close positions in the same semantic vector space, thereby enabling cross-modal understanding and searching during the search process and helping to improve the comprehensiveness of search results.

[0034] S103. Based on the semantic similarity between the initial search results and the target keywords, perform a first sorting operation on each initial search result to obtain a first search result set.

[0035] In this step, each initial search result can be ranked and scored based on its semantic similarity to the target keyword. Then, a first ranking operation is performed on each initial search result according to its first ranking score, and a certain number or proportion of initial search results with higher first ranking scores are selected to form a first search result set. The specific number or proportion can be a preset value or can be adaptively adjusted based on the number of initial search results.

[0036] S104. Obtain the second ranking model and filter the target search results set from the first search results set.

[0037] In this step, the first search results in the first search results set can be sorted again based on the second sorting model, and the target search results can be selected from them according to the second sorting results to form the target search results set.

[0038] Since the second sorting is based on the results of the first sorting, the first sorting can be regarded as a coarse-grained search sorting, and the second sorting as a fine-grained search sorting. That is, by superimposing the two sorting operations and the filtering operation, the effect of selecting the best among the best can be achieved, ensuring that the final filtered target search results have a high degree of relevance to the search instructions and meet the search intent, thereby improving the accuracy of search results and enhancing search quality.

[0039] Here, the second ranking model can be a rule-based model or a pre-trained machine learning model. For machine learning models, training data can be constructed based on different search optimization objectives, enabling the trained model to achieve the corresponding search optimization effect during ranking operations. For example, training data can be constructed based on the characteristics of historically adopted search results, allowing the trained second ranking model to predict search results with a high probability of adoption and form a target search result set.

[0040] This application provides a data search method that improves the semantic understanding of search instructions and enhances the accuracy and comprehensiveness of search results by performing semantic similarity searches on target keywords in search instructions in various source databases. By combining a first sorting operation based on semantic similarity with a second sorting operation based on a second sorting model, the method ensures the relevance of the final filtered target search results to the search instructions, thereby improving the accuracy and quality of search results.

[0041] In one embodiment of this application, in step S102, for the semantic similarity search of the target keyword, a single search strategy can be used, such as a spatial distance calculation strategy or an inverted index exact matching strategy; or a combination of multiple search strategies can be used, such as using a spatial distance calculation strategy and an inverted index exact matching strategy to search, and integrating the results of the two to obtain the initial search results under each information source.

[0042] Specifically, the spatial distance calculation strategy involves transforming the target keyword to be queried into a high-dimensional vector y, and then finding the K1 high-dimensional vectors x that are closest in spatial distance to the high-dimensional vector y in each source database D by calculating the spatial distance.

[0043] K1 represents the initial search quantity. If the K1 value is too small, similar answers or closely related pages may be missed. However, if the K1 value is too large, the search time will increase non-linearly. Therefore, a balance needs to be struck between the two. In specific implementations, K1 can be a value set based on experience. Preferably, another embodiment of this application takes into account that different information sources may have drastically different data distributions and content characteristics. A parameter calculation method that is effective on one information source may not be applicable to another. Therefore, a K1 value is configured separately for each information source through analysis and calculation.

[0044] In another embodiment of this application, step S102 may include: S1021. Determine the feature values ​​of the target keywords and / or the data content feature values ​​of each information source.

[0045] In one possible implementation, the target keyword features include at least one of the following: the rarity of the target keyword relative to historical search processes in each source and the word segmentation length of the target keyword. The data content features of each source include at least: the distribution density of pre-search results under that source.

[0046] To determine the rarity feature S1 of a target keyword relative to historical search data in each source, we first obtain the historical keyword database for each source, which includes the user's historical search commands or keywords. We then query the frequency of the target keyword corresponding to the current search task in the historical keyword database. The rarity feature of each target keyword in each source is then expressed by the formula S1i = log(N / n), where N represents the total number of historical search keywords, n represents the number of times the target keyword appears in the historical keyword database, and i represents the source number. The rarity feature value is positively correlated with the number of searches performed in the first search.

[0047] For the word segmentation length feature S2 of the target keyword, existing word segmentation algorithms can be used to segment the target keyword (group) and determine the word segmentation length feature value. For example, the word segmentation result of "blue sun umbrella" is "blue", "sunshade", and "umbrella", that is, S2=3. The word segmentation length feature value is positively correlated with the number of first searches.

[0048] For each data content feature S3 of a source, the feature value of the distribution density feature of the pre-search results under that source can be determined, which may include: Step b1: Based on the target keyword, perform a pre-search on the data in the information source to determine the second spatial distance between the target keyword and each piece of data in the information source in the first space.

[0049] In this step, a pre-retrieval is performed in each source database based on the target keywords to determine the second spatial distance between each piece of data in each source database and the target keywords in the first space (semantic vector space).

[0050] Step b2: Select the data of the second search quantity that is closest to the second spatial distance as the relevant data.

[0051] Wherein, the second retrieval quantity M is a preset value, and the second retrieval quantity M is greater than the first retrieval quantity K1. That is, the M data points with the closest second spatial distance to the target keyword are selected as the relevant data in the information source.

[0052] Step b3: Calculate the third spatial distance between every two related data points and the average of all third spatial distances. Determine the average of the third spatial distances as the feature value of the distribution density of the pre-search results under this information source.

[0053] In this step, the third spatial distance between each pair of these M relevant data points is calculated, and the average third spatial distance is obtained. This average third spatial distance is then used as the feature value for the distribution density of the pre-search results under this information source. The formula is expressed as:

[0054] in, This represents a function for calculating spatial distance. This represents the vector data obtained from the pre-retrieval; This represents the number of combinations of choosing 2 vectors from M vectors, used to calculate all vector pairs.

[0055] S1022. For each information source, determine the first retrieval quantity corresponding to the information source based on the feature value of the target keyword and / or the data content feature value of the information source.

[0056] In this step, for each information source, based on the feature values ​​of the target keywords and / or the data content feature values ​​of the information source, the first retrieval quantity corresponding to that information source can be determined through a weighted calculation of indicators. The formula is expressed as: K1_i=K0+W1*S1+W2*S2+W3*S3. K1_i represents the first retrieval quantity of the i-th information source, K0 is a preset hyperparameter, and W1, W2, and W3 are the preset weights corresponding to each feature.

[0057] S1023. Determine the first spatial distance between the target keyword and each piece of data in the current information source in the first space, and select the data with the closest first spatial distance as the initial search result in the current information source.

[0058] The first spatial distance is used to characterize the semantic similarity between the target keyword and each piece of data in the current information source. The smaller the spatial distance, the higher the semantic similarity between the target keyword and the data. The first spatial distance, the second spatial distance, and the third spatial distance all use the same spatial distance calculation function. Preferably, in this embodiment, the cosine similarity function is used to characterize the spatial distance.

[0059] For inverted index exact matching strategies, the target keywords can first be expanded semantically, such as through synonym expansion or near-synonym expansion. The initial search results corresponding to this search strategy are then obtained by matching the expanded keywords with the indexes of various source databases.

[0060] It is worth noting that, considering that using multiple search strategies within a single source can lead to duplicate initial search results, or that data duplication can also occur in cross-source search scenarios, in order to save subsequent data processing overhead, deduplication can be performed on the initial search results of each source before the first sorting operation.

[0061] In another embodiment of this application, step S103 may include: S1031. Determine the index value of each initial search result with respect to at least one search quality index.

[0062] It should be noted that, unlike traditional search engine coarse-grained ranking, in order to achieve precise content delivery to users or provide rich and diverse relevant data to artificial intelligence models, and to ensure the comprehensiveness of the final search results or LLM-generated answers, this application embodiment sets search quality indicators from multiple dimensions. In one example, the search quality indicators include at least one of the following: credibility indicators, diversity indicators, real-time indicators, and user preference indicators.

[0063] S1032. Based on the index value of each initial search result for each search quality index, determine the correction coefficient corresponding to each initial search result.

[0064] S1033. Determine the first ranking score for each initial search result based on the correction coefficient corresponding to each initial search result and the semantic similarity with the target keyword.

[0065] For steps S1032 and S1033, the formula for calculating the first ranking score can be expressed as: ; That is, semantic similarity, which can be represented by the first spatial distance; let... ,in, For hyperparameters, the preferred ones are... ; This represents the credibility index value. Indicates the diversity index value, This represents the real-time performance index value. This represents the value of a user preference metric. This represents the correction factor corresponding to the initial search result.

[0066] S1034. Perform a first sorting operation based on the first sorting score of each initial search result, select the first search result from the initial search results after the first sorting operation according to a preset method, and aggregate them to form the first search result set.

[0067] In this step, the score is based on the first ranking. Perform the first sorting operation on all initial search results, selecting the top results. The initial search result is % and is used as the first search result. The value can be determined based on the number of initial search results. If the number of initial search results is large, the value can be appropriately reduced. When the number of initial search results is small, increase the value. In a preferred embodiment, ,in, This indicates the maximum number of search results that can be found in the first search result. This indicates the lower limit of the number of search results for the first result. This indicates the standard number of the first search result. This indicates the initial number of search results.

[0068] Alternatively, you can choose the previous option. The first initial search result is used as the first search result. , The function represents the function to round to an integer.

[0069] In one possible implementation, for diversity indicators Its calculation includes source coverage index values ​​and / or opinion coverage index values; that is, diversity encompasses the diversity of source coverage and / or the diversity of opinion coverage; therefore, the methods for calculating diversity index values ​​include: Method A1: For the source coverage index value, the initial search results can be converted into a sequence, and a sliding window can be configured in each sequence to record the source to which each initial search result in the sliding window belongs.

[0070] As the sliding window slides across the sequence, for each initial search result entering the range of the sliding window, a penalty is applied to the original score corresponding to the initial search result using a set penalty factor based on the number of times the source to which the initial search result belongs within the sliding window, thereby determining the source coverage index value of the initial search result. The original score can be determined based on the semantic similarity between the initial search result and the target keyword.

[0071] Specifically, the formula for calculating the source coverage index value is as follows:

[0072]

[0073] in, This refers to a manually set penalty factor. This indicates the number of times the information source appears within a sliding window. It is a normalized calculation function. The sliding window length is a preset value, for example, 20~50.

[0074] Method A2: For the opinion coverage index value, sub-topics related to the search instruction can be determined; based on the mean semantic similarity between each initial search result and each sub-topic, the opinion coverage index value of each initial search result is determined.

[0075] In this approach, existing topic models, such as LDA or BERT-based topic models, can be used to identify sub-topics related to the search query. A sub-vector corresponding to each sub-topic is calculated, and the mean semantic similarity between each initial search result and each sub-topic is used as the opinion coverage metric. .

[0076] It should be noted that the above two methods can be used individually, that is, the source coverage index value can be used as the diversity index value, or the opinion coverage index value can be used as the diversity index value. Preferably, the source coverage index value and the opinion coverage index value can be combined to determine the diversity index value. For example, a weighted summation method can be used, and the formula can be expressed as follows: ;in, Indicates the diversity index value, and These represent the weights corresponding to the source coverage index value and the opinion coverage index value, respectively.

[0077] In another possible implementation, for the credibility index The methods for calculating the credibility index value include: The weight adjustment value of a specific type of information source is determined according to the search instruction; the weight value of each information source is determined according to the preset weight value of each information source and the weight adjustment value of the specific type of information source, and the weight value of the information source is determined as the index value of the credibility index of each initial search result generated in the information source.

[0078] The preset weight value of each information source can be determined using existing algorithms, such as the PageRank algorithm; the weight adjustment value Δr (0≤Δr<1) can fluctuate according to the user's instructions. For example, if the semantic meaning of the search instruction is to want to know about travel guides for a certain place, the weight of social media (especially those that are travel-oriented) can be increased. Thus, a positive Δr (0<Δr<1) can be assigned to ordinary social media, a larger Δr can be assigned to travel-oriented social media / forums, and Δr=0 for other information sources.

[0079] In another possible implementation, for real-time metrics The methods for calculating real-time performance metrics include: Based on the time difference between the publication time of the source to which each initial search result belongs and the time of generation of the search query, and the timeliness type of the source to which the source belongs, determine the indicator value of each initial search result regarding the real-time performance metric.

[0080] For example, the formula for calculating the real-time performance index value can be expressed as follows: ,in, This indicates the time difference between the publication time and the time the search query was generated, in hours. It is an adjustable attenuation coefficient, related to the timeliness type of the information source. For example, for real-time information sources such as news and social media, You can set it higher to make the timeliness score decay more quickly; for sources with slower updates, such as historical documents, official websites, and academic journals, It can be set to a smaller value.

[0081] In another possible implementation, for user preference metrics This allows us to determine a user's preferences for information sources and / or certain content based on their historical data, in order to calculate a user preference index value. It should be noted that the historical user data used here is obtained with the user's prior authorization. Specific calculation methods include: Method B1: Determine the user's source preference score for each initial search result based on the user's historical interaction behavior with the source.

[0082] Here, historical interaction behavior can include all of the user's historical operations on the information source, such as clicks, favorites, likes, shares, and dwell time. Different types of operations can correspond to different weights or scores, for example, from highest to lowest: share, favorite, like, click. For dwell time scores, the dwell time can be multiplied by a certain correction coefficient, or the score can be calculated with a certain interval granularity, for example, 0~30s is 0.1, 30~60s is 0.2, and so on. In addition, to prevent the model from focusing too much on old interactions and ignoring new interactions, a time decay factor can also be set. Multiply by all operation scores, where It is the time elapsed since the interaction occurred. It is a preset parameter.

[0083] Method B2: Determine the user's content preference score for each initial search result based on the semantic similarity between each initial search result and the pre-built user profile.

[0084] Here, the user profile includes the user's preferences for specific subjects, specific keywords, specific information media, etc., and the user profile can also be determined through the user's historical interaction behavior with the information source database. By calculating the cosine similarity between each initial search result and the user profile, the user's content preference score for each initial search result is determined.

[0085] Similarly, the two methods mentioned above can be used separately to determine user preference index values, or they can be combined by normalization and weighted summation to obtain user preference index values.

[0086] In one embodiment of this application, the second ranking model includes an input layer, a shared network, a gated network, a branch network, and an output layer. The main task of the second ranking model is to calculate the priority of each first search result, rank the first search results according to their priorities, and finally select the N highest-priority first search results as the target search results. The second ranking model also has at least one branch task, which corresponds one-to-one with a branch network. The branch task refers to an auxiliary task used to assist in priority calculation.

[0087] Then, regarding step S104, it may include: S1041. Receive the first search result set and the search instruction through the input layer, and generate a feature vector for each first search result.

[0088] In this step, the input layer receives input data such as the first search result set and search instructions, and generates a feature vector of each first search result with a unified structure through operations such as transformation and concatenation.

[0089] S1042. Extract the general feature representation of the feature vector of each first search result through the shared underlying network.

[0090] In this step, the shared underlying network serves as the bottom layer of the second ranking model, responsible for processing features common to all business operations, such as user profiles, query term information, and contextual information. Through this shared layer, the model can learn common knowledge across different business operations, improving the model's generalization ability and efficiency.

[0091] S1043. Calculate the branch weights corresponding to each branch network through the gating network.

[0092] In this step, the gating network learns the user's interests and preferences and dynamically assigns different weights to different branch networks (branch tasks). For example, based on the user's characteristics and target keywords, a weight is output for each branch network to determine which branch networks are more important to the ranking results. It should be noted that the user interests and preferences here are also obtained with the user's prior authorization. User interests and preferences can be included in the search instructions or used as separate input variables for the second ranking model.

[0093] S1044. Different branch networks are used to execute different sub-tasks. Each branch network is connected to the shared underlying network through the gating network. Each branch network determines the branch feature representation corresponding to each sub-task.

[0094] Here, the branch feature representation can be extracted from the general feature representation, and the branch representation and the general feature representation can also be independent of each other. In one example, the subtasks include predicting the adoption probability of the search result and predicting the user dwell time of the search result. Then, the branch feature representation can be seen as the prediction scores for adoption and dwell time, respectively, which are based on changes in user preferences. The general feature representation can be seen as a secondary score of the first search result.

[0095] S1045. By combining the general feature representation, the feature representation of each branch, and the corresponding weights of each branch through the output layer, the second ranking score of each first search result is determined.

[0096] In this step, the output layer performs a weighted summation of the general feature representation of the shared underlying network output and the branch feature representation of each branch network based on the weight values ​​given by the gating network, to obtain the second ranking score for each first search result.

[0097] It is worth noting that in scenarios requiring the simultaneous satisfaction of one primary metric and multiple sub-metrics, existing technologies often exhibit a "seesaw phenomenon." This means that the model performs better than a single-task model on one metric dimension, but performs worse than a model with a single primary metric on other metric dimensions. Better performance on one dimension inevitably leads to worse performance on one or more other metrics. The embodiments of this application, through the cooperation of gating networks and sub-networks, can reduce the predictive impact of the seesaw phenomenon.

[0098] S1046. Perform a second sorting operation based on the second sorting score of each first search result, select the target search result from the first search results after the second sorting operation according to a preset method, and form the target search result set.

[0099] Here, you can select results whose second ranking score is greater than or equal to the score threshold, or select the top m% of the first search results as the target search results. Alternatively, you can refer to the first ranking operation method and set a minimum number of target search results, such as 50.

[0100] In one example, the methods for training the second ranking model include: Step c1: Obtain training data.

[0101] For example, each piece of training data may include: user ID, feature vector, inverted index, metric score, and label; the label may include an adoption label and / or a user dwell time label; for the scenario of artificial intelligence model, if the model adopts the adoption label, the adoption label is 1, and if it does not adopt the adoption label, the adoption label is 0.

[0102] Step c2: Input the training data into the second ranking model, and determine the first loss function of the second ranking model regarding priority ranking, the second loss function regarding adoption status, and the third loss function regarding dwell time.

[0103] The first loss function for priority ranking This requires constructing pairs of samples (A, B). Assuming the large model adopts webpage A but not webpage B, or the user clicked on A but not B, then A is considered superior to B in reality. During prediction, a predicted ranking score will be given for both samples A and B. , Then the first loss function is expressed as:

[0104] in, This represents the sigmoid function. If the model predicts... In cases where the prediction contradicts reality, it indicates that the model's ranking prediction is incorrect, and the loss function will output a large loss value. When the prediction... When the model predicts correctly, the loss value approaches 0.

[0105] The second loss function for the adoption state A binary cross-entropy loss function can be used to measure the difference between the adoption probability predicted by the model and the actual adoption, so that the adoption probability predicted by the model is close to the actual probability. Here, the adoption probability is a value between 0 and 1, and the actual adoption is 0 or 1.

[0106] The third loss function for dwell time The mean squared error loss function can be used to measure the difference between the dwell time predicted by the model and the actual dwell time (both in seconds), so that the dwell time predicted by the model is close to the actual dwell time.

[0107] Step c3: Determine the target loss function based on the first loss function, the second loss function, the third loss function, and the weights corresponding to each loss function.

[0108] Preferred, , The weights of the first loss function, The weights of the second loss function, These are the weights for the third loss function. Priority ranking is the most important task as the primary task, and among the auxiliary tasks, adoption rate, which characterizes whether the model adopts a model, is also relatively important. The three weights can be predicted by the model or preset.

[0109] The final target loss function is then expressed as: .

[0110] Step c4: Train the parameters of the second ranking model in an end-to-end manner based on the target loss function.

[0111] In this step, the specific implementation of training the parameters of the second ranking model in an end-to-end manner based on the target loss function can refer to existing technologies, such as updating the model parameters by backpropagation after determining the target loss value, etc. This application does not impose any restrictions here.

[0112] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a data search device provided in an embodiment of this application. Figure 2 As shown, the data search device 200 includes: The acquisition module 210 is used to acquire a search instruction; the search instruction includes at least one target keyword; Search module 220 is used to acquire at least one source database, perform semantic similarity search on the at least one target keyword in each source database, and obtain the initial search results under each source. The first sorting module 230 is used to perform a first sorting operation on the initial search results based on the semantic similarity between the initial search results and the target keyword, so as to obtain a first search result set; The second sorting module 240 is used to obtain a second sorting model and filter the target search results set from the first search results set.

[0113] Furthermore, when the search module 220 performs semantic similarity searches on the at least one target keyword in each source database to obtain initial search results for each source, the search module 220 is used to: Determine the feature values ​​of the target keywords and / or the data content feature values ​​of each information source; For each information source, the first retrieval quantity corresponding to that information source is determined based on the feature value of the target keyword and / or the data content feature value of the information source; Determine the first spatial distance between the target keyword and each piece of data in the current information source within a first space, and select the data with the closest first spatial distance as the initial search result in the current information source; wherein, the first spatial distance is used to characterize the semantic similarity between the target keyword and each piece of data in the current information source.

[0114] Furthermore, the characteristics of the target keyword include at least one of the following: the rarity of the target keyword compared to the historical search process in each source and the word segmentation length of the target keyword; the data content characteristics of each source include at least the distribution density of the pre-search results under that source.

[0115] Furthermore, the data search device 200 further includes: a determining module; the determining module is used to determine the feature values ​​of the distribution density characteristics of the pre-search results under the information source; the determining module is specifically used for: Based on the target keywords, a pre-search is performed on the data of the information source to determine the second spatial distance between the target keywords and each piece of data in the information source in the first space; The data with the second closest spatial distance and the second number of searches are selected as the relevant data; wherein, the second number of searches is a preset value, and the second number of searches is greater than the first number of searches; Calculate the third spatial distance between every two related data points, and the average of all third spatial distances. Determine the average of the third spatial distances as the feature value of the distribution density of the pre-search results under this information source.

[0116] Furthermore, when the first sorting module performs a first sorting operation on the initial search results based on the semantic similarity between the initial search results and the target keyword to obtain a first search result set, the first sorting module is used to: Determine the value of each initial search result for at least one search quality metric; the search quality metric includes: credibility metric, diversity metric, real-time metric, and user preference metric; Based on the index value of each initial search result for each search quality metric, determine the correction factor corresponding to each initial search result; The first ranking score of each initial search result is determined based on the correction coefficient corresponding to each initial search result and the semantic similarity with the target keyword. A first sorting operation is performed based on the first sorting score of each initial search result. The first search result is selected from the initial search results after the first sorting operation according to a preset method and aggregated to form the first search result set.

[0117] Furthermore, the calculation of the diversity index includes source coverage index values ​​and / or viewpoint coverage index values; the determining module is also used for: The initial search results are converted into sequences, and sliding windows are configured in each sequence. The information source to which each initial search result belongs in the sliding window is recorded. As the sliding window slides on the sequence, for each initial search result that enters the range of the sliding window, a penalty is applied to the original score corresponding to the initial search result based on the number of times the information source to which the initial search result belongs has appeared in the sliding window, and the information source coverage index value of the initial search result is determined. Identify subtopics related to the search instruction; determine the opinion coverage index value for each initial search result based on the mean semantic similarity between each initial search result and each subtopic.

[0118] Furthermore, the second ranking model includes an input layer, a shared network, a gated network, a branching network, and an output layer; when the second ranking module 240 is used to obtain the second ranking model and filter the target search result set from the first search result set, the second ranking module 240 is used to: The input layer receives the first search result set and the search instruction, and generates a feature vector for each first search result. The common feature representation of the feature vector of each first search result is extracted through the shared underlying network; The branch weights corresponding to each branch network are calculated using the gating network. Different branch networks are used to execute different subtasks. Each branch network is connected to the shared underlying network through the gating network. Each branch network determines the branch feature representation corresponding to each subtask. By combining the general feature representation, the feature representation of each branch, and the corresponding weights of each branch in the output layer, the second ranking score of each first search result is determined. A second sorting operation is performed based on the second sorting score of each first search result. The target search result is selected from the first search results after the second sorting operation according to a preset method, and the target search result set is formed.

[0119] Furthermore, the sub-tasks include predicting the adoption probability of search results and predicting the user dwell time of search results; the data search device 200 further includes: a training module; the training module trains the second ranking model in the following ways: Acquire training data, input the training data into the second ranking model, and determine the first loss function of the second ranking model regarding priority ranking, the second loss function regarding adoption status, and the third loss function regarding dwell time; The target loss function is determined based on the first loss function, the second loss function, the third loss function, and the weights corresponding to each loss function. The parameters of the second ranking model are trained in an end-to-end manner based on the target loss function.

[0120] Furthermore, the data search device 200 also includes: a construction module; the construction module constructs the information source database in the following ways: For each information source, each data in the information source is preprocessed, and each data is decomposed into modal data corresponding to each modal type according to the modal type; Extract the semantic vectors of the modal data corresponding to each modality type, and map each modality data to the same semantic vector space; Based on each semantic vector in the semantic vector space, construct the source database corresponding to the source.

[0121] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0122] The memory 320 stores machine-readable instructions executable by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, they can perform the operations described above. Figure 1 The steps of the data search method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0123] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the data search method in the illustrated method embodiment can be found in the method embodiment for specific implementation, and will not be repeated here.

[0124] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0128] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data search method, characterized in that, The method includes: Obtain a search instruction; the search instruction includes at least one target keyword; Obtain at least one source database, and perform semantic similarity search on the at least one target keyword in each source database to obtain the initial search results for each source. Based on the semantic similarity between the initial search results and the target keywords, a first sorting operation is performed on each initial search result to obtain a first search result set; Obtain the second ranking model and filter the target search results set from the first search results set.

2. The method according to claim 1, characterized in that, Semantic similarity searches are performed on at least one target keyword in each source database to obtain initial search results for each source, including: Determine the feature values ​​of the target keywords and / or the data content feature values ​​of each information source; For each information source, the first retrieval quantity corresponding to that information source is determined based on the feature value of the target keyword and / or the data content feature value of the information source; Determine the first spatial distance between the target keyword and each piece of data in the current information source within a first space, and select the data with the closest first spatial distance as the initial search result in the current information source; wherein, the first spatial distance is used to characterize the semantic similarity between the target keyword and each piece of data in the current information source.

3. The method according to claim 2, characterized in that, The target keyword features include at least one of the following: the rarity of the target keyword compared to the historical search process in each source and the word segmentation length of the target keyword; The data content features of each information source should include at least the distribution density features of the pre-search results under that information source.

4. The method according to claim 3, characterized in that, The feature values ​​for determining the distribution density of pre-search results under this information source include: Based on the target keywords, a pre-search is performed on the data of the information source to determine the second spatial distance between the target keywords and each piece of data in the information source in the first space; The data with the second closest spatial distance and the second number of searches are selected as the relevant data; wherein, the second number of searches is a preset value, and the second number of searches is greater than the first number of searches; Calculate the third spatial distance between every two related data points, and the average of all third spatial distances. Determine the average of the third spatial distances as the feature value of the distribution density of the pre-search results under this information source.

5. The method according to claim 1, characterized in that, Based on the semantic similarity between the initial search results and the target keywords, a first sorting operation is performed on each initial search result to obtain a first search result set, including: Determine the value of each initial search result for at least one search quality metric; the search quality metric includes: credibility metric, diversity metric, real-time metric, and user preference metric; Based on the index value of each initial search result for each search quality metric, determine the correction factor corresponding to each initial search result; The first ranking score of each initial search result is determined based on the correction coefficient corresponding to each initial search result and the semantic similarity with the target keyword. A first sorting operation is performed based on the first sorting score of each initial search result. The first search result is selected from the initial search results after the first sorting operation according to a preset method and aggregated to form the first search result set.

6. The method according to claim 5, characterized in that, The calculation of the diversity index includes the source coverage index value and / or viewpoint coverage index value: The initial search results are converted into a sequence, and a sliding window is configured in each sequence to record the information source to which each initial search result belongs in the sliding window. As the sliding window slides on the sequence, for each initial search result that enters the range of the sliding window, a penalty is applied to the original score corresponding to the initial search result based on the number of times the information source to which the initial search result belongs has appeared in the sliding window, and the information source coverage index value of the initial search result is determined. Identify subtopics related to the search instruction; determine the opinion coverage index value for each initial search result based on the mean semantic similarity between each initial search result and each subtopic.

7. The method according to claim 1, characterized in that, The second ranking model includes an input layer, a shared network, a gated network, a branching network, and an output layer; obtaining the second ranking model and filtering the target search result set from the first search result set includes: The input layer receives the first search result set and the search instruction, and generates a feature vector for each first search result. The common feature representation of the feature vector of each first search result is extracted through the shared underlying network; The branch weights corresponding to each branch network are calculated using the gating network. Different branch networks are used to execute different subtasks. Each branch network is connected to the shared underlying network through the gating network. Each branch network determines the branch feature representation corresponding to each subtask. By combining the general feature representation, the feature representation of each branch, and the corresponding weights of each branch in the output layer, the second ranking score of each first search result is determined. A second sorting operation is performed based on the second sorting score of each first search result. The target search result is selected from the first search results after the second sorting operation according to a preset method, and the target search result set is formed.

8. The method according to claim 7, characterized in that, The subtasks include predicting the adoption probability of search results and predicting the user dwell time of search results; The methods for training the second ranking model include: Acquire training data, input the training data into the second ranking model, and determine the first loss function of the second ranking model regarding priority ranking, the second loss function regarding adoption status, and the third loss function regarding dwell time; The target loss function is determined based on the first loss function, the second loss function, the third loss function, and the weights corresponding to each loss function. The parameters of the second ranking model are trained in an end-to-end manner based on the target loss function.

9. The method according to claim 1, characterized in that, Methods for constructing a source database include: For each information source, each data in the information source is preprocessed, and each data is decomposed into modal data corresponding to each modal type according to the modal type; Extract the semantic vectors of the modal data corresponding to each modality type, and map each modality data to the same semantic vector space; Based on each semantic vector in the semantic vector space, construct the source database corresponding to the source.

10. A data search device, characterized in that, The device includes: The acquisition module is used to acquire search instructions; the search instructions include at least one target keyword; The search module is used to acquire at least one source database, perform semantic similarity search on the at least one target keyword in each source database, and obtain the initial search results for each source. The first sorting module is used to perform a first sorting operation on the initial search results based on the semantic similarity between the initial search results and the target keyword, so as to obtain a first search result set; The second sorting module is used to obtain a second sorting model and filter the target search results set from the first search results set.

Images