Search model training method and device, equipment, medium and program product

Abstract

The invention discloses a search model training method and device, equipment, a medium and a program product, and relates to the technical field of computers, the method comprises the following steps: obtaining a plurality of search contents and a plurality of multimedia contents, the search contents being search conditions during data search, and the multimedia contents being contents in search results; correlation analysis is conducted between the multiple pieces of search content and the multiple pieces of multimedia content, multiple search data pairs are obtained, and the search data pairs comprise correlation labels between the search content and the multimedia content; the first model is trained on the basis of the correlation labels corresponding to the multiple pieces of search data, a search model is obtained, the search model is used for converting the input search content and the multimedia content into feature representation, and the correlation degree between the input search content and the multimedia content is obtained. The intention and expected search results when the user inputs the search content can be fully understood by using the search model obtained by training.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, device, medium and program product for training a search model. Background Art

[0002] Search engines in the music field can provide music search services for users. Users enter query words into the search engine, and the search engine will feedback music content related to the query words, such as: song information, singer information, playlist information, etc.

[0003] In the related art, search engines in the field of music can provide a playlist search function. After the user enters a query word, the search engine performs keyword matching on the playlist content in the database based on the query word, and feeds back the playlist content containing the query word.

[0004] However, by relying solely on keyword matching, the search engine will return all playlist content containing the keyword, ignoring the user's intention, resulting in a large amount of irrelevant content in the search results and low search result accuracy. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device, medium and program product for training a search model, which can train a search model, use the search model to convert input search content and multimedia content into feature representations, determine the correlation between the search content and multimedia content based on the feature representations, and fully understand the user's intention when inputting the search content and the expected search results. The technical solution is as follows:

[0006] In one aspect, a method for training a search model is provided, the method comprising:

[0007] Acquire multiple search contents and multiple multimedia contents, wherein the search contents are search conditions during data search, and the multimedia contents are contents in the search results;

[0008] Performing correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple search data pairs, wherein the search data pairs include correlation labels between the search contents and the multimedia contents, and the correlation labels are used to describe the degree of association between the search contents and the multimedia contents;

[0009] The first model is trained based on the correlation labels corresponding to the multiple search data pairs to obtain the search model, which is used to convert the input search content and multimedia content into feature representations to obtain the degree of association between the input search content and the multimedia content.

[0010] In another aspect, a training device for a search model is provided, the device comprising:

[0011] An acquisition module, used to acquire multiple search contents and multiple multimedia contents, wherein the search contents are search conditions during data search, and the multimedia contents are contents in the search results;

[0012] a correlation analysis module, configured to perform correlation analysis on the plurality of search contents and the plurality of multimedia contents to obtain a plurality of search data pairs, wherein the search data pairs include correlation labels between the search contents and the multimedia contents, and the correlation labels are used to describe the degree of association between the search contents and the multimedia contents;

[0013] A training module is used to train a first model based on the correlation labels corresponding to the multiple search data pairs to obtain the search model, and the search model is used to convert the input search content and multimedia content into feature representations to obtain the degree of association between the input search content and the multimedia content.

[0014] In an optional embodiment, the acquisition module is also used to acquire search records within a historical time period, wherein the historical time period refers to a time period of a preset length from a current moment, and the search record contains the multiple search contents received; based on the search record, candidate multimedia contents corresponding to the multiple search contents are acquired, wherein the candidate multimedia content corresponding to the mth search content is at least one content included in the search results when data is searched based on the mth search content, and m is a positive integer; for the mth search content, click events corresponding to the at least one content are counted; based on the number of click events, at least one target content is determined from the candidate multimedia contents corresponding to the mth search content; based on the at least one target content corresponding to the multiple search contents, the multiple multimedia contents are obtained.

[0015] In an optional embodiment, the acquisition module is also used to sort the at least one content based on the number of click events for the candidate multimedia content corresponding to the m-th search content, and determine the first a contents to be sorted as the at least one target content corresponding to the m-th search content, where a is a positive integer.

[0016] In an optional embodiment, the correlation analysis module is also used to perform correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple first search data pairs; perform matching analysis on multiple contents in the multimedia content library based on the multiple search contents to obtain multiple second search data pairs; and obtain the multiple data search pairs based on the multiple first search data pairs and the multiple second search data pairs.

[0017] In an optional embodiment, the matching analysis includes at least one of the following methods:

[0018] Perform keyword matching on multiple contents in the multimedia content library based on the multiple search contents, and the multimedia contents in the second search data pair include the search contents; or

[0019] Based on the multiple search contents, content tag matching is performed on the multiple contents in the multimedia content library, the multiple contents in the multimedia content library are respectively marked with preset content tags, and the preset content tags are used to describe the content theme of the multimedia content. The preset content tags marked by the multimedia content in the second search data pair match the search contents.

[0020] In an optional embodiment, the correlation analysis module is further used to integrate the multiple search contents and the multiple multimedia contents to obtain multiple initial data pairs, wherein the initial data pairs include a search content and a multimedia content; input the multiple initial data pairs into a pre-trained correlation analysis model, and output the correlation scores corresponding to the multiple initial data pairs respectively; determine the correlation labels corresponding to the multiple initial data pairs respectively based on the correlation scores, and obtain the multiple first search data pairs, wherein the i-th first search data pair includes the i-th initial data pair and the corresponding correlation label.

[0021] In an optional embodiment, the training module is also used to input the kth search data pair into the first model; perform feature extraction on the i-th search content and the j-th multimedia content contained in the k-th search data pair through the first model to obtain a first feature representation corresponding to the i-th search content and a second feature representation corresponding to the j-th multimedia content; i, j, k are positive integers; calculate the similarity between the first feature representation and the second feature representation to obtain the similarity corresponding to the k-th search data pair; train the first model based on the loss value between the similarity corresponding to the k-th search data pair and the correlation label to obtain the search model.

[0022] In an optional embodiment, the device further comprises:

[0023] The feature extraction module is used to obtain a multimedia content library, which contains multiple contents; extract features from the multimedia content library through the search model to obtain a feature representation library, which contains feature representations corresponding to the multiple contents respectively.

[0024] In an optional embodiment, the device further comprises:

[0025] The feature extraction module is further configured to receive a first search request, wherein the first search request includes a first search content; extract features of the first search content using the search model to obtain a feature representation corresponding to the first search content;

[0026] The search module is used to calculate the similarity between the feature representation corresponding to the first search content and the feature representation in the feature representation library to obtain a similarity result; sort the multiple contents in the multimedia content library based on the similarity result to obtain a sorting result; and determine at least one multimedia content based on the sorting result as a search result corresponding to the first search content.

[0027] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a training method for a search model as described in any of the above-mentioned embodiments of the present application.

[0028] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a training method for a search model as described in any of the above-mentioned embodiments of the present application.

[0029] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the training method of the search model described in any of the above embodiments.

[0030] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0031] By performing a correlation analysis on the search content and multimedia content, determining the correlation labels between the search content and the multimedia content, and integrating the search content and the multimedia content into a data pair as training data for training the search model, the trained search model can learn the correlation between the search content and the multimedia content, and convert the search content and the multimedia content into a more accurate feature representation. When the search model is subsequently applied, it can extract features from the multimedia content already in the database and the newly received search content, fully understand the user's intention when entering the search content, obtain an accurate feature representation, and index the matching multimedia content as the search result based on the degree of correlation between the feature representations, thereby improving the accuracy of the search results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 is a schematic diagram of a search model training system provided by an exemplary embodiment of the present application;

[0034] Figure 2 is a flow chart of a method for training a search model provided by an exemplary embodiment of the present application;

[0035] Figure 3 is a flowchart of a method for obtaining search data pairs provided by an exemplary embodiment of the present application;

[0036] Figure 4 is a flow chart of a method for performing data search based on a search model provided by an exemplary embodiment of the present application;

[0037] Figure 5 is a structural block diagram of a training device for a search model provided by an exemplary embodiment of the present application;

[0038] Figure 6 is a structural block diagram of a training device for a search model provided by another exemplary embodiment of the present application;

[0039] Figure 7 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0042] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be noted that the information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0044] It should be understood that, although the terms first, second, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] First, a brief introduction is given to the terms involved in the embodiments of this application:

[0046] Large Language Models (LLM): A type of technology in the field of natural language processing. These models can understand and generate text similar to human language by analyzing and learning large amounts of text data.

[0047] The core technologies of LLM include self-supervised learning, natural language processing, and pre-trained language models. Self-supervised learning allows the model to learn the intrinsic structure and patterns of language on a large amount of unlabeled data, and trains the model by predicting the next word or sentence in the text. Natural language processing technology enables computers to understand and generate human language text, involving text preprocessing, feature extraction, and model training. Pre-trained language models learn the intrinsic structure and semantic information of language by pre-training on a large amount of unlabeled text data, so that contextual information and semantic coherence can be better considered when generating text.

[0048] The search model and relevance analysis model mentioned in this application all belong to LLM, wherein the architecture of the first model used in training the search model and the architecture of the pre-trained model used in training the relevance analysis model are both determined based on LLM.

[0049] The trained search model can extract features from the input search content and multimedia content to obtain corresponding feature representations. The similarity between the feature representations corresponding to the search content and the feature representations corresponding to the multimedia content can reflect the degree of association between the search content and the multimedia content, indicating the association between the search content and the multimedia content at the content theme and semantic level.

[0050] The trained correlation analysis model can calculate the correlation scores between the input data pairs (each data pair contains a search content and a multimedia content), and generate corresponding correlation labels for the data pairs based on the intervals to which the correlation scores belong. There is a preset correspondence between the divided score intervals and the correlation labels.

[0051] For example, data pair A includes search content A and multimedia content A. When data pair A is input into the correlation analysis model, the correlation score between search content A and multimedia content A is calculated to be 70. The corresponding relationship between the score range and the correlation label is as follows:

[0052] The correlation label corresponding to the score interval [0, 25] is [weak correlation], the correlation label corresponding to the score interval (25, 50] is [relatively weak correlation], the correlation label corresponding to the score interval (50, 75] is [relatively strong correlation], and the correlation label corresponding to the score interval (75-100] is [strong correlation].

[0053] Among them, the correlation degrees corresponding to the correlation labels are ranked from low to high as follows: [weak correlation] < [weaker correlation] < [stronger correlation] < [strong correlation].

[0054] Then, the correlation score of 70 belongs to the score interval (50, 75], and the corresponding correlation label is [strong correlation].

[0055] In actual application scenarios, when a user enters search content to search for data, the correlation between the search content and the multimedia content is calculated, and the content with a higher correlation degree is preferentially fed back to the user as the search result.

[0056] As a key tool for Internet information retrieval, search engines play the role of a bridge connecting users with the ocean of network information. Search engines can index the massive multimedia content (including text, pictures, videos, audio and other forms of multimedia content) on the Internet or in databases based on search conditions, filter out the most relevant results and present the results to users.

[0057] The design of search engines aims to simplify the user's operation process. By providing a user-friendly input interface, users can easily enter query requests and efficiently retrieve and locate the information of their interest.

[0058] For example, when using a search engine in a music scene to search for a playlist, a user can send a query instruction to the search engine by entering search content such as the name of the song, the name of the singer, or the theme of the playlist.

[0059] After receiving the search content entered by the user, the search engine will match it in its huge database and finally return the playlist content related to the search content as the search result.

[0060] However, current search engine technologies often rely solely on keyword matching mechanisms when processing user queries.

[0061] For example, the search content input by the user is text content, and the search engine performs keyword matching based on the text content, and feeds back the playlist content whose playlist name contains the text content to the user as the search result, and the playlist content contains at least one song to be played.

[0062] Although this method can quickly return content containing specific keywords, due to the lack of in-depth understanding of user query intent, it often leads to search results filled with a large amount of information that is irrelevant to the user's actual needs, resulting in low accuracy of search results.

[0063] Next, the training system of the search model involved in the embodiment of the present application is described. For illustration, please refer to Figure 1 The system involves a client 110 and a server 120 , and there is a communication connection between the client 110 and the server 120 .

[0064] The server 120 can provide a data search service. For example, a search application or a search engine is deployed in the server 120 , and the client 110 can provide a user with an interface entry for invoking the data search service.

[0065] In some embodiments, a search application or a search engine is deployed in the client 110, which can directly provide data search services to users.

[0066] The client 110 displays a search interface, which can receive the first search content input by the user. Exemplarily, the first search content may be in the form of text, audio, image, video, animation, or a file in a specific format.

[0067] After receiving the first search content, the client 110 generates a search request and sends the search request to the server 120 .

[0068] The search request includes a first search content, and the search request is used to request the server 120 to perform a data search based on the first search content, obtain search results from a database corresponding to the server 120, and feed back the search results to the user.

[0069] A pre-trained search model is deployed in the server 120, and the first search content is input into the search model for feature extraction to obtain a first feature representation corresponding to the first search content.

[0070] The server 120 corresponds to a database and a feature representation library. The database contains multiple multimedia contents. The multiple feature representations contained in the feature representation library are obtained by extracting features of the multimedia contents respectively through a search model. The i-th multimedia content in the database corresponds to the i-th feature representation in the feature representation library, where i is a positive integer.

[0071] Among them, similarity calculation is performed based on the first feature representation and multiple feature representations in the feature representation library to obtain the similarity between the first feature representation and each feature representation, and the similarity can reflect the degree of association between the first search content and each multimedia content in the database.

[0072] The multimedia contents are sorted in descending order based on the similarity values, and the first n multimedia contents are used as search results, where n is a positive integer.

[0073] The server 120 sends the search results to the client 110 , and the client 110 displays the multimedia content in the search results according to the ranking results.

[0074] The training phase of the search model and the feature extraction phase of the multimedia content in the database are performed before receiving the first search content.

[0075] The training phase of the search model includes the following steps:

[0076] Step 1: Acquire multiple search contents and multiple multimedia contents.

[0077] The search records within the historical time period are obtained, where the search records contain multiple search contents, and the multiple search contents are search contents input by at least one user who uses the same search engine or data search service when searching within the historical time period.

[0078] The historical time period refers to a time period of a preset length from the current moment. Each time a user enters a search content, the search results fed back by the search engine contain at least one multimedia content.

[0079] Based on the search records, historical tracing is performed to determine the clicks of users on multimedia content displayed in the search results after entering the search content, the number of times each multimedia content is clicked by the user is recorded, and multimedia content whose click times meet preset requirements (for example, reach a preset threshold) is filtered out to obtain multiple multimedia content.

[0080] There is a click event association between the multiple search contents and the multiple multimedia contents.

[0081] Step 2: Perform correlation analysis between multiple search contents and multiple multimedia contents to obtain multiple search data pairs.

[0082] Multiple search contents and multiple multimedia contents are grouped into data pairs in pairs, thereby obtaining multiple data pairs, each of which contains one search content and one multimedia content.

[0083] For example, there are 2 search contents: search content A and search content B, and 3 multimedia contents: multimedia content 1, multimedia content 2, and multimedia content 3, which constitute 2*3=6 data pairs, as follows: Data pair 1: search content A and multimedia content 1, data pair 2: search content A and multimedia content 2, data pair 3: search content A and multimedia content 3, data pair 4: search content B and multimedia content 1, data pair 5: search content B and multimedia content 2, data pair 6: search content B and multimedia content 3.

[0084] The six data pairs are input into the pre-trained relevance analysis model, which can output the relevance label between the search content and the multimedia content in each data pair.

[0085] Based on the correlation tags corresponding to the multiple data pairs, multiple search data pairs are obtained, and each search data pair includes a search content, a multimedia content, and a correlation tag between the two.

[0086] Step 3: Train the first model based on the correlation labels corresponding to the multiple search data pairs to obtain a search model.

[0087] A plurality of search data pairs are input as training data into the first model, and the first model learns the correlation between the search content and the multimedia content in the search data pairs, and converts the search content and the multimedia content into feature representations.

[0088] The similarity between the feature representation of the search content and the feature representation of the multimedia content is calculated, and the first model is trained with loss based on the difference between the similarity and the relevance label to obtain the search model.

[0089] The trained search model is used to extract features from multimedia content in the database to obtain feature representations of each multimedia content.

[0090] The devices involved in the above-mentioned client 110 can be various forms of terminal devices such as mobile phones, tablet computers, desktop computers, portable laptops, smart TVs, car terminals, smart home devices, etc., and the embodiments of the present application are not limited to this.

[0091] It is worth noting that the above-mentioned server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0092] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0093] In some embodiments, the server 120 may also be implemented as a node in a blockchain system.

[0094] In combination with the above-mentioned noun introduction and application scenarios, the training method of the search model provided in the present application is described. The method can be executed by a server or a terminal, or can be executed by a server and a terminal together. In the embodiment of the present application, the method is executed by a server as an example. Figure 2 As shown, Figure 2 1 is a flowchart of a method for training a search model provided by an exemplary embodiment of the present application. The method comprises the following steps.

[0095] Step 210: Acquire multiple search contents and multiple multimedia contents.

[0096] Among them, the search content is the search condition when searching for data, and the multimedia content is the content in the search results.

[0097] An application program that can provide data search services is running on the server and the terminal / a search engine is deployed. The user enters the search content through the interface entrance of the data search service provided by the terminal to call the data search service. The terminal sends the search content to the server, and the server indexes from its corresponding database based on the search content, obtains the search results and returns them to the terminal, which displays them to the user.

[0098] The search content may be in the form of text, audio, image, video, animation, or file content in a specific format.

[0099] The search results include at least one multimedia content. When there are multiple multimedia contents in the search results, the multiple multimedia contents are displayed based on a preset arrangement order.

[0100] The multimedia content includes but is not limited to at least one of the following forms of content: (1) text content: such as information flow articles, academic reports, news content, and e-book content; (2) audio content: such as song content, playlist content, dubbing content, recording content, and accompaniment audio content; (3) picture content: such as photos of people, landscape photos, animal photos, paintings, promotional pictures, and poster content; (4) video content: such as filmed videos, recorded videos, surveillance videos, and film and television content;

[0101] (5) Animation content: such as dynamic image content and animated film and television content; (6) File content in a specific format, etc.

[0102] Exemplarily, the multimedia content is a playlist content, which includes audio content (at least one song to be played), picture content (playlist cover), and text content (playlist introduction).

[0103] The information contained in the search content may be arbitrary, and data search services in different application fields correspond to different databases, which can provide corresponding search results according to the search content.

[0104] Optionally, the data search service can be applied to areas including but not limited to the following:

[0105] (1) Music: Data search services can be used to retrieve information such as songs, playlists, and albums.

[0106] (2) Financial field: Data search services are used in financial data retrieval, market analysis, and investment decision-making.

[0107] (3) Academic field: Data search services are used in literature retrieval, academic research, and knowledge management.

[0108] (4) Medical field: Data search services are used in medical literature retrieval, disease diagnosis, and drug search.

[0109] (5) Manufacturing industry: Data search services are used in production process monitoring, quality control, and fault diagnosis.

[0110] (6) Media and entertainment: Data search services are used for content retrieval, copyright management, and user recommendations.

[0111] (7) Education and training: Data search services are used in learning resource retrieval, course management, and learning analysis.

[0112] For example, in a data search service in the field of music, the multimedia content that the user expects to search is a playlist content, and the playlist content contains at least one song. According to the expression form of the search content, the search content may include the following information:

[0113] 1. Text content: The search content is the text content containing the playlist theme / playlist name / lyrics / song name / singer name / song theme / multimedia work (the multimedia work contains song content) name.

[0114] For example, the user enters the text "name of singer A - playlist" and expects to search for playlist content related to singer A.

[0115] 2. Audio content: The search content is audio content containing songs / instrumental music / voice clips.

[0116] For example, a user uploads audio content obtained when recording song A, and hopes to search for playlist content containing song A.

[0117] 3. Image content: The search content includes images downloaded from the Internet / album cover images / singer photos / promotional images of certain film and television works / image content.

[0118] For example, a user uploads a photo of singer A and hopes to search for playlist content related to singer A.

[0119] 4. Video content: The search content is video content that includes a certain song MV (Music Video) clip / a certain film or TV work clip / a certain music performance clip.

[0120] For example, a user uploads a video clip containing a recording of film or TV work A, and expects to search for a playlist content related to film or TV work A.

[0121] 5. Animation content: The search content is animation content that contains animation clips or dynamic graphics related to a certain musical work.

[0122] For example, a user uploads an animation clip related to animation work A, hoping to search for playlist content related to the animation work.

[0123] 6. File content in a specific format: The search content is a file in a specific format, and the file content in this format can be parsed to obtain relevant information.

[0124] For example, a user uploads a music file in MP3 format (an audio encoding format), and the music file can be parsed. The parsed result indicates that the music file is the audio file of song A, and it is expected to search for playlist content related to song A.

[0125] For example, in a data search service in the field of education and training, the search content may include the following information according to the presentation form of the search content:

[0126] 1. Text content: The search content is the text content containing subject name / textbook name / teacher name / university name.

[0127] For example, the user enters "tutoring institution A-high mathematics" and expects to search for high mathematics course content published by tutoring institution A.

[0128] 2. Audio content: The search content is the audio content including course recordings / listening test content.

[0129] For example, a user uploads audio content containing an English listening test segment and expects to search for complete listening audio content corresponding to the listening segment.

[0130] 3. Image content: The search content is the image content containing the cover of a textbook / a teacher’s photo / a university emblem / a logo of an educational institution.

[0131] For example, a user uploads a cover image of a university advanced mathematics textbook, hoping to search for course content corresponding to the advanced mathematics textbook.

[0132] 4. Video content: The search content is video content that includes clips of a teacher’s teaching / clip clips of an online course / clip clips of a school’s public lecture.

[0133] For example, a user uploads a video clip of an advanced mathematics online course and expects to search for online course content related to the video clip.

[0134] 5. Animation content: The search content includes animation content mentioned in teaching courseware / teaching courses.

[0135] For example, a user uploads a playable animation content included in a teaching courseware of a high-level mathematics course, and hopes to search for content related to the teaching courseware.

[0136] 6. File content in a specific format: The search content is a file in a specific format, and the file content in this format can be parsed to obtain relevant information.

[0137] For example, a user uploads a teaching courseware in the format of an e-book and expects to search for content related to the teaching courseware.

[0138] Optionally, the process of acquiring multiple search contents and multiple multimedia contents is as follows.

[0139] S1: Get the search records within the historical time period.

[0140] The search history contains multiple search contents received. The historical time period refers to a time period of a preset length from the current moment. The determination methods of the historical time period include but are not limited to the following.

[0141] (1) The preset duration is the specified duration, and the time period from the current time to the specified duration is the historical time period. For example, if the current time is 12:00:00 on January 20, 2020, and the specified duration is 240 hours, then the historical time period is the time period between 12:00:00 on January 10, 2020 and 12:00:00 on January 20, 2020.

[0142] (2) Determine the preset duration based on the most recent N search records generated, obtain N search records generated before the current time, and determine the historical time period based on the time corresponding to the earliest generated record in the search records, where N is a positive integer. For example, if N is 100, obtain the 100 search records generated before the current time, and determine the historical time period with the time corresponding to the earliest generated record among the 100 search records (12:00:00 on January 10, 2020) as the starting point and the current time (12:00:00 on January 20, 2020) as the end point.

[0143] S2: Acquire candidate multimedia contents respectively corresponding to the plurality of search contents based on the search records.

[0144] The candidate multimedia content corresponding to the m-th search content is at least one content included in the search results when data search is performed based on the m-th search content, and m is a positive integer.

[0145] Exemplarily, the historical time period includes 10 search records, and the 10 search records are generated when different users input search content to perform data search.

[0146] The 10 search records contain 3 different search contents: search content A, search content B, and search content C.

[0147] The search results corresponding to search content A include 1 candidate multimedia content: multimedia content 1; the search results corresponding to search content B include 2 candidate multimedia contents: multimedia content 2 and multimedia content 3; the search results corresponding to search content C include 2 candidate multimedia contents: multimedia content 1 and multimedia content 4.

[0148] S3: for the mth search content, count the click events corresponding to at least one content, and determine at least one target content from the candidate multimedia contents corresponding to the mth search content based on the number of click events.

[0149] Exemplarily, for candidate multimedia content corresponding to the mth search content, at least one content is sorted based on the number of click events, and the first a contents in the sorting are determined as at least one target content corresponding to the mth search content, where a is a positive integer.

[0150] Among them, click events include the following: (1) direct click events, in which the user directly clicks on the multimedia content displayed in the search results after entering the search content; (2) indirect click events, in which the user enters the first search content but does not directly click on the multimedia content displayed in the search results corresponding to the first search content; after entering the second search content within a preset time after entering the first search content, the user directly clicks on the multimedia content displayed in the search results corresponding to the second search content. Since the second search content is regarded as an associated search event for the first search content, the multimedia content with a click event in the associated search event is regarded as the multimedia content corresponding to the first search content.

[0151] For example, for search content A, multimedia content 1 in its candidate multimedia content corresponds to 3 click events in 10 search records.

[0152] a is 1, and multimedia content 1 is determined as the target content corresponding to search content A.

[0153] For example, for search content B, multimedia content 2 in its candidate multimedia content corresponds to 3 click events in 10 search records, and multimedia content 3 in its candidate multimedia content corresponds to 2 click events in 10 search records. The order based on the number of click events is: multimedia content 2, multimedia content 3.

[0154] a is 1, and multimedia content 2 is determined as the target content corresponding to search content B.

[0155] For example, for search content C, multimedia content 1 in the candidate multimedia content corresponds to 2 click events in 10 search records, and multimedia content 4 in the candidate multimedia content corresponds to 1 click event in 10 search records. The order based on the number of click events is: multimedia content 1, multimedia content 4.

[0156] a is 1, and multimedia content 1 is determined as the target content corresponding to the search content C.

[0157] S4: Obtain multiple multimedia contents based on at least one target content corresponding to each of the multiple search contents.

[0158] The target contents corresponding to all the search contents are integrated, and the duplicate parts are removed to obtain multiple multimedia contents.

[0159] For example, the target content corresponding to search content A is multimedia content 1, the target content corresponding to search content B is multimedia content 2, and the target content corresponding to search content C is multimedia content 1. Remove the repeated parts and determine that the multiple multimedia contents include multimedia content 1 and multimedia content 2.

[0160] Step 220: performing correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple search data pairs.

[0161] The search data pair includes a correlation tag between the search content and the multimedia content, and the correlation tag is used to describe the degree of association between the search content and the multimedia content.

[0162] Exemplarily, the number of types of correlation tags is preset, and each correlation tag corresponds to a different degree of association.

[0163] For example, the correlation labels include the following four categories: weak correlation, relatively weak correlation, relatively strong correlation, and strong correlation. The correlation degrees corresponding to the correlation labels are ranked from low to high as follows: [weak correlation] < [relatively weak correlation] < [relatively strong correlation] < [strong correlation].

[0164] A search data pair refers to a data pair obtained by pairing multiple search contents and multiple multimedia contents in pairs. Each search data pair contains a search content and a multimedia content. The same search content can form a search data pair with different multimedia contents, and the same multimedia content can also form a search data pair with different search contents.

[0165] Exemplarily, the number of search contents is a first number, and the number of multimedia contents is a second number, and the number of search data pairs is determined based on the product of the first number and the second number, and the first number and the second number are positive integers.

[0166] For example, there are two search contents: search content A and search content B. Search content A corresponds to multimedia content 1 and multimedia content 2, and search content B corresponds to multimedia content 3 and multimedia content 4. Then, 2*4=8 search data pairs are formed, which are as follows:

[0167] Search data pair 1: Search content A and multimedia content 1, corresponding to relevance label 1;

[0168] Search data pair 2: Search content A and multimedia content 2, corresponding to relevance label 2;

[0169] Search data pair 3: Search content A and multimedia content 3, corresponding to relevance label 3;

[0170] Search data pair 4: search content A and multimedia content 4, corresponding to relevance label 4;

[0171] Search data pair 5: Search content B and multimedia content 1, corresponding to relevance label 5;

[0172] Search data pair 6: search content B and multimedia content 2, corresponding to relevance label 6;

[0173] Search data pair 7: Search content B and multimedia content 3, corresponding to relevance tag 7;

[0174] Search data pair 8: search content B and multimedia content 4, corresponding to correlation tag 8.

[0175] The correlation labels between the search content and the multimedia content contained in the search data pair are obtained in the following manner: multiple search contents and multiple multimedia contents are paired in pairs and input into a pre-trained correlation analysis model.

[0176] The pre-trained relevance analysis model is a relevance classification LLM, and the relevance analysis model can analyze the semantics of the search content and the multimedia content in each initial data pair.

[0177] For example, when a piece of text is input into the relevance analysis model, the model will split the text into at least one word and convert each word into a vector form, learning the basic semantics of the word during the conversion process. The model's self-attention mechanism identifies the correlation between words and the overall meaning of each word in the text content.

[0178] A vector corresponding to the text content is formed based on the vectors of the words and the correlation between the words. Vectors corresponding to different text contents are obtained, and the degree of correlation between the vectors is calculated through the correlation analysis model. The correlation score between the search content and the multimedia content is output through the correlation analysis model, and the type of correlation label is determined based on the score interval to which the correlation score belongs.

[0179] There is a preset corresponding relationship between the score range of the correlation score and the type of the correlation label.

[0180] Exemplarily, the value range of the relevance score is [0, 100], which is divided into 4 different score intervals, and each score interval corresponds to a relevance label.

[0181] For example, the correlation label corresponding to the score interval [0, 25] is [weak correlation], the correlation label corresponding to the score interval (25, 50] is [relatively weak correlation], the correlation label corresponding to the score interval (50, 75] is [relatively strong correlation], and the correlation label corresponding to the score interval (75-100] is [strong correlation].

[0182] The correlation analysis model outputs the correlation score and correlation label between the search content and the multimedia content in each search data pair.

[0183] For example, search data pair 1: the correlation score between search content A and multimedia content 1 is 80, and the correlation label is [strong correlation].

[0184] Step 230: Train a first model based on correlation labels corresponding to a plurality of search data pairs to obtain a search model.

[0185] The search model is used to convert the input search content and multimedia content into feature representations to obtain the degree of association between the input search content and the multimedia content.

[0186] The search model converts the search content into the corresponding search content feature representation, converts the multimedia content into the corresponding multimedia content feature representation, and maps the search content feature representation and the multimedia content feature representation into the same feature representation space. By calculating the vector distance between the search content feature representation and the multimedia content feature representation, the degree of association between the search content and the multimedia content can be obtained.

[0187] Optionally, when training the first model, multiple search data pairs are sequentially input into the first model for training, with correlation labels contained in the search data pairs being used as training targets.

[0188] Exemplarily, the training process is described by taking the training of the first model when the k-th search data pair is used as training data as an example.

[0189] For the kth search data pair, the kth search data pair is input into the first model.

[0190] The first model is used to extract features from the i-th search content and the j-th multimedia content included in the k-th search data pair to obtain a first feature representation corresponding to the i-th search content and a second feature representation corresponding to the j-th multimedia content.

[0191] Exemplarily, when the first model extracts features from input content (including search content and multimedia content), the input content is divided into different units, which are called tokens. Each token is mapped to a pre-trained high-dimensional vector, which is called a word embedding / word vector and contains the semantic information of the token. Position information is added to each token, and the position information is used to indicate the order of different tokens in the input content (sentence). The relationship between each token and other tokens is analyzed through the self-attention mechanism, the importance of each token in the context is determined, and the vector of the token is updated in the process. A more comprehensive contextual information is provided for each token through a multi-head attention mechanism. The word vector output by the self-attention layer is processed using a feedforward network, and the word vector is further nonlinearly transformed to extract deeper features. The output of the feedforward network is normalized and residually connected. After repeating the above process for a preset number of times, the final feature representation is output. The first feature representation and the second feature representation are high-dimensional vectors, and the number of dimensions of the high-dimensional vector is a preset positive integer.

[0192] For example, the dimensions of the first feature representation and the second feature representation are both 256.

[0193] For example, the dimension of the first feature representation is 200, and the dimension of the second feature representation is 256.

[0194] The similarity between the first feature representation and the second feature representation is calculated to obtain the similarity corresponding to the kth search data pair.

[0195] Among them, the methods of obtaining the similarity between the first feature representation and the second feature representation include but are not limited to the following: (1) calculating the Euclidean distance between the first feature representation and the second feature representation as the similarity; (2) calculating the cosine similarity between the first feature representation and the second feature representation as the similarity; (3) calculating the Manhattan distance between the first feature representation and the second feature representation as the similarity, etc.

[0196] Exemplarily, the cosine similarity between the first feature representation and the second feature representation is calculated as the similarity.

[0197] Cosine similarity is a measure of the similarity of the angle between two non-zero vectors, and its value ranges from -1 to 1. Cosine similarity is determined by calculating the ratio of the dot product of two vectors and the product of their moduli. If the two vectors are exactly the same, their cosine similarity is 1; if the two vectors are completely opposite, the cosine similarity is -1; if the two vectors are orthogonal (i.e. unrelated), the cosine similarity is 0.

[0198] In this embodiment, the cosine similarity between the first feature representation and the second feature representation has a value range of 0 to 1. When the cosine similarity approaches 0, it indicates that the correlation between the search content and the multimedia content is lower. When the cosine similarity approaches 1, it indicates that the correlation between the search content and the multimedia content is higher.

[0199] For example, when the cosine similarity value range is [0, 0.25], the corresponding correlation label is [weak correlation], when the cosine similarity value range is (0.25, 0.5], the corresponding correlation label is [relatively weak correlation], when the cosine similarity value range is (0.5, 0.75], the corresponding correlation label is [relatively strong correlation], and when the cosine similarity value range is (0.75, 1], the corresponding correlation label is [relatively weak correlation].

[0200] The first model is trained based on the loss values ​​between the similarity and relevance labels corresponding to the k-th search data pair to obtain a search model.

[0201] Exemplarily, an actual output label output by the first model is determined based on the similarity corresponding to the k-th search data pair, and a loss value is determined based on a difference between the actual output label and the correlation label.

[0202] If the actual output label is the same as the correlation label, no loss training is required; if the actual output label is different from the correlation label, the loss value is determined according to the preset loss value comparison table.

[0203] For example, the loss value comparison table indicates: the loss value between [weak correlation] and [weaker correlation] is 0.25, the loss value between [weak correlation] and [stronger correlation] is 0.5, the loss value between [weak correlation] and [strong correlation] is 0.75, the loss value between [weaker correlation] and [stronger correlation] is 0.25, the loss value between [weak correlation] and [strong correlation] is 0.5, and the loss value between [stronger correlation] and [strong correlation] is 0.25.

[0204] In some embodiments, in order to improve the accuracy of adjusting the training of the first model based on the loss value, the correlation score corresponding to the correlation label is normalized to obtain a normalized score with the same similarity value range, and the loss value is determined based on the difference between the normalized score and the similarity.

[0205] For example, the correlation label between search content A and multimedia content 1 in search data pair 1 is [strong correlation], the correlation score is 80, and the correlation score is normalized based on the maximum value of the correlation score range, and the normalized score is 80 / 100=0.8.

[0206] The first model extracts features from the search content A and the multimedia content 1 in the search data pair 1 and obtains a cosine similarity of 0.7 between the feature representations, and thus determines the loss value to be 0.8-0.7=0.1.

[0207] A preset loss value threshold is obtained. When the loss value reaches the loss value threshold, loss training is performed on the first model based on the loss value; when the loss value does not reach the loss value threshold, loss training on the first model is stopped. The loss training process is a process in which the first model adjusts its internal parameters.

[0208] For example, the preset loss value threshold is 0.05. When the loss value is 0.1, the loss value threshold is reached, and loss training is performed on the first model based on the loss value.

[0209] In some embodiments, when the first model is trained based on the loss values ​​corresponding to different search data pairs, the parameters may be adjusted multiple times. In order to improve the training efficiency, it can be determined whether the first model has been trained based on the following method. When training the first model, the cosine similarity value corresponding to the correlation label is used as the training target. The first model calculates the cosine similarity of the input multiple search data pairs respectively, and determines the search data pairs whose cosine similarity value range matches the type of its own correlation label as the target data pairs. The proportion of the target data pairs in all search data pairs is calculated to obtain the first proportion value. If the first proportion value reaches the preset proportion threshold, the training of the first model is stopped to obtain the search model.

[0210] For example, the preset proportion threshold is 95%, and there are 100 search data pairs in total. After training the first model based on the 100 search data pairs in sequence, it is determined that the number of target data pairs is 98, and the first proportion value is 98 / 100=98%. When the preset proportion threshold is reached, the training is stopped to obtain the search model.

[0211] In summary, the search model training method provided by the present application determines the correlation labels between the search content and the multimedia content by performing correlation analysis on the search content and the multimedia content, and integrates the search content and the multimedia content into a data pair as training data for training the search model, so that the trained search model can learn the correlation between the search content and the multimedia content, and convert the search content and the multimedia content into a more accurate feature representation. When the search model is subsequently applied, it is possible to extract features from the multimedia content already in the database and the newly received search content, fully understand the user's intention when entering the search content, obtain an accurate feature representation, and index the matching multimedia content as the search result based on the degree of correlation between the feature representations, thereby improving the accuracy of the search results.

[0212] In some embodiments, in addition to obtaining search data pairs by allocating them in pairs based on the multiple search contents and multiple multimedia contents obtained in the above steps, search data pairs can also be obtained by other methods, and the data pairs obtained by different methods can be integrated to obtain multiple search data pairs.

[0213] For example, Figure 3 As shown, Figure 3 It is a flowchart of a method for obtaining search data pairs provided by an exemplary embodiment of the present application. Step 220 can also be implemented as the following steps 310 to 330.

[0214] Step 310: Perform correlation analysis between a plurality of search contents and a plurality of multimedia contents to obtain a plurality of first search data pairs.

[0215] The first search data pair includes a correlation tag between the search content and the multimedia content, and the correlation tag is used to describe the degree of association between the search content and the multimedia content.

[0216] Exemplarily, the number of types of correlation tags is preset, and each correlation tag corresponds to a different degree of association.

[0217] For example, the relevance labels include the following two categories: relevant and irrelevant. The relevance degrees corresponding to the relevance labels are ranked from low to high as follows: [irrelevant] < [relevant].

[0218] Exemplarily, multiple search contents and multiple multimedia contents are integrated to obtain multiple initial data pairs, wherein each initial data pair includes a search content and a multimedia content. The same search content can form an initial data pair with different multimedia contents, and the same multimedia content can also form an initial data pair with different search contents.

[0219] Exemplarily, the number of search contents is a first number, and the number of multimedia contents is a second number, and the number of initial data pairs is determined based on the product of the first number and the second number, and the first number and the second number are positive integers.

[0220] For example, there are 2 search contents: search content A and search content B, and there are 3 multimedia contents: multimedia content 1, multimedia content 2, and multimedia content 3, thus forming 2*3=6 initial data pairs.

[0221] Multiple initial data pairs are input into a pre-trained correlation analysis model, and the correlation scores corresponding to the multiple initial data pairs are output.

[0222] The pre-trained relevance analysis model is a relevance classification LLM, which can analyze the semantics of the search content and multimedia content in each initial data pair and output a relevance score, which describes the degree of relevance between the search content and the multimedia content through a numerical value. The training process of the relevance analysis model is not described in this application.

[0223] Based on the correlation scores, correlation labels respectively corresponding to the multiple initial data pairs are determined to obtain multiple first search data pairs.

[0224] The i-th first search data pair includes the i-th initial data pair and the corresponding correlation label.

[0225] Optionally, the value range of the relevance score is [0, 100], which is divided into different score intervals, and each score interval corresponds to a relevance label.

[0226] Exemplarily, the example in which the relevance label includes two types (irrelevant and relevant) is used for description.

[0227] For example, the score interval [0, 50] corresponds to the relevance label [irrelevant], and the score interval (50, 100] corresponds to the relevance label [relevant].

[0228] The initial data pair 1 (search content A and multimedia content 1) is analyzed by the correlation analysis model, and the output correlation score is 60, which belongs to the score interval (50, 100], so the correlation label corresponding to the initial data pair 1 is [Related].

[0229] Step 320: Perform matching analysis on multiple contents in the multimedia content library based on the multiple search contents to obtain multiple second search data pairs.

[0230] When a user searches for data based on search content, the search results obtained include at least one multimedia content. There are some multimedia contents that are highly relevant to the search content but are not clicked by the user. Screening out this part of multimedia content as training data for training the first model can enable the first model to better learn the correlation between the search content and the multimedia content.

[0231] Optionally, the matching analysis includes at least one of the following methods:

[0232] Method 1: keyword matching is performed on multiple contents in the multimedia content library based on multiple search contents, and the multimedia contents in the second search data pair include the search contents.

[0233] Exemplarily, the search content input by the user is used as a keyword to perform keyword matching with text information appearing in the multimedia content, and the multimedia content in the library that matches the search content is determined as the multimedia content included in the second search data pair.

[0234] Taking the multimedia content as a playlist content as an example, the playlist content includes the playlist introduction information and the information of at least one song in the playlist. The playlist introduction information includes the playlist description content and the playlist name. The playlist description content is used to describe the type and theme of the songs included in the playlist. The information of at least one song includes the name of the song, the name of the singer who sang the song, the album to which the song belongs, etc.

[0235] For example, if search content A is "name of singer A", and the playlist name of playlist content 1 (multimedia content 4) contains "name of singer A", the keyword match is successful.

[0236] For example, if search content A is "name of singer A", and playlist content 2 (multimedia content 5) contains songs sung by singer A, the keyword match is successful.

[0237] For example, if the search content B is "the name of song B", and the playlist content 3 (multimedia content 6) contains song B, the keyword match is successful.

[0238] Then, multimedia content 4, multimedia content 5, and multimedia content 6 are used as multimedia contents in the second search data pair.

[0239] The number of second search data pairs corresponding to the multiple search contents is determined to be the number of multimedia contents corresponding to the multiple search contents obtained based on method 1. The correlation labels between the search contents and the multimedia contents in each second search data pair are all [correlated]. The second search data pairs are as follows:

[0240] The second search data pair 1: search content A and multimedia content 4, corresponding to the relevance tag [Related];

[0241] The second search data pair 2: search content A and multimedia content 5, corresponding to the relevance tag [Related];

[0242] The second search data pair 3: search content A and multimedia content 6, corresponding to the relevance label [Related].

[0243] Method 2: content tag matching is performed on multiple contents in the multimedia content library based on multiple search contents. The multiple contents in the multimedia content library are respectively marked with preset content tags, and the preset content tags are used to describe the content theme of the multimedia contents. The preset content tags marked with the multimedia contents in the second search data pair match the search contents.

[0244] Optionally, the preset content labels for multimedia content annotation may be manually annotated or annotated by a pre-trained classification model.

[0245] For example, the subject content of the multimedia content is manually analyzed, and when the frequency of click events of the multimedia content reaches a preset frequency, the click search content corresponding to the click event is analyzed.

[0246] The clicked search content is the search content input by the user. After the user inputs a certain search content 001, the user clicks on the multimedia content 002. This click operation is called a click event. The search content 001 is the clicked search content corresponding to the click event of the multimedia content 002.

[0247] A preset content tag is manually determined based on at least one of the subject content and the click search content, and the multimedia content is annotated.

[0248] For another example, a pre-trained classification model can analyze input content and output a classification label for the content, obtaining a preset classification label set, wherein the classification label set includes at least one classification label, and the classification label can describe the subject content of the multimedia content.

[0249] A certain multimedia content is input into the classification model, the classification model analyzes the multimedia content, outputs a classification label for the multimedia content, and annotates the classification label onto the multimedia content as a preset content label.

[0250] Exemplarily, the search content input by the user is used as a keyword to perform keyword matching with the preset content tag annotated with the multimedia content, and the multimedia content in the library that matches the search content is determined as the multimedia content included in the second search data pair.

[0251] Taking the multimedia content as playlist content as an example, the playlist content is annotated with preset content tags for describing the style / theme of the songs in the playlist content.

[0252] For example, the search content A is "rock", and the playlist content 4 (multimedia content 7) is marked with the following preset content tags: "rock", "heavy metal", and the preset content tags contain the search content 1, so the match is successful.

[0253] For example, the search content B is "the name of film and television work C", and the playlist content 5 (multimedia content 8) is marked with the preset content tags as follows: "the name of film and television work C", "the original soundtrack of film and television", and the preset content tags contain the search content 3, so the match is successful.

[0254] Then, multimedia content 7 and multimedia content 8 are used as multimedia contents in the second search data pair.

[0255] The number of second search data pairs corresponding to the multiple search contents is determined to be the number of multimedia contents corresponding to the multiple search contents obtained based on method 2. The correlation labels between the search contents and the multimedia contents in each second search data pair are all [correlated]. The second search data pairs are as follows:

[0256] The second search data pair 1: search content A and multimedia content 7, corresponding to the relevance tag [Related];

[0257] The second search data pair 2: search content A and multimedia content 8, corresponding to the relevance label [Related].

[0258] Step 330: obtaining a plurality of data search pairs based on the plurality of first search data pairs and the plurality of second search data pairs.

[0259] There may be overlapping parts in the first search data pair and the second search data pair. The overlapping parts are removed to obtain multiple search data pairs. The multiple search data pairs are used to train the first model to obtain the search model.

[0260] For example, the first search data pair includes 6 data pairs, and the second search data pair includes 4 data pairs obtained based on method 1.

[0261] Among them, there are 2 data pairs in the second search data pairs that are the same as 2 data pairs in the first search data pairs. After removing the overlapping parts, the multiple search data pairs obtained include 6+4-2=8 data pairs.

[0262] For example, the first search data pair includes 6 data pairs, and the second search data pair includes 3 data pairs obtained based on method 2.

[0263] Among them, there is one data pair in the second search data pair that is the same as one data pair in the first search data pair. After removing the overlapping parts, the multiple search data pairs obtained include 6+3-1=8 data pairs.

[0264] For example, the first search data pair includes 6 data pairs, the second search data pair includes 4 data pairs obtained based on method 1, and 3 data pairs obtained based on method 2.

[0265] Among them, there are 2 data pairs in the data pairs obtained based on method 1, which are the same as the 2 data pairs in the data pairs obtained based on method 1. After removing the overlapping parts, the second search data pairs after deduplication contain 4+3-2=5 data pairs.

[0266] There are 2 data pairs in the second search data pairs after deduplication that are the same as 2 data pairs in the first search data pairs. After removing the overlapping parts, the multiple search data pairs obtained include 6+5-2=9 data pairs.

[0267] In summary, the search model training method provided by the present application determines the correlation labels between the search content and the multimedia content by performing correlation analysis on the search content and the multimedia content, and integrates the search content and the multimedia content into a data pair as training data for training the search model, so that the trained search model can learn the correlation between the search content and the multimedia content, and convert the search content and the multimedia content into a more accurate feature representation. When the search model is subsequently applied, it is possible to extract features from the multimedia content already in the database and the newly received search content, fully understand the user's intention when entering the search content, obtain an accurate feature representation, and index the matching multimedia content as the search result based on the degree of correlation between the feature representations, thereby improving the accuracy of the search results.

[0268] Figure 4 It is a flow chart of a method for performing data search based on a search model provided by an exemplary embodiment of the present application, and the method includes the following steps.

[0269] Step 410, obtain a multimedia content library, extract features from the multimedia content library through a search model, and obtain a feature representation library.

[0270] The multimedia content library contains a plurality of contents, and the feature representation library contains feature representations corresponding to the plurality of contents.

[0271] The multimedia content library is a database used by the data search service / search engine for data retrieval. The server indexes the corresponding multimedia content from the multimedia content library based on the received search content and feeds it back to the user as search results.

[0272] After the search model is trained, feature extraction is performed on each multimedia content in the multimedia content library to obtain feature representations corresponding to each multimedia content, wherein the number of multimedia contents in the multimedia content library is the same as the number of feature representations in the feature representation library.

[0273] For example, the multimedia content library contains 10,000 different multimedia contents. After extracting features one by one using the search model, 10,000 feature representations are obtained and stored in the feature representation library.

[0274] Optionally, when new multimedia content is added to the multimedia content library, feature extraction is immediately performed on the new multimedia content through the search model to obtain a new feature representation and store it in the feature representation library.

[0275] Optionally, when multimedia content in the multimedia content library is modified, feature extraction is immediately performed again on the modified multimedia content through the search model to obtain an updated feature representation and replace the feature representation corresponding to the multimedia content in the library.

[0276] Optionally, when multimedia content in the multimedia content library is deleted, the feature representation corresponding to the deleted content in the feature representation library is deleted. In some embodiments, the feature representation corresponding to the deleted content in the feature representation library may not be deleted temporarily, but stored at a designated location in the feature representation library, and the index mark is retained. When multimedia content identical to the originally deleted content is added / restored to the multimedia content library, the feature representation corresponding to the deleted content is moved from the designated location to the location corresponding to the multimedia content in the feature representation library.

[0277] Step 420: Receive a first search request.

[0278] An application program capable of providing data search service is run in the server and the terminal / a search engine is deployed thereon. A user inputs a first search content through an interface entrance of the data search service provided by the terminal to invoke the data search service.

[0279] After receiving the first search content, the terminal generates a first search request based on the first search content, wherein the first search request includes the first search content.

[0280] The first search content is the search condition when searching for data, and the search condition is used to match in the database corresponding to the data search service.

[0281] The expression form of the first search content includes, but is not limited to: text content, audio content, picture content, video content, animation content, file content in a specific format, etc.

[0282] Step 430: extract features from the first search content using the search model to obtain a feature representation corresponding to the first search content.

[0283] The feature corresponding to the first search content is represented as a high-dimensional vector, and the vector dimension can be any positive integer.

[0284] For example, after the search model extracts features from the first search content, a first input feature representation is obtained. The dimension of the first input feature representation is 256, indicating that the feature representation contains 256-bit numbers.

[0285] Step 440: Calculate the similarity between the feature representation corresponding to the first search content and the feature representation in the feature representation library to obtain a similarity result.

[0286] Exemplarily, the cosine similarities between the feature representation corresponding to the first search content and each feature representation in the feature representation library are calculated, and the obtained multiple cosine similarities are used as similarity results.

[0287] For example, the feature representation library contains 1000 feature representations, the first search content corresponds to the first input feature representation, and the cosine similarity between the first input feature representation and the 1000 feature representations is calculated to obtain a similarity result.

[0288] Step 450 , sorting the multiple contents in the multimedia content library based on the similarity result to obtain a sorting result.

[0289] The feature representations in the feature representation library are sorted in descending order based on multiple similarity values ​​contained in the similarity result. Since each feature representation uniquely corresponds to a multimedia content in the multimedia content library, the multiple multimedia contents in the multimedia content library are sorted based on the result of sorting the feature representations to obtain a sorting result.

[0290] For example, the feature representations in the feature representation library are sorted from high to low according to the similarity values, and the results are as follows: feature representation 90, feature representation 56, feature representation 34...feature representation 77.

[0291] Then, the sorting results obtained by sorting the multimedia contents are as follows: multimedia content 90, multimedia content 56, multimedia content 34... multimedia content 77.

[0292] Step 460: determine at least one multimedia content based on the ranking result as a search result corresponding to the first search content.

[0293] Optionally, the first b multimedia contents in the sorted results are used as search results corresponding to the first search content, where b is a positive integer.

[0294] For example, b is 10, and the sorting results are as follows: multimedia content 90, multimedia content 56, multimedia content 34, multimedia content 10, multimedia content 44, multimedia content 5, multimedia content 1, multimedia content 2, multimedia content 12, multimedia content 8...multimedia content 77.

[0295] The top 10 multimedia contents in the sorted results are used as search results, and the search results include:

[0296] [1] Multimedia content 90, [2] Multimedia content 56, [3] Multimedia content 34, [4] Multimedia content 10, [5] Multimedia content 44, [6] Multimedia content 5, [7] Multimedia content 1, [8] Multimedia content 2, [9] Multimedia content 12,

[10] Multimedia content 8.

[0297] In some embodiments, the number of click events for each multimedia content in the multimedia content library is counted, a weighted operation is performed based on the number of click events and the serial number mapping value of each multimedia content in the sorting result to obtain an operation result, and at least one multimedia content is determined based on the operation result as the search result corresponding to the first search content.

[0298] The sum of the serial number mapping value of the multimedia content and its serial number in the sorting result is the specified value.

[0299] For example, there are 100 multimedia contents in total, the designated value is 100, the multimedia content with the serial number 1 is mapped to the serial number 99; the multimedia content with the serial number 2 is mapped to the serial number 98, and so on.

[0300] Obtain a first weight and a second weight, and for the i-th multimedia content, obtain a first product based on the product of the first weight and the number of click events, obtain a second product based on the product of the second weight and the sequence mapping value, and obtain a calculation value based on the sum of the first product and the second product.

[0301] The multimedia contents are sorted based on the operation values ​​corresponding to all the multimedia contents, and the first c multimedia contents in the sorting are used as the search results corresponding to the first search content, where c is a positive integer.

[0302] For example, the number of click events of multimedia content 1 is 100, and the sequence number of multimedia content 1 is 7, then its sequence number mapping value is 93, the first weight is 0.5, and the second weight is 0.5.

[0303] Then the calculation value corresponding to the multimedia content 1 is 100*0.5+93*0.5=96.5.

[0304] Based on the above method, the operation value of each multimedia content is calculated and arranged in order. The resulting operation value arrangement results are as follows: Multimedia content 90 (operation value is 120), Multimedia content 10 (operation value is 100.9), Multimedia content 1 (operation value is 96.5), Multimedia content 50 (operation value is 88.8), Multimedia content 40 (operation value is 77.9)... Multimedia content 20 (operation value is 8.5).

[0305] When C is 4, the first four multimedia contents in the calculation value arrangement result are determined as the search results corresponding to the first search content: [1] Multimedia content 90, [2] Multimedia content 10, [3] Multimedia content 1, [4] Multimedia content 50.

[0306] In summary, the method provided by the present application determines the correlation labels between the search content and the multimedia content, and integrates the search content and the multimedia content into a data pair as training data for training the search model, so that the trained search model can learn the correlation between the search content and the multimedia content, and convert the search content and the multimedia content into a more accurate feature representation. When the search model is subsequently applied, it is possible to extract features from the multimedia content already in the database and the newly received search content, fully understand the user's intention when entering the search content, obtain an accurate feature representation, and index the matching multimedia content as the search result based on the degree of correlation between the feature representations, thereby improving the accuracy of the search results.

[0307] Figure 5 is a structural block diagram of a training device for a search model provided by an exemplary embodiment of the present application, such as Figure 5 As shown, the device includes the following parts.

[0308] An acquisition module 510 is used to acquire a plurality of search contents and a plurality of multimedia contents, wherein the search contents are search conditions during data search, and the multimedia contents are contents in the search results;

[0309] A correlation analysis module 520 is used to perform correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple search data pairs, wherein the search data pairs contain correlation labels between the search contents and the multimedia contents, and the correlation labels are used to describe the degree of association between the search contents and the multimedia contents;

[0310] The training module 530 is used to train a first model based on the correlation labels corresponding to the multiple search data pairs to obtain the search model, and the search model is used to convert the input search content and multimedia content into feature representations to obtain the degree of association between the input search content and the multimedia content.

[0311] In an optional embodiment, the acquisition module 510 is also used to obtain search records within a historical time period, wherein the historical time period refers to a time period of a preset length from the current moment, and the search record contains the multiple search contents received; based on the search record, candidate multimedia contents corresponding to the multiple search contents are obtained, wherein the candidate multimedia content corresponding to the mth search content is at least one content included in the search results when data search is performed based on the mth search content, and m is a positive integer; for the mth search content, click events corresponding to the at least one content are counted; based on the number of click events, at least one target content is determined from the candidate multimedia content corresponding to the mth search content; based on the at least one target content corresponding to the multiple search contents, the multiple multimedia contents are obtained.

[0312] In an optional embodiment, the acquisition module 510 is also used to sort the at least one content based on the number of click events for the candidate multimedia content corresponding to the m-th search content, and determine the first a contents to be sorted as the at least one target content corresponding to the m-th search content, where a is a positive integer.

[0313] In an optional embodiment, the correlation analysis module 520 is further used to perform correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple first search data pairs; perform matching analysis on multiple contents in the multimedia content library based on the multiple search contents to obtain multiple second search data pairs; and obtain the multiple data search pairs based on the multiple first search data pairs and the multiple second search data pairs.

[0314] In an optional embodiment, the matching analysis includes at least one of the following methods:

[0315] Perform keyword matching on multiple contents in the multimedia content library based on the multiple search contents, and the multimedia contents in the second search data pair include the search contents; or

[0316] Based on the multiple search contents, content tag matching is performed on the multiple contents in the multimedia content library, the multiple contents in the multimedia content library are respectively marked with preset content tags, and the preset content tags are used to describe the content theme of the multimedia content. The preset content tags marked by the multimedia content in the second search data pair match the search contents.

[0317] In an optional embodiment, the correlation analysis module 520 is further used to integrate the multiple search contents and the multiple multimedia contents to obtain multiple initial data pairs, wherein the initial data pairs include a search content and a multimedia content; input the multiple initial data pairs into a pre-trained correlation analysis model, and output the correlation scores corresponding to the multiple initial data pairs respectively; determine the correlation labels corresponding to the multiple initial data pairs respectively based on the correlation scores, and obtain the multiple first search data pairs, wherein the i-th first search data pair includes the i-th initial data pair and the corresponding correlation label.

[0318] In an optional embodiment, the training module 530 is also used to input the kth search data pair into the first model; perform feature extraction on the i-th search content and the j-th multimedia content contained in the k-th search data pair through the first model to obtain a first feature representation corresponding to the i-th search content and a second feature representation corresponding to the j-th multimedia content; i, j, k are positive integers; calculate the similarity between the first feature representation and the second feature representation to obtain the similarity corresponding to the k-th search data pair; train the first model based on the loss value between the similarity corresponding to the k-th search data pair and the correlation label to obtain the search model.

[0319] In an optional embodiment, if Figure 6 As shown, the device also includes:

[0320] The feature extraction module 540 is used to obtain a multimedia content library, which contains multiple contents; extract features from the multimedia content library through the search model to obtain a feature representation library, which contains feature representations corresponding to the multiple contents.

[0321] In an optional embodiment, the device further comprises:

[0322] The feature extraction module 540 is further configured to receive a first search request, wherein the first search request includes a first search content; extract features of the first search content using the search model to obtain a feature representation corresponding to the first search content;

[0323] The search module 550 is used to calculate the similarity between the feature representation corresponding to the first search content and the feature representation in the feature representation library to obtain a similarity result; sort the multiple contents in the multimedia content library based on the similarity result to obtain a sorting result; and determine at least one multimedia content based on the sorting result as a search result corresponding to the first search content.

[0324] In summary, the training device for the search model provided by the present application determines the correlation labels between the search content and the multimedia content by performing correlation analysis on the search content and the multimedia content, and integrates the search content and the multimedia content into a data pair as training data for training the search model, so that the trained search model can learn the correlation between the search content and the multimedia content, and convert the search content and the multimedia content into a more accurate feature representation. When the search model is subsequently applied, it is possible to extract features from the multimedia content already in the database and the newly received search content, fully understand the user's intention when entering the search content, obtain an accurate feature representation, and index the matching multimedia content as the search result based on the degree of correlation between the feature representations, thereby improving the accuracy of the search results.

[0325] It should be noted that the search model training device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the search model training device provided in the above embodiment and the search model training method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0326] Figure 7 The block diagram of the structure of a computer device 700 provided by an exemplary embodiment of the present application is shown. The computer device 700 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a laptop computer or a desktop computer. The computer device 700 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0327] Typically, the computer device 700 includes a processor 701 and a memory 702 .

[0328] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0329] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one instruction, which is used to be executed by the processor 701 to implement the training method of the search model provided in the method embodiment of the present application.

[0330] In some embodiments, the computer device 700 further includes some other components 703, and the type and quantity of the other components 703 can be selected based on the functional requirements of the computer device 700. Those skilled in the art will appreciate that Figure 7 The structure shown in the figure does not constitute a limitation on the computer device 700, and the computer device 700 may include more or less components than shown in the figure, or combine some components, or adopt a different arrangement of components.

[0331] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0332] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a search model training method as described in any of the above embodiments of the present application.

[0333] An embodiment of the present application also provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a search model training method as described in any of the above embodiments of the present application.

[0334] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the training method of the search model described in any of the above embodiments.

[0335] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0336] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for training a search model, characterized in that: The method comprises: Acquire multiple search contents and multiple multimedia contents, wherein the search contents are search conditions during data search, and the multimedia contents are contents in the search results; Performing correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple search data pairs, wherein the search data pairs include correlation labels between the search contents and the multimedia contents, and the correlation labels are used to describe the degree of association between the search contents and the multimedia contents; The first model is trained based on the correlation labels corresponding to the multiple search data pairs to obtain the search model, which is used to convert the input search content and multimedia content into feature representations to obtain the degree of association between the input search content and the multimedia content.

2. The method according to claim 1, characterized in that The obtaining of the plurality of search contents and the plurality of multimedia contents comprises: Acquire search records within a historical time period, where the historical time period refers to a time period of a preset length from a current moment, and the search records include the received multiple search contents; Acquire candidate multimedia content corresponding to the multiple search contents respectively based on the search record, wherein the candidate multimedia content corresponding to the m-th search content is at least one content included in the search results when data search is performed based on the m-th search content, and m is a positive integer; For the mth search content, counting click events corresponding to the at least one content respectively; Determine at least one target content from the candidate multimedia content corresponding to the m-th search content based on the number of the click events; The plurality of multimedia contents are obtained based on at least one target content respectively corresponding to the plurality of search contents.

3. The method according to claim 2, characterized in that The determining at least one target content from the candidate multimedia content corresponding to the mth search content based on the number of the click events includes: For the candidate multimedia content corresponding to the mth search content, the at least one content is sorted based on the number of the click events, and the first a contents are determined as the at least one target content corresponding to the mth search content, where a is a positive integer.

4. The method according to any one of claims 1 to 3, characterized in that: The performing correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple search data pairs includes: Performing correlation analysis between the multiple search contents and the multiple multimedia contents to obtain multiple first search data pairs; Perform matching analysis on multiple contents in the multimedia content library based on the multiple search contents to obtain multiple second search data pairs; Based on the plurality of first search data pairs and the plurality of second search data pairs, the plurality of data search pairs are obtained.

5. The method according to claim 4, characterized in that The matching analysis includes at least one of the following methods: Perform keyword matching on multiple contents in the multimedia content library based on the multiple search contents, and the multimedia contents in the second search data pair include the search contents; or Based on the multiple search contents, content tag matching is performed on the multiple contents in the multimedia content library, the multiple contents in the multimedia content library are respectively marked with preset content tags, and the preset content tags are used to describe the content theme of the multimedia content. The preset content tags marked by the multimedia content in the second search data pair match the search contents.

6. The method according to claim 4, characterized in that The performing correlation analysis between the plurality of search contents and the plurality of multimedia contents to obtain a plurality of first search data pairs includes: Integrate the multiple search contents and the multiple multimedia contents to obtain multiple initial data pairs, wherein the initial data pair includes one search content and one multimedia content; Inputting the multiple initial data pairs into a pre-trained correlation analysis model, and outputting correlation scores corresponding to the multiple initial data pairs respectively; Based on the correlation scores, correlation tags respectively corresponding to the multiple initial data pairs are determined to obtain the multiple first search data pairs, wherein the i-th first search data pair includes the i-th initial data pair and the corresponding correlation tag.

7. The method according to any one of claims 1 to 3, characterized in that: The step of training a first model based on the correlation labels respectively corresponding to the plurality of search data pairs to obtain the search model comprises: For the k-th search data pair, input the k-th search data pair into the first model; Performing feature extraction on the i-th search content and the j-th multimedia content contained in the k-th search data pair through the first model to obtain a first feature representation corresponding to the i-th search content and a second feature representation corresponding to the j-th multimedia content; i, j, and k are positive integers; Calculating the similarity between the first feature representation and the second feature representation to obtain the similarity corresponding to the kth search data pair; The first model is trained based on the loss values ​​between the similarity and relevance labels corresponding to the kth search data pair to obtain the search model.

8. The method according to any one of claims 1 to 3, characterized in that: After the first model is trained based on the correlation labels respectively corresponding to the plurality of search data pairs to obtain the search model, the method further includes: Acquire a multimedia content library, wherein the multimedia content library contains a plurality of contents; The search model is used to extract features from the multimedia content library to obtain a feature representation library, wherein the feature representation library contains feature representations corresponding to the plurality of contents.

9. The method according to claim 8, characterized in that The method further comprises: Receiving a first search request, wherein the first search request includes a first search content; Extracting features of the first search content by using the search model to obtain a feature representation corresponding to the first search content; Calculating the similarity between the feature representation corresponding to the first search content and the feature representation in the feature representation library to obtain a similarity result; Sorting the plurality of contents in the multimedia content library based on the similarity result to obtain a sorting result; At least one multimedia content is determined based on the ranking result as a search result corresponding to the first search content.

10. A training device for a search model, characterized in that: The device comprises: An acquisition module, used to acquire multiple search contents and multiple multimedia contents, wherein the search contents are search conditions during data search, and the multimedia contents are contents in the search results; a correlation analysis module, configured to perform correlation analysis on the plurality of search contents and the plurality of multimedia contents to obtain a plurality of search data pairs, wherein the search data pairs include correlation labels between the search contents and the multimedia contents, and the correlation labels are used to describe the degree of association between the search contents and the multimedia contents; A training module is used to train a first model based on the correlation labels corresponding to the multiple search data pairs to obtain the search model, and the search model is used to convert the input search content and multimedia content into feature representations to obtain the degree of association between the input search content and the multimedia content.

11. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the training method of the search model as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the search model training method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the training method of the search model as described in any one of claims 1 to 9.

Images