Training method for recall model, recall method and related device

By pre-training and fine-tuning the recall model, and utilizing the descriptive information of multimedia and the resource identifiers of related multimedia, the problem of low accuracy in multimedia recall is solved, achieving efficient multimedia recall and reducing the number of interactions.

WO2024230337A1PCT designated stage expired Publication Date: 2024-11-14TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Patent Information

Application Number
PCT/CN2024/083436
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2024-03-25
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

The accuracy of multimedia retrieval in existing technologies is not high, which requires users to make multiple queries and searches, increasing the number of interactions between the terminal and the multimedia server.

Method used

The recall model is pre-trained by acquiring multiple first text pairs. Question and answer texts are generated using the descriptive information of multimedia, and the association between multimedia resource identifiers and descriptive information is learned. Then, the pre-trained recall model is fine-tuned using the resource identifiers of related multimedia to learn the common features between multimedia and related multimedia.

Benefits of technology

It improves the accuracy of multimedia recall, reduces the number of user queries, and enhances the interaction efficiency between terminals and servers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024083436_14112024_PF_FP_ABST
    Figure CN2024083436_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present application relates to the technical field of artificial intelligence. Disclosed are a training method for a recall model, a recall method and a related device. The training method comprises: acquiring a plurality of first text pairs which comprise first question text and first answer text; pre-training a recall model by means of the first question text and the first answer text among the plurality of first text pairs; acquiring a plurality of second text pairs, wherein the second text pairs comprise second question text and second answer text, the second question text is text taking as a question target a resource identifier of relevant multimedia corresponding to multimedia, and the second answer text is the resource identifier of the relevant multimedia about which questions in the second question text are raised; and performing fine adjustment training on the pre-trained recall model by means of the second question text and the second answer text among the plurality of second text pairs. By means of the trained recall model, the relevant multimedia corresponding to the multimedia can be accurately recalled, and thus the multimedia recall accuracy is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Recall model training methods, recall methods, and related equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 11, 2023, with application number 2023105250958 and application name “Training method of recall model, recall method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and more specifically, to the training of recall models and multimedia recall. Background Art

[0003] With the development of multimedia technology, the amount of multimedia, such as audio and video, is rapidly increasing. Accurately recalling multimedia from this massive amount of multimedia can effectively shorten the time it takes for users to search for multimedia. However, if the recalled multimedia is not highly relevant to the multimedia the user actually wants, the multimedia recall accuracy is low, requiring the user to perform multiple queries and searches, resulting in multiple interactions between the user's terminal and the multimedia server. Therefore, improving multimedia recall accuracy is a technical problem that needs to be solved in related technologies.

[0004] Summary of the Invention

[0005] In view of the above problems, the embodiments of the present application propose a recall model training method, a recall method and related equipment to improve the recall accuracy of multimedia.

[0006] According to one aspect of an embodiment of the present application, a method for training a recall model is provided, comprising: obtaining a plurality of first text pairs, the first text pair comprising a first question text and a first answer text, the first question text being generated based on description information of multimedia and being a text with a resource identifier of multimedia as a question target, the first answer text being the resource identifier of the question asked by the first question text; pre-training a recall model based on the first question texts and the first answer texts in the plurality of first text pairs; obtaining a plurality of second text pairs, the second text pair comprising a second question text and a second answer text, the second question text being a text with a resource identifier of related multimedia as a question target, the second answer text being the resource identifier of the related multimedia asked by the second question text, and the related multimedia involved in the second text pair corresponding to the multimedia involved in the first text pair; and fine-tuning the pre-trained recall model based on the second question texts and the second answer texts in the plurality of second text pairs.

[0007] According to one aspect of an embodiment of the present application, a recall method is provided, comprising: obtaining a resource identifier of a target multimedia; generating a target question text with a relevant resource identifier as a question target based on the resource identifier of the target multimedia, wherein the relevant resource identifier refers to the resource identifier of the relevant multimedia corresponding to the target multimedia; generating a target answer text corresponding to the target question text based on the target question text by a recall model, wherein the target answer text includes the resource identifier of the relevant multimedia corresponding to the target multimedia; the recall model is trained according to the above recall model training method; and determining a recall result of the target multimedia based on the resource identifier in the target answer text.

[0008] According to one aspect of an embodiment of the present application, a training device for a recall model is provided, comprising: a first acquisition module for acquiring multiple first text pairs, wherein the first text pair includes a first question text and a first answer text, the first question text is generated based on description information of the multimedia, and is a text with a resource identifier of the multimedia as a question target, and the first answer text is the resource identifier of the question asked by the first question text; a pre-training module for pre-training the recall model based on the first question texts and the first answer texts in the multiple first text pairs; a second acquisition module for acquiring multiple second text pairs, wherein the second text pairs include a second question text and a second answer text, the second question text is a text with a resource identifier of the relevant multimedia as a question target, and the second answer text is the resource identifier of the relevant multimedia asked by the second question text, and the relevant multimedia involved in the second text pair corresponds to the multimedia involved in the first text pair; and a fine-tuning training module for fine-tuning the pre-trained recall model based on the second question texts and the second answer texts in the multiple second text pairs.

[0009] According to one aspect of an embodiment of the present application, a recall device is provided, comprising: a third acquisition module for acquiring a resource identifier of a target multimedia; a target question text generation module for generating a target question text with a relevant resource identifier as a question target based on the resource identifier of the target multimedia, wherein the relevant resource identifier refers to the resource identifier of the relevant multimedia corresponding to the target multimedia; a target answer text determination module for generating a target answer text corresponding to the target question text based on the target question text by a recall model, wherein the target answer text includes the resource identifier of the relevant multimedia corresponding to the target multimedia; the recall model is trained according to the training method of the recall model as described above; and a recall result determination module for determining the recall result of the target multimedia based on the resource identifier in the target answer text.

[0010] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the training method of the recall model as described above is implemented, or the recall method as described above is implemented.

[0011] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the training method of the recall model as described above is implemented, or the recall method as described above is implemented.

[0012] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the training method of the recall model as described above, or implements the recall method as described above.

[0013] In this application, the recall model is first pre-trained through multiple first text pairs. Since the first question text in the first text pair is generated based on the description information of the multimedia, and the first question text takes the resource identifier of the multimedia as the question target, and the first answer text is the resource identifier of the multimedia, through pre-training, the recall model can learn the association between the resource identifier of the multimedia and the description information of the multimedia, so as to determine the feature representation of the multimedia resource identifier through the characteristics of the multimedia description information. After the pre-training is completed, the pre-trained recall model is fine-tuned using the second question text with the resource identifier of the multimedia corresponding to the multimedia as the question target, and the second answer text including the resource identifier of the multimedia corresponding to the multimedia. In this way, the recall model can use the association between the resource identifier of the multimedia and the description information of the multimedia learned in the pre-training stage, and learn the common characteristics between the multimedia used as a reference and the multimedia corresponding to the multimedia based on the resource identifier of the multimedia and the resource identifier of the multimedia corresponding to the multimedia in the fine-tuning training stage. Therefore, in the subsequent application process, the recall model can accurately predict the resource identifier of the multimedia corresponding to the multimedia based on the resource identifier of the multimedia used as a reference, and then accurately recall the multimedia corresponding to the multimedia, effectively ensuring the correlation between the recalled related multimedia and the multimedia used as a reference. Moreover, the solution of the present application realizes the conversion of the multimedia recall task into a text generation task. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG1 is a schematic diagram illustrating an application scenario of the present application according to an embodiment of the present application;

[0015] FIG2 is a flowchart of a method for training a recall model according to an embodiment of the present application;

[0016] FIG3 is a schematic diagram illustrating training a recall model according to an embodiment of the present application;

[0017] FIG4 is a schematic diagram of the structure of a recall model according to an embodiment of the present application;

[0018] FIG5 exemplarily shows a schematic diagram of encoding and decoding processing of a BART model;

[0019] FIG6 exemplarily shows a schematic diagram of a converter model;

[0020] FIG7 is a flowchart of step 220 in one embodiment according to one embodiment of the present application;

[0021] FIG8 is a flow chart showing step 240 in one embodiment according to one embodiment of the present application;

[0022] FIG9 is a flowchart of a recall method according to an embodiment of the present application;

[0023] FIG10 is a flowchart of a recall method according to another embodiment of the present application;

[0024] FIG11 is a block diagram of a training device for a recall model according to an embodiment of the present application;

[0025] FIG12 is a block diagram of a recall device according to an embodiment of the present application;

[0026] FIG13 shows a schematic structural diagram of a computer system suitable for implementing an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0028] It should be noted that the term "plurality" used in this document refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0029] Before describing the solution of this application in detail, the terms involved in this application are explained as follows:

[0030] Sequence-to-sequence (Seq2Seq) models are neural network models that map sequences to sequences. Sequence-to-sequence models were originally used to improve machine translation technology, mapping sentences (sequences of words) in one language to corresponding sentences in another language.

[0031] Text generation: This refers to generating understandable text from non-verbal representations. Based on the different classifications of non-verbal representations, text generation can be divided into "text → text", "data → text", and "image → text".

[0032] Pre-training: It uses as much training data as possible to extract as many common features as possible, thereby reducing the learning burden of the model for specific tasks.

[0033] Fine tuning: The principle of fine tuning is to use the known model structure and known model parameters to modify the parameters of the output layer to serve as the output layer of the current task, and to fine-tune the parameters of several network layers before the last layer (i.e., the output layer). This can effectively utilize the powerful generalization ability of deep neural networks, avoid the design of complex models, and reduce training time.

[0034] Prompt Learning: The core of prompt learning is to transform the problem to be solved into a form similar to the pre-trained task using a template. For example, for the text "I missed the bus today.", you can construct the template "I missed the bus today. I felt so [MASK][MASK]" and use the Masked Language Model (MLM) to predict sentiment words to identify its sentiment polarity. You can also construct the prefix "English: I missed the bus today. Chinese: [MASK][MASK]" and use a generative model to obtain its corresponding Chinese translation.

[0035] The Transformer model is a deep learning model that uses a self-attention mechanism to assign different weights to different parts of the input data based on their importance. This model is primarily used in the fields of natural language processing (NLP) and computer vision (CV).

[0036] The attention mechanism is a problem-solving method that mimics human attention. Simply put, it quickly filters out high-value information from a large amount of information. It is primarily used to address the difficulty of obtaining a reasonable final vector representation for long input sequences in Long Short-Term Memory (LSTM) and Recursive Neural Network (RNN) models. The main approach is to retain the intermediate results of the LSTM, learn them with a new model, and associate the intermediate results with the output to achieve the purpose of information filtering.

[0037] The Bidirectional Encoder Representations from Transformers (BERT) model is a pre-trained language representation model that replaces the traditional unidirectional language model or the shallow concatenation of two unidirectional language models for pre-training. Instead, it uses a new language model with a masking mechanism to generate deep bidirectional language representations.

[0038] Generative Pre-trained Transformer (GPT): is an autoregressive language model that aims to use deep learning to generate natural language that humans can understand.

[0039] Knowledge Graph: Essentially, it is a knowledge base called a semantic network, that is, a knowledge base with a directed graph structure. A knowledge graph is a data structure composed of entities, relationships, and attributes.

[0040] Overfitting: This refers to a situation where the gap between training error and test error is too large. In other words, the model complexity is higher than the actual problem, and the model performs well on the training set but poorly on the test set.

[0041] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology and machine learning / deep learning. The solution of this application uses natural language processing technology to convert the task of multimedia recall into a text generation task to improve the recall accuracy of multimedia.

[0042] FIG1 is a schematic diagram of an application scenario of the present application according to an embodiment of the present application. As shown in FIG1 , the application scenario includes a server 120 . The server 120 may be a physical server or a cloud server, which is not specifically limited here.

[0043] The server 120 can train the recall model according to the training method of the recall model provided in this application, and the training process includes: step S1, pre-training the recall model through multiple first text pairs; step S2, fine-tuning the pre-trained recall model through multiple second text pairs. Based on the recall model completed with fine-tuning training, step S3 can be executed to call the recall model to perform recall processing on each multimedia in the multimedia library, that is, the recall result corresponding to the multimedia is determined according to the resource identifier corresponding to the multimedia through the recall model, and the resource identifier of the multimedia is associated with the recall result corresponding to the multimedia and stored in the recall data set. The recall result corresponding to the multimedia indicates related multimedia related to the multimedia. In a specific embodiment, step S3 can be performed offline.

[0044] Based on the recall data set, the server 120 can provide a search and recall service to the terminal. In this case, the application scenario also includes the terminal 110, which can be a smartphone, tablet computer, laptop computer, desktop computer, car terminal, smart TV, etc., which is not specifically limited here.

[0045] The terminal 110 is connected to the server 120 via a wired or wireless network, and the server 120 performs search and recall according to the following process: step S41, receiving a multimedia search request; step S42, matching the second multimedia; specifically, the multimedia search request sent by the terminal 110 includes a search keyword, and the search keyword is matched with the description information of each multimedia in the multimedia library to determine the second multimedia that matches the search keyword; step S43, determining the recall result corresponding to the second multimedia; specifically, obtaining the recall result corresponding to the second multimedia from the recall data set; step S44, sending the second multimedia and the recall result corresponding to the second multimedia.

[0046] In some embodiments, the training of the recall model, calling the recall model to determine the recall results of each multimedia, and providing search recall services to the terminal can be performed by the same electronic device, for example, all performed by the server 120, or by different electronic devices, which is not specifically limited here.

[0047] The following is a detailed description of the implementation details of the technical solution of the embodiment of the present application:

[0048] FIG2 is a flowchart of a method for training a recall model according to an embodiment of the present application. The method can be executed by an electronic device with processing capabilities, such as a server, which is not specifically limited here. Referring to FIG2 , the method includes at least steps 210 to 240, which are described in detail as follows:

[0049] Step 210, obtain multiple first text pairs, the first text pair includes a first question text and a first answer text, the first question text is generated based on the description information of the multimedia, and takes the resource identifier of the multimedia as the question target, and the first answer text is the resource identifier targeted by the first question text.

[0050] The first text pair is a text pair used for pre-training the recall model. The question text in the first text pair representing the question is called the first question text, and the answer text in the first text pair representing the answer to the question is called the first answer text.

[0051] Multimedia can include video (e.g., long or medium-length videos), audio, or audio / video. Examples include TV series, documentaries, movies, variety shows, animations, and cartoons. Multimedia description information indicates basic attributes of the multimedia, such as the title (e.g., TV series, documentary, movie, animation), release date, language, screenwriter, director, synopsis, starring actors, and genre.

[0052] In some embodiments, multimedia description information may be stored in the form of a knowledge graph, that is, the multimedia description information is presented through a directed graph structure. Taking Chen Xling's movies as an example, the schema of the multimedia description information may be as follows:

[0053] As can be seen above, in the schema corresponding to the multimedia Chen Xling Movie, different types of descriptive information are represented by description fields (a description field is used to represent a basic attribute) and the values ​​of the description fields. For example, the description field "year" represents the release date, the description field "summary" represents the introduction, and the description field "actors" represents the actors. In other embodiments, the multimedia description information may also include more or fewer description fields than those listed above, which are not specifically limited here.

[0054] A resource identifier is a text string used to uniquely identify multimedia content. It can be a multimedia content identifier (CID). A multimedia content identifier is generated by encrypting the multimedia content (e.g., performing a cryptographic hash process). The content identifier is equivalent to the multimedia's "content fingerprint."

[0055] The first question text targets the multimedia resource identifier. That is, the first question text is intended to inquire about the multimedia resource identifier. For example, the first question text may be: What is the CID of XX? "XX" in the first question text may be at least one qualifier used to define the multimedia. For example, "XX" may be a basic attribute such as the multimedia title, starring actor, release date, screenwriter, director, or genre.

[0056] In some embodiments, the first question text and the first answer text in each first text pair may be determined according to the following steps A1 to A3:

[0057] Step A1: Obtain multimedia description information and resource identifiers corresponding to the multimedia.

[0058] Step A2: generating a first question text with the multimedia resource identifier as the question target according to the value of at least one description field in the description information.

[0059] As described above, a description field in the multimedia description information is used to represent a basic attribute. The value of a description field in the description information represents the attribute content of the multimedia under the corresponding attribute. In step A2, the multimedia is defined by the value of at least one description field in the description information, and a first question text is generated based on this, with the resource identifier of the multimedia as the question target. For example, the first question text may be: "What is the CID corresponding to Chen Xling's extra-curricular movie?" In this first question text, "Chen Xling's extra-curricular movie" is the value of the description field representing the movie title.

[0060] In some embodiments, step A2 includes the following steps B1 to B3, which are described in detail as follows:

[0061] Step B1: Obtain a first question template. The first question template uses a resource identifier as a question target and indicates at least one description field.

[0062] In this application, the question template used to generate the first question text is called the first question template. The first question template can be pre-set. The number of first question templates is not limited. To ensure the richness of the generated first question text, there can be multiple first question templates.

[0063] The first question template indicates the common question content in the first question text, such as "The corresponding CID is." Furthermore, the first question template also indicates at least one description field used to define multimedia. For example, the description field indicated by the first question template for defining multimedia may include a description field indicating the multimedia name, a description field indicating the corresponding starring actor of the multimedia, a description field indicating the corresponding director of the multimedia, and so on. The common question content in different first question templates can be the same, while the indicated description fields can be different. In this way, based on the description information of the same multimedia, multiple first question texts can be generated using different first question templates.

[0064] Step B2: Obtain the values ​​of the description fields indicated by the first question template from the multimedia description information.

[0065] Based on the first question template, the values ​​of each description field indicated by the first question template can be obtained from the description information of each multimedia in the multimedia library. For example, if the description field indicated by the first question template includes a description field representing the leading actor, then the value of the description field representing the leading actor is obtained from the description information of the multimedia, and this value represents the leading actor corresponding to the multimedia.

[0066] Step B3: Combine the acquired value of the description field with the first question template to obtain the first question text.

[0067] In step B3, the acquired value of the description field is filled into the position of the description field in the first question template, and the first question text is obtained accordingly.

[0068] For example, the first question template might be: "[Description Field I] starring [Description Field II], what is the corresponding CID?" Here, Description Field I refers to the description field representing the starring actor, and Description Field II refers to the description field representing the multimedia title. Based on this first question template, the following first question text might be generated: "What is the corresponding CID for YY Dijiu starring Liu XX?"

[0069] For example, the first question template can be: [Description Field III] What is the CID corresponding to the director's [Description Field II]? Where Description Field III refers to the description field representing the director. Based on this first question template, the following first question text can be generated: What is the CID corresponding to Director Zhang XX's ZZZ?

[0070] It is worth mentioning that the first question templates listed above are merely illustrative examples and cannot be considered as limiting the scope of use of this application. In a specific embodiment, in order to enrich the form of the first question text, more first question templates can be set.

[0071] It can be seen that through the value of the description field indicated by the first question template, the first question text can be efficiently generated based on multimodal description information. Moreover, since there can be more than one first question template, the richness of the generated first question text is further guaranteed, thereby improving the pre-training quality of the recall model.

[0072] Step A3: Use the multimedia resource identifier as the first answer text corresponding to the first question text.

[0073] For example, for the first question text: What is the CID corresponding to Director Zhang XX's ZZZ? The first answer text corresponding to the first question text is the resource identifier corresponding to the multimedia Director Zhang XX's ZZZ, for example: klv6811ljzbhs8k.

[0074] For another example, the first question text is: What is the CID corresponding to YYYY starring Liu XX? The corresponding first answer text is: mzc0020035l5vcf; "mzc0020035l5vcf" is the resource identifier of the multimedia YYYY starring Liu XX.

[0075] In some embodiments, according to the different number of question targets of the first question text, the first question template can be divided into a main requirement scenario template and an intention scenario template, wherein the main requirement scenario template has one question target, that is, it uses a certain multimedia resource identifier as the question target, and the intention scenario template uses at least two multimedia resource identifiers as the question target. It can also be understood that the multimedia defined by the description field in the main requirement scenario template is one, and the multimedia defined by the description field in the intention scenario template is at least two.

[0076] For example, in the above text, the first question templates corresponding to the two first question texts, “What is the CID corresponding to YYYY starring Liu XX?” and “What is the CID corresponding to ZZZ directed by Zhang XX?” can be regarded as main demand scene templates.

[0077] For example, an intent scenario template might look like this: [Description Field IV] What is the CID corresponding to the movie series? Description Field IV refers to the description field that indicates the series name of the multimedia series. Based on this intent scenario template, the following first question text can be generated: What is the CID corresponding to the Chen Xling movie series? The first answer text corresponding to this first question text is: mzc00200lpxf8uq.

[0078] For each multimedia in the multimedia library, multiple first text pairs can be generated for each multimedia according to the above process using the description information of the multimedia.

[0079] It can be seen that since the description field can identify the description information through different dimensions, the multimedia can be accurately expressed through the value of at least one description field in the description information, so that the constructed first question text is more closely related to the multimedia, thereby improving the generation quality of the first question text.

[0080] Step 220 : Pre-training the recall model based on the first question texts and the first answer texts in the plurality of first text pairs.

[0081] The recall model is a sequence-to-sequence model constructed through a neural network, that is, the recall model can map the input sequence to the output sequence. In this application, during the pre-training process, the input sequence of the recall model is the first question text, and the output sequence is the answer text representing the resource identifier of the multimedia.

[0082] During the pre-training process, the first question text is input into the recall model, and the recall model performs semantic encoding on the first question text, and then decodes and outputs the predicted multimedia resource identifier based on the semantic encoding result; thereafter, the first loss is calculated based on the predicted multimedia resource identifier and the resource identifier in the first answer text corresponding to the first question text, and then the weight parameters of the recall model are reversely adjusted based on the first loss.

[0083] In some embodiments, a pre-training termination condition may be pre-set. The pre-training termination condition may be when the number of pre-training iterations reaches a first threshold, or when the loss function in the pre-training phase converges, etc., which are not specifically limited herein. During the pre-training process, if it is determined that the pre-training termination condition has been met, the pre-training is stopped.

[0084] Since the first question text is generated based on the description information of the multimedia, the first answer text is the resource identifier of the multimedia. By pre-training the recall model with the first question text and the first answer text, the recall model can learn the characteristic representation of the multimedia resource identifier, that is, learn the association between the multimedia resource identifier and the multimedia description information, so that the recall model can perceive and remember the description information corresponding to the multimedia with different resource identifiers, and then, the multimedia resource identifier can be described through the description information of the multimedia.

[0085] Step 230, obtain multiple second text pairs, the second text pairs include a second question text and a second answer text, the second question text is a text with the resource identifier of the related multimedia corresponding to the multimedia as the question target, and the second answer text is the resource identifier of the related multimedia targeted by the second question text.

[0086] The second text pair refers to a text pair used for fine-tuning the recall model. The question text representing the question in the second text pair is called the second question text, and the answer text representing the answer in the second text pair is called the second answer text.

[0087] The related multimedia corresponding to a multimedia A refers to multimedia that is associated with the multimedia A in one or more dimensions, such as multimedia that is highly similar to the multimedia A (for example, high content similarity, the same type, etc.), or multimedia that is associated with the content of the multimedia A, or multimedia that is of high user interest together with multimedia A, or multimedia that is highly likely to be paid attention to by users together with multimedia A.

[0088] The second question text uses the resource identifier of the related multimedia corresponding to the multimedia as the question target, that is, the second question text is intended to inquire about the resource identifier of the related multimedia corresponding to the multimedia. In some embodiments, the second question text indicates the multimedia used as a reference; for example, if the second question text uses the resource identifier of the related multimedia corresponding to multimedia A as the question target, then the multimedia used as a reference is multimedia A. In some embodiments, the second question text can indicate the multimedia used as a reference by the resource identifier of the multimedia used as a reference. For example, the second question text can be: Which CID do users who search for CID: mzc002000mqs1cp also tend to click? The resource identifier "mzc002000mqs1cp" in the second question text is used to limit the multimedia used as a reference. The second question text is intended to inquire about the resource identifier of the related multimedia corresponding to the multimedia with the resource identifier "mzc002000mqs1cp".

[0089] In some embodiments, the second question text and the second answer text in the second text pair may be determined according to the following steps C1 to C3, which are described in detail as follows:

[0090] Step C1: Acquire multimedia feedback data, where the multimedia feedback data indicates at least two multimedia whose feedback operations are triggered within a set time period.

[0091] The feedback operation can be an interactive operation on the multimedia when the multimedia is displayed, which can be a click operation, a like operation, a favorite operation, a forwarding operation, etc., which is not specifically limited here. In some embodiments, after the multimedia cover is presented on the user interface, the client's operation log for the multimedia cover can be collected. The operation log indicates the user's feedback operation triggered on the multimedia cover. Afterwards, at least two multimedia that are triggered to perform feedback operations within a set time period are determined based on the operation log over a period of time. It is worth mentioning that in the case where it is necessary to collect the client's operation log for the multimedia cover, any one of them needs to obtain user permission or consent separately first, and the collection, use and processing of the operation log need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0092] In some embodiments, in the scenario of multimedia searched by the user, based on matching the searched multimedia (for ease of distinction, the searched multimedia is referred to as the third multimedia), at least one fourth multimedia with a high similarity to the third multimedia is further matched from the multimedia library, and the third multimedia and the at least one fourth multimedia are pushed to the initiator of the search, so as to display the third multimedia and the at least one fourth multimedia on the search results display page. Subsequently, the operation log collected from the client can be used to determine the other multimedia clicked by the user within a set time period after clicking on the third multimedia. In the search scenario, for the user, the third multimedia that matches the search term is the one that they are most concerned about and interested in. If the user also clicks on the fourth multimedia displayed on the search results display page, it indicates that the user is also concerned about the fourth multimedia that was triggered by the click behavior. In this case, the multimedia feedback data is a log of multiple click operations on the search results display page, and the multimedia feedback data corresponds to at least two multimedia that were triggered by the click operation within the set time period.

[0093] Step C2: Generate a second question text based on the resource identifier corresponding to the first multimedia in at least two multimedia, with the resource identifier of the related multimedia corresponding to the first multimedia as the question target; the related multimedia corresponding to the first multimedia includes at least one multimedia in the at least two multimedia except the first multimedia.

[0094] In other words, in step C2, one of the at least two multimedia indicated by the multimedia feedback data that triggers the feedback operation within the set time period is regarded as the first multimedia, and the other multimedia except the first multimedia in the at least two multimedia is regarded as the related multimedia corresponding to the first multimedia.

[0095] In the second question text, the resource identifier corresponding to the first multimedia is used to characterize or limit the first multimedia. Based on this, the resource identifiers of related multimedia corresponding to the first multimedia are used as the question target. For example, the second question text may be: What is the CID of the multimedia that the user interested in CID XX is also interested in? "XX" in the second question text refers to the CID of the first multimedia.

[0096] In the search scenario, that is, when the multimedia feedback data is a log of multiple click operations on the search results display page, the third multimedia matched by the search can be regarded as the first multimedia, and correspondingly, the fourth multimedia indicated by the click operation log as being clicked can be regarded as the related multimedia of the third multimedia.

[0097] In some embodiments, step C1 includes the following steps D1 and D2, which are described in detail as follows:

[0098] Step D1: Obtain a second question template, where the second question template uses the resource identifier of the relevant multimedia as the question target.

[0099] The second question template refers to a question template set for the second question text. The second question template can be pre-set, and can be one or more. Similarly, the second question template indicates the common question content in the second question text, that is, if multiple second question texts are generated according to the same second question template, then the common question content in the multiple second question texts is the same. The second question template also indicates the position of the resource identifier of the multimedia used as a reference in the second question text. If a second prompt text uses the resource identifier of the relevant multimedia corresponding to multimedia A as the question target, then the resource identifier of multimedia A is used as the reference.

[0100] The second question template could be: "What CID is the multimedia resource that users interested in CID XX are also interested in?" Another example could be: "What CID do users who search for CID XX also tend to click on?" Another example could be: "What CID do users who follow CID XX also follow?" The location of "XX" in the second question template shown above represents the location of the multimedia resource identifier used as a reference. Of course, the above is merely an illustrative example of the second question template and should not be considered a limitation on the scope of use of this application.

[0101] Step D2: combining the resource identifier corresponding to the first multimedia of the at least two multimedia with the second question template to obtain a second question text.

[0102] As described above, the second question template indicates the position of the resource identifier of the reference multimedia in the second question text. Therefore, the resource identifier corresponding to the first multimedia is used as a reference and filled into the corresponding position in the second question template to obtain the second question text.

[0103] It can be seen that the second question text can be generated quickly and conveniently through the second question template. Moreover, since there can be more than one second question template, the richness of the generated second question text is further guaranteed, thereby improving the pre-training quality of the recall model.

[0104] Step C3: Using the resource identifier of the related multimedia corresponding to the first multimedia as the second answer text corresponding to the second question text.

[0105] According to the multimedia feedback data, the first multimedia can be correspondingly determined, and at least one other multimedia except the first multimedia among the at least two multimedia indicated by the multimedia feedback data that are triggered to perform the feedback operation can be used as the related multimedia of the first multimedia. Based on the resource identifiers of the stored multimedia, the resource identifiers of the related multimedia corresponding to the first multimedia can be correspondingly determined, thereby correspondingly determining the second answer text corresponding to the second question text.

[0106] Following steps C1 to C3, the following second question-answer pair can be determined, where the second question is: Which CID do users who search for cid:mzc002000mqs1cp also tend to click on? The second answer is: mzc0020028aguo0. In the second question, "mzc002000mqs1cp" is the resource identifier of the reference multimedia (i.e., the resource identifier of the first multimedia); and in the second answer, "mzc0020028aguo0" is the resource identifier of the related multimedia corresponding to the first multimedia.

[0107] Because the second text pair helps the recall model learn the relationships between different multimedia during training, and at least two multimedia items in the multimedia feedback data precisely reflect the relationship between these multimedia items based on feedback operations within a set time period, for example, the first multimedia item and the other multimedia items are related. Therefore, the second text pair determined based on this can effectively improve the training quality of the recall model.

[0108] In some embodiments, to ensure the number of second text pairs, multimedia feedback data in multiple time periods (eg, multiple time periods in the past 30 days) can be used to determine the second text pairs to ensure sufficient training samples in the fine-tuning training phase.

[0109] Step 240 : fine-tune the pre-trained recall model based on the second question text and the second answer text in the plurality of second text pairs.

[0110] During the fine-tuning training process, the input sequence of the pre-trained recall model is the second question text, and the output sequence is the answer text representing the resource identifier of the relevant multimedia corresponding to the multimedia. During the fine-tuning training process, the second question text is input into the pre-trained recall model, which performs semantic encoding on the second question text. The pre-trained recall model then decodes the semantic encoding result and outputs the predicted resource identifier of the relevant multimedia. Subsequently, a second loss is calculated based on the predicted resource identifier of the relevant multimedia and the resource identifier in the second answer text corresponding to the second question text, and the weight parameters of the recall model are then adjusted inversely based on the second loss.

[0111] After pre-training, the recall model learns the association between the multimedia resource identifier and the multimedia's description information, and uses the features corresponding to the multimedia's description information to construct a feature representation of the multimedia's resource identifier. On this basis, the pre-trained recall model is fine-tuned using a second question text with the multimedia's corresponding resource identifier as the question target and a second answer text including the multimedia's corresponding resource identifier. This allows the recall model to utilize the association between the multimedia's resource identifier and the multimedia's description information learned during the pre-training phase. During the fine-tuning phase, based on the multimedia's resource identifier and the multimedia's corresponding resource identifier, it learns the commonalities in features between the reference multimedia and the multimedia's corresponding related multimedia. Consequently, in subsequent applications, the recall model can accurately recall the multimedia's corresponding related multimedia based on the multimedia's resource identifier, i.e., accurately predict the multimedia's corresponding related multimedia resource identifier.

[0112] In some embodiments, based on the pre-training of the recall model, in order to shorten the training time and improve the training efficiency, during the fine-tuning training process, the weight parameters of some network layers in the recall model can be reversely adjusted according to the second loss. Specifically, since the parameters of the multiple network layers closest to the output of the recall model are directly related to the recall task of the recall model, the weight parameters of the last network layer (i.e., the output layer) and the multiple network layers before the last network layer in the recall model can be reversely adjusted according to the second loss during the fine-tuning training stage. In this way, compared with adjusting the weight parameters of all network layers of the recall model, the time spent on adjusting the weight parameters of some network layers in the recall model is reduced, and the length of fine-tuning training can be reduced accordingly. By reversely adjusting the weight parameters of the last network layer (i.e., the output layer) and the multiple network layers before the last network layer in the recall model, not only the powerful generalization ability of the deep neural network is utilized, but also the design of complex models and time-consuming training are avoided.

[0113] In some embodiments, a fine-tuning training termination condition can be pre-set. The fine-tuning training termination condition can include, for example, the number of fine-tuning training iterations reaching a second threshold, or the loss function convergence during the fine-tuning training phase, etc., which are not specifically limited herein. During the fine-tuning training process, if it is determined that the fine-tuning training termination condition has been met, fine-tuning training is stopped.

[0114] Figure 3 is a schematic diagram of training a recall model according to an embodiment of the present application. Figure 3 exemplifies two first text pairs and two second text pairs, wherein, in a first text pair, the text on the left is the first question text, and the text on the right is the first answer text. The two texts on the left and right that are located in the same dotted box (or in the same row) belong to the same text pair (i.e., the first text pair or the second text pair). The process of pre-training the recall model using the first text pair and fine-tuning the pre-trained recall model using the second text pair is described above and will not be repeated here.

[0115] In this application, a recall model is pre-trained using multiple first text pairs. Since the first question text in the first text pair is generated based on multimedia description information, and the first question text uses the multimedia resource identifier as the question target, and the first answer text is the multimedia resource identifier, it is equivalent to pre-training the recall model using the prior knowledge of the multimedia graph, so that the recall model can perceive and remember the multimedia description information. Through pre-training, the recall model can learn the association between the multimedia resource identifier and the multimedia description information, and determine the feature representation of the multimedia resource identifier through the characteristics of the multimedia description information. After the pre-training is completed, the pre-trained recall model is fine-tuned using a second question text using the multimedia resource identifier of the multimedia corresponding to the multimedia as the question target, and a second answer text including the multimedia resource identifier of the multimedia corresponding to the multimedia. In this way, the recall model can use the association between the multimedia resource identifier and the multimedia description information learned in the pre-training stage, and learn the feature commonality between the multimedia used as a reference and the multimedia corresponding to the multimedia in the fine-tuning training stage based on the multimedia resource identifier and the multimedia resource identifier of the multimedia corresponding to the multimedia. In subsequent applications, the recall model can accurately recall related multimedia corresponding to the multimedia based on the resource identifier of the multimedia used as a reference, that is, accurately predict the resource identifier of the related multimedia corresponding to the multimedia. Moreover, through the solution of this application, the multimedia recall task is converted into a text generation task. This eliminates the need to process the video or audio frames in the multimedia, simplifies the multimedia recall task, and improves recall efficiency.

[0116] In some embodiments, to facilitate the construction of the first and second text pairs, the multimedia data table can be adjusted. The original multimedia data table includes the values ​​of the multimedia under various description fields, but does not include the multimedia's resource identifiers. Therefore, the multimedia's resource identifiers can be added to the multimedia data table before pre-training, and the information in the multimedia data table can be periodically updated based on the multimedia in the multimedia library. For example, when multimedia is added or removed from the multimedia library, the data in the multimedia data table is updated accordingly. Subsequently, the first and second text pairs can be constructed based on the data in the multimedia data table. After pre-training the recall model using the first text pairs, the recall model can learn feature representations for each resource identifier in the multimedia data table. The feature representations of the resource identifiers are reflected by the features of the description information of the multimedia corresponding to the resource identifiers. It is worth noting that if the multimedia library has a large number of multimedia, then the number of resource identifiers in the multimedia data table is also large. This may result in slow training and online inference speeds, and difficulty in cold starting. In practice, it has been found that if the number of resource identifiers is maintained at the millions, learning feature representations of the resource identifiers can meet the training time and online inference speed requirements.

[0117] FIG4 is a schematic diagram of the structure of a recall model according to an embodiment of the present application. As shown in FIG4 , the recall model includes an encoder network 410 and a decoder network 420; wherein the encoder network 410 is used to semantically encode the input sequence and output a semantically encoded sequence, and the decoder network is used to decode the semantically encoded sequence output by the encoder network to obtain an output sequence. Specifically, in the present application, in the pre-training phase, the input sequence is the first question text, and the output sequence is the resource identifier of the predicted multimedia; in the fine-tuning training phase, the input sequence is the second question text, and the output sequence is the resource identifier of the related multimedia corresponding to the predicted multimedia.

[0118] In some embodiments, the recall model can be a BART (Bidirectional and Auto-Regressive Transformers) model. The BART model absorbs the bidirectional encoding of the BERT model and the left-to-right decoding of the GPT model, and is built on the basis of a standard sequence-to-sequence transformer model. This makes the BART model more suitable for text generation scenarios than the BERT model; and compared to the GPT model, it also has more bidirectional context information. Figure 5 shows a schematic diagram of the encoding and decoding processing of the BART model. As shown in Figure 5, after the input sequence is input into the encoder network 410, the encoder network 410 performs bidirectional encoding on the input sequence and outputs a semantic encoding sequence. Afterwards, the decoder network 420 performs autoregressive decoding (i.e., unidirectional decoding from left to right) to obtain an output sequence. In the BART model, the input sequence of the encoder network does not need to be aligned with the output sequence of the decoder network, allowing the input sequence of the encoder network to be preprocessed. For example, the preprocessing can replace the characters in some positions in the input sequence with mask symbols. For example, in the input sequence in Figure 5, the characters after "A" and the characters after "B" are both replaced with mask symbols.

[0119] The BART model adopts the Attention mechanism and the Transformer model structure. In the application scenario of the present application, considering that the data size of the multimedia library is in the millions and the resource consumption, the encoder network in the recall model includes the encoder in the 3-layer transformer model, and the decoder network in the recall model includes the decoder in the 3-layer transformer model. Figure 6 exemplifies a schematic diagram of the transformer model. As shown in Figure 6, the encoder in the transformer model includes a multi-head attention layer, a first summation and normalization layer, a feedforward neural network layer and a second summation and normalization layer, and a residual connection is established between the multi-head attention layer and the first summation and normalization layer, and a residual connection is established between the input of the feedforward neural network layer and the second summation and normalization layer. The decoder in the transformer model includes a masked multi-head attention layer, a third summation and normalization layer, a multi-head attention layer, a fourth summation and normalization layer, a feedforward neural network layer, and a fifth summation and normalization layer, wherein a residual connection is established between the input of the masked multi-head attention layer and the third summation and normalization layer, a residual connection is established between the input of the multi-head attention layer and the fourth summation and normalization layer, and a residual connection is established between the input of the feedforward neural network layer and the fifth summation and normalization layer.

[0120] In some embodiments, as shown in FIG7 , step 220 includes the following steps 710 to 740 , which are described in detail below:

[0121] In step 710 , the encoder network performs semantic encoding on the first question text to obtain a first semantic encoding sequence corresponding to the first question text.

[0122] Specifically, the encoder network can semantically encode the first question text based on an attention mechanism (such as a multi-head attention mechanism), so as to fully utilize the contextual information in the first question text and ensure the accuracy of the obtained first semantic encoding sequence.

[0123] In step 720 , the decoder network decodes the first semantic coding sequence to obtain a predicted answer text corresponding to the first question text.

[0124] The predicted answer text decoded and output by the decoder network includes the resource identifier of the multimedia targeted by the predicted first question text.

[0125] Step 730 : Calculate a first loss based on the predicted answer text corresponding to the first question text and the corresponding first answer text.

[0126] The loss function for the recall model during the pre-training phase can be pre-set. For ease of distinction, the loss function set for the recall model during the pre-training phase is referred to as the first loss function. The first loss function can be a cross-entropy loss function, an absolute value loss function, a mean square error loss function, or the like, and is not specifically limited here. Based on this, the predicted answer text and the corresponding first answer text corresponding to the first question text can be substituted into the first loss function to calculate a first loss, which reflects the difference between the predicted answer text corresponding to the first question text and the first answer text corresponding to the first question text.

[0127] In a specific embodiment, the first loss function may be a cross entropy loss function, and the expression of the cross entropy loss function is shown in the following formula 1:

[0128] Where K represents the number of all categories. i represents the true label of the sample, p i represents the predicted probability of belonging to category i. If category i refers to the category indicating that the predicted answer text is the same as the actual answer text, then p i represents the probability that the predicted answer text for the first question text is the same as the first answer text corresponding to the first question text. i It can be determined by the softmax function according to the following formula 2:

[0129] Among them, z j represents the confidence score that the predicted answer text corresponds to category j.

[0130] Step 740: Inversely adjust the weight parameters of the encoder network and the decoder network according to the first loss.

[0131] In a specific embodiment, the weight parameters of the encoder network and the decoder network can be adjusted according to the first loss using a gradient descent method to minimize the first loss function.

[0132] For each first sample pair, the recall model is iteratively pre-trained according to the process shown in step 710 to step 740 until the pre-training end condition is met.

[0133] In some embodiments, as shown in FIG8 , step 240 includes steps 810 to 840 , which are described in detail below:

[0134] Step 810 : The pre-trained encoder network performs semantic encoding on the second question text to obtain a second semantic encoding sequence corresponding to the second question text.

[0135] Step 820: The pre-trained decoder network decodes the second semantic coding sequence to obtain a predicted answer text corresponding to the second question text.

[0136] The predicted answer text corresponding to the second question text includes the predicted resource identifier of the relevant multimedia targeted by the second question text.

[0137] Step 830 : Calculate a second loss based on the predicted answer text corresponding to the second question text and the corresponding second answer text.

[0138] Similarly, a second loss function can be pre-set for the recall model during the fine-tuning training phase. This second loss function can be set based on actual needs and is not specifically defined here. Subsequently, the predicted answer text and the corresponding second answer text corresponding to the second question text are substituted into the second loss function to calculate the second loss. This second loss reflects the difference between the predicted answer text and the corresponding second answer text corresponding to the second question text.

[0139] In some embodiments, to alleviate the overfitting problem, the second loss function may be a cross entropy loss function with label smoothing. The cross entropy loss function with label smoothing is the same as Formula 1, but q in the cross entropy loss function with label smoothing is i Determine according to the following formula 3:

[0140] Among them, K represents the total number of multi-classification categories, ε is a small hyperparameter that can be set in advance; y represents a positive sample, that is, if the first answer text in the first text pair is the answer to the first question text, then the q corresponding to the sample is i=1-ε. Based on the cross entropy loss function with label smoothing, the recall model can have stronger generalization ability by suppressing the difference between positive and negative sample outputs.

[0141] Label smoothing is a regularization technique used to mitigate overfitting. In the label-smoothed cross-entropy loss function, the probability distribution of the true label is smoothed, preventing the model from overconfidently predicting a particular category during training, thereby reducing the risk of overfitting. Specifically, the label-smoothed cross-entropy loss function can be viewed as transforming the probability distribution of the true label from a one-hot vector into a smoothed probability distribution. This smoothed probability distribution causes the recall model to focus more on the data distribution during training, rather than focusing too much on a specific category. This makes the model more robust and reduces the risk of overfitting. Furthermore, the label-smoothed cross-entropy loss function can also provide a certain degree of regularization. In the label-smoothed cross-entropy loss function, the smoothed probability distribution makes the model smoother during training, thereby reducing model complexity and, in turn, the risk of overfitting.

[0142] Step 840: reversely adjust the weight parameters of some network layers in the pre-trained recall model according to the second loss.

[0143] In some embodiments, the weight parameters of some network layers in the recall model can be adjusted according to the second loss using a gradient descent method to minimize the second loss function. In some embodiments, in step 840, only the weight parameters of the decoder network in the recall model can be adjusted, or the weight parameters of the output layer and a set number of network layers before the output layer in the decoder network can be adjusted. This can reduce the amount of parameter adjustment and shorten the training time of the recall model.

[0144] For each second sample pair, the recall model is iteratively fine-tuned according to the process shown in steps 810-840 until the fine-tuning training end condition is met. After the fine-tuning training is completed, the recall model can be used in online applications to accurately recall the corresponding multimedia based on the multimedia resource identifier.

[0145] FIG9 is a flowchart of a recall method according to an embodiment of the present application. The recall method may be executed by an electronic device such as a server. As shown in FIG9 , the method includes steps 910 to 940 , which are described in detail as follows:

[0146] Step 910: Obtain the resource identifier of the target multimedia.

[0147] The target multimedia refers to the multimedia for which the recall result is to be determined. In some embodiments, each multimedia in the multimedia library can be used as the target multimedia to determine the recall result corresponding to each multimedia in the multimedia library according to the method of the present application.

[0148] Step 920 : Generate a target question text with the relevant resource identifier as the question target according to the resource identifier of the target multimedia. The relevant resource identifier refers to the resource identifier of the relevant multimedia corresponding to the target multimedia.

[0149] In some embodiments, a target question text can be generated based on the second question template and the resource identifier of the target multimedia. Specifically, the resource identifier of the target multimedia is added to the location of the resource identifier of the reference multimedia in the second question template to obtain the target question text. The format of the second question template can be as described above and will not be repeated here.

[0150] In step 930, the recall model generates a target answer text corresponding to the target question text based on the target question text, and the target answer text includes the resource identifier of the relevant multimedia corresponding to the target multimedia; the recall model is trained according to the training method of the recall model in any of the above embodiments.

[0151] In step 930, the target question text is input into the encoder network of the recall model, and the encoder network performs semantic encoding on the target question text to obtain a semantic encoding sequence corresponding to the target question text. Thereafter, the decoder network in the recall model decodes the semantic encoding sequence corresponding to the target question text to obtain a target answer text.

[0152] Step 940: Determine the recall result of the target multimedia according to the resource identifier in the target answer text.

[0153] Based on the correspondence between multimedia and resource identifiers, the multimedia corresponding to the resource identifier in the target answer text can be determined. The multimedia corresponding to the resource identifier in the target answer text is the related multimedia corresponding to the target multimedia. In step 940, the related multimedia corresponding to the target multimedia determined based on the resource identifier in the target answer text is used as the recall result of the target multimedia.

[0154] In the present application, the recall task for multimedia is converted into a text generation task, that is, for the target multimedia to be recalled, a target question text with the relevant resource identifier as the question target is generated according to the resource identifier of the target multimedia. Afterwards, the trained recall model is called to generate a target answer text for the target question text according to the target question text. Since the target question text uses the relevant resource identifier as the question target, the target answer text generated by the trained recall model includes the resource identifier corresponding to the relevant multimedia corresponding to the target multimedia. In this way, the relevant multimedia corresponding to the target multimedia can be determined according to the resource identifier corresponding to the relevant multimedia corresponding to the target multimedia, thereby realizing the recall of the relevant multimedia corresponding to the target multimedia.

[0155] Since the recall model is obtained by pre-training with multiple first text pairs and fine-tuning with multiple second text pairs, the recall model accurately learns the association between the resource identifier and the description information of the multimedia, and realizes the use of the characteristics of the description information of the multimedia as the feature representation of the resource identifier. In this way, when the recall model is used for multimedia recall, multimedia with a high correlation with the feature representation of the resource identifier (i.e., the characteristics of the description information of the multimedia) can be recalled based on the learned association between the resource identifier and the description information of the multimedia. In this way, the correlation between the recalled multimedia and the target multimedia used as a reference can be guaranteed, thereby improving the recall accuracy of the multimedia.

[0156] In some embodiments, after step 940 , the method further includes: associating the resource identifier of the target multimedia with the recall result of the target multimedia and storing them in a recall data set.

[0157] In some embodiments, as shown in FIG10 , the method includes:

[0158] Step 1010: Obtain a multimedia search request; the multimedia search request includes a search keyword.

[0159] The search keywords may be words used to limit the multimedia to be searched, such as the starring actor, multimedia name, director, screenwriter, type, and the like.

[0160] Step 1020: Perform multimedia matching based on the search keyword to determine a second multimedia that matches the search keyword.

[0161] In some embodiments, multimedia matching can be performed based on a maintained multimedia data table. This multimedia data table includes at least the values ​​of each multimedia under each description field, i.e., the multimedia data table includes descriptive information for each multimedia. Based on this, a search keyword can be matched with the multimedia description information in the multimedia data table to determine multimedia that matches the search keyword, i.e., the second multimedia. It is worth noting that the second multimedia determined to match the search keyword in a multimedia search request can be one or more.

[0162] Step 1030: Obtain a recall result corresponding to the second multimedia from the recall data set.

[0163] The recall data set stores recall results corresponding to multiple multimedia. Therefore, after determining the first media, the recall result corresponding to the first multimedia can be obtained from the recall data set according to the resource identifier of the first multimedia.

[0164] In some embodiments, steps 910 to 940 as described above, and the process of storing the recall results of multimedia in the recall data set can be performed offline. In this way, in the process of providing search and recall services online, it is not necessary to call the recall model to determine the recall results of the second multimedia only after the second multimedia is determined. Instead, the recall model is called in advance offline to determine the recall results of each multimedia and store them. In this way, in the process of providing search and recall services online, the recall results of multimedia can be read directly from the recall data set. This can improve the online service efficiency of the search and recall service and shorten the response time.

[0165] In other embodiments, if the computing power of the server is sufficient to meet the response time requirement, the second multimedia can be used as the target multimedia after the second multimedia is matched and determined, and then the recall result of the second multimedia can be determined according to the process of steps 920 to 940.

[0166] Step 1040: Send the second multimedia and the recall result corresponding to the second multimedia to the initiator of the multimedia search request.

[0167] Based on the embodiment corresponding to Figure 10, by initiating a multimedia search request, not only the second multimedia matched based on the search keyword is returned to the initiator of the multimedia search request, but also the related multimedia corresponding to the second multimedia is returned. In this way, it is possible to avoid initiating a multimedia search request again when the user needs to search for multimedia similar to the second multimedia or the same type as the second multimedia. This reduces the number of interactions between the terminal and the server and improves the user experience.

[0168] In some embodiments, the user can be prompted in advance whether it is necessary to add related multimedia corresponding to the second multimedia to be searched to the search results in the multimedia search scenario. With the user's permission or consent, in accordance with the embodiment corresponding to Figure 10, when searching for multimedia, the second multimedia and the recall results corresponding to the second multimedia are sent to the initiator of the multimedia search request; if the user does not agree to add related multimedia corresponding to the second multimedia to be searched to the search results, the matched second multimedia is sent to the initiator of the multimedia search request without sending the recall results of the second multimedia.

[0169] The following describes an embodiment of the device of the present application, which can be used to perform the method described in the above embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the above method embodiment of the present application.

[0170] FIG11 is a block diagram of a recall model training device according to an embodiment of the present application. The recall model training device can be configured in an electronic device to implement the recall model training method provided in the present application. As shown in Figure 11, the training device of the recall model includes: a first acquisition module 1110, used to acquire multiple first text pairs, the first text pair includes a first question text and a first answer text, the first question text is generated based on the description information of the multimedia, and is a text with the resource identifier of the multimedia as the question target, and the first answer text is the resource identifier of the question asked by the first question text; a pre-training module 1120, used to pre-train the recall model based on the first question text and the first answer text in the multiple first text pairs; a second acquisition module 1130, used to acquire multiple second text pairs, the second text pair includes a second question text and a second answer text, the second question text is a text with the resource identifier of the relevant multimedia as the question target, and the second answer text is the resource identifier of the relevant multimedia asked by the second question text, and the relevant multimedia involved in the second text pair corresponds to the multimedia involved in the first text pair; a fine-tuning training module 1140, used to fine-tune the pre-trained recall model based on the second question text and the second answer text in the multiple second text pairs.

[0171] In some embodiments, the recall model includes an encoder network and a decoder network; the pre-training module 1120 includes: a first semantic encoding unit, which is used to perform semantic encoding processing on the first question text by the encoder network to obtain a first semantic encoding sequence corresponding to the first question text; a first decoding unit, which is used to perform decoding processing on the first semantic encoding sequence by the decoder network to obtain a predicted answer text corresponding to the first question text; a first loss determination unit, which is used to calculate the first loss based on the predicted answer text corresponding to the first question text and the corresponding first answer text; and a first adjustment unit, which is used to reversely adjust the weight parameters of the encoder network and the decoder network according to the first loss.

[0172] In some embodiments, the fine-tuning training module 1140 includes: a second semantic encoding unit, which is used to perform semantic encoding processing on the second question text by a pre-trained encoder network to obtain a second semantic encoding sequence corresponding to the second question text; a second decoding unit, which is used to perform decoding processing on the second semantic encoding sequence by a pre-trained decoder network to obtain a predicted answer text corresponding to the second question text; a second loss determination unit, which is used to calculate the second loss based on the predicted answer text corresponding to the second question text and the corresponding second answer text; and a second adjustment unit, which is used to reversely adjust the weight parameters of some network layers in the pre-trained recall model according to the second loss.

[0173] In some embodiments, the training device of the recall model also includes: a fourth acquisition module, which acquires the description information of the multimedia and the resource identifier corresponding to the multimedia; a first question text determination module, which is used to generate a first question text with the resource identifier of the multimedia as the question target based on the value of at least one description field in the description information; and a first answer text determination module, which is used to use the resource identifier of the multimedia as the first answer text corresponding to the first question text.

[0174] In some embodiments, the first question text determination module includes: a first question template acquisition unit, used to obtain the first question template, the first question template uses the resource identifier as the question target, and the first question template indicates at least one description field; a first acquisition unit, used to obtain the values ​​of each description field indicated by the first question template from the description information of the multimedia; a first combination unit, used to combine the obtained description field value with the first question template to obtain the first question text.

[0175] In some embodiments, the training device of the recall model also includes: a second acquisition unit, used to acquire multimedia feedback data, the multimedia feedback data indicates at least two multimedia that are triggered to perform feedback operations within a set time period; a second question text determination unit, used to generate a second question text with the resource identifier of the related multimedia corresponding to the first multimedia as the question target based on the resource identifier corresponding to the first multimedia in the at least two multimedia; the related multimedia corresponding to the first multimedia includes at least one multimedia in the at least two multimedia except the first multimedia; a second answer text determination unit, used to use the resource identifier of the related multimedia corresponding to the first multimedia as the second answer text corresponding to the second question text.

[0176] In some embodiments, the second question text determination unit includes: a second question template acquisition unit, used to obtain a second question template, the second question template uses the resource identifier of the relevant multimedia as the question target; a second combination unit, used to combine the resource identifier corresponding to the first multimedia in at least two multimedia with the second question template to obtain a second question text.

[0177] Figure 12 is a block diagram of a recall device according to an embodiment of the present application. The recall device can be configured in an electronic device to implement the recall method provided by the present application. As shown in Figure 12, the recall device includes: a third acquisition module 1210, which is used to obtain the resource identifier of the target multimedia. A target question text generation module 1220 is used to generate a target question text with a relevant resource identifier as the question target based on the resource identifier of the target multimedia. The relevant resource identifier refers to the resource identifier of the relevant multimedia corresponding to the target multimedia. A target answer text determination module 1230 is used to generate a target answer text corresponding to the target question text based on the target question text by a recall model. The target answer text includes the resource identifier of the relevant multimedia corresponding to the target multimedia; the recall model is trained according to the training method of the recall model in any of the above embodiments. A recall result determination module 1240 is used to determine the recall result of the target multimedia based on the resource identifier in the target answer text.

[0178] In some embodiments, the recall device further includes: an association storage module, configured to associate the resource identifier of the target multimedia with the recall result of the target multimedia and store them in the recall data set.

[0179] In some embodiments, the recall device also includes: a fifth acquisition module, used to obtain a multimedia search request; the multimedia search request includes a search keyword; a matching module, used to perform multimedia matching based on the search keyword, and determine the second multimedia that matches the search keyword; a recall result acquisition module, used to obtain the recall result corresponding to the second multimedia from the recall data set; and a sending module, used to send the second multimedia and the recall result corresponding to the second multimedia to the initiator of the multimedia search request.

[0180] FIG13 shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application. It should be noted that the computer system 1300 of the electronic device shown in FIG13 is only an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application. The electronic device can be used to execute the training method of the recall model provided in the present application, and can also be used to execute the recall method provided in the present application.

[0181] As shown in Figure 13, computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 1302 or programs loaded from storage unit 1308 into random access memory (RAM) 1303, such as executing the methods in the above embodiments. Various programs and data required for system operation are also stored in RAM 1303. CPU 1301, ROM 1302, and RAM 1303 are connected to each other via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0182] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, and the like; an output section 1307 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1308 including a hard disk; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. Removable media 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1310 as needed, so that computer programs read from the removable media can be installed in the storage section 1308 as needed.

[0183] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1309, and / or installed from a removable medium 1311. When the computer program is executed by the central processing unit (CPU) 1301, the various functions defined in the system of the present application are executed.

[0184] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0186] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0187] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable storage medium carries a computer program, and when the computer-readable storage instructions are executed by a processor, the recall model training method or recall method described in any of the above embodiments is implemented.

[0188] According to one aspect of an embodiment of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the recall model training method or recall method described in any of the above embodiments.

[0189] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0190] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0191] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0192] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for training a recall model, the method being performed by an electronic device, the method comprising: Acquire a plurality of first text pairs, wherein the first text pairs include a first question text and a first answer text, wherein the first question text is generated according to the description information of the multimedia and takes the resource identifier of the multimedia as the question target, and the first answer text is the resource identifier targeted by the first question text; Pre-training a recall model according to a first question text and a first answer text in a plurality of first text pairs; Acquire a plurality of second text pairs, the second text pairs comprising a second question text and a second answer text, the second question text being a text with a resource identifier of related multimedia as a question target, the second answer text being a resource identifier of related multimedia targeted by the second question text, and the related multimedia involved in the second text pair corresponding to the multimedia involved in the first text pair; The pre-trained recall model is fine-tuned according to the second question texts and the second answer texts in the plurality of second text pairs.

2. The method according to claim 1, wherein the recall model comprises an encoder network and a decoder network; The pre-training of the recall model according to the first question text and the first answer text in the plurality of first text pairs includes: The encoder network performs semantic encoding processing on the first question text to obtain a first semantic encoding sequence corresponding to the first question text; The decoder network decodes the first semantic coding sequence to obtain a predicted answer text corresponding to the first question text; Calculating a first loss based on the predicted answer text corresponding to the first question text and the corresponding first answer text; The weight parameters of the encoder network and the decoder network are reversely adjusted according to the first loss.

3. The method according to claim 2, wherein fine-tuning the pre-trained recall model according to the second question text and the second answer text in the plurality of second text pairs comprises: The second question text is semantically encoded by the pre-trained encoder network to obtain a second semantic encoding sequence corresponding to the second question text; Decoding the second semantic coding sequence using a pre-trained decoder network to obtain a predicted answer text corresponding to the second question text; Calculating a second loss according to the predicted answer text corresponding to the second question text and the corresponding second answer text; The weight parameters of some network layers in the pre-trained recall model are reversely adjusted according to the second loss.

4. The method according to any one of claims 1 to 3, before obtaining the plurality of first text pairs, the method further comprises: Obtaining description information of multimedia and resource identifiers corresponding to the multimedia; generating, according to a value of at least one description field in the description information, the first question text with the resource identifier of the multimedia as the question target; The resource identifier of the multimedia is used as the first answer text corresponding to the first question text.

5. The method according to claim 4, wherein generating the first question text with the resource identifier of the multimedia as the question target according to the value of at least one description field in the description information comprises: Acquire a first question template, wherein the first question template takes a resource identifier as a question target, and the first question template indicates at least one description field; Acquire the values ​​of the description fields indicated by the first question template from the multimedia description information; The acquired value of the description field is combined with the first question template to obtain the first question text.

6. The method according to any one of claims 1 to 3, before obtaining a plurality of second text pairs, the method further comprises: Acquire multimedia feedback data, where the multimedia feedback data indicates at least two multimedia for which feedback operations are triggered within a set time period; generating, according to the resource identifier corresponding to the first multimedia of the at least two multimedia, a second question text with the resource identifier of the related multimedia corresponding to the first multimedia as the question target; The related multimedia corresponding to the first multimedia includes at least one multimedia among the at least two multimedia except the first multimedia; The resource identifier of the related multimedia corresponding to the first multimedia is used as the second answer text corresponding to the second question text.

7. The method according to claim 6, wherein generating, based on the resource identifier corresponding to the first multimedia of the at least two multimedia, a second question text with the resource identifier of the related multimedia corresponding to the first multimedia as the question target comprises: Acquire a second question template, wherein the second question template uses the resource identifier of the relevant multimedia as a question target; The resource identifier corresponding to the first multimedia of the at least two multimedia is combined with the second question template to obtain the second question text.

8. A recall method, the method being performed by an electronic device, the method comprising: Get the resource identifier of the target multimedia; Generate a target question text with a related resource identifier as a question target according to the resource identifier of the target multimedia, wherein the related resource identifier refers to the resource identifier of the related multimedia corresponding to the target multimedia; The recall model generates a target answer text corresponding to the target question text according to the target question text, wherein the target answer text includes a resource identifier of related multimedia corresponding to the target multimedia; the recall model is trained according to the method described in any one of claims 1 to 7; The recall result of the target multimedia is determined according to the resource identifier in the target answer text.

9. The method according to claim 8, after determining the recall result of the target multimedia according to the resource identifier in the target answer text, the method further comprises: The resource identifier of the target multimedia is associated with the recall result of the target multimedia and stored in a recall data set.

10. The method according to claim 9, comprising: Get multimedia search request; The multimedia search request includes a search keyword; Perform multimedia matching according to the search keyword to determine a second multimedia that matches the search keyword; Acquire a recall result corresponding to the second multimedia from the recall data set; The second multimedia and the recall result corresponding to the second multimedia are sent to the initiator of the multimedia search request.

11. A training device for a recall model, comprising: A first acquisition module is used to acquire a plurality of first text pairs, wherein the first text pairs include a first question text and a first answer text, wherein the first question text is generated according to the description information of the multimedia and takes the resource identifier of the multimedia as the question target, and the first answer text is the resource identifier targeted by the first question text; A pre-training module, used for pre-training a recall model according to a first question text and a first answer text in a plurality of first text pairs; A second acquisition module is used to acquire a plurality of second text pairs, wherein the second text pairs include a second question text and a second answer text, wherein the second question text is a text that takes a resource identifier of related multimedia as a question target, and the second answer text is a resource identifier of related multimedia that the second question text asks about, and the related multimedia involved in the second text pair corresponds to the multimedia involved in the first text pair; The fine-tuning training module is used to perform fine-tuning training on the pre-trained recall model according to the second question text and the second answer text in the plurality of second text pairs.

12. A recall device, comprising: A third acquisition module is used to acquire a resource identifier of a target multimedia; A target question text generation module, used to generate a target question text with a related resource identifier as a question target according to the resource identifier of the target multimedia, wherein the related resource identifier refers to the resource identifier of the related multimedia corresponding to the target multimedia; a target answer text determination module, configured to generate a target answer text corresponding to the target question text by a recall model according to the target question text, wherein the target answer text includes a resource identifier of related multimedia corresponding to the target multimedia; the recall model is trained according to the method described in any one of claims 1 to 7; The recall result determination module is used to determine the recall result of the target multimedia according to the resource identifier in the target answer text.

13. An electronic device, comprising: processor; A memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented, or the method according to any one of claims 8 to 10 is implemented.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of the computer, implements the method according to any one of claims 1 to 7, or implements the method according to any one of claims 8 to 10.

15. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented, or the method according to any one of claims 8 to 10 is implemented.

Citation Information

Patent Citations

  • Model training method, data processing method, equipment and computer readable storage medium

    CN113822444A

  • Intelligent question and answer recall method and device, computer equipment and storage medium

    CN114358023A

  • Method for generating multi-task generation model, question and answer pair generation method and related device

    CN115391499A

  • Training method of recall model, recall method and related equipment

    CN116303975A

  • Question and answer pair generation using machine learning

    US20190228099A1

Cited By

  • Multi-modal knowledge graph construction method and device

    CN119578526A