Methods, devices, equipment and storage media for searching multimedia data

Abstract

This application provides a method, apparatus, device, and computer-readable storage medium for searching multimedia data. The method includes: based on an acquired search request, determining the multimedia data and text data to be processed, and acquiring a trained cross-modal representation model, wherein the trained cross-modal representation model is obtained through two training phases using multimedia training data, the identifier information of the multimedia training data, and the search text corresponding to the multimedia training data; inputting the multimedia data and the text data into the trained cross-modal representation model to obtain the semantic similarity of each element of the multimedia data and the text data; and determining and outputting search results based on the semantic similarity and each element of the multimedia data. This application can improve the relevance between the search results and the search content itself.

Description

Technical Field

[0001] This application relates to machine learning techniques, and more particularly to a method, apparatus, device, and computer-readable storage medium for searching multimedia data. Background Technology

[0002] With the development of internet and terminal technologies, mobile terminals, as a communication tool, have far exceeded their original functions, bringing users more entertainment and interactivity. For example, users can use mobile terminals to watch live streams or videos, listen to music, etc., anytime and anywhere.

[0003] When users need to search for videos or music they are interested in, they can use video apps or music apps, or enter search keywords in a browser to obtain related videos or music. Since the search object is multimedia, there is a semantic gap between multimedia and natural language retrieval. Therefore, multimodal data beyond textual information such as titles is needed to obtain more accurate search results. In related technologies, multimedia semantic information extraction methods and semantic vector convex combination methods in cross-modal representation models require converting temporal information into textual information, which can lead to loss of detail. Furthermore, the training data for cross-modal representation models such as the dual-tower model mostly comes from manually labeled classifications, tags, and text fragments describing video or image content, requiring substantial annotation resources, incurring high costs, and limiting the availability and application scope of training data. Summary of the Invention

[0004] This application provides a method, apparatus, and computer-readable storage medium for searching multimedia data, which can improve the relevance between search results and the search content itself.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for searching multimedia data, including:

[0007] Based on the obtained search request, determine the multimedia data and text data to be processed;

[0008] A trained cross-modal representation model is obtained, wherein the trained cross-modal representation model is obtained through two training stages using multimedia training data, the identification information of the multimedia training data, and the search text corresponding to the multimedia training data;

[0009] The multimedia data and the text data are input into the trained cross-modal representation model to obtain the semantic similarity between the multimedia data and the text data;

[0010] Based on the semantic similarity and the multimedia data, the search results are determined and output.

[0011] This application provides a multimedia data search device, including:

[0012] The first determining module is used to determine the multimedia data and text data to be processed based on the obtained search request;

[0013] The first acquisition module is used to acquire a trained cross-modal representation model, wherein the trained cross-modal representation model is obtained through two training stages using multimedia training data, the identification information of the multimedia training data, and the search text corresponding to the multimedia training data;

[0014] The similarity calculation module is used to input the multimedia data and the text data into the trained cross-modal representation model to obtain the semantic similarity between the multimedia data and the text data;

[0015] The second determining module is used to determine and output search results based on the semantic similarity and the multimedia data.

[0016] In some embodiments, the device further includes:

[0017] The second acquisition module is used to acquire a preset cross-modal representation model and acquire the training dataset of the preset cross-modal representation model, wherein the training dataset includes a first training data subset and a second training data subset.

[0018] The first training data subset includes multimedia training data and the identification information of the multimedia training data; the second training data subset includes the multimedia training data and the search text corresponding to the multimedia training data.

[0019] The first training module is used to train the preset cross-modal representation model using the first training data subset to obtain a pre-trained cross-modal representation model.

[0020] The second training module is used to train the pre-trained cross-modal representation model using the second training data subset to obtain the trained cross-modal representation model.

[0021] In some embodiments, the second acquisition module is further configured to:

[0022] Obtain search logs within a preset time period, the search logs including multiple search texts and click event information corresponding to the multiple search texts;

[0023] Obtain the click playback duration and actual duration of the multimedia data corresponding to each click event;

[0024] Based on the click playback duration and actual duration of each multimedia data, the target multimedia data is determined, and the search text corresponding to the target multimedia data and the target multimedia data identification information are obtained.

[0025] The target multimedia data is determined as the multimedia training data, the identification information of the target multimedia data is determined as the identification information of the multimedia training data, and the search text corresponding to the target multimedia data is determined as the search text corresponding to the multimedia training data.

[0026] Multiple supplementary multimedia data with actual durations less than a duration threshold are selected from the multimedia database, and the identification information of the supplementary multimedia data is obtained; the first training data subset also includes the supplementary multimedia data and the identification information of the supplementary multimedia data.

[0027] In some embodiments, the first training module is further configured to:

[0028] First training positive samples and first training negative samples are determined based on the first training data subset, wherein the first training positive samples include each multimedia data and the corresponding identification information of each multimedia data, and the first training negative samples include randomly determined multimedia data and randomly determined identification information.

[0029] Each first training positive sample is input into the preset cross-modal representation model to obtain the predicted similarity of each first positive sample;

[0030] Each first training negative sample is input into the preset cross-modal representation model to obtain the predicted similarity of each first negative sample;

[0031] The predicted similarity of each first positive sample and the predicted similarity of each second negative sample are backpropagated to the preset cross-modal representation model. The preset cross-modal representation model is trained using the first loss function to adjust the parameters of the preset cross-modal representation model.

[0032] In some embodiments, the second training module is further configured to:

[0033] Obtain the category labels and predicted categories of each multimedia data in the second training data subset;

[0034] The second training positive sample and the second training negative sample are determined based on the second training data subset. The second training positive sample includes each multimedia data and the search text corresponding to each multimedia data. The second training negative sample includes randomly determined multimedia data and randomly determined search text.

[0035] Each second training positive sample is input into the preset cross-modal representation model to obtain the predicted similarity of each second positive sample;

[0036] Each second training negative sample is input into the preset cross-modal representation model to obtain the predicted similarity of each second negative sample;

[0037] The predicted similarity of each second positive sample, the predicted similarity of each second negative sample, each category label, and the predicted category are backpropagated to the preset cross-modal representation model. The preset cross-modal representation model is jointly trained using the first loss function and the second loss function to adjust the parameters of the preset cross-modal representation model.

[0038] In some embodiments, the preset cross-modal representation model includes a multimedia sub-model and a text sub-model, and the second training module is further configured to:

[0039] Each multimedia data is input into the multimedia sub-model to obtain the feature vector of each multimedia.

[0040] The search text corresponding to each multimedia data is input into the text sub-model to obtain the feature vector of each search text.

[0041] Clustering is performed on the multimedia feature vectors to obtain the multimedia categories;

[0042] The feature vectors of each search text are classified to obtain each text category;

[0043] The multimedia category corresponding to each multimedia data is determined as the predicted category, and the text category of the search text corresponding to each multimedia data is determined as the category label.

[0044] In some embodiments, the trained cross-modal representation model includes a trained multimedia sub-model and a trained text sub-model. Correspondingly, the similarity calculation module is further used for:

[0045] The multimedia data is input into the multimedia sub-model to obtain the multimedia feature vectors corresponding to the multimedia data;

[0046] The text data is input into the text sub-model to obtain the text feature vector corresponding to the text data, wherein the multimedia feature vector and the text feature vector belong to the same vector space;

[0047] The semantic similarity between the multimedia data and the text data is determined using the multimedia feature vector and the text feature vector.

[0048] In some embodiments, the first determining module is further configured to:

[0049] Obtain the search request sent by the terminal, and determine the search text based on the search request;

[0050] Based on the search text and the title information of each multimedia data in the multimedia database, the multimedia data to be processed is determined.

[0051] The search text is identified as the text data to be processed.

[0052] In some embodiments, the first determining module is further configured to:

[0053] Obtain the search information carried in the search request;

[0054] When the search information is text information, the search information is determined as the search text;

[0055] When the search information is voice information, the search information is subjected to voice recognition to obtain the search text.

[0056] In some embodiments, the second determining module is further configured to:

[0057] Multiple multimedia data to be processed are sorted based on semantic similarity to obtain sorted multimedia data. Among them, the multimedia data with the highest similarity to the search text is the first multimedia data.

[0058] Determine search results based on the sorted multimedia data;

[0059] The search results are returned to the terminal that sent the search request.

[0060] This application provides a multimedia data search device, including:

[0061] Memory, used to store executable instructions;

[0062] A processor, when executing executable instructions stored in the memory, implements the method provided in the embodiments of this application.

[0063] This application provides a computer-readable storage medium storing executable instructions for inducing a processor to execute and implement the method provided in this application.

[0064] The embodiments of this application have the following beneficial effects:

[0065] Based on the acquired search request, the multimedia data and text data to be processed are determined, and a trained cross-modal representation model is obtained. The multimedia data and text data are then input into the trained cross-modal representation model to obtain the semantic similarity between the multimedia data and the text data. Finally, based on the semantic similarity and the multimedia data, the search results are determined and output. The trained cross-modal representation model is obtained through two training phases using multimedia training data, the identifier information of the multimedia training data, and the corresponding search text. In other words, this cross-modal representation model can effectively map multimedia data and text data to the same space, thereby establishing a connection between text and multimedia data during the search process, improving the relevance of the search results to the input search information, and thus improving the relevance of the search results. Furthermore, the training process of this cross-modal representation model does not require additional manually labeled data, thus reducing training costs and improving training efficiency. Attached Figure Description

[0066] Figure 1A This is a schematic diagram of a convex combination model of semantic vectors in related technologies;

[0067] Figure 1B This is a schematic diagram of a neural network model with a dual-tower structure in related technologies;

[0068] Figure 2 A schematic diagram of the network architecture of the multimedia data search system provided in the embodiments of this application;

[0069] Figure 3 This is a schematic diagram of the structure of the server 400 provided in an embodiment of this application;

[0070] Figure 4 A schematic diagram illustrating an implementation flow of the multimedia data search method provided in this application embodiment;

[0071] Figure 5A A schematic diagram illustrating the implementation process of performing a first-stage training of a preset cross-modal representation model using a first training subset, as provided in an embodiment of this application.

[0072] Figure 5B A schematic diagram illustrating the implementation process of using a second training subset to perform a second-stage training of a preset cross-modal representation model, as provided in an embodiment of this application.

[0073] Figure 6 A schematic diagram illustrating another implementation flow of the multimedia data search method provided in this application embodiment;

[0074] Figure 7 This is a schematic diagram of the neural network model with a dual-tower structure provided in an embodiment of this application. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0077] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0078] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0079] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0080] 1) Cross-modal embedding: The inputs of multiple modalities such as video, image, and text are mapped to fixed-length low-dimensional feature vectors in the same vector space. The distance between them can measure the semantic similarity between different inputs.

[0081] 2) NextVLAD: A temporal convergence method that converts video frame features into video-level features, a module of a video recognition model.

[0082] 3) The loss function, also known as the cost function, is a function that maps the values ​​of a random event or its related random variables to non-negative real numbers to represent the "risk" or "loss" of that random event. In applications, the loss function is often used as a learning criterion in relation to optimization problems; that is, the model is solved and evaluated by minimizing the loss function. For example, in statistics and machine learning, it is used for parameter estimation of models and serves as the optimization objective of machine learning models.

[0083] 4) Word vectors, also known as word embeddings or word space embedding representations, are the representations of natural language word segments in the word space. They are vectors obtained by mapping words to a semantic space.

[0084] 5) Triplet ranking loss: The idea is to make the distance between negative sample pairs greater than the distance between positive sample pairs.

[0085] To better understand the multimedia data search method provided in the embodiments of this application, we will first explain the multimedia data search methods in related technologies and their shortcomings. Video search will be used as an example for this explanation.

[0086] Because video platform searches involve multimedia content, there is a semantic gap between them and natural language searches. Therefore, more accurate search results require the use of multimodal data beyond textual information such as titles. To meet the needs of multimodal search, the following search methods are currently available:

[0087] First, video semantic information extraction: Automatically predicts tags for videos and frames, identifies and indexes content such as celebrities, objects, and actions in visual modalities, and then matches the retrieved text with the predicted text tags to retrieve the information after video deconstruction, which can support users' needs for accurate video content segments.

[0088] Second, image search: Users can take photos or upload images to search for people, programs, and videos with similar scenes. The search steps can be to recall programs by name after facial recognition, or to recall similar scenes directly by the image's representation vector. That is, the visual module can use the representation vector of a single module for recall.

[0089] Third, cross-modal representation models: Tasks associated with multimodal search include converting videos to text, locating and searching video clips, and retrieving video clips related to input text. Models learned from these tasks can perform cross-modal matching between videos and text. The structure of cross-modal representation models includes convex combinations of semantic vectors and dual-tower models. A schematic diagram of the convex combination model of semantic vectors is shown below. Figure 1A As shown, the video predicts labels by using a pre-trained multi-label model 101 and a pre-trained word vector model 102; the retrieved text predicts labels by using a pre-trained word vector model 103 and a language model 104. The video predicts labels and the text predicts labels are combined as semantic vectors and added together to obtain a corresponding semantic vector, which is then matched with the semantic vector of the retrieved text.

[0090] The structural diagram of the twin-tower model is as follows: Figure 1BAs shown, the retrieved text is input into the word vector + text model 112, and the video is input into the visual model 111. Then, the outputs of the two neural network models are transformed in dimension to map them into a vector space, and then similarity matching and ranking are performed.

[0091] The method of using video semantic information extraction to assist multimodal retrieval compresses multimedia content such as videos and images into text information. While it can utilize multimedia content to some extent, it loses many rich details and relies on a predefined tag system, making it unable to capture new semantic information in a timely manner.

[0092] Image-based search methods can only calculate similarity within a single-modal visual space, limiting the input format for retrieval. In contrast, natural language is more natural and prevalent in everyday retrieval interactions. Similarly, semantic vector convex combination methods in cross-modal representation models rely on converting visual information into textual information for cross-modal representation, sharing the same limitations as semantic information extraction.

[0093] Multimodal retrieval aided by cross-modal representation models such as the dual-tower model can make fuller use of multimedia information and reduce the semantic gap between vision and language. However, the training data for current cross-modal representation models mostly comes from manually annotated classifications, labels, and text fragments describing video or image content, requiring a large amount of annotation resources, incurring high costs, and limiting the availability and application scope of training data. Furthermore, directly training models using weakly correlated video and text data from the internet often results in models that struggle to converge.

[0094] A large number of sample pairs can be constructed from user search and click data to train cross-modal embeddings, effectively utilizing user behavioral data to obtain a better representation model. The method proposed in this invention is based on a dual-tower model and optimizes it by introducing a multi-stage training approach. The first stage of training utilizes textual information such as video titles, followed by a second stage using search logs. This allows for the training of a representation model that effectively maps videos and text to the same space without requiring additional manually labeled data. Furthermore, the use of clustering pseudo-labels during training accelerates the model's convergence speed and improves its performance.

[0095] The following describes an exemplary application of the multimedia data search device provided in the embodiments of this application. The device provided in the embodiments of this application can be implemented as a terminal or as a server.

[0096] See Figure 2 , Figure 2 This is a schematic diagram of the network architecture of the multimedia data search system 100 provided in the embodiments of this application, as shown below. Figure 2As shown, the network architecture includes a terminal 200, a network 300, and a server 400. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0097] Terminal 200 has various applications (Apps) installed, such as shopping Apps, music Apps, video Apps, instant messaging Apps, and browser Apps. Users can search for videos they want to watch through the video App on terminal 200, or search for music or songs they want to listen to through the music App. Taking video search as an example, terminal 200 can obtain search information based on received input and send it to server 400 via network 300, carrying the search information in a search request. After receiving the search request, server 400 determines the search text based on the search information and filters multiple videos matching the search text from the video library. It then obtains a trained cross-modal representation model and inputs each video and search text into the model to determine the semantic similarity between the search text and each video. Based on the semantic similarity, the videos are ranked, and the entry information of the ranked videos is returned to terminal 200 as the search results. Terminal 200 receives and presents the search results. Thus, the videos in the search results presented by terminal 200 are sorted from high to low semantic similarity, ensuring the accuracy of the search results.

[0098] In some embodiments, server 400 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart TV, smart vehicle device, etc., but is not limited thereto. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0099] See Figure 3 , Figure 3 This is a schematic diagram of the structure of the multimedia data search device provided in this application embodiment. In this embodiment, the multimedia data search device is described as server 400. Figure 3The server 400 shown includes at least one processor 410, memory 450, at least one network interface 420, and a user interface 430. The various components in server 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 3 The general labeled all buses as Bus System 440.

[0100] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0101] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0102] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0103] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0104] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0105] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0106] The network communication module 452 is used to reach other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0107] The input processing module 453 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0108] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 3 A multimedia data search device 454 stored in memory 450 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a first determination module 4541, a first acquisition module 4542, a similarity calculation module 4543, and a second determination module 4544. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.

[0109] The functions of each module will be explained below.

[0110] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimedia data search method provided in this application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0111] The method for searching multimedia data provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0112] To better understand the methods provided in the embodiments of this application, we will first explain artificial intelligence, its various branches, and the application fields involved in the methods provided in the embodiments of this application.

[0113] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0114] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning. The solutions provided in this application mainly relate to natural language processing and machine learning technologies in AI, which will be described separately below.

[0115] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0116] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0117] This application provides a method for searching multimedia data. This method can be implemented by a server or a terminal, and in some embodiments, it can be implemented by both a terminal and a server. In this application, the method is described using a server as an example. See also... Figure 4 , Figure 4 This is a schematic diagram illustrating an implementation flow of the multimedia data search method provided in this application embodiment, which will be combined with... Figure 4 The steps shown are explained.

[0118] Step S101: Based on the obtained search request, determine the multimedia data and text data to be processed.

[0119] Here, the search request can be sent from the terminal to the server in response to a received search operation. This search request carries search information, which can be text information (e.g., "braised pork recipe") or voice information. When the search information is text, it is directly identified as the text data to be processed. When the search information is voice, speech recognition is performed, and the recognized text is identified as the text data to be processed. The multimedia data to be processed can be determined based on the text data. In implementation, multiple multimedia data sets whose titles match the text data in the multimedia database can be identified as the multimedia data to be processed. This multimedia data can be video data or audio data.

[0120] Step S102: Obtain the trained cross-modal representation model.

[0121] Here, the trained cross-modal representation model is obtained through two training phases using multimedia training data, the identifier information of the multimedia training data, and the search text corresponding to the multimedia training data. The identifier information of the multimedia training data can be the title of the multimedia training data. In implementation, the multimedia training data and its identifier information can be used as the training data for the first phase, and the multimedia training data and its corresponding search text can be used as the training data for the second phase. The multimedia training data and its corresponding search text can be obtained by filtering search logs within a certain time period. In this embodiment, the training data used in the training process of the cross-modal representation model does not require manual annotation, thereby reducing the manual cost of training data and improving training efficiency. Furthermore, the cross-modal representation model includes two sub-models: a multimedia sub-model and a text sub-model, used to map data from different modalities to the same vector space.

[0122] Step S103: Input the multimedia data and the text data into the trained cross-modal representation model to obtain the semantic similarity between the multimedia data and the text data.

[0123] In this embodiment of the application, the trained cross-modal representation model includes a trained multimedia sub-model and a trained text sub-model. The multimedia data to be processed is generally multiple. In step S103, each multimedia data is input into the trained multimedia sub-model to obtain each multimedia feature vector, and the text data is input into the text sub-model to obtain the text feature vector. The multimedia feature vectors and the text feature vectors are in the same vector space. Then, based on the multimedia feature vectors and the text feature vectors, the semantic similarity between each multimedia data and the text data is determined.

[0124] Step S104: Based on the semantic similarity and multimedia data, determine and output the search results.

[0125] Here, step S104 can be implemented by sorting multiple multimedia data sets in descending order of semantic similarity, and then determining the search result based on the sorted multimedia data. The search result can be the entry address corresponding to the sorted multimedia data, such as the Uniform Resource Locator (URL) address corresponding to the multimedia data. Outputting the search result can be implemented by the server sending the search result to the terminal corresponding to the search request; in some embodiments, the search result can also be presented on the server's display interface.

[0126] In the multimedia data search method provided in this application embodiment, after determining the various multimedia data and text data to be processed based on the obtained search request and obtaining a trained cross-modal representation model, the various multimedia data and text data are input into the trained cross-modal representation model to obtain the semantic similarity of the various multimedia data and text data. Finally, based on the semantic similarity and the various multimedia data, the search results are determined and output. The trained cross-modal representation model is obtained through two training stages using multimedia training data, the identifier information of the multimedia training data, and the search text corresponding to the multimedia training data. In other words, this cross-modal representation model can effectively map multimedia data and text data to the same space, thereby establishing a connection between text and multimedia data during the search process, improving the relevance between the search results and the input search information, and thus improving the accuracy of the search results. Furthermore, the training process of this cross-modal representation model does not require additional collection of manually labeled data, thus reducing training costs and improving training efficiency.

[0127] In some embodiments, prior to step S102, a trained cross-modal representation model needs to be obtained through the following steps:

[0128] Step S001: Obtain a preset cross-modal representation model and obtain the training dataset of the preset cross-modal representation model.

[0129] Here, the preset cross-modal representation model can be a neural network model, such as a deep learning neural network model. The cross-modal representation model includes a multimedia sub-model and a text sub-model. The training dataset includes a first training data subset and a second training data subset; the first training data subset includes multimedia training data and the identification information of the multimedia training data, which may be the title of the multimedia training data; the second training data subset includes the multimedia training data and the search text corresponding to the multimedia training data, and the second training data subset is filtered from the search logs of the multimedia platform.

[0130] Step S002: Use the first training data subset to train the preset cross-modal representation model to obtain a pre-trained cross-modal representation model.

[0131] Here, in implementing step S002, positive and negative samples can be constructed based on the first training data subset, and the cross-modal representation model can be trained based on the preset loss function and the positive and negative samples to obtain the pre-trained cross-modal representation model.

[0132] Step S003: Use the second training data subset to train the pre-trained cross-modal representation model to obtain the trained cross-modal representation model.

[0133] In step S003, positive and negative samples can be constructed based on the second training data subset. In order to improve the convergence speed of the model, in step S003, the search text corresponding to the multimedia data can also be clustered, and the clustered categories can be used as the category labels of the multimedia data. The classification loss function and the first loss function can be jointly trained to improve the stability of training convergence.

[0134] While the data from the retrieval logs better meets the requirements of the target retrieval task, the retrieval logs contain a lot of noisy data and the vocabulary coverage is not comprehensive enough, making direct training difficult to converge. Therefore, in steps S001 to S003 above, the cross-modal representation model is first trained using multimedia data and the corresponding label information. Then, the model is trained in the second stage using multimedia data and the corresponding search text. In the second stage of training, video pseudo-labels are added to accelerate the convergence speed and improve the model's prediction performance.

[0135] In some embodiments, the step S001 of "obtaining the training dataset of the preset cross-modal representation model" can be achieved through the following steps:

[0136] Step S0011: Obtain the search logs within a preset time period.

[0137] Here, the search log includes multiple search texts and corresponding click event information. In this embodiment, the click event information includes the identifier of the clicked multimedia data and the playback duration. The playback duration refers to the duration of playback after clicking on a specific multimedia data.

[0138] Step S0012: Obtain the click playback duration and actual duration of the multimedia data corresponding to each click event.

[0139] Here, the actual duration of multimedia data refers to the total time from the start to the end of playback. The click-to-play duration of the same multimedia data is less than or equal to the actual duration. For example, if the actual duration of a multimedia file is 5 minutes and 40 seconds, the click-to-play duration could be 5 minutes and 20 seconds.

[0140] Step S0013: Based on the click playback duration and actual duration of each multimedia data, determine the target multimedia data, and obtain the search text corresponding to the target multimedia data and the target multimedia data identification information.

[0141] In implementing step S0013, candidate multimedia data with actual duration less than a duration threshold can be selected from multiple multimedia data first. Then, target multimedia data with a duration difference between actual duration and click playback duration less than a difference threshold can be selected from the candidate multimedia data. The search text corresponding to the target multimedia data and the target multimedia data identification information can be obtained.

[0142] In this embodiment, the duration threshold and the difference threshold can be preset. For example, the duration threshold can be 5 minutes or 10 minutes; the difference threshold can be 0 or 5 seconds, etc. When the difference threshold is 0, it means that the multimedia data that has been played completely should be selected from the candidate multimedia data.

[0143] Step S0014: The target multimedia data is determined as the multimedia training data, the identification information of the target multimedia data is determined as the identification information of the multimedia training data, and the search text corresponding to the target multimedia data is determined as the search text corresponding to the multimedia training data.

[0144] Step S0015: Select multiple supplementary multimedia data with actual durations less than the duration threshold from the multimedia database, and obtain the identification information of the supplementary multimedia data.

[0145] In this embodiment, in addition to determining the target multimedia data from the search log, multiple supplementary multimedia data sets with actual durations less than a duration threshold can be filtered from the multimedia database to increase the amount of training data. That is, the first training data subset also includes the supplementary multimedia data and its identification information. In some embodiments, a preset number of supplementary multimedia data sets can be filtered from the multimedia database, for example, filtering 1 million multimedia data sets.

[0146] In this embodiment, since the supplementary multimedia data is obtained from the multimedia database, and only the identification information of the supplementary multimedia data is obtained, but not the search text of the supplementary multimedia data, the supplementary multimedia data and the identification information of the supplementary multimedia data are added to the first training data subset, thereby increasing the amount of data in the first training data subset and ensuring that there is sufficient training data in the first training stage. In the second training stage, the target multimedia data and the search text corresponding to the target multimedia data are used as training data, so that the cross-modal representation model finally trained is more in line with the needs of the retrieval target task.

[0147] In some embodiments, step S002 above, "training the preset cross-modal representation model using the first training data subset to obtain a pre-trained cross-modal representation model," can be achieved through methods such as... Figure 5A Steps S0021 to S0024 shown are implemented as follows:

[0148] Step S0021: Determine the first training positive sample and the first training negative sample based on the first training data subset.

[0149] The first training positive sample includes each multimedia training data and its corresponding label information. The first training negative sample includes randomly determined multimedia training data and randomly determined label information. Taking video data as an example, if a video is about cooking braised pork and its label information is "How to make braised pork that is fatty but not greasy and lean but not dry," then the video data and the label information constitute a positive training sample. The video data and the label information "This autumn's most popular outfit" constitute a negative training sample.

[0150] Step S0022: Input each first training positive sample into the preset cross-modal representation model to obtain the predicted similarity of each first positive sample.

[0151] Here, the multimedia training data of each first training positive sample can be input into the multimedia sub-model of the preset cross-modal representation model to obtain the feature vector of the multimedia training data. Further, the multimedia sub-model can include a feature extraction layer to extract the feature vector of the multimedia data. When the multimedia data is video data, the feature vector of each sample frame in the video data is extracted to obtain a vector matrix. Then, it passes through a temporal pooling layer and a context gating layer to obtain a one-dimensional feature vector. Finally, it passes through a fully connected layer to obtain the final multimedia feature vector. The identification information of the first training sample is input into the text sub-model. First, it passes through a word vector extraction layer to obtain the word vector of the identification information. Then, it is input into a fully connected layer and a ReLU activation layer for nonlinear transformation. Then, it passes through a max pooling layer and a fully connected layer to obtain the final text feature vector. At this time, the multimedia feature vector and the text feature vector are mapped in the same vector space. At this time, the multimedia feature vector and the text feature vector can be L2 normalized and the dot product can be performed to calculate the cosine distance between the two vectors, thereby determining the semantic similarity between the multimedia data and the identification information in the first training sample.

[0152] Step S0023: Input each first training negative sample into the preset cross-modal representation model to obtain the predicted similarity of each first negative sample.

[0153] Here, similar to step S0022, the multimedia data in each of the first training negative samples is input into the multimedia sub-model of the preset cross-modal representation model to obtain multimedia feature vectors, the identification information is input into the text sub-model to obtain text feature vectors, and the semantic similarity between the multimedia data and the identification information in the first training negative samples is determined based on the multimedia feature vectors and the text feature vectors.

[0154] Step S0024: Backpropagate the predicted similarity of each first positive sample and the predicted similarity of each second negative sample to the preset cross-modal representation model, and train the preset cross-modal representation model using the first loss function to adjust the parameters of the preset cross-modal representation model.

[0155] Here, the first loss function can be the triple ranking loss function, which aims to make the distance between negative sample pairs greater than the distance between positive sample pairs. In other words, it aims to make the semantic similarity between multimedia data and labeling information in negative samples less than the semantic similarity between multimedia data and labeling information in positive samples.

[0156] Through the above steps S0021 to S0024, the construction of the first training positive sample and the first training negative sample using the first training data subset is completed. Multiple first training positive samples and first training negative samples are then input into the preset cross-modal representation model to obtain the predicted semantic similarity corresponding to the first training positive sample and the predicted semantic similarity corresponding to the first training negative sample. Then, the cross-modal representation model is trained using the preset first loss function, thereby adjusting the parameters of the cross-modal representation model to obtain a preliminarily trained cross-modal representation model.

[0157] In some embodiments, step S003 above, "training the pre-trained cross-modal representation model using the second training data subset to obtain a trained cross-modal representation model," can be achieved through... Figure 5B The following steps are implemented as shown:

[0158] Step S0031: Obtain the category label and predicted category of each multimedia data in the second training data subset.

[0159] In this embodiment of the application, the category label of each multimedia data in the second training data subset can be determined by the classification result obtained by classifying the search text corresponding to each multimedia data, and the predicted category of the multimedia data can be obtained by predicting the content of the multimedia data.

[0160] Step S0032: Determine the second training positive sample and the second training negative sample based on the second training data subset.

[0161] The second training positive sample includes each multimedia training data and its corresponding search text. The second training negative sample includes randomly determined multimedia training data and randomly determined search text. For example, if the multimedia data content is a video about Michelin chefs in China, and the corresponding search text is "Michelin chefs in China," then the multimedia data and search text constitute a second training positive sample. Conversely, the multimedia data and the randomly determined search text "children's radio gymnastics" constitute a second training negative sample.

[0162] Step S0033: Input each second training positive sample into the preset cross-modal representation model to obtain the predicted similarity of each second positive sample.

[0163] Here, the multimedia data in the second training positive sample is input into the multimedia sub-model of the preset cross-modal representation model to obtain the multimedia feature vector, and the search text is input into the text sub-model of the preset cross-modal representation model to obtain the text feature vector. Then, the prediction similarity of each second positive sample is determined based on the multimedia feature vector and the text feature vector.

[0164] Step S0034: Input each second training negative sample into the preset cross-modal representation model to obtain the predicted similarity of each second negative sample.

[0165] Similar to step S0033, in step S0034, the multimedia data in the second training negative sample is input into the multimedia sub-model of the preset cross-modal representation model to obtain the multimedia feature vector, and the search text is input into the text sub-model of the preset cross-modal representation model to obtain the text feature vector. Then, the predicted similarity of each second negative sample is determined based on the multimedia feature vector and the text feature vector.

[0166] Step S0035: Backpropagate the predicted similarity of each second positive sample, the predicted similarity of each second negative sample, each category label and the predicted category to the preset cross-modal representation model, and jointly train the preset cross-modal representation model using the first loss function and the second loss function to adjust the parameters of the preset cross-modal representation model.

[0167] Here, the first loss function can be a triplet ranking loss function, used to limit the predicted similarity of the second negative sample to be lower than that of the predicted similarity of the second positive sample. The second loss function is a classification loss function, such as the cross-entropy loss function. Due to the difference in the numerical values ​​of the loss functions, when jointly training the cross-modal representation model using the first and second loss functions, different weights can be assigned to the first and second loss functions. For example, the weight of the first loss function can be 0.9, and the weight of the second loss function can be 0.1. This constructs a multi-loss function for training, thereby using the predicted similarity of each second positive sample, the predicted similarity of each second negative sample, each class label and predicted class, and the multi-loss function to adjust the parameters of the cross-modal representation model, resulting in the final trained cross-modal representation function.

[0168] In steps S0031 to S0035, the category labels and predicted categories of each multimedia training data in the second training data subset are first obtained. Then, second training positive samples and second training negative samples are constructed. Multiple second training positive samples and second training negative samples are input into the preset cross-modal representation model to obtain the predicted similarity of the second positive samples and the predicted similarity of the second negative samples. A joint loss function is constructed based on the first loss function and the second loss function. Then, the cross-modal representation model is jointly trained using the joint loss function, multiple predicted similarities of the second positive samples, predicted similarities of the second negative samples, category labels, and predicted categories. Since category labels, predicted categories, and the second loss function (classification loss function) are introduced, the model convergence speed can be accelerated and the model training efficiency can be improved.

[0169] Since the preset cross-modal representation model includes a multimedia sub-model and a text sub-model, during the training of the cross-modal representation model, the search text in the second training data subset is clustered to obtain the labels of each multimedia training data. Correspondingly, the above step S0031 "obtaining the category labels and predicted categories of each multimedia data in the second training data subset" can be achieved through the following steps:

[0170] Step S311: Input each multimedia data into the multimedia sub-model to obtain each multimedia feature vector.

[0171] Here, in step S311, multimedia data is input into the multimedia sub-model. The multimedia data can be sampled to obtain sampled multimedia data. Then, feature vectors of each sampled multimedia data are extracted to obtain a feature matrix. Then, a temporal pooling layer is used to combine the feature matrix into a single expression, that is, to obtain a feature vector. Then, the feature vector is passed through a context gating layer, which captures its feature association information and readjusts the weights to obtain a multimedia feature vector.

[0172] Step S312: Input the search text corresponding to each multimedia data into the text sub-model to obtain the feature vector of each search text.

[0173] Here, in step S312, after inputting the search text into the text sub-model, the search text is first segmented into words, and then the word vectors of each segmented word are extracted. The multiple word vectors are then averaged to obtain a vector of a fixed length, which is the feature vector of each search text.

[0174] Step S313: Classify the multimedia feature vectors to obtain the multimedia categories.

[0175] In implementing step S313, K-Means clustering can be performed on the feature vectors of each multimedia element to obtain the various multimedia categories. For example, these could be categories such as food, education, or entertainment gossip.

[0176] Step S314: Classify the feature vectors of each search text to obtain each text category.

[0177] In implementation, K-Means clustering can be performed on the feature vectors of each search text to determine the text category corresponding to each search text.

[0178] Step S315: Determine the multimedia category corresponding to each multimedia data as the predicted category, and determine the text category of the search text corresponding to each multimedia data as the category label.

[0179] Because the search logs contain a lot of noisy data and the vocabulary coverage is not comprehensive, directly using the second training data subset extracted from the search logs to train the multimodal representation model results in slow convergence. Furthermore, the training effect depends on the construction of positive and negative sample sets; simple negative samples limit the network's generalization ability. Therefore, in steps S311 to S315, a clustering method is used, employing the cluster category of the search text corresponding to the multimedia data as the label for each multimedia data point. This accelerates the convergence speed of the multimodal representation model and improves its prediction performance without requiring additional manually labeled data.

[0180] In this embodiment of the application, the trained cross-modal representation model includes a trained multimedia sub-model and a trained text sub-model, correspondingly, Figure 4 Step S103, "Inputting the multimedia data and the text data into the trained cross-modal representation model to obtain the semantic similarity between the multimedia data and the text data," can be achieved through the following steps:

[0181] Step S1031: Input the multimedia data into the multimedia sub-model to obtain the multimedia feature vector corresponding to the multimedia data.

[0182] In implementation, multimedia data is input into the multimedia sub-model to obtain feature vectors of multimedia training data. Further, the multimedia data is first sampled to obtain sampled multimedia data. Then, the feature extraction layer in the multimedia sub-model is used to extract feature vectors of multiple sampled multimedia data to obtain a vector matrix. This feature extraction layer can be a backbone network. Then, it goes through a temporal pooling layer and a context gating layer to obtain a one-dimensional feature vector. Finally, it goes through a fully connected layer to obtain the final multimedia feature vector.

[0183] Step S1032: Input the text data into the text sub-model to obtain the text feature vector corresponding to the text data.

[0184] The text data is input into the text sub-model. First, the text data is segmented into words. Then, the word vector extraction layer is used to obtain the word vectors of each segmented word, thus obtaining the word vector matrix. When the number of segmented words is less than the preset word count threshold, zero vector padding is used. Then, the word vector matrix is ​​input into a fully connected layer and a ReLU activation layer for non-linear transformation. After passing through a max pooling layer and a fully connected layer, the final text feature vector is obtained. At this point, the multimedia feature vector and the text feature vector are mapped in the same vector space.

[0185] Step S1033: Determine the semantic similarity between the multimedia data and the text data using the multimedia feature vector and the text feature vector.

[0186] Here, the multimedia feature vector and the text feature vector can be L2 normalized, and the cosine distance between the two vectors can be calculated by dot product, thereby determining the semantic similarity between multimedia data and identifier information in the first training sample.

[0187] In some embodiments, the semantic similarity between multimedia data and text data can also be determined by calculating the Hamming distance, Euclidean distance, etc., between multimedia feature vectors and text feature vectors.

[0188] Based on the foregoing embodiments, this application further provides a method for determining cross-modal semantic similarity, applicable to... Figure 2 The network architecture shown is Figure 6 A schematic diagram illustrating another implementation flow of the multimedia data search method provided in this application embodiment is shown below. Figure 6 As shown, the process includes:

[0189] In step S601, the terminal responds to the input operation on the search input control and obtains the input search information.

[0190] Here, the search input control can receive text information entered by the user. The search input control can also switch the type of input information. The input search information can be text information or voice message.

[0191] In step S602, the terminal responds to the search operation by sending a search request to the server.

[0192] Here, the search operation can be a touch or click operation on the search control. After receiving the search operation, the terminal sends a search request to the server, which carries the search information.

[0193] Step S603: The server receives the search request and determines the search text based on the search request;

[0194] Here, the search request carries search information. The server obtains the search request sent by the terminal, parses the search request, and obtains the search information. When the search information is text information, it directly determines the search information as search text. When the search information is voice information, it performs voice recognition on the search information to obtain the search text.

[0195] In step S604, the server determines multiple multimedia data to be processed based on the search text and the title information of each multimedia data in the multimedia database.

[0196] In this implementation, the search text can be segmented into words, and then the multimedia data whose titles in the multimedia database contain these segmented words can be identified as the multimedia data to be processed.

[0197] In step S605, the server determines the searched text as text data to be processed.

[0198] Step S606: The server obtains the trained cross-modal representation model.

[0199] Here, the trained cross-modal representation model includes a trained multimedia sub-model and a trained text sub-model. This cross-modal representation model is obtained through at least two training phases, utilizing multimedia training data, the identifier information of the multimedia training data, and the search text corresponding to the multimedia training data.

[0200] In step S607, the server inputs each multimedia data to be processed into the trained multimedia sub-model to obtain the multimedia feature vectors corresponding to each multimedia data.

[0201] In step S608, the server inputs the text data to be processed into the trained text sub-model to obtain the text feature vector corresponding to the text data.

[0202] Here, the various multimedia feature vectors and the text feature vectors belong to the same vector space.

[0203] In step S609, the server uses the multimedia feature vectors and the text feature vectors to determine the semantic similarity between the multimedia data and the text data.

[0204] In step S610, the server sorts the multiple multimedia data to be processed based on their semantic similarity to obtain sorted multimedia data.

[0205] The multimedia data with the highest similarity to the search text is designated as the first multimedia data. The semantic similarity scores are then sorted from highest to lowest to obtain the sorted multimedia data.

[0206] Step S611: The server determines the search results based on the sorted multimedia data.

[0207] Here, the search results may include the URLs of the top N multimedia data and the identification information (i.e., titles) of the top N multimedia data. When the multimedia data is video data, the search results may also include the video cover.

[0208] In step S612, the server sends the search results to the terminal.

[0209] In step S613, the terminal receives and presents the search result.

[0210] Here, after receiving the search result, the terminal displays the title of the multimedia data on its own display interface. When the multimedia data is video data, the cover of the video data can also be displayed.

[0211] In the multimedia data search method provided in this application embodiment, when the terminal receives a search operation, it sends a search request carrying search information to the server. The server determines the multimedia data and text data to be processed based on the search request, and uses a trained cross-modal representation model to determine the semantic similarity between each multimedia data and text data. The search results are then determined based on the semantic similarity, which ensures that search results with higher semantic relevance to the search information can be recalled, thereby improving the accuracy of the search results.

[0212] In some embodiments, the purpose of determining the cross-modal semantic relevance of multimedia data and text data can also be to review the title of the multimedia data to be reviewed, in order to determine whether the title matches the actual content of the multimedia data. In this scenario, the server obtains a title review request for multimedia data sent by the terminal. The multimedia data review request carries the multimedia data to be reviewed and the title information of the multimedia data. The server identifies the multimedia data to be reviewed as the multimedia data to be processed, identifies the title information as the text data to be processed, and inputs the multimedia data to be processed and the text data into a trained cross-modal representation model to determine the semantic similarity between the multimedia data to be processed and the text data. Then, the server determines whether the semantic similarity between the multimedia data and the text data is greater than a preset similarity threshold. When the semantic similarity threshold is greater than the similarity threshold, the server determines that the title of the multimedia data has passed the review and sends a title review response indicating that the review has been passed to the terminal.

[0213] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0214] Video representation (Embedding) maps videos to fixed-length, low-dimensional, floating-point feature vectors, and the distance between them can measure the semantic similarity between videos. Cross-modal representation maps natural language (text) and video to the same vector space, in which the similarity between text and video can be directly compared, thus using this similarity as an important feature in the recall and ranking stages of search and recommendation.

[0215] The multimedia data search method provided in this application embodiment employs a dual-tower neural network model, the structure of which is as follows: Figure 7 As shown, it includes a video-side structure 701 and a text-side structure 702, wherein:

[0216] The video-side structure 701 includes: a feature extraction layer 7011, a temporal pooling layer 7012, a gated context layer 7013, and a linear transformation layer 7014;

[0217] The text-side structure 702 includes: a word vector model 7021, a fully connected layer (Linear Layer) + ReLU activation layer 7022, a max pooling layer 7023, and a linear transformation layer 7024.

[0218] The training and prediction process of this dual-tower model is explained below, combining video-side and text-side structures.

[0219] The input to the video-side structure 701 is multiple video frames (8 frames are sampled from each video during training). The feature extraction layer 7011 extracts features from these multiple video frames. In practice, a ResNet50 backbone network can be used to extract frame features, resulting in a vector matrix of [8, 2048]. This vector matrix is ​​then processed by a temporal pooling layer 7012 and a context gating layer 7013 to obtain a fused video feature matrix of [1, 2048]. In implementation, the temporal pooling layer can be a NextVLAD fusion layer. In the NextVLAD layer, the feature maps (i.e., the aforementioned vector matrix) from the multiple video input frames are combined into a single representation using learnable parameters, resulting in a feature vector of [1, 2048]. This feature vector is then input to the Context Gating layer, which captures the feature association information and readjusts the weights to obtain the final video feature representation.

[0220] In this embodiment, a preprocessing module 703 is included before the text-side structure. This preprocessing module 703 is used to segment the input retrieval text into words. The input to the text-side structure 702 is the segmented retrieval words. A word vector list, i.e., a word vector matrix of [8, 64], is obtained using a pre-trained word embedding model 7021. For sentences with fewer than eight words, zero vectors are used for padding. Subsequently, the input is fed into a module 7022 consisting of two fully connected layers (Linear Layer) and a ReLU activation layer for nonlinear transformation. Then, a MaxPooling layer 7023 aggregates the vectors to obtain a text feature vector of [1, 256].

[0221] After obtaining video and text features, the dual-tower model inputs them into two fully connected layers respectively. Specifically, the video features are input into fully connected layer 7014, and the text features are input into fully connected layer 7024. In this way, the features of the two modalities are mapped into a vector space of the same dimension. The resulting vectors are then normalized, and the cosine distance is calculated by dot product in this vector space to measure the similarity between the two inputs.

[0222] While language models can utilize large amounts of natural language data for training, cross-modal models of visual language require high-quality image and video descriptions, and the quality and size of the data are crucial for model training. In this embodiment, video titles are used as pre-training data, and data constructed from user search click logs is used as data for the second-stage training to train the cross-modal representation model.

[0223] Assume that the video platform's search logs averaged approximately 65 million queries and 100 million clicks per day during the sampling period, with 20 million clicks resulting in complete video playback. In training, the experimental data used consisted of search click pairs from a two-week period where the video was played in its entirety, the video length was less than 10 minutes, and the frequency was at least 2. This yielded approximately 11 million search query-video pairs, corresponding to 6.9 million different videos with an average video length of 117 seconds. The pre-training data included all videos corresponding to the selected search data, plus 10 million additional videos with a duration of less than 10 minutes sampled from the video library, along with their corresponding title text information, as multimodal training data. 5% of the search log dataset was allocated as a validation set.

[0224] The training objective of cross-modal representation models is to make the distance between the vectors of matched text and video closer in the space. Therefore, triplet ranking loss is typically used as the training loss function. In this embodiment, the video and its title are first used as pre-training data for the first stage, and the video and its search query are used as training data for the second stage. The dual-tower structure model is trained through these two stages.

[0225] While retrieval log data better matches the requirements of the target retrieval task, it contains a lot of noisy data and its vocabulary coverage is not comprehensive enough, making direct training difficult to converge. In the first training phase, a video and its corresponding title are considered a positive sample pair, while other videos and titles within the same random training batch are considered negative samples. The triplet ranking loss function is used as the training objective. In the second training phase, when using retrieval logs as training data, the user's search query and its corresponding clicked and fully played video are considered a positive sample pair, while other videos and titles within the same random training batch are considered negative samples.

[0226] Because training using the triplet ranking loss function directly is slow and its effectiveness depends on the construction of positive and negative sample sets, simple negative samples limit the network's generalization ability. In the second training stage, the clustering category of the search text corresponding to the video is used as the pseudo-label for each video. This is added to the classification loss function and trained together with the triplet ranking loss function, labeled as a multi-loss model, which greatly increases the stability of training convergence. In implementation, the text input (i.e., the search text) is segmented and processed by a pre-trained word vector model to obtain multiple word vectors. These multiple word vectors are then averaged and pooled to obtain a fixed-length vector, which serves as the initial feature of the search text. This initial feature of the search text is then subjected to k-means clustering, with 6400 clusters used empirically. The clustering category ID corresponding to each search text is used as the video's classification label, and the cross-entropy loss function is used to calculate the classification loss. Due to the difference in the numerical values ​​of the loss functions, the classification loss function and the triplet ranking loss function are added together with a weight of 1:10, which is the multi-loss function used for training.

[0227] When using a trained dual-tower model to predict the similarity between a video and input text, the trained dual-tower model can obtain the features of the two modal inputs in a vector space. The cosine distance is calculated by dot product in this vector space, thereby predicting the similarity between the two inputs.

[0228] Table 1 presents a comparison of the experimental results. In Table 1, R@k represents the percentage of the answer video ranked among the top k most relevant videos retrieved on the validation set, i.e., R@5 represents the proportion of the answer video actually clicked in the top 5 videos of the retrieved video list sorted by relevance. medR represents the median ranking of the answer video in the retrieved results, and meanR represents the average ranking of the answer video in the retrieved results. Both multiple loss functions and multi-stage training effectively improved the retrieval performance.

[0229] Table 1

[0230]

[0231] In practice, different basic network structures, different video recognition, language models, and different distance metric functions can be used to achieve similar cross-modal representation models.

[0232] In this embodiment of the application, without the need to collect additional manually labeled data, an effective representation model that maps videos and text to the same space can be trained. Using this cross-modal representation model, a connection between language and vision can be established in the search, more relevant videos can be recalled, and the relevance between the search results and the content itself can be improved.

[0233] Meanwhile, the multi-loss function and multi-stage training method that introduce cluster pseudo-labels proposed in the embodiments of this application effectively improve the stability of the training process and enhance the model performance.

[0234] The following description continues to illustrate the exemplary structure of the multimedia data search device 454 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 3 As shown, the software module in the multimedia data search device 454 stored in the memory 440 may include:

[0235] The first acquisition module is used to acquire a trained cross-modal representation model, wherein the training dataset of the preset cross-modal representation model includes multimedia training data, the identification information of the multimedia training data, and the search text corresponding to the multimedia training data.

[0236] The first acquisition module is used to acquire multimedia data and text data to be processed;

[0237] The similarity calculation module is used to input the multimedia data and the text data into the trained cross-modal representation model to obtain the semantic similarity between the multimedia data and the text data.

[0238] In some embodiments, the device further includes:

[0239] The second acquisition module is used to acquire a preset cross-modal representation model and acquire the training dataset of the preset cross-modal representation model, wherein the training dataset includes a first training data subset and a second training data subset.

[0240] The first training data subset includes multimedia training data and the identification information of the multimedia training data; the second training data subset includes the multimedia training data and the search text corresponding to the multimedia training data.

[0241] The first training module is used to train the preset cross-modal representation model using the first training data subset to obtain a pre-trained cross-modal representation model.

[0242] The second training module is used to train the pre-trained cross-modal representation model using the second training data subset to obtain the trained cross-modal representation model.

[0243] In some embodiments, the second acquisition module is further configured to:

[0244] Obtain search logs within a preset time period, the search logs including multiple search texts and click event information corresponding to the multiple search texts;

[0245] Obtain the click playback duration and actual duration of the multimedia data corresponding to each click event;

[0246] Based on the click playback duration and actual duration of each multimedia data, the target multimedia data is determined, and the search text corresponding to the target multimedia data and the target multimedia data identification information are obtained.

[0247] The target multimedia data is determined as the multimedia training data, the identification information of the target multimedia data is determined as the identification information of the multimedia training data, and the search text corresponding to the target multimedia data is determined as the search text corresponding to the multimedia training data.

[0248] Multiple supplementary multimedia data with actual durations less than a duration threshold are selected from the multimedia database, and the identification information of the supplementary multimedia data is obtained; the first training data subset also includes the supplementary multimedia data and the identification information of the supplementary multimedia data.

[0249] In some embodiments, the first training module is further configured to:

[0250] First training positive samples and first training negative samples are determined based on the first training data subset, wherein the first training positive samples include each multimedia data and the corresponding identification information of each multimedia data, and the first training negative samples include randomly determined multimedia data and randomly determined identification information.

[0251] Each first training positive sample is input into the preset cross-modal representation model to obtain the predicted similarity of each first positive sample;

[0252] Each first training negative sample is input into the preset cross-modal representation model to obtain the predicted similarity of each first negative sample;

[0253] The predicted similarity of each first positive sample and the predicted similarity of each second negative sample are backpropagated to the preset cross-modal representation model. The preset cross-modal representation model is trained using the first loss function to adjust the parameters of the preset cross-modal representation model.

[0254] In some embodiments, the second training module is further configured to:

[0255] Obtain the category labels and predicted categories of each multimedia data in the second training data subset;

[0256] The second training positive sample and the second training negative sample are determined based on the second training data subset. The second training positive sample includes each multimedia data and the search text corresponding to each multimedia data. The second training negative sample includes randomly determined multimedia data and randomly determined search text.

[0257] Each second training positive sample is input into the preset cross-modal representation model to obtain the predicted similarity of each second positive sample;

[0258] Each second training negative sample is input into the preset cross-modal representation model to obtain the predicted similarity of each second negative sample;

[0259] The predicted similarity of each second positive sample, the predicted similarity of each second negative sample, each category label, and the predicted category are backpropagated to the preset cross-modal representation model. The preset cross-modal representation model is jointly trained using the first loss function and the second loss function to adjust the parameters of the preset cross-modal representation model.

[0260] In some embodiments, the preset cross-modal representation model includes a multimedia sub-model and a text sub-model, and the second training module is further configured to:

[0261] Each multimedia data is input into the multimedia sub-model to obtain the feature vector of each multimedia.

[0262] The search text corresponding to each multimedia data is input into the text sub-model to obtain the feature vector of each search text.

[0263] Clustering is performed on the multimedia feature vectors to obtain the multimedia categories;

[0264] The feature vectors of each search text are classified to obtain each text category;

[0265] The multimedia category corresponding to each multimedia data is determined as the predicted category, and the text category of the search text corresponding to each multimedia data is determined as the category label.

[0266] In some embodiments, the trained cross-modal representation model includes a trained multimedia sub-model and a trained text sub-model. Correspondingly, the similarity calculation module is further used for:

[0267] The multimedia data is input into the multimedia sub-model to obtain the multimedia feature vector corresponding to the multimedia data;

[0268] The text data is input into the text sub-model to obtain the text feature vector corresponding to the text data, wherein the multimedia feature vector and the text feature vector belong to the same vector space;

[0269] The semantic similarity between the multimedia data and the text data is determined using the multimedia feature vector and the text feature vector.

[0270] In some embodiments, the first determining module is further configured to:

[0271] Obtain the search request sent by the terminal, and determine the search text based on the search request;

[0272] Based on the search text and the title information of each multimedia data in the multimedia database, the multimedia data to be processed is determined.

[0273] The search text is identified as the text data to be processed.

[0274] In some embodiments, the first determining module is further configured to:

[0275] Obtain the search information carried in the search request;

[0276] When the search information is text information, the search information is determined as the search text;

[0277] When the search information is voice information, the search information is subjected to voice recognition to obtain the search text.

[0278] In some embodiments, the second determining module is further configured to:

[0279] Multiple multimedia data to be processed are sorted based on semantic similarity to obtain sorted multimedia data. Among them, the multimedia data with the highest similarity to the search text is the first multimedia data.

[0280] Determine search results based on the sorted multimedia data;

[0281] The search results are returned to the terminal that sent the search request.

[0282] It should be noted that the description of the above-described multimedia data search device embodiments is similar to the method description above, and has the same beneficial effects as the method embodiments. For technical details not disclosed in the multimedia data search device embodiments of this application, those skilled in the art should refer to the description of the method embodiments of this application for understanding.

[0283] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in this application.

[0284] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to perform the method provided in this application, for example, as shown in the figure.Figure 4 , Figure 5A , Figure 5B and Figure 6 The method shown.

[0285] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0286] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0287] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0288] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0289] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A search method of multimedia data, characterized by, The method comprises the following steps: Based on the obtained search request, determine the multimedia data and text data to be processed; Obtain a preset cross-modal representation model and a training data set of the preset cross-modal representation model, wherein the training data set comprises a first training data subset and a second training data subset; The first training data subset comprises multimedia training data and identification information of the multimedia training data, and the second training data subset comprises the multimedia training data and search text corresponding to the multimedia training data; The first training data subset is used to train the preset cross-modal representation model to obtain a preliminarily trained cross-modal representation model; The second training data subset is used to train the preliminarily trained cross-modal representation model to obtain the trained cross-modal representation model; The multimedia data and the text data are input into the trained cross-modal representation model to obtain the semantic similarity of the multimedia data and the text data; Based on the semantic similarity and the multimedia data, determine and output the search result.

2. The method of claim 1, wherein, The method comprises the following steps: Obtain search logs within a preset time period, wherein the search logs comprise a plurality of search texts and a plurality of search text corresponding click event information; Obtain the click play duration and actual duration of the multimedia data corresponding to each click event information; Based on the click play duration and actual duration of each multimedia data, determine target multimedia data and obtain the search text corresponding to the target multimedia data and the identification information of the target multimedia data; Determine the target multimedia data as the multimedia training data, determine the identification information of the target multimedia data as the identification information of the multimedia training data, and determine the search text corresponding to the target multimedia data as the search text corresponding to the multimedia training data; From the multimedia database, filter a plurality of supplementary multimedia data with an actual duration less than a duration threshold, and obtain the identification information of the supplementary multimedia data; the first training data subset further comprises the supplementary multimedia data and the identification information of the supplementary multimedia data.

3. The method of claim 2, wherein, The method comprises the following steps: Based on the first training data subset, determine first training positive samples and first training negative samples, wherein the first training positive samples comprise each multimedia data and the identification information corresponding to each multimedia data, and the first training negative samples comprise randomly determined multimedia data and randomly determined identification information; Input each first training positive sample into the preset cross-modal representation model to obtain a first positive sample predicted similarity; Input each first training negative sample into the preset cross-modal representation model to obtain a first negative sample predicted similarity; The first positive sample prediction similarity and the second negative sample prediction similarity are back propagated to the preset cross-modal feature model, and the preset cross-modal feature model is trained by using a first loss function to adjust parameters of the preset cross-modal feature model.

4. The method of claim 1, wherein, The training of the preliminary trained cross-modal feature model by using the second training data subset to obtain a trained cross-modal feature model comprises: obtaining a category label and a prediction category of each multimedia data in the second training data subset; determining second training positive samples and second training negative samples based on the second training data subset, wherein the second training positive samples comprise each multimedia data and a search text corresponding to each multimedia data, and the second training negative samples comprise randomly determined multimedia data and randomly determined search texts; inputting each second training positive sample into the preset cross-modal feature model to obtain a second positive sample prediction similarity; inputting each second training negative sample into the preset cross-modal feature model to obtain a second negative sample prediction similarity; back propagating the second positive sample prediction similarity, the second negative sample prediction similarity, the category label and the prediction category to the preset cross-modal feature model, and jointly training the preset cross-modal feature model by using a first loss function and a second loss function to adjust parameters of the preset cross-modal feature model.

5. The method of claim 4, wherein, The preset cross-modal feature model comprises a multimedia sub-model and a text sub-model, and the obtaining of the category label and the prediction category of each multimedia data in the second training data subset comprises: inputting each multimedia data into the multimedia sub-model to obtain a multimedia feature vector; inputting a search text corresponding to each multimedia data into the text sub-model to obtain a search text feature vector; performing classification processing on the multimedia feature vector to obtain a multimedia category; performing classification processing on the search text feature vector to obtain a text category; determining the multimedia category corresponding to each multimedia data as the prediction category, and determining the text category of the search text corresponding to each multimedia data as the category label.

6. The method as claimed in claim 1, wherein, The trained cross-modal feature model comprises a trained multimedia sub-model and a trained text sub-model, and correspondingly, the inputting of the multimedia data and the text data into the trained cross-modal feature model to obtain a semantic similarity of the multimedia data and the text data comprises: inputting the multimedia data into the multimedia sub-model to obtain a multimedia feature vector corresponding to the multimedia data; inputting the text data into the text sub-model to obtain a text feature vector corresponding to the text data, wherein the multimedia feature vector and the text feature vector belong to a same vector space; determining the semantic similarity of the multimedia data and the text data by using the multimedia feature vector and the text feature vector.

7. The method according to any one of claims 1 to 6, characterized in that, The determination of each multimedia data and text data to be processed based on the obtained search request comprises: acquire a search request sent by a terminal, and determine a search text based on the search request; determine multimedia data to be processed based on the search text and title information of each multimedia data in a multimedia database; determine the search text as text data to be processed.

8. The method of claim 7, wherein, The determination of the search text based on the search request comprises: acquiring search information carried in the search request; when the search information is text information, determining the search information as the search text; when the search information is voice information, performing voice recognition on the search information to obtain the search text.

9. The method according to any one of claims 1 to 6, characterized in that, The determination and output of the search result based on the semantic similarity and the multimedia data comprises: sorting the multimedia data to be processed based on the semantic similarity, to obtain sorted multimedia data, wherein the multimedia data with the highest similarity to the search text is the first multimedia data; determining a search result based on the sorted multimedia data; returning the search result to the terminal sending the search request.

10. A search apparatus for multimedia data, characterized by comprising: comprise: a first determination module configured to determine multimedia data and text data to be processed based on the acquired search request; a second acquisition module configured to acquire a preset cross-modal representation model and a training data set of the preset cross-modal representation model, wherein the training data set comprises a first training data subset and a second training data subset; the first training data subset comprises multimedia training data and identification information of the multimedia training data, and the second training data subset comprises the multimedia training data and search text corresponding to the multimedia training data; a first training module configured to train the preset cross-modal representation model using the first training data subset to obtain a preliminarily trained cross-modal representation model; a second training module configured to train the preliminarily trained cross-modal representation model using the second training data subset to obtain the trained cross-modal representation model; a similarity calculation module configured to input the multimedia data and the text data into the trained cross-modal representation model to obtain a semantic similarity of the multimedia data and the text data; a second determination module configured to determine and output a search result based on the semantic similarity and each multimedia data.

11. The apparatus according to claim 10, wherein the second acquisition module is further configured to acquire search logs within a preset time length, the search logs comprising a plurality of search texts and a plurality of click event information corresponding to the search texts; acquire a click play duration and an actual duration of multimedia data corresponding to each click event information; determine target multimedia data based on the click play duration and the actual duration of each multimedia data, and acquire search text corresponding to the target multimedia data and identification information of the target multimedia data; determine the target multimedia data as the multimedia training data, determine the identification information of the target multimedia data as the identification information of the multimedia training data, and determine the search text corresponding to the target multimedia data as the search text corresponding to the multimedia training data. Filtering a plurality of supplementary multimedia data with an actual duration less than the duration threshold from the multimedia database, and obtaining identification information of the supplementary multimedia data; the first training data subset further includes the supplementary multimedia data and the identification information of the supplementary multimedia data.

12. The apparatus of claim 11, wherein, The first training module is further configured to determine first training positive samples and first training negative samples based on the first training data subset, wherein the first training positive samples include each multimedia data and identification information corresponding to the multimedia data, and the first training negative samples include randomly determined multimedia data and randomly determined identification information; input each first training positive sample into the preset cross-modal representation model to obtain a first positive sample predicted similarity; and input each first training negative sample into the preset cross-modal representation model to obtain a first negative sample predicted similarity. The first positive sample predicted similarity and the second negative sample predicted similarity are back propagated to the preset cross-modal representation model, and the preset cross-modal representation model is trained by using a first loss function to adjust parameters of the preset cross-modal representation model.

13. A search apparatus for multimedia data, characterized by comprising: comprising: a memory configured to store executable instructions; a processor configured to execute the executable instructions stored in the memory to implement the method of any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that, executable instructions stored therein, and configured to, when executed by a processor, implement the method of any one of claims 1 to 9.

15. A computer program product, characterised in that, computer instructions stored therein, and configured to, when executed by a processor, implement the method of any one of claims 1 to 9.

Images