Multimedia processing method, device and equipment

Abstract

The embodiment of the present application discloses a multimedia processing method, device and equipment, and the multimedia processing method relates to the field of computer technology. The method includes: obtaining a multimedia file set, using a multimedia file tag prediction model to predict the file content of each multimedia file respectively, and obtaining a candidate word set for each multimedia file. When an intention retrieval request carrying a target search word is received, a search prompt word matching the target search word is searched in the candidate word set of N multimedia files, and the search prompt word is output. It can be seen that by pre-constructing a candidate word set for a multimedia file set, the coverage of the candidate words can be expanded; based on the pre-constructed candidate word set, the user is prompted, and the user can directly select the prompt content for search without entering the complete video search information, which can simplify the video search process and thus improve the efficiency of video search.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multimedia processing method, device and equipment. Background Art

[0002] With the development of computer technology, the number of videos included in video platforms is increasing. When users search for videos they want on video platforms, they mainly search for videos through video search information (such as video titles). The search prompt function can assist users in entering video search information. In practice, it is found that the current search prompt function is mainly based on the user's historical search word records. Once the number of historical search word records of the user is small, the user needs to enter the complete video search information (such as video title), and the user's retrieval cost is high. Summary of the invention

[0003] The embodiments of the present invention provide a multimedia processing method, apparatus and device, which can simplify the video search process and improve the video search efficiency.

[0004] On the one hand, an embodiment of the present application provides a multimedia processing method, including:

[0005] Acquire a multimedia file set, where the multimedia file set includes N multimedia files, where N is a positive integer;

[0006] The multimedia file tag prediction model is used to predict the file content of each multimedia file to obtain a set of candidate words for each multimedia file;

[0007] When receiving an intention search request, searching for a search prompt word matching a target search word in a candidate word set of N multimedia files, wherein the intention search request carries the target search word;

[0008] The search hint word is outputted.

[0009] On the one hand, an embodiment of the present application provides a multimedia processing device, the multimedia processing device comprising:

[0010] An acquisition unit, configured to acquire a multimedia file set, wherein the multimedia file set includes N multimedia files, where N is a positive integer;

[0011] A processing unit is used to use a multimedia file tag prediction model to predict the file content of each multimedia file respectively to obtain a set of candidate words for each multimedia file; and when receiving an intention retrieval request, search for a search prompt word matching a target search word in the candidate word sets of N multimedia files, wherein the intention retrieval request carries the target search word; and output the search prompt word.

[0012] In one aspect, the present application provides a multimedia processing device, the device comprising:

[0013] A processor for loading and executing computer programs;

[0014] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned multimedia processing method is implemented.

[0015] On the one hand, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for being loaded by a processor and executing the above-mentioned multimedia processing method.

[0016] On the one hand, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned multimedia processing method.

[0017] In the embodiment of the present application, a multimedia file set is obtained, and the file content of each multimedia file is predicted and processed respectively using a multimedia file tag prediction model to obtain a candidate word set for each multimedia file. When an intended search request carrying a target search word is received, a search prompt word matching the target search word is searched in the candidate word set of N multimedia files, and the search prompt word is output. It can be seen that by pre-constructing a candidate word set of a multimedia file set, the coverage of the candidate words can be expanded so that the candidate word set can cover all multimedia files. When a user intends to retrieve a multimedia file, the user is prompted based on the pre-constructed candidate word set, and the user can directly select the prompt content for search without having to enter the complete video search information, which can simplify the video search process and thus improve the efficiency of video search. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1a A scene architecture diagram of multimedia processing provided in an embodiment of the present application;

[0020] Figure 1b A main flow chart of multimedia processing provided by an embodiment of the present application;

[0021] Figure 2 A flowchart of a multimedia processing method provided in an embodiment of the present application;

[0022] Figure 3a A schematic diagram of a process for generating a candidate word set provided in an embodiment of the present application;

[0023] Figure 3b A flowchart of finding a search prompt word that matches a target search word provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of another multimedia processing method provided in an embodiment of the present application;

[0025] Figure 5a An architecture diagram of a multimedia file tag prediction model provided in an embodiment of the present application;

[0026] Figure 5b An architecture diagram of another multimedia file tag prediction model provided in an embodiment of the present application;

[0027] Figure 6 A flowchart of another multimedia processing method provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of the structure of a multimedia processing device provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of a multimedia processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] The embodiments of the present application relate to artificial intelligence (AI) and machine learning (ML). Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0032] AI technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, processing technologies for large applications, operating / interactive systems, and mechatronics. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. Among them, computer vision technology (Computer Vision, CV) is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further performing graphics processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, trying to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The embodiments of this application mainly involve extracting features of image modality data of multimedia files through image recognition technology in computer vision technology, and recognizing subtitle text in images through OCR.

[0033] ML is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. ML is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. ML and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The embodiments of the present application mainly involve optimizing and training the initial encoding and decoding model using corpus sample data to obtain an encoding and decoding model, and optimizing and training the initial multimedia file label prediction model using a training data set to obtain a multimedia file label prediction model.

[0034] The key technologies of speech technology include automatic speech recognition technology (ASR), text to speech (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech becomes one of the most promising human-computer interaction methods in the future. The embodiments of this application mainly involve extracting the features of audio modal data in multimedia files through speech technology; and converting audio data into corresponding text data through ASR.

[0035] In addition, the present application also relates to natural language processing (NLP). NLP is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question and answer, knowledge graph and other technologies. The embodiments of the present application mainly relate to text features extracted from the data of the text modality of multimedia files by NLP, and the splitting and processing of sentences by NIP.

[0036] See also Figure 1a , Figure 1a This is a scene architecture diagram of a multimedia processing provided by an embodiment of the present application. Figure 1a As shown, the scenario architecture diagram includes a terminal device 101 and a server 102. The terminal device 101 is a device used by a user, and the terminal device 101 may include but is not limited to: a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a portable personal computer, a mobile Internet device (Mobile Internet Devices, MID) and other devices; the terminal device is often equipped with a display device, which may also be a display, a display screen, a touch screen, etc., and the touch screen may also be a touch screen, a touch panel, etc., which is not limited in the embodiment of the present invention.

[0037] Server 102 refers to a background device that can provide video (search) services for terminal device 101; in one embodiment, server 102 can be a background server of a video client in terminal device 101. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired communication or wireless communication, and this application does not limit this.

[0038] It should be noted that Figure 1a The number of terminal devices and servers in the multimedia processing scenario shown is only an example. For example, the number of terminal devices and servers can be multiple, and this application does not limit the number of terminal devices and servers.

[0039] Figure 1b The main flow chart of multimedia processing provided by the embodiment of the present application is as follows. Figure 1bAs shown, the multimedia processing flow mainly includes the following steps: (1) The server 102 pre-constructs candidate words for multimedia files in the multimedia file library: First, a multimedia file set is obtained from the video library, which can be stored in the server 102 or exist independently in the database; the multimedia file set includes N multimedia files, where N is a positive integer; each multimedia file (such as a short video uploaded by a user, a film and television work released by a film and television company, etc.) includes data in multiple modes (media forms) such as text, sound and image. Then, the server 102 uses a multimedia file tag prediction model to predict the file content of each multimedia file respectively to obtain a candidate word set for each multimedia file; the file content of the multimedia file includes one or more of the following: image modality data (such as video frames), audio modality data (such as audio stream data), text modality data (such as subtitles, titles, etc.); it should be noted that the text modality data can be carried in the file content of the multimedia file, or can be extracted from the image modality data or audio modality data; for example, converting the audio stream data into corresponding text, extracting the subtitles in the video frame, etc. The multimedia file tag prediction model includes a multimedia key information extraction model and a multimedia candidate word generation model; wherein the multimedia key information extraction model (including at least one of a keyword extraction model and a coding model) is used to generate a first candidate word subset according to the data of the text mode of the multimedia file; the multimedia candidate word generation model is used to generate a second candidate word subset according to the data of P modes of the multimedia file, and the candidate word set is obtained by combining the first candidate word subset and the second candidate word subset. (2) When the terminal device 101 detects the user's intention to search, it sends an intention search request carrying the target search term to the server 102. (3) The server 102 determines the candidate search prompt word based on the target search word input by the user: search for the search prompt word matching the target search word in the candidate word set of N multimedia files (such as by prefix query, fuzzy query, etc.); the server 102 obtains the characteristic information of the current user, and determines the weight of each candidate search prompt word in the candidate search prompt word set according to the characteristic information, wherein the characteristic information includes at least one of the following: user annotation information, user historical behavior information; sorts the candidate search prompt words in the candidate search prompt word set in descending order of weight, and determines the search prompt word according to the sorting result. (4) The server 102 returns the search prompt word corresponding to the target search word to the terminal device 101, so that the terminal device 101 outputs (such as displays on the screen) the search prompt word.

[0040] In the embodiment of the present application, the server obtains a multimedia file set, and uses a multimedia file tag prediction model to predict the file content of each multimedia file, and obtains a candidate word set for each multimedia file. When receiving an intention search request carrying a target search word, the server searches for a search prompt word that matches the target search word in the candidate word set of N multimedia files, and outputs the search prompt word. It can be seen that by pre-constructing a candidate word set for a multimedia file set, the coverage of the candidate words can be expanded, so that the candidate word set can cover all multimedia files. When the user intends to retrieve a multimedia file, the user is prompted based on the pre-constructed candidate word set. The user can directly select the prompt content for search, without having to enter the complete video search information, which can simplify the video search process and thus improve the efficiency of video search.

[0041] The multimedia processing solution provided in the embodiments of the present application is described in detail below in conjunction with the accompanying drawings.

[0042] See also Figure 2 , Figure 2 A flowchart of a multimedia processing method provided in an embodiment of the present application. The multimedia processing solution can be executed by the server 102 described above; the solution includes steps S201-S204, wherein:

[0043] S201: The server obtains a multimedia file set.

[0044] The multimedia file set includes N multimedia files, where N is a positive integer; the multimedia file set can be obtained by the server from the platform video library, or uploaded by the user in a certain period of time; for example, multimedia file set 1 is composed of videos uploaded by users received by the video platform within 1 minute.

[0045] S202: The server uses a multimedia file tag prediction model to predict the file content of each multimedia file to obtain a candidate word set for each multimedia file.

[0046] The file content of a multimedia file includes data in at least two modes among text mode, image mode and audio mode; for example, a multimedia file can be a short video uploaded by a user, which includes a title (i.e., data in the text mode of the multimedia file), a video frame (i.e., data in the image mode of the multimedia file) and a dubbing (i.e., data in the audio mode of the multimedia file). It should be noted that the data in the text mode of a multimedia file can be carried in the file content of the multimedia file (such as a title, content overview, etc.), or can be extracted by a server from the data in the image mode or the data in the audio mode through a text recognition model; for example, the audio stream data can be converted into the corresponding text mode data through ASR, and subtitles in the video frame can be extracted.

[0047] The multimedia file label prediction model is used to extract the features of each multimedia file in the multimedia file set, and predict the candidate word set of the pair of media files based on the features of each multimedia file. Specifically, the multimedia file label prediction model extracts the features of the data of P modes of the target multimedia file (any multimedia file in the multimedia file set) through P feature extraction modules to obtain P sub-features of the target multimedia file, where P is an integer greater than 1; the P sub-features of the target multimedia file are fused to obtain the fused features of the target multimedia file, and prediction is performed based on the fused features to obtain the candidate word set of the target multimedia file. For example, assuming that the file content of multimedia file 1 includes data of text mode, data of sound mode and data of image mode; the multimedia file label prediction model extracts the features of the data of text mode to obtain the text features of multimedia file 1; similarly, the features of the data of sound mode are extracted to obtain the audio features of multimedia file 1; the features of the data of image mode are extracted to obtain the image features of multimedia file 1; the text features, audio features and image features are fused to obtain the fused features of multimedia file 1, and prediction is performed based on the fused features to obtain the candidate word set of multimedia file 1.

[0048] It can be understood that the candidate word set of the target multimedia file is obtained based on the feature prediction of the target multimedia file in P modal data, which is more comprehensive and accurate than the candidate word set predicted based on the feature prediction of data in a single modality.

[0049] The candidate word set of the target multimedia file includes at least one candidate word, which is predicted by the multimedia file tag prediction model based on the file content of the target multimedia file; for example, the candidate words are the main events, main characters, content summary, etc. of the video.

[0050] Figure 3a A schematic diagram of a process for generating a candidate word set provided in an embodiment of the present application. Figure 3a As shown, the multimedia file tag prediction model includes a multimedia key information extraction model and a multimedia alternative word generation model; wherein the multimedia key information extraction model (including at least one of a keyword extraction model and a coding and decoding model) is used to generate a first alternative word subset based on the data of the text mode of the multimedia file; the multimedia alternative word generation model is used to generate a second alternative word subset based on the data of P modes of the multimedia file, and obtain an alternative word set by combining the first alternative word subset and the second alternative word subset.

[0051] S203: When receiving the intended search request, the server searches for a search prompt word that matches the target search word in the candidate word set of N multimedia files.

[0052] The intention retrieval request is sent to the server after the terminal device obtains the target search term input by the user, and the intention retrieval request carries the target search term. Search prompt words that match the target search term include search prompt words that match the target search term in form (for example, the search prompt word that matches "new crown" is "new coronavirus"); and search prompt words that are semantically associated with the target search term (for example, the search prompt word that matches "new crown" is "pneumonia"). In a specific implementation, the server can search for search prompt words that match the target search term through one or more methods of index table, prefix query and fuzzy query.

[0053] Figure 3b A flowchart of searching for search prompt words that match a target search word is provided in an embodiment of the present application. Figure 3b As shown, S301: when the terminal device detects that the user intends to search for multimedia files, it obtains the target search term input by the user and sends an intention search request to the server, in which the target search term is carried; S302: after receiving the intention search request, the server determines the search prompts to be selected based on the target search term input by the user; S303: outputs the sorted search prompt terms based on the user's feature information and the target search term information.

[0054] In one embodiment, the server obtains a search vocabulary set (the search vocabulary set can be constructed based on multimedia-related vocabulary or historical search terms), and constructs a prompt index based on the association between each search vocabulary in the search vocabulary set and all the alternative terms in the N alternative word sets; specifically, the server constructs a prompt index for each search vocabulary based on the association between the vocabulary and all the alternative terms in the N alternative word sets, and determines the set of search prompt terms to be selected based on the target search terms and the prompt index. Alternative terms that have an association relationship with the search vocabulary include alternative terms that have an association relationship with the search vocabulary in form (for example, the alternative terms that have an association relationship with "new crown" include "new crown prevention"); and alternative terms that have an association relationship with the search vocabulary in semantics (for example, the search prompt terms that match "new crown" include "pneumonia"); and alternative terms that have an indirect relationship with the search vocabulary (for example, the alternative terms that have an association relationship with "new crown" include "new crown prevention", "new crown prevention" and "epidemic" are obtained by splitting the same sentence, so "new crown" and "epidemic" also have an association relationship). Table 1 is a prompt index table provided in an embodiment of the present application:

[0055] Table 1

[0056]

[0057] As can be seen from Table 1, when the target search term is "new crown", the server determines the set of candidate search prompt terms based on "new crown" and Table 1 as: new coronavirus, pneumonia, the number of new crown cases today, new crown prevention, epidemic situation...

[0058] Furthermore, the server obtains feature information of the current user, and determines the weights of each candidate search prompt word in the candidate search prompt word set based on the feature information, wherein the feature information includes at least one of the following: user annotation information (used to annotate the user's features) and user historical behavior information; for example, the user annotation information indicates that the feature of the current user is "student", assuming that the order of students' attention is: new crown prevention>epidemic>pneumonia>new coronavirus, and the order of doctors' attention is: new coronavirus>epidemic>pneumonia>new crown prevention; then, according to the order of students' attention, the weight of "new crown prevention" is set to 0.9, the weight of "epidemic" is set to 0.7, the weight of "pneumonia" is set to 0.6, and the weight of "new coronavirus" is set to 0.5; for another example, assuming that the user has historically searched for "epidemic" 8 times, "new crown prevention" 10 times, "new coronavirus" 2 times, and "pneumonia" 0 times, then according to the user's historical behavior information, the weight of "new crown prevention" is set to 0.8, the weight of "epidemic" is set to 0.7, the weight of "pneumonia" is set to 0.2, and the weight of "new coronavirus" is set to 0.3. The server sorts the candidate search prompt words in the candidate search prompt word set in descending order of weight, and determines the search prompt word according to the sorting result (e.g., determining the first Q candidate search prompt words as search prompt words in order, where Q is a positive integer; or determining the candidate search prompt words whose weights are greater than a weight threshold as search prompt words in order).

[0059] Optionally, the server may also determine the weight of the candidate search prompt term based on the degree of correlation between the candidate search prompt term and the target search term; for example, the target search term is "new crown prevention", the candidate search prompt term 1 is "new coronavirus prevention", and the candidate search prompt term 2 is "pneumonia prevention"; since "new crown prevention" is included in "new coronavirus prevention", the degree of correlation between "new coronavirus prevention" and "new crown prevention" is higher than the degree of correlation between "pneumonia prevention" and "new crown prevention".

[0060] S204: The server outputs the search prompt word.

[0061] In one implementation, the server sends a search prompt word matching the target search word to the terminal device.

[0062] In the embodiment of the present application, the server obtains a multimedia file set, and uses a multimedia file tag prediction model to predict the file content of each multimedia file, and obtains a candidate word set for each multimedia file. When receiving an intention search request carrying a target search word, the server searches for a search prompt word that matches the target search word in the candidate word set of N multimedia files, and outputs the search prompt word. It can be seen that by pre-constructing a candidate word set for a multimedia file set, the coverage of the candidate words can be expanded, so that the candidate word set can cover all multimedia files. When the user intends to retrieve a multimedia file, the user is prompted based on the pre-constructed candidate word set. The user can directly select the prompt content for search, without having to enter the complete video search information, which can simplify the video search process and thus improve the efficiency of video search.

[0063] See also Figure 4 , Figure 4 A flowchart of another multimedia processing method provided in an embodiment of the present application. The multimedia processing solution can be executed by the server 102 described above; the solution includes steps S401-S405, wherein:

[0064] S401: The server obtains a multimedia file set.

[0065] The specific implementation of step S401 can be found in Figure 2 The implementation of step S201 will not be repeated here.

[0066] S402: The server obtains fusion features of the target multimedia file.

[0067] Among them, the fusion feature is obtained by fusing the features of at least two modal data of the target multimedia file after extracting the features respectively, and the file content of the target multimedia file includes data of at least two modalities of text modality, image modality and audio modality; the multimedia file label prediction model includes an image feature extraction module, an audio feature extraction module, and at least two feature extraction modules of a text feature extraction module, and the file content of the multimedia file includes data of text modality, data of image modality and data of at least two modalities of audio modality. The multimedia file label prediction model also includes a fully connected layer; the fully connected layer is used to fuse the features of the data of P modalities extracted by P feature extraction modules to obtain the fusion feature.

[0068] The following takes a multimedia file (referred to as a target multimedia file) as an example to illustrate determining a target candidate word set for the media file:

[0069] Figure 5a This is a diagram of the architecture of a multimedia file tag prediction model provided in an embodiment of the present application. Figure 5aAs shown, the multimedia file tag prediction model extracts the image frame sequence of the target multimedia file from the data of the image modality; and extracts the image features of the target multimedia file through the image feature extraction module. Specifically, firstly, a single-frame representation of each image frame is constructed (such as constructing a single-frame representation of each image frame based on the feature extraction network inception), and then the image content representation (i.e., image features) is obtained by fusing multiple frames of temporal features through the encoding model based on the Transformer (transformer). Similarly, the multimedia file tag prediction model extracts the audio frame sequence of the target multimedia file from the data of the audio modality; and extracts the audio features of the target multimedia file through the audio feature extraction module. Specifically, firstly, a single-frame representation of each audio frame is constructed (such as constructing a single-frame representation of each image frame based on the feature extraction model VGGish), and then the audio content representation (i.e., image features) is obtained by fusing multiple frames of temporal features through the encoding model based on the Transformer. The multimedia file tag prediction model obtains the text modality data of the target multimedia file (such as title, dialogue, etc.), and performs word segmentation and vectorization on the text modality data of the target multimedia file, and then performs vector concatenation through the Transformer-based encoding model to obtain the text content representation (i.e., text features). The encoded features are then fused through the fully connected layer, so that the dimension of the obtained fused features (decoding initial state) is the same as the dimension of the Transformer-based decoding model.

[0070] S403: The server decodes the fused features to obtain one or more predicted search phrases; concatenates the one or more predicted search phrases into predicted search text, and adds the predicted search text to a candidate word set of the target multimedia file.

[0071] In one embodiment, one or more predicted search phrases include a first search phrase and a second search phrase. The server decodes the fused feature to obtain the first search phrase; and decodes the first search phrase and the fused feature to obtain the second search phrase. Among them, the specific implementation method of the server decoding the fused feature to obtain the first search phrase is: suppose that the data of the text modality of the target multimedia file includes P text phrases, and P is a positive integer; the server obtains the text feature of the data of the text modality of the target multimedia file, and obtains the feature vectors of the P text phrases included in the text feature; based on the attention mechanism, the attention coefficient between the fused feature and the feature vector of each text phrase is determined (such as calculating the inner product of the fused feature and the feature vector of each text phrase to obtain the attention coefficient of each feature vector, or determining the attention coefficient through K, Q, and V in the attention mechanism, In the process of generating P feature vectors, K matrix and V matrix will be generated, and K matrix and V matrix can be the same. In the decoding process, the input feature sequence is used as Q matrix, and the attention coefficient is determined by the inner product between K matrix and Q matrix); the feature vectors of P text phrases and the attention coefficient of each feature vector are weighted summed to obtain the features to be decoded, and the features to be decoded are decoded by the decoding model of Transformer to obtain the probability values ​​corresponding to K label words; the label word or text phrase corresponding to the maximum value is selected as the first search phrase among the K probability values ​​and P attention coefficients.

[0072] As can be seen from the above, the decoding process of the present application is an iterative decoding process, and the input at each moment is related to the search phrase that has been decoded previously.

[0073] In another embodiment, the fused features are predicted based on the Transformer decoding model to obtain one or more predicted search phrases; the one or more predicted search phrases are spliced ​​into predicted search text, and the predicted search text is added to the alternative word set of the target multimedia file; for example, assuming that the predicted search phrases are "today", "new crown", "newly added", and "number of people", the spliced ​​predicted search text is "the number of new crown cases today".

[0074] Figure 5b This is another architecture diagram of a multimedia file tag prediction model provided in an embodiment of the present application. Figure 5b As shown, in Figure 5a On the basis of the multimedia file tag prediction model shown, the text features output by the Transformer-based encoding model are a feature matrix. The data in each row of the feature matrix is ​​used to represent a word vector. Suppose the feature matrix has n rows, then based on the feature matrix, n feature vector representations of word 1-word n ​​can be obtained (that is, the text modality data of the target multimedia file contains n phrases).

[0075] The attention mechanism is involved in the decoding model of Transformer. The attention mechanism is used to determine the contribution of the feature vector of each phrase to the current decoding position, so that different decoding positions can pay attention to the feature vectors of different phrases. Specifically, when decoding the i-th position, the attention mechanism determines the attention coefficient of the fused feature (the fused feature can be regarded as the input feature of the 0-th decoding position) to the feature vector of each phrase, determines the attention coefficient of the predicted word vector of the first search phrase y1 to the feature vector of each phrase, determines the attention coefficient of the predicted word vector of the second search phrase y2 to the feature vector of each phrase..., determines the attention coefficient of the predicted word vector of the i-1-th search phrase y(i-1) to the feature vector of each phrase. By weighted summing the attention coefficient and the feature vectors of n phrases, the feature to be decoded at the i-th position can be determined. It can be seen that the feature to be decoded at the i-th position calculated in this way not only contains context information, but also includes the contribution of the feature vector of the phrase to the predicted phrase at the i-th position. The Transformer decoding model decodes the feature to be decoded at the i-th position and obtains the probability values ​​corresponding to the K label words. The probability values ​​of the K label words can be regarded as the prediction results of the simple Transformer decoding model. The probability values ​​of the K label words are compared with all the aforementioned attention coefficients, and the maximum value is selected. The label word corresponding to the maximum value or the phrase corresponding to the feature vector is used as the predicted word at the i-th position (that is, the predicted i-th search phrase yi).

[0076] This is because, if the attention coefficient of the feature vector of a certain phrase is particularly large, it means that the phrase corresponding to the feature vector of the word has a high probability of being the correct decoding result, and the phrase corresponding to the feature vector can be directly used as the predicted word.

[0077] Of course, in addition to comparing the probability values ​​and attention coefficients of the label words, the label word corresponding to the maximum probability value among the probability values ​​of the K label words can also be directly used as the predicted word for the i-th position.

[0078] It can be seen that the attention module can be used to filter the important words in the important word set and the label words in the label word set, so that the keywords in the candidate word set of the target multimedia file are more consistent with the file content of the multimedia file.

[0079] In another embodiment, the multimedia file tag prediction model is obtained by optimizing and training the initial multimedia file tag prediction model using a training data set, wherein the training data set is constructed by the server based on user search records, and the training data set includes training videos and search terms associated with the training videos; for example, the search term associated with the XX TV series is YY actor. Specifically, the server extracts the first feature of the image modality data of the training video through the image feature extraction module in the initial multimedia file tag prediction model; extracts the second feature of the audio modality data of the training video through the audio feature extraction module in the initial multimedia file tag prediction model; extracts the third feature of the text modality data of the training video through the text feature extraction module in the initial multimedia file tag prediction model, and obtains a word vector set based on the feature matrix corresponding to the third feature. The first feature, the second feature, and the third feature are fused using the fully connected layer in the initial multimedia file tag prediction model to obtain the target fused feature. Subsequently, the target fusion features are predicted by the prediction layer in the initial multimedia file label prediction model to obtain a predicted retrieval phrase, each predicted retrieval term in the predicted retrieval phrase carries a probability value, and the label word is determined according to the probability value of each word vector and the probability value of each predicted retrieval term, and the label word is added to the predicted word set to obtain a predicted word set. The loss function is used to calculate the loss of the predicted word set and the retrieval terms associated with the training video, and the parameters of the initial multimedia file label prediction model are adjusted according to the loss calculation result to obtain a multimedia file label prediction model according to the adjusted parameters.

[0080] S404: When receiving the intended search request, the server searches for a search prompt word that matches the target search word in the candidate word set of N multimedia files.

[0081] S405: The server outputs the search prompt word.

[0082] The specific implementation of step S404 and step S405 can be referred to Figure 2 The implementation of step S203 and step S204 will not be repeated here.

[0083] The embodiments of this application are Figure 2 On the basis of the embodiment, the features of the data of P modes of the target multimedia file are extracted through the multimedia file tag prediction model, and the candidate word set is predicted based on the features of the data of P modes, which is more comprehensive and accurate than the candidate word set predicted based on the features of the data of a single mode. In addition, the important words in the important word set and the label words in the label word set are screened through the attention module, so that the keywords in the third keyword set are more consistent with the file content of the multimedia file, thereby improving the prediction accuracy of the multimedia file tag prediction model.

[0084] See also Figure 6 , Figure 6 A flowchart of another multimedia processing method provided in an embodiment of the present application. The multimedia processing solution can be executed by the server 102 described above; the solution includes steps S601-S606, wherein:

[0085] S601: The server obtains a multimedia file set.

[0086] The specific implementation of step S601 can be found in Figure 2 The implementation of step S201 will not be repeated here.

[0087] S602: The server extracts keywords from the text modality data of the target multimedia file to obtain a first keyword set.

[0088] In one embodiment, the file content of the target multimedia file includes text modality. The server obtains data of the text modality of the target multimedia file, and the data of the text modality includes one or more of the following: data of the text modality carried in the file content (such as the title and overview of the target multimedia file), data of the text modality recognized from the data of the image modality by image recognition technology (such as OCR) (such as subtitles), and data of the text modality recognized from the data of the audio modality by audio recognition technology (such as ASR) (such as character dialogue).

[0089] First, the server splits the data of the text mode of the acquired target multimedia file (such as splitting the sentence through the maximum match algorithm) to obtain a vocabulary set (that is, the vocabulary in the vocabulary set is obtained by the server splitting the title, subtitle, dialogue, etc. in the text mode data). Then, the server constructs the nodes in the relationship graph according to the vocabulary set (each node in the relationship graph corresponds to a vocabulary in the vocabulary set), and constructs the edges of each node in the relationship graph according to the association relationship between each vocabulary in the vocabulary set; wherein, the association relationship between each vocabulary includes: vocabulary A and vocabulary B are obtained by splitting the same sentence (such as "new coronavirus can cause pneumonia" is split into "new coronavirus" and "pneumonia", then there is an association relationship between "new coronavirus" and "pneumonia"), or the interval distance between vocabulary A and vocabulary B (such as the number of words in the middle, the time interval in the video / audio, etc.) is less than the interval threshold. Then, the server iterates the relationship graph (such as by constructing a similarity matrix and iterating the similarity matrix until the iteration converges) to obtain the weight of each node (that is, the weight of each vocabulary in the vocabulary set). Finally, the words corresponding to the nodes whose weights are greater than the threshold are added to the first keyword set to obtain the first keyword set.

[0090] S603: The server performs encoding and decoding processing on the text mode data of the target multimedia file to obtain a second keyword set.

[0091] In one embodiment, the file content of the target multimedia file includes data in text mode. The server performs data conversion on the data in text mode to obtain a word vector sequence of the data in text mode, that is, the data in text mode is represented by the word vector sequence. After obtaining the word vector sequence, the word vector sequence is encoded based on the association relationship between each word vector in the word vector sequence to obtain a hidden feature sequence; the hidden feature sequence can be specifically represented by a feature matrix. After obtaining the hidden feature sequence, the hidden feature sequence is decoded to obtain a second keyword set.

[0092] The encoding and decoding processing can be implemented through a codec model, which is a model with a codec structure, and the encoding structure or the decoding structure can be specifically a convolutional neural network (CNN), a recurrent neural network (RNN), a gate recurrent unit (GRU), or a long short-time memory model (LSTM). The encoding and decoding model can be specifically a transformer, wherein the transformer is composed of an encoder (encoding) model and a decoder (decoding) model.

[0093] The codec model is used to convert text data of the multimedia file into tags related to the file content of the multimedia file.

[0094] In one embodiment, the codec model is obtained by optimizing and training the initial codec model using corpus sample data, and the corpus sample data includes the text content of the annotated video and the annotated label set corresponding to the video; for example, the text content of video 1: Luo XX ridiculed himself on the talk show that he owed 600 million yuan, and the way he tried to pay off his debts was so cute; the annotated label set of video 1: talk show Luo XX owes debt funny variety show. Specifically, the labels in the label set are fine-grained semantic units of the video, such as "California epidemic in the United States", "talk show", "Luo XX owes debt", etc. The labels in the label set can be considered as an intuitive reflection of the video content. Users can search for videos through the labels in the label set, so the labels in the label set are used as the training targets of the codec model. Taking the codec model as Transformer, the text content of the annotated video is input into the Encoder model in the Transformer, and the predicted label set is generated through the Decoder model, and then the predicted label set and the annotated label set are calculated by the loss function, and the parameters in the model are adjusted according to the results of the loss calculation, so that the model has the ability to input video text and output video label sequences.

[0095] It should be noted that the determination of the second keyword set here and the determination of the alternative word set based on the multimodal data of multimedia files mentioned above both involve Transformers, but the input and output of the two Transformers are completely different, that is, the model parameters of the two Transformers are inconsistent, which only shows that the model structures of the two Transformers are similar.

[0096] S604: The server performs a fusion process on the first keyword set and the second keyword set to obtain a candidate word set for the target multimedia file.

[0097] In an optional implementation, the server combines the first keyword set and the second keyword set into an alternative word set, and removes repeated alternative words in the alternative word set. In another implementation, the server combines the first keyword set and the second keyword set into a candidate keyword set, and counts the number of repetitions of each keyword in the candidate keyword set, and adds keywords with a number of repetitions greater than a threshold to the alternative word set to obtain an alternative word set for the target multimedia file. It is understandable that by adding keywords that appear multiple times to the alternative word set, the alternative words in the alternative word set can be made more consistent with the file content of the multimedia file.

[0098] S605: When receiving the intended search request, the server searches for a search prompt word that matches the target search word in the candidate word set of N multimedia files.

[0099] S606: The server outputs the search prompt word.

[0100] The specific implementation of step S605 and step S606 can refer to the implementation of step S203 and step S204 in 2, and will not be repeated here.

[0101] In an optional implementation, the candidate word set of the target multimedia file obtained in step S403 is determined as the third keyword set, the first keyword set, the second keyword set and the third keyword set are combined into a candidate word set, and repeated candidate words in the candidate word set are removed to obtain the candidate word set of the target multimedia file. In another implementation, the server combines the first keyword set, the second keyword set and the third keyword set into a candidate keyword set, and counts the number of repetitions of each keyword in the candidate keyword set, and adds keywords with a number of repetitions greater than a threshold value to the candidate word set to obtain the candidate word set of the target multimedia file. It can be understood that by adding keywords that appear multiple times to the candidate word set, the candidate words in the candidate word set can be made more consistent with the file content of the multimedia file.

[0102] In an embodiment of the present application, the target multimedia file is processed in a variety of ways to obtain multiple keyword sets, and the multiple keyword sets are combined to obtain an alternative word set to improve the file content coverage of the target multimedia file by the alternative word set.

[0103] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0104] See also Figure 7 , Figure 7 The structure diagram of a multimedia processing device provided in an embodiment of the present application is as follows. The multimedia processing device 700 includes an acquisition unit 701 and a processing unit 702. The device can be mounted on Figure 1a On server 102 as shown. Figure 7 The multimedia processing device shown can be used to perform the above Figure 2 , Figure 4 and Figure 6 Part or all of the functions of the method embodiment described. The detailed description of each unit is as follows:

[0105] An acquisition unit 701 is used to acquire a multimedia file set, where the multimedia file set includes N multimedia files, where N is a positive integer;

[0106] The processing unit 702 is used to use the multimedia file tag prediction model to predict the file content of each multimedia file respectively to obtain a set of candidate words for each multimedia file; and when receiving an intention retrieval request, search for a search prompt word matching the target search word in the candidate word set of N multimedia files, wherein the intention retrieval request carries the target search word; and output the search prompt word.

[0107] In one embodiment, the target multimedia file is one of N multimedia files; the target multimedia file is one of N multimedia files, and the file content of the target multimedia file includes data of P modes, where P is a positive integer; the multimedia file tag prediction model includes: a keyword extraction model, a coding model and a multimodal model; the processing unit 702 is used to use the multimedia file tag prediction model to predict the file content of the target multimedia file to obtain a process of a candidate word set for the target multimedia file, specifically for:

[0108] Acquire a fusion feature of the target multimedia file, wherein the fusion feature is obtained by fusing features extracted from data of at least two modalities of the target multimedia file, wherein the file content of the target multimedia file includes data of at least two modalities of a text modality, an image modality, and an audio modality;

[0109] Decoding the fused features to obtain one or more predicted search phrases;

[0110] splicing one or more predicted search phrases into predicted search text, and adding the predicted search text to the candidate word set of the target multimedia file. Processing the data of the text modality of the target multimedia file using the keyword extraction model to obtain a first keyword set, wherein the text modality belongs to the P modalities;

[0111] Processing the text modality data of the target multimedia file using the encoding and decoding model to obtain a second keyword set;

[0112] Processing the data of P modes of the target multimedia file using the multimodal model to obtain a third keyword set;

[0113] The first keyword set, the second keyword set and the third keyword set are combined into a candidate word set for the target multimedia file.

[0114] In one embodiment, the one or more predicted search phrases include a first search phrase and a second search phrase; the multimodal model includes an image feature extraction module, an audio feature extraction module, a text feature extraction module, a fully connected layer and a prediction layer; the P modalities also include an image modality and an audio modality; the processing unit 6702 is used to decode the fusion feature to obtain one or more predicted search phrases; the multimodal model is used to process the data of the P modalities of the target multimedia file to obtain a third keyword set, which is specifically used to:

[0115] Decoding the fused features to obtain the first search phrase;

[0116] The word vector of the first search phrase is obtained, and the word vector of the first search phrase and the fusion feature are decoded to obtain the second search phrase. The image feature of the image modality data of the target multimedia file is extracted by the image feature extraction module;

[0117] Extracting audio features of data of the audio mode of the target multimedia file by the audio feature extraction module;

[0118] Extracting text features of data in the text modality of the target multimedia file by the text feature extraction module;

[0119] Using the fully connected layer to fuse the image feature, the audio feature and the text feature to obtain a fused feature;

[0120] A third keyword set is obtained according to the prediction result, wherein the prediction result is obtained by the prediction layer performing prediction processing on the fusion feature.

[0121] In one embodiment, the data of the text modality of the target multimedia file includes P text phrases, where P is a positive integer; the prediction result includes a tag word set, and the tag word set carries a probability value corresponding to each tag word; the processing unit 6702 is further used to decode the fusion feature to obtain the first search phrase, specifically for:

[0122] Acquire text features of data in a text mode of the target multimedia file, wherein the text features include feature vectors of P text phrases;

[0123] Determining an attention coefficient between the fusion feature and a feature vector of each text phrase based on an attention mechanism;

[0124] Performing a weighted summation on the feature vectors of the P text phrases and the attention coefficient of each feature vector to obtain features to be decoded, and performing a decoding process on the features to be decoded to obtain probability values ​​corresponding to the K label words respectively;

[0125] Select the label word or text phrase corresponding to the maximum value from the K probability values ​​and the P attention coefficients as the first search phrase. Generate an important word set of the text modality data of the target multimedia file through the text feature extraction module, and the important word set carries the probability value corresponding to each important word;

[0126] The step of obtaining a third keyword set according to the prediction result includes:

[0127] The important words in the important word set whose probability values ​​are higher than a first probability threshold and the tag words in the tag word set whose probability values ​​are higher than a second probability threshold are added to the third keyword set.

[0128] In one embodiment, the target multimedia file is one of N multimedia files, and the file content of the target multimedia file includes data in text mode; the processing unit 6702 is used to use the multimedia file tag prediction model to predict the file content of the target multimedia file to obtain a candidate word set of the target multimedia file; use the keyword extraction model to process the text mode data of the target multimedia file to obtain a first keyword set, specifically for:

[0129] Extracting keywords from the text modality data of the target multimedia file to obtain a first keyword set;

[0130] Performing encoding and decoding processing on the data of the text mode of the target multimedia file to obtain a second keyword set;

[0131] The first keyword set and the second keyword set are merged to obtain a candidate word set of the target multimedia file. The text modality data of the target multimedia file is split to obtain a vocabulary set;

[0132] A relationship graph is constructed according to the vocabulary set and the association relationship between each word in the vocabulary set, wherein the nodes in the relationship graph correspond to the words in the vocabulary set one by one, and the edge between the i-th node and the j-th node in the relationship graph is determined according to the association relationship between the word corresponding to the i-th node and the word corresponding to the j-th node, i and j are positive integers, and i≠j;

[0133] Iteratively process the relationship graph to obtain the weight of each node;

[0134] The words corresponding to the nodes whose weights are greater than the threshold are added to the first keyword set.

[0135] In one embodiment, the encoding and decoding model is constructed based on the attention mechanism; the processing unit 6702 is used to extract keywords from the data of the text modality of the target multimedia file to obtain a first keyword set, and use the encoding and decoding model to process the data of the text modality of the target multimedia file to obtain a second keyword set, specifically for:

[0136] Splitting the text modality data of the target multimedia file to obtain a vocabulary set;

[0137] A relationship graph is constructed according to the vocabulary set and the association relationship between each word in the vocabulary set, wherein the nodes in the relationship graph correspond to the words in the vocabulary set one by one, and the edge between the i-th node and the j-th node in the relationship graph is determined according to the association relationship between the word corresponding to the i-th node and the word corresponding to the j-th node, i and j are positive integers, and i≠j;

[0138] Iteratively process the relationship graph to obtain the weight of each node;

[0139] The words corresponding to the nodes whose weights are greater than the threshold are added to the first keyword set. The data of the text mode of the target multimedia file is encoded to obtain a word vector sequence of the data of the text mode, wherein the word vector sequence includes M subsets, where M is a positive integer;

[0140] Based on the attention mechanism, k subsets are selected from the M subsets for decoding processing to obtain a second keyword set, where k is a positive integer and k≤M.

[0141] In one embodiment, the processing unit 6702 is used to perform encoding and decoding processing on the data of the text mode of the target multimedia file to obtain a second keyword set, and search for a search prompt word matching the target search word in the candidate word set of N multimedia files, specifically for:

[0142] Converting the text modality data of the target multimedia file into a word vector sequence;

[0143] Encoding the word vector sequence to obtain a hidden feature sequence;

[0144] The hidden feature sequence is decoded to obtain the second keyword set.

[0145] In one embodiment, the processing unit 702 is used to search for a search prompt word matching the target search word in the candidate word set of N multimedia files, specifically to:

[0146] Acquire a search word set, and construct a prompt index according to the association relationship between each search word in the search word set and all the candidate words in the N candidate word sets;

[0147] Determine a set of search prompt words to be selected according to the target search word and the prompt index;

[0148] Acquire characteristic information of the current user, and determine the weight of each candidate search prompt word in the candidate search prompt word set according to the characteristic information, wherein the characteristic information includes one or more of user annotation information and user historical behavior information;

[0149] The candidate search prompt words in the candidate search prompt word set are sorted in descending order of weight, and the search prompt word is determined according to the sorting result. A search word set is obtained, and a prompt index is constructed according to the association relationship between each search word in the search word set and all the candidate words in the N candidate word sets;

[0150] Determine a set of search prompt words to be selected according to the target search word and the prompt index;

[0151] Acquire characteristic information of the current user, and determine the weight of each candidate search prompt word in the candidate search prompt word set according to the characteristic information, wherein the characteristic information includes at least one of the following: user annotation information, user historical behavior information;

[0152] The candidate search prompt words in the candidate search prompt word set are sorted in descending order of weight, and the search prompt word is determined according to the sorting result.

[0153] According to one embodiment of the present application, Figure 2 , Figure 4 and Figure 6 Some of the steps involved in the multimedia processing method shown can be represented by Figure 7 The various units in the multimedia processing device shown are executed. For example, Figure 2 The step S201 shown in FIG. 1 can be performed by Figure 7 The acquisition unit 701 shown in FIG. 1 is executed, and steps S202 to S204 can be performed by Figure 7 Processing unit 702 is shown executing. Figure 4 Steps S401 and S402 shown in FIG. 4 can be represented by Figure 7 The acquisition unit 701 shown in FIG. 1 is executed, and steps S403 to S405 can be performed by Figure 7 Processing unit 702 is shown executing. Figure 6 The step S601 shown in FIG. Figure 7 The acquisition unit 701 shown in FIG. 1 is executed, and steps S602 to S606 can be performed by Figure 7 Processing unit 702 is shown executing. Figure 7 The various units in the multimedia processing device shown can be separately or all combined into one or several other units to constitute, or some of the units can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the multimedia processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0154] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM) and other processing elements and storage elements. Figure 2 , Figure 4 and Figure 6 A computer program (including program code) for each step involved in the corresponding method shown in Figure 7 The multimedia processing device shown in and the multimedia processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0155] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the multimedia processing device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the multimedia processing device in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0156] See also Figure 8 , Figure 8A structural diagram of a multimedia processing device provided in an embodiment of the present application, the multimedia processing device 800 at least includes a processor 801, a communication interface 802 and a memory 803. Among them, the processor 801, the communication interface 802 and the memory 803 can be connected via a bus or other means. Among them, the processor 801 (or the central processing unit (CPU)) is the computing core and control core of the terminal, which can parse various instructions in the terminal and process various data of the terminal. For example, the CPU can be used to parse the power on and off instructions sent by the user to the terminal, and control the terminal to perform power on and off operations; for example, the CPU can transmit various types of interactive data between the internal structures of the terminal, and so on. The communication interface 802 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 801; the communication interface 802 can also be used for the transmission and interaction of internal data of the terminal. The memory 803 (Memory) is a memory device in the terminal for storing programs and data. It is understandable that the memory 803 here may include the built-in memory of the terminal, and of course may also include the extended memory supported by the terminal. The memory 803 provides a storage space, which stores the operating system of the terminal, including but not limited to: Android system, iOS system, Windows Phone system, etc., which is not limited in this application.

[0157] In the embodiment of the present application, the processor 801 is used to perform the following operations by running the executable program code in the memory 803:

[0158] Acquire a multimedia file set through the communication interface 802, wherein the multimedia file set includes N multimedia files, where N is a positive integer;

[0159] The multimedia file tag prediction model is used to predict the file content of each multimedia file to obtain a set of candidate words for each multimedia file;

[0160] When receiving an intention search request, searching for a search prompt word matching a target search word in a candidate word set of N multimedia files, wherein the intention search request carries the target search word;

[0161] The search hint word is outputted.

[0162] It should be understood that the multimedia processing device described in the embodiments of the present application can execute the above Figure 2 , Figure 4 and Figure 6 The description of the multimedia processing method in the corresponding embodiment can also be performed as described above. Figure 7The description of the multimedia processing device 700 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0163] Based on the same inventive concept, the principles and beneficial effects of solving the problems of the multimedia processing device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems of the multimedia processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0164] An embodiment of the present application also provides a computer-readable storage medium, in which one or more instructions are stored. The one or more instructions are suitable for being loaded by a processor and executing the multimedia processing method described in the above method embodiment.

[0165] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the multimedia processing method described in the above method embodiment.

[0166] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the above-mentioned multimedia processing method.

[0167] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0168] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0169] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the readable storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0170] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present application are still within the scope of the invention.

Claims

1. A multimedia processing method, characterized in that: The method comprises: Acquire a multimedia file set, where the multimedia file set includes N multimedia files, where N is a positive integer; The multimedia file tag prediction model is used to predict the file content of each multimedia file to obtain a set of candidate words for each multimedia file; When receiving an intent search request, obtaining a search vocabulary set, wherein the intent search request carries a target search term; Constructing a prompt index according to the association relationship between each search word in the search word set and all the candidate words in the N candidate word sets; Determine a set of search prompt words to be selected according to the target search word and the prompt index; Acquire characteristic information of the current user, and determine the weight of each candidate search prompt word in the candidate search prompt word set according to the characteristic information; Sorting the candidate search prompt words in the candidate search prompt word set in descending order of weight, and determining the search prompt word according to the sorting result; The search hint word is outputted.

2. The method according to claim 1, characterized in that The target multimedia file is one of the N multimedia files; The method of using a multimedia file tag prediction model to predict the file content of a target multimedia file to obtain a candidate word set for the target multimedia file includes: Acquire a fusion feature of the target multimedia file, wherein the fusion feature is obtained by fusing features extracted from data of at least two modalities of the target multimedia file, wherein the file content of the target multimedia file includes data of at least two modalities of a text modality, an image modality, and an audio modality; Decoding the fused features to obtain one or more predicted search phrases; One or more predicted search phrases are spliced ​​into predicted search text, and the predicted search text is added to a candidate word set of a target multimedia file.

3. The method according to claim 2, characterized in that The one or more predicted search phrases include a first search phrase and a second search phrase; The decoding process of the fusion feature to obtain one or more predicted search phrases includes: Decoding the fused features to obtain the first search phrase; The word vector of the first search phrase is obtained, and the word vector of the first search phrase and the fusion feature are decoded to obtain the second search phrase.

4. The method according to claim 3, characterized in that The data of the text mode of the target multimedia file includes P text phrases, where P is a positive integer; The decoding process of the fused feature to obtain the first search phrase includes: Acquire text features of data in a text mode of the target multimedia file, wherein the text features include feature vectors of P text phrases; Determining an attention coefficient between the fusion feature and a feature vector of each text phrase based on an attention mechanism; Performing a weighted summation on the feature vectors of the P text phrases and the attention coefficient of each feature vector to obtain features to be decoded, and performing a decoding process on the features to be decoded to obtain probability values ​​corresponding to the K label words respectively; The label word or text phrase corresponding to the maximum value is selected from the K probability values ​​and the P attention coefficients as the first search phrase.

5. The method according to claim 1, characterized in that The target multimedia file is one of the N multimedia files, and the file content of the target multimedia file includes data in a text mode; The multimedia file tag prediction model is used to predict the file content of the target multimedia file to obtain a set of candidate words for the target multimedia file, including: Extracting keywords from the text modality data of the target multimedia file to obtain a first keyword set; Performing encoding and decoding processing on the data of the text mode of the target multimedia file to obtain a second keyword set; The first keyword set and the second keyword set are fused to obtain a candidate word set for the target multimedia file.

6. The method according to claim 5, characterized in that The step of extracting keywords from the text modality data of the target multimedia file to obtain a first keyword set includes: Splitting the text modality data of the target multimedia file to obtain a vocabulary set; A relationship graph is constructed according to the vocabulary set and the association relationship between each word in the vocabulary set, wherein the nodes in the relationship graph correspond to the words in the vocabulary set one by one, and the edge between the i-th node and the j-th node in the relationship graph is determined according to the association relationship between the word corresponding to the i-th node and the word corresponding to the j-th node, i and j are positive integers, and i≠j; Iteratively process the relationship graph to obtain the weight of each node; The words corresponding to the nodes whose weights are greater than the threshold are added to the first keyword set.

7. The method according to claim 5, characterized in that The encoding and decoding of the text modality data of the target multimedia file to obtain a second keyword set includes: Converting the text modality data of the target multimedia file into a word vector sequence; Encoding the word vector sequence to obtain a hidden feature sequence; The hidden feature sequence is decoded to obtain the second keyword set.

8. The method according to claim 1, characterized in that: The feature information includes one or more of user annotation information and user historical behavior information.

9. A multimedia processing device, characterized in that: The multimedia processing device comprises: An acquisition unit, configured to acquire a multimedia file set, wherein the multimedia file set includes N multimedia files, where N is a positive integer; A processing unit, used to use a multimedia file tag prediction model to predict the file content of each multimedia file, and obtain a candidate word set for each multimedia file; The processing unit is further configured to, when receiving an intention search request, obtain a search vocabulary set, the intention search request carrying a target search term; construct a prompt index according to the association relationship between each search vocabulary in the search vocabulary set and all the candidate terms in the N candidate term sets; determine a candidate search prompt term set according to the target search term and the prompt index; obtain feature information of the current user, and determine a weight of each candidate search prompt term in the candidate search prompt term set according to the feature information; sort the candidate search prompt terms in the candidate search prompt term set in descending order of weight, and determine a search prompt term according to the sorting result; The processing unit is further configured to output the search prompt word.

10. A multimedia processing device, characterized in that: include: storage devices and processors; The storage device stores a computer program; A processor, configured to load and execute the computer program to implement the multimedia processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the multimedia processing method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads and executes the computer instructions from the computer-readable storage medium, so that the computer device executes the multimedia processing method according to any one of claims 1 to 8.

Images