Method for generating multimedia title, method and apparatus for pushing multimedia title

Abstract

The present disclosure relates to a multimedia title generation method, a multimedia title pushing method and device, the method comprising: obtaining a to-be-processed multimedia, and extracting target multi-modal information of the to-be-processed multimedia; taking the target multi-modal information as an input of a multi-modal title generation model, and outputting a plurality of multimedia titles; the training sample of the multi-modal title generation model comprises preset multi-modal information of a plurality of preset multimedia; evaluating the plurality of multimedia titles, and determining a target multimedia title based on the obtained evaluation result. By extracting the target multi-modal information of the to-be-processed multimedia, the title generated based on the target multi-modal information and the multi-modal title generation model can more comprehensively reflect the content of the to-be-processed multimedia, and thus a target multimedia title with higher quality can be obtained. And by pushing the target multimedia title to the user, more users can be attracted to watch the multimedia content corresponding to the target multimedia title.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a method and apparatus for generating multimedia titles and pushing multimedia titles. Background Technology

[0002] With the increasing popularity of mobile devices, users can easily watch their favorite videos on these devices. However, because many video titles only partially reflect the content of the video itself, the quality of the generated video titles may be poor. Summary of the Invention

[0003] This disclosure provides a multimedia title generation method, a multimedia title push method, and an apparatus.

[0004] According to a first aspect of this disclosure, a multimedia title generation method is provided, the method comprising:

[0005] Acquire the multimedia to be processed and extract the target multimodal information of the multimedia to be processed; wherein, the target multimodal information includes information of multiple dimensions in the multimedia to be processed;

[0006] The target multimodal information is used as input to the multimodal title generation model to output multiple multimedia titles; wherein, the multimodal text generation model is trained by training samples to obtain the multimodal title generation model, and the training samples include preset multimodal information of multiple preset multimedia.

[0007] The plurality of multimedia titles are evaluated, and the target multimedia title of the multimedia to be processed is determined from the plurality of multimedia titles based on the evaluation results.

[0008] According to a second aspect of this disclosure, a multimedia title push method is provided, characterized in that the method includes:

[0009] Obtain the target multimedia title provided in the above method;

[0010] The target multimedia title is pushed to the target terminal.

[0011] According to a third aspect of this disclosure, a multimedia title generation apparatus is provided, the apparatus comprising:

[0012] A multimodal information acquisition module is used to acquire multimedia to be processed and extract target multimodal information of the multimedia to be processed; wherein, the target multimodal information includes information of multiple dimensions in the multimedia to be processed;

[0013] The multimedia title acquisition module is used to take the target multimodal information as input to the multimodal title generation model and output multiple multimedia titles; wherein, the multimodal title generation model is obtained by training the multimodal text generation model through training samples, and the training samples include preset multimodal information of multiple preset multimedia.

[0014] An evaluation module is used to evaluate the multiple multimedia titles;

[0015] A multimedia title determination module is used to determine the target multimedia title of the multimedia to be processed from the plurality of multimedia titles based on the obtained evaluation results.

[0016] According to a fourth aspect of this disclosure, a multimedia title push device is provided, the device comprising:

[0017] A multimedia title acquisition module is used to acquire the target multimedia title in the aforementioned device;

[0018] The multimedia title push module is used to push the target multimedia title to the target terminal.

[0019] According to a fifth aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0020] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the methods described above.

[0021] The multimedia title generation method, multimedia title push method, and apparatus provided in this disclosure acquire multimedia to be processed and extract target multimodal information from the multimedia to be processed. This target multimodal information is used as input to a multimodal title generation model to output multiple multimedia titles. The multiple multimedia titles are evaluated, and a target multimedia title is determined based on the evaluation results. A multimodal text generation model is trained using preset multimodal information containing multiple preset multimedia elements to obtain a multimodal title model. Because the embodiments extract target multimodal information from the multimedia to be processed, the titles generated based on the target multimodal information and the multimodal title generation model can more comprehensively reflect the content of the multimedia to be processed, thus obtaining higher-quality target multimedia titles. Furthermore, by pushing this target multimedia title to users, more users can be attracted to watch the multimedia content corresponding to the target multimedia title. Attached Figure Description

[0022] Further details, features, and advantages of this disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram of the model structure provided for an exemplary embodiment of this disclosure;

[0024] Figure 2 A flowchart of a multimedia title generation method provided as an exemplary embodiment of this disclosure;

[0025] Figure 3 A flowchart of a multimedia title generation method provided as another exemplary embodiment of this disclosure;

[0026] Figure 4 A flowchart of a multimedia title push method provided as another exemplary embodiment of this disclosure;

[0027] Figure 5 A schematic block diagram of the functional modules of a multimedia title generation apparatus provided in an exemplary embodiment of the present disclosure;

[0028] Figure 6 A schematic block diagram of the functional modules of a multimedia title push device provided in an exemplary embodiment of the present disclosure;

[0029] Figure 7 A structural block diagram of an electronic device provided as an exemplary embodiment of this disclosure;

[0030] Figure 8 A block diagram of a computer system provided for an exemplary embodiment of this disclosure. Detailed Implementation

[0031] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0032] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0033] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc., used in this disclosure are only used to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0034] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0036] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0037] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0038] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understood that the above notification and user authorization process is merely illustrative and does not constitute a limitation on the implementation of this disclosure; other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0039] A good video title can comprehensively reflect the core content of a video, allowing users to quickly obtain the important information they need. Therefore, in this era of information overload, generating video titles based on video content is an important technology. However, currently, in many user-posted videos, the video titles merely describe the user's mood or life status at the time of posting, resulting in poor-quality video titles.

[0040] In addition, video titles in related technologies are mainly generated through template configuration or by training text generation models through manually constructed datasets. The generated effect is limited by the size of the template or manually constructed dataset, and information from modalities such as audio and video is not fully utilized.

[0041] Therefore, in order to solve the above-mentioned technical problems, the embodiments of this disclosure optimize the generation effect of video titles by pre-training a multimodal generative model, automatic data augmentation of large models and fine-tuning of multimodal generative models, as well as attractiveness ranking evaluation, which can greatly improve the generation quality of video titles.

[0042] In this embodiment, short videos are used as an example for illustration. By acquiring a set of short videos containing multiple short videos, text information, image information, audio information, and user input information of each short video in the set are extracted. The text information can include dialogue from characters in the short video; the image information can include video frames from the short video, and text information in the video frames can be extracted using OCR (Optical Character Recognition); text information corresponding to audio information in the short video can also be extracted using ASR (Automatic Speech Recognition); the user input information can include at least one of subtitle information, bullet screen information, and user comment information. This allows for the acquisition of multimodal information based on the aforementioned text information, image information, audio information, and user input information from the short videos, thus providing a more comprehensive understanding of the content within the short videos.

[0043] In the embodiments, combined with Figure 1 As shown, Figure 1 This is a schematic diagram of the model structure provided in an embodiment of this disclosure. By training the network model, a multimodal generation model capable of generating multimodal information is obtained. The model structure of this network model can be similar to that of the Transformers model. The input includes encoders for three modalities: text, image, and audio. After encoding the information from these three modalities, the multimodal representation information is modeled through an attention mechanism. The decoder structure then generates plain text, thereby obtaining multiple title words.

[0044] In the embodiments, the multimodal generation model can be pre-trained using a large-scale multimodal dataset, allowing the model to learn knowledge from a large-scale corpus.

[0045] The aforementioned user input information may include user comments and bullet screen messages entered by the user in the short video. By segmenting the user input information into words and then ID-coding the segmented data, such as converting it into multiple uint32determ_ids and multiple word vectors, multiple word representations can be obtained through text encoder processing, including word 1 representation, word 2 representation, ... word m representation, ... word n ​​representation, etc.

[0046] By extracting the video frame sequence from the short video, processing it through a deep convolutional neural network model, and then processing its output through an image encoder, representations of each frame can be obtained, such as frame 1 representation, frame 2 representation, ... frame m representation, ..., frame n representation, etc.

[0047] In the embodiments, the audio frame sequence in the short video can also be extracted, processed by a deep neural network, and its output can be processed by an audio encoder to obtain, for example, a representation of frame 1, a representation of frame 2, ..., a representation of frame m, ..., a representation of frame n, etc.

[0048] The word representations, video frame representations, and audio frame representations obtained above are modeled using an attention mechanism and then processed by a decoder structure to generate plain text with multimodal characteristics.

[0049] In this embodiment, to obtain more comprehensive information from the short video, the text data obtained by this disclosure is not limited to various text sources such as video ASR text, OCR text, bullet comments, subtitles, or reviews. Simultaneously, video encoding and audio encoding data can also be incorporated to improve the title generation effect through implicit modeling of multimodal information.

[0050] like Figure 2 As shown, a dataset is constructed to pre-train a first preset model. This first preset model can be a deep learning model or a neural network model, etc., and the embodiments are not limited to this. In the embodiments, the dataset can be constructed using general data, and the first preset model is pre-trained using the constructed dataset, so that the trained first preset model has a preliminary text generation capability, and the trained first preset model can be used as a multimodal text generation model. In the embodiments, the general data can include data from three modalities: audio, video, and images, and the embodiments are not limited to this.

[0051] In the embodiments provided in this disclosure, the multimodal text generation model described above can be trained using augmented data to obtain a multimodal title generation model. This augmented data can include short videos of multiple categories, with each category's short video dataset corresponding to a data label, i.e., a category identifier, such as lifestyle, automotive, or comedy. By training the multimodal text generation model using datasets of different categories or a single category, a multimodal title generation model can be obtained. For example, by extracting multimodal information corresponding to each short video in the dataset and training the multimodal text generation model using this extracted information, the trained multimodal text generation model possesses the ability to generate short video titles. Therefore, the trained multimodal text generation model can be used as a multimodal title generation model.

[0052] In this embodiment, the constructed dataset may include large-scale publicly available data and multimodal data title corpora, such as high-quality videos and corresponding title data within the platform. The model structure described above can be an autoregressive model structure, and the loss function (Loss) can be designed as a maximum likelihood estimate based on the conditional probabilities of the original multimodal representations and the first n-1 tokens. By initializing the model weights and starting training, training can be stopped when the loss function meets the convergence condition.

[0053] In this embodiment, a title evaluation model can also be obtained by training a second preset model using a constructed dataset. This second preset model can be specifically modified based on the BERT model. The BERT model itself can act as an encoder, and by adding a prediction head to the BERT model, it can be used as a regression model. However, this embodiment is not limited to this. Taking short videos as an example, the dataset can include a set of target short videos within the platform. For instance, short videos with a multimedia popularity exceeding a threshold within the platform can be grouped into this target short video set. Specifically, the evaluation model in this embodiment can be a scoring model.

[0054] In this embodiment, the aforementioned multimedia popularity can be determined based on video CTR (Click-Through-Rate), DAU (Daily Active User), number of shares, or number of likes. Alternatively, multimedia popularity can be determined by weighted summation based on the aforementioned CTR, DAU, number of shares, and number of likes, along with their respective weights.

[0055] Since the short videos in the target short video set correspond to high multimedia popularity, it indicates that the titles of these short videos are also more likely to be accepted by users. Therefore, the BERT model can be trained using the short videos from the aforementioned target short video set as a dataset to obtain a title evaluation model.

[0056] By concatenating the multimodal title generation model and the title evaluation model obtained above, relevant information of the short video is input into the multimodal title generation model during title generation, and then evaluated by the title evaluation model. For example, the title evaluation model can score the titles generated by the multimodal title generation model, and the title with the highest score can be selected as the push title for the short video. Alternatively, the top N titles with the highest scores can be pushed, and one of them can be selected as the title for the short video based on the user's choice. Here, N is a positive integer.

[0057] Based on the above embodiments, in another embodiment provided in this disclosure, a multimedia title generation method is also provided, such as... Figure 3 As shown, the method may include the following steps:

[0058] In step S310, the multimedia to be processed is acquired, and the target multimodal information of the multimedia to be processed is extracted.

[0059] The target multimodal information may include information from multiple dimensions of the multimedia to be processed. For example, the target multimodal information may include two or more of the following information from the multimedia to be processed: subtitle information, bullet screen information, comment information, video information, and audio information.

[0060] In this embodiment, the multimedia to be processed can be video, audio, or image data. This embodiment uses a short video as an example for illustration, but the embodiment is not limited to this.

[0061] Several types of information can be extracted from short videos, including subtitles, bullet comments, comments, video information, and audio information, as target multimodal information. Since bullet comments and comments are input from other users, they provide feedback on the short video from their perspective. Subtitles, video information, and audio information are part of the short video's content itself, providing insights into the video's content from its own perspective. Combining this target multimodal information with the short video's own content and other user-generated content allows for a more comprehensive reflection of the short video's content, enabling the generation of higher-quality titles.

[0062] Therefore, in this embodiment, the multimedia to be processed may specifically include a video to be processed. By acquiring video information, audio information, and user input information from the video to be processed, and based on the video information, audio information, and user input information, target multimodal information is obtained. For example, multimodal text information can be obtained by extracting text information corresponding to user input information from the video to be processed, obtaining text information from audio information through audio recognition, and obtaining corresponding text information from the video through OCR recognition, thereby obtaining multimodal text information.

[0063] In step S320, the target multimodal information is used as input to the multimodal title generation model, and multiple multimedia titles are output.

[0064] Among them, a multimodal title generation model can be obtained by training the multimodal text generation model with training samples. The training samples can include preset multimodal information of multiple preset multimedia.

[0065] In this embodiment, the first preset model can be initially trained using the aforementioned general data, and the trained first preset model can be used as a multimodal text generation model. The trained first preset model will have preliminary text generation capabilities. Then, by training the multimodal text generation model again using preset multimodal information including multiple preset multimedia resources, a multimodal title generation model with multimedia title generation capabilities can be obtained. By using the target multimodal information as input to the multimodal title generation model, the multimodal title generation model can generate multiple multimedia titles that more comprehensively reflect the multimedia content to be processed. The preset multimodal information can include user input information, audio information, and video information. The user input information can include bullet comments and reviews, and the video information can include relevant information from the video obtained through OCR recognition.

[0066] In this embodiment, the first preset model is initially trained using general data, and then trained again using training samples containing multimedia data. This reduces the amount of data in the training samples containing multimedia data, thereby improving the training efficiency of the model.

[0067] In step S330, multiple multimedia titles are evaluated, and the target multimedia title of the multimedia to be processed is determined from the multiple titles based on the evaluation results.

[0068] In this embodiment, a second preset model can be trained using a constructed dataset. The trained second preset model is then used as a title evaluation model, which can be a Bert model or similar. This title evaluation model is used to evaluate the multiple multimedia titles, which are then sorted according to their scores. The title with the highest score can be selected as the target multimedia title.

[0069] The multimedia title generation method provided in this disclosure acquires multimedia to be processed and extracts target multimodal information from the multimedia. This target multimodal information is then used as input to a multimodal title generation model to output multiple multimedia titles. These multiple multimedia titles are evaluated, and a target multimedia title is determined based on the evaluation results. A multimodal text generation model is trained using preset multimodal information containing multiple preset multimedia elements to obtain a multimodal title model. Furthermore, because the target multimodal information of the multimedia to be processed is extracted in this embodiment, the title generated based on the target multimodal information and the multimodal title generation model can more comprehensively reflect the content of the multimedia to be processed, thus resulting in a higher-quality target multimedia title.

[0070] Based on the above embodiments, in another embodiment provided in this disclosure, the above method may further include the following steps:

[0071] In step S340, the target category to which the multimedia to be processed belongs is obtained, and multiple preset multimedia corresponding to the target category are obtained.

[0072] In step S350, preset multimodal information of multiple preset multimedia is obtained respectively, and the multimodal text generation model is trained based on the preset multimodal information, and the trained multimodal text generation model is used as the multimodal title generation model.

[0073] In this embodiment, since multimedia can include multiple categories, the title types corresponding to different categories of multimedia will also be different. For example, for serious multimedia, its title should also be serious; while for humorous or lifestyle multimedia, its title can be non-serious, such as entertainment, etc.

[0074] Therefore, by obtaining the target category of the multimedia to be processed, and training the multimodal text generation model with the multimodal information of multiple preset multimedia corresponding to the target category, the resulting multimodal title model can combine the target multimodal information of the multimedia to be processed and the target category to generate higher quality multimedia titles.

[0075] Based on the above embodiments, in another embodiment provided in this disclosure, the above method may further include the following steps:

[0076] In step S360, a multimedia set is obtained. This multimedia set includes multiple multimedia media.

[0077] In step S370, multiple multimedias in the multimedia set are classified to obtain multiple multimedia categories, and each multimedia category carries a category identifier.

[0078] In step S380, preset multimodal information of multiple multimedia categories is obtained respectively, and the multimodal text generation model is trained based on the preset multimodal information and category identifiers, and the trained multimodal text generation model is used as the multimodal title generation model.

[0079] In this embodiment, if there are a large number of short videos on the platform, and they are of various types, it is necessary to generate multimedia titles for each category of multimedia to be processed. This can be achieved by obtaining a multimedia set within the platform and classifying the multiple multimedia videos in that set, thus obtaining multiple multimedia types. For example, multiple short videos in a single-ticket set can be divided into categories such as lifestyle, efficiency, automobiles, and travel.

[0080] Each type of multimedia can be associated with a category label, i.e., a category identifier. This allows the multimodal text generation model to be trained using multimedia from multiple categories, resulting in a multimodal title generation model. When generating multimedia titles using this model, the generated titles will better match the category of the multimedia being processed, thus producing higher-quality titles.

[0081] Based on the above embodiments, in another embodiment provided in this disclosure, multiple multimedia titles can be used as input to an evaluation model, outputting scores corresponding to each multimedia title, and the scores are used as the evaluation results of the multiple multimedia titles. The scores represent the multimedia popularity corresponding to each multimedia title.

[0082] In this embodiment, multiple multimedia titles are evaluated using a title evaluation model. Since the scores in the evaluation results can reflect the user's enthusiasm for the multimedia titles, the evaluation results can better reflect the user's acceptance of the multimedia titles, thereby improving the evaluation efficiency.

[0083] Therefore, based on the above embodiments, the method provided in this disclosure may further include the following steps:

[0084] In step S391, historical multimedia content is acquired, and the multimedia popularity corresponding to the title of the historical multimedia content is acquired. The historical multimedia content includes the corresponding historical multimedia title.

[0085] In step S392, the historical multimedia titles and multimedia popularity are used as training samples to train the second preset model to obtain the title evaluation model.

[0086] In this embodiment, historical multimedia content containing multimedia popularity data can be selected. Since multimedia popularity reflects, to some extent, users' acceptance of multimedia content, such as their liking for it, it can also reflect their overall acceptance of the multimedia content. Therefore, by acquiring historical multimedia content containing multimedia popularity data and training the second preset model using this historical multimedia content, a title evaluation model is obtained. The evaluation results obtained through this title evaluation model can better reflect users' acceptance of multiple multimedia titles, thus improving evaluation efficiency.

[0087] Specifically, the multimedia popularity can include CTR, or it can be obtained by weighted summation based on CTR, DAU, number of reposts and number of likes and their respective weights. The weights can be set as needed, and the embodiments are not limited to this.

[0088] Based on the above embodiments, in another embodiment provided in this disclosure, such as Figure 4 As shown, a multimedia title push method is also provided, which may include the following steps:

[0089] In step S410, the target multimedia title is obtained. This target multimedia title can be the target multimedia title obtained in the above embodiments.

[0090] In step S420, the target multimedia title is pushed to the target terminal.

[0091] In this embodiment, target multimedia titles can be randomly pushed to each user terminal, and relevant user information corresponding to each terminal user can be obtained to push multimedia titles that match the user information to each terminal user.

[0092] For example, the user information may include the user's location. By obtaining the target location information carried by the target multimedia title, the target multimedia can be pushed to user terminals near the target location. For instance, if the content of the target multimedia title is about a car show, and the target location carried by the content may be the location of the car show, pushing the target multimedia title to user terminals near that location is more likely to pique the user's interest, leading them to click on the title and view the content. Therefore, the target terminal can be determined based on the target location, and user terminals near that location can be considered the target terminals.

[0093] In this embodiment, target multimedia titles that match user behavior characteristics can also be obtained. For example, some users like cars, and target multimedia titles related to cars can be recommended to these users to increase their consumption of the target multimedia content corresponding to the target multimedia title.

[0094] In this embodiment, since the target multimedia title is generated based on multimodal information, it can better reflect the content of the multimedia itself. Therefore, when the target multimedia title is pushed to other users, the other users will be more interested when they see the target multimedia title, and then watch the multimedia content corresponding to the target multimedia title. This can attract more users to watch the multimedia content.

[0095] In this embodiment, the target multimedia file may also carry a jump link. When a user clicks on the target multimedia file's title or performs other triggering actions, the user is redirected to the target multimedia file's content display interface based on the jump link. Because the target multimedia file's title carries a jump link, when a user is interested in the title, they can click on the target multimedia file to jump to the corresponding content interface, facilitating viewing.

[0096] By dividing each functional module according to its corresponding function, this disclosure provides a multimedia title generation device, which can be a server, a terminal, or a chip applied to a server. Figure 5 This is a schematic block diagram of the functional modules of a multimedia title generation apparatus provided in an exemplary embodiment of this disclosure. Figure 5 As shown, the multimedia title generation device includes:

[0097] The multimodal information acquisition module 10 is used to acquire multimedia to be processed and extract target multimodal information of the multimedia to be processed; wherein, the target multimodal information includes information of multiple dimensions in the multimedia to be processed;

[0098] The multimedia title acquisition module 20 is used to take the target multimodal information as input to the multimodal title generation model and output multiple multimedia titles; wherein, the multimodal title generation model is obtained by training the multimodal text generation model through training samples, and the training samples include preset multimodal information of multiple preset multimedia.

[0099] Evaluation module 30 is used to evaluate the plurality of multimedia titles;

[0100] The multimedia title determination module 40 is used to determine the target multimedia title of the multimedia to be processed from the plurality of multimedia titles based on the obtained evaluation results.

[0101] In another embodiment provided in this disclosure, the apparatus further includes: a first training module, the first training module being specifically used for:

[0102] Obtain the target category to which the multimedia to be processed belongs, and obtain multiple preset multimedia corresponding to the target category;

[0103] The preset multimodal information of the multiple preset multimedia is obtained respectively, and the multimodal text generation model is trained based on the preset multimodal information. The trained multimodal text generation model is then used as the multimodal title generation model.

[0104] In another embodiment provided in this disclosure, the apparatus further includes: a second training module, the second training module being specifically used for:

[0105] Obtain a multimedia collection, which includes multiple multimedia resources;

[0106] The multimedia in the multimedia collection is classified to obtain multiple multimedia categories, and each multimedia category carries a category identifier.

[0107] The preset multimodal information of the multiple multimedia categories is obtained respectively, and the multimodal text generation model is trained based on the preset multimodal information and the category identifier, and the trained multimodal text generation model is used as the multimodal title generation model.

[0108] In yet another embodiment provided in this disclosure, the apparatus further includes:

[0109] The third training module is used to acquire general data and train the first preset model using the general data, and use the trained first preset model as the multimodal text generation model.

[0110] In another embodiment provided in this disclosure, the evaluation module is specifically used for:

[0111] The multiple multimedia titles are used as input to the title evaluation model, and the scores corresponding to the multiple multimedia titles are output. The scores are used as the evaluation results of the multiple multimedia titles; wherein, the scores represent the multimedia popularity corresponding to the multimedia title.

[0112] In another embodiment provided in this disclosure, the apparatus further includes: a fourth training module, the fourth training module being used for:

[0113] The information acquisition module is used to acquire historical multimedia and acquire the multimedia popularity corresponding to the historical multimedia; the historical multimedia includes the corresponding historical multimedia title;

[0114] The title evaluation model is obtained by training the second preset model with the historical multimedia titles and the multimedia popularity as training samples.

[0115] In another embodiment provided in this disclosure, the multimedia to be processed includes video to be processed; the multimodal information acquisition module is specifically used for:

[0116] The video information, audio information, and user input information in the video to be processed are obtained, and the target multimodal information is obtained based on the video information, audio information, and user input information.

[0117] For details regarding the apparatus, please refer to the descriptions corresponding to the above method embodiments; they will not be repeated here.

[0118] The multimedia title generation apparatus provided in this disclosure acquires multimedia to be processed and extracts target multimodal information from the multimedia. This target multimodal information is then used as input to a multimodal title generation model to output multiple multimedia titles. These multiple multimedia titles are evaluated, and a target multimedia title is determined based on the evaluation results. A multimodal text generation model is trained using preset multimodal information containing multiple preset multimedia elements to obtain a multimodal title model. Furthermore, because the target multimodal information of the multimedia to be processed is extracted in this embodiment, the title generated based on the target multimodal information and the multimodal title generation model can more comprehensively reflect the content of the multimedia to be processed, thereby obtaining a higher-quality target multimedia title.

[0119] By dividing each functional module according to its corresponding function, this disclosure provides a multimedia title push device, which can be a server, a terminal, or a chip applied to a server. Figure 6 This is a schematic block diagram of the functional modules of a multimedia title generation apparatus provided in an exemplary embodiment of this disclosure. Figure 6 As shown, the multimedia title generation device includes:

[0120] The multimedia title acquisition module 50 is used to acquire a target multimedia title; wherein, the target multimedia title may be the target multimedia title obtained in the above embodiments.

[0121] The multimedia title push module 60 is used to push the target multimedia title to the target terminal.

[0122] Based on the above embodiments, in another embodiment provided in this disclosure, the target multimedia carries a jump link to the target multimedia; the device further includes:

[0123] The interface redirection module is used to respond to a trigger operation on the target multimedia and redirect to the content display interface of the target multimedia based on the redirection link.

[0124] For details, please refer to the descriptions corresponding to the above method embodiments, which will not be repeated here.

[0125] The multimedia title push device provided in this embodiment generates target multimedia titles based on multimodal information, which better reflects the content of the multimedia itself. Therefore, when the target multimedia title is pushed to other users, it will pique their interest and encourage them to watch the corresponding multimedia content, thus attracting more users to view the multimedia content. Furthermore, since the target multimedia title carries a jump link, users who are interested in the target multimedia title can jump to the corresponding content interface by clicking on it, facilitating viewing.

[0126] This disclosure also provides an electronic device, including: at least one processor; a memory for storing processor-executable instructions; wherein the at least one processor is configured to execute the instructions to implement the methods disclosed in this disclosure.

[0127] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this disclosure. For example... Figure 7 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801. The processor 1801 can perform the corresponding steps in the methods disclosed in the embodiments of this disclosure.

[0128] The processor 1801 described above can also be called a central processing unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method disclosed in this embodiment can be implemented by the integrated logic circuitry in the processor 1801 or by software instructions. The processor 1801 can be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 1802, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 1801 reads information from the memory 1802 and, in conjunction with its hardware, completes the steps of the method described above.

[0129] Furthermore, various operations / processes according to this disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 8 The computer system 1900 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above. Figure 8 A block diagram of a computer system provided for an exemplary embodiment of this disclosure.

[0130] Computer System 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 8As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 into a random access memory (RAM) 1903. The RAM 1903 may also store various programs and data required for the operation of the computer system 1900. The computing unit 1901, ROM 1902, and RAM 1903 are interconnected via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0132] Multiple components in computer system 1900 are connected to I / O interface 1905, including: input unit 1906, output unit 1907, storage unit 1908, and communication unit 1909. Input unit 1906 can be any type of device capable of inputting information into computer system 1900. Input unit 1906 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of the electronic device. Output unit 1907 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1908 may include, but is not limited to, hard disks and optical disks. Communication unit 1909 allows computer system 1900 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0133] The computing unit 1901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the methods disclosed in this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1908. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 can be configured to perform the methods disclosed in this disclosure by any other suitable means (e.g., by means of firmware).

[0134] This disclosure also provides a computer-readable storage medium, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the methods disclosed in this disclosure.

[0135] The computer-readable storage medium in this disclosure can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. The aforementioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specifically, the aforementioned computer-readable storage medium may include electrical connections based on one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0136] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0137] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the methods disclosed in the embodiments of this disclosure.

[0138] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer.

[0139] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0140] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0141] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0142] The above description is merely an embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0143] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A multimedia title generation method, characterized in that, The method includes: Acquire the multimedia to be processed and extract the target multimodal information of the multimedia to be processed; wherein, the target multimodal information includes information of multiple dimensions in the multimedia to be processed; The target multimodal information is used as input to the multimodal title generation model to output multiple multimedia titles; wherein, the multimodal text generation model is trained by training samples to obtain the multimodal title generation model, and the training samples include preset multimodal information of multiple preset multimedia. The multiple multimedia titles are used as input to the title evaluation model, and the scores corresponding to the multiple multimedia titles are output. The scores are used as the evaluation results of the multiple multimedia titles; wherein, the scores represent the multimedia popularity corresponding to the multimedia title. Based on the obtained evaluation results, the target multimedia title of the multimedia to be processed is determined from the plurality of multimedia titles; The method further includes: Obtain historical multimedia content and the corresponding multimedia popularity; the historical multimedia content includes the corresponding historical multimedia content title; The historical multimedia titles and multimedia popularity are used as training samples to train the second preset model to obtain the title evaluation model; wherein, the second preset model is based on the BERT model and modified, the BERT model is used as an encoder, and a prediction head is added to the BERT model to use it as a regression model.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the target category to which the multimedia to be processed belongs, and obtain multiple preset multimedia corresponding to the target category; The preset multimodal information of the multiple preset multimedia is obtained respectively, and the multimodal text generation model is trained based on the preset multimodal information. The trained multimodal text generation model is then used as the multimodal title generation model.

3. The method according to claim 1, characterized in that, The method further includes: Obtain a multimedia collection, which includes multiple multimedia resources; The multimedia in the multimedia collection is classified to obtain multiple multimedia categories, and each multimedia category carries a category identifier. The preset multimodal information of the multiple multimedia categories is obtained respectively, and the multimodal text generation model is trained based on the preset multimodal information and the category identifier, and the trained multimodal text generation model is used as the multimodal title generation model.

4. The method according to claim 2 or 3, characterized in that, The method further includes: Acquire general data, pre-train a first preset model using the general data, and fine-tune the pre-trained first preset model to obtain the multimodal text generation model.

5. The method according to claim 1, characterized in that, The multimedia to be processed includes video to be processed; The extraction of the target multimodal information of the multimedia to be processed includes: The video information, audio information, and user input information in the video to be processed are obtained, and the target multimodal information is obtained based on the video information, audio information, and user input information.

6. A multimedia title push method, characterized in that, The method includes: Obtain the target multimedia title as described in any one of claims 1 to 5; The target multimedia title is pushed to the target terminal.

7. The method according to claim 6, characterized in that, The target multimedia file carries a redirect link to the target multimedia file; the method further includes: In response to a trigger operation targeting the target multimedia, the user is redirected to the content display interface of the target multimedia via the redirect link.

8. A multimedia title generation device, characterized in that, The device includes: A multimodal information acquisition module is used to acquire multimedia to be processed and extract target multimodal information of the multimedia to be processed; wherein, the target multimodal information includes information of multiple dimensions in the multimedia to be processed; The multimedia title acquisition module is used to take the target multimodal information as input to the multimodal title generation model and output multiple multimedia titles; wherein, the multimodal title generation model is obtained by training the multimodal text generation model through training samples, and the training samples include preset multimodal information of multiple preset multimedia. An evaluation module is used to take the multiple multimedia titles as input to a title evaluation model, output scores corresponding to the multiple multimedia titles respectively, and use the scores as the evaluation results of the multiple multimedia titles; wherein, the scores represent the multimedia popularity corresponding to the multimedia title; A multimedia title determination module is used to determine the target multimedia title of the multimedia to be processed from the plurality of multimedia titles based on the obtained evaluation results; The fourth training module is used to: acquire historical multimedia and acquire the multimedia popularity corresponding to the historical multimedia; the historical multimedia includes the corresponding historical multimedia title; use the historical multimedia title and the multimedia popularity as training samples to train the second preset model to obtain the title evaluation model; wherein, the second preset model is based on the BERT model and modified, the BERT model is used as an encoder, and by adding a prediction head to the BERT model, it is used as a regression model.

9. A multimedia title push device, characterized in that, The device includes: A multimedia title acquisition module is used to acquire the target multimedia title as described in claim 8; The multimedia title push module is used to push the target multimedia title to the target terminal.

10. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-7.

Images