Video polyphonic ringtone generation method and device, equipment, medium and computer program product

By using a dual-tower audio and video model and a user audio sorting model, the problem of the inability to personalize video ringback tones has been solved, enabling the generation of video ringback tones based on the caller's characteristics, thereby improving user experience and communication smoothness.

CN119011726APending Publication Date: 2024-11-22MIGU CO LTD +1

Patent Information

Application Number
CN202411083906.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In existing technologies, recorded videos cannot generate audio that matches the video material based on the caller's personalized characteristics, resulting in fixed video ringback tones shared by the called user. This makes it impossible to play different audio for different callers, leading to a poor user experience.

Method used

By using a dual-tower audio and video model and a user audio sorting model, personalized video ringback tones are generated based on the video frame vector of the called user and the personalized characteristics of the calling user. This includes storing the video frame vector, filtering and sorting audio candidates, thus realizing personalized generation of video ringback tones.

Benefits of technology

It enables the generation of video ringback tones based on the caller's personalized characteristics, ensuring that the video material shared by the called user plays different audio for different callers, thereby improving user experience and a sense of intimacy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011726A_ABST
    Figure CN119011726A_ABST
Patent Text Reader

Abstract

The invention discloses a video polyphonic ringtone generation method, device and equipment, a medium and a computer program product, the method is applied to a video polyphonic ringtone platform, and the method comprises the following steps: when a calling user initiates a call request to a called user, searching a corresponding video frame and a video frame vector according to the information of the called user; searching a plurality of candidate audios corresponding to the video frame based on the video frame vector, and determining a target audio from the plurality of candidate audios according to personalized features of the calling user; and generating a video polyphonic ringtone according to the video frame and the target audio, and issuing the video polyphonic ringtone to the calling user for playing. According to the invention, for the video material shared by the called user, the corresponding video polyphonic ringtone can be generated according to the personalized characteristics of the calling user, so that the video material shared by the called user can play different audios for different calling users.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video ringback tone, and in particular to a method, device, equipment, medium and computer program product for generating a video ringback tone. Background Art

[0002] With the development of smart phones, mobile phones have gradually become an indispensable electronic product in people's lives. People can not only make calls to communicate with friends, but also browse news anytime and anywhere to understand social trends. When encountering interesting things or beautiful scenery, they can also use their mobile phones to record videos for future memories and sharing of the beauty of life. However, the original sound (noise) of the recorded video is not what people need. At this time, how to configure pleasant audio for the recorded video is an urgent problem to be solved. Although there are various audio and video editing applications on the Internet for users to edit videos, the audio materials provided by the application are not necessarily closely related to the video materials, or users need to manually select from a large amount of audio materials, which brings great difficulties to users.

[0003] In addition, in the era of online social networking, users can not only watch the videos they record themselves, but also share them with friends to watch, thereby enhancing the topic of conversation. Although there are various forms of social apps on the Internet for users to share interesting videos of daily life, this offline sharing method is stereotyped, that is, the video played for different friends is the same (that is, it has nothing to do with friends). The audio of the video ringback tone set by the traditional called user for a certain video material is fixed, and it is impossible to play different audio video ringback tones according to different calling users. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method, device, equipment, medium and computer program product for generating video ringback tone, which can generate corresponding video ringback tone according to the personalized characteristics of the calling user for the video material shared by the called user, so as to achieve the effect of playing different audio for different calling users for the video material shared by the called user.

[0005] In order to achieve the above object, an embodiment of the present invention provides a method for generating a video ring back tone, which is applied to a video ring back tone platform, comprising:

[0006] When the calling user initiates a call request to the called user, the corresponding video frame and video frame vector are searched according to the called user's information;

[0007] Searching for multiple candidate audios corresponding to the video frame based on the video frame vector, and determining a target audio from the multiple candidate audios according to the personalized characteristics of the calling user;

[0008] A video ring back tone is generated according to the video frame and the target audio, and the video ring back tone is sent to the calling user for playing.

[0009] As an improvement of the above solution, the video frame and the video frame vector are stored in a video material library, the video frame is obtained by parsing the video uploaded by the called user, and the video frame vector is predicted by a trained audio and video dual-tower model.

[0010] As an improvement of the above solution, the training method of the audio and video dual-tower model includes:

[0011] Acquire video ring back tone material, parse the video ring back tone material to obtain audio material and video frame material, and use the audio material and the video frame material as training samples of the audio and video dual-tower model; wherein the audio and video dual-tower model includes an audio material tower and a video material tower;

[0012] Input the audio material into the audio material tower to obtain the audio vector output by the audio material tower;

[0013] Input the video frame material into the video material tower to obtain a video frame vector output by the video material tower;

[0014] The cosine distance between the audio vector and the video frame vector is calculated, and the model parameters of the audio and video dual-tower model are optimized using the In-batch softmax objective function and returned to continue training until the training end condition is met to obtain a trained audio and video dual-tower model.

[0015] As an improvement of the above solution, the method further includes:

[0016] After parsing the video ring back tone material to obtain the audio material and the video frame material, the audio material is stored in an audio material library, and the video frame material is stored in the video material library.

[0017] As an improvement of the above solution, searching for multiple candidate audios corresponding to the video frame based on the video frame vector, and determining the target audio from the multiple candidate audios according to the personalized characteristics of the calling user, includes:

[0018] Calculate the similarity between the video frame vector and the audio vector stored in the audio Faiss index library, select the audio vector whose similarity is greater than a preset threshold, and obtain multiple candidate audio vectors; wherein the audio Faiss index library stores the audio vector predicted by the audio and video dual tower model;

[0019] Searching for corresponding audio from the audio material library according to the candidate audio vector to obtain a plurality of candidate audios;

[0020] Based on the trained user audio sorting model, multiple candidate audios are sorted according to the personalized characteristics of the calling user, and the target audio is determined from the multiple candidate audios.

[0021] As an improvement of the above solution, the training method of the user audio sorting model includes:

[0022] Constructing a training sample of a binary classification model based on the historical listening data of the calling user, and performing feature extraction; wherein the user-side features include user ID, gender, age, interest tags, and user listening ID sequence; the item-side features include song ID and song audio modality features;

[0023] Vectorizing the user-side features and horizontally splicing them to generate a user input vector;

[0024] Vectorizing the song ID and horizontally concatenating it with the audio modal feature to generate an item input vector;

[0025] The user input vector and the item input vector are simultaneously input into a multi-layer DNN network for feature interaction, and after obtaining the output vector of the DNN network, binary classification is performed through softmax;

[0026] The cross entropy loss function is used to calculate the loss of the binary classification model, and the model parameters are updated in reverse until the training end condition is met to obtain a trained user audio sorting model.

[0027] The embodiment of the present invention further provides a video ring back tone generation device, which is applied to a video ring back tone platform, comprising:

[0028] A target video search module, used to search for corresponding video frames and video frame vectors according to the called user's information when the calling user initiates a call request to the called user;

[0029] A target audio determination module, configured to search for a plurality of candidate audios corresponding to the video frame based on the video frame vector, and determine a target audio from the plurality of candidate audios according to the personalized characteristics of the calling user;

[0030] The video ring back tone generation module is used to generate a video ring back tone according to the video frame and the target audio, and send the video ring back tone to the calling user for playing.

[0031] An embodiment of the present invention further provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any of the above-mentioned video ringback tone generation methods when executing the computer program.

[0032] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any of the above-mentioned video ringback tone generation methods.

[0033] An embodiment of the present invention further provides a computer program product, which includes a computer program or a computer instruction. When the computer program or the computer instruction is executed by a processor, any of the above-mentioned methods for generating video ringback tone is implemented.

[0034] Compared with the prior art, the beneficial effects of a video ringback tone generation method, device, equipment, medium and computer program product provided by the embodiments of the present invention are: when a calling user initiates a call request to a called user, the video ringback tone platform searches for the corresponding video frame and video frame vector according to the information of the called user; searches for multiple candidate audios corresponding to the video frame based on the video frame vector, and determines the target audio from the multiple candidate audios according to the personalized characteristics of the calling user; generates a video ringback tone according to the video frame and the target audio, and sends the video ringback tone to the calling user for playback. The embodiments of the present invention can generate a corresponding video ringback tone according to the personalized characteristics of the calling user for the video material shared by the called user, so as to achieve the effect of playing different audios for different calling users for the video material shared by the called user. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a preferred embodiment of a method for generating a video ring back tone provided by the present invention;

[0036] Figure 2 It is an interactive flow chart of a method for generating a video ring back tone provided by the present invention;

[0037] Figure 3 It is a structural schematic diagram of an audio and video dual-tower model in a video ringback tone generation method provided by the present invention;

[0038] Figure 4 It is a structural schematic diagram of a user audio sorting model in a video ringback tone generation method provided by the present invention;

[0039] Figure 5 It is a structural schematic diagram of a preferred embodiment of a video color ring back tone generating device provided by the present invention;

[0040] Figure 6 It is a structural schematic diagram of a preferred embodiment of a terminal device provided by the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1 , Figure 1 The figure is a flow chart of a preferred embodiment of a method for generating a video ring back tone provided by the present invention. The method for generating a video ring back tone is applied to a video ring back tone platform and comprises:

[0043] S1, when the calling user initiates a call request to the called user, the corresponding video frame and video frame vector are searched according to the called user's information;

[0044] S2, searching for multiple candidate audios corresponding to the video frame based on the video frame vector, and determining a target audio from the multiple candidate audios according to the personalized characteristics of the calling user;

[0045] S3, generating a video ring back tone according to the video frame and the target audio, and sending the video ring back tone to the calling user for playing.

[0046] Specifically, the video ring back tone generation method provided in the embodiment of the present invention is applied to the video ring back tone platform, please refer to Figure 2 , Figure 2 It is an interactive flow chart of a video ring back tone generation method provided by the present invention. In an embodiment of the present invention, the video ring back tone platform may include a video ring back tone distribution platform and a video ring back tone synthesis platform. The video ring back tone distribution platform is used to receive a video ring back tone request instruction sent by a calling user when initiating a call request to a called user, send a video ring back tone to the calling user, and call a video material library, and search for video frames and video frame vectors corresponding to the called user according to the called user's information, such as a mobile phone number. The video ring back tone synthesis platform is used to receive the video frame and video frame vector of the called user sent by the video ring back tone distribution platform, call the audio Faiss index library, query the topK audio vectors that are most similar to the video frame vector of the called user, perform preliminary screening, query the audio material library, obtain the audio material associated with the topK audio vectors, and obtain multiple candidate audios. The video ring back tone synthesis platform sends the calling user ID and the topK audio, and calls the user audio sorting model to perform personalized sorting on the topK audio, thereby finely screening the audio that is most relevant to the calling user as the target audio. The video ringback tone synthesis platform generates the video ringback tone most relevant to the calling user based on the called user's video frame and model, and sends it to the video ringback tone distribution platform, which then sends it to the calling user for playback.

[0047] It should be noted that all the above steps can also be executed by the video ring back tone synthesis platform, and the structure of the video ring back tone platform is not limited to the above embodiment.

[0048] In the embodiment of the present invention, when a calling user dials the mobile phone number of a called user, the background sound of the video material set by the called user is determined through a multi-level screening method, and a call video ringtone that is closely related to the two is dynamically synthesized, thereby achieving the effect of playing different audios for different calling users for the video material shared by the called user.

[0049] In another preferred embodiment, the video frame and the video frame vector are stored in a video material library, the video frame is obtained by parsing the video uploaded by the called user, and the video frame vector is predicted by a trained audio and video dual-tower model.

[0050] Specifically, in the embodiment of the present invention, the user uploads video material offline (the video ringback tone synthesis platform will use the video material to synthesize video ringback tone, and when others call the user, it will be shown to others) to the video material management platform, and the video material uploaded by the user is parsed through the FFmpeg tool to obtain a video frame list, and the corresponding video frame vector is predicted through the trained audio and video dual-tower model, and the <user mobile phone number, video frame, video frame vector> 3-tuple is written into the video material library for storage, so that the video ringback tone distribution platform can query and use the video frame and video frame vector.

[0051] In another preferred embodiment, the training method of the audio and video dual tower model includes:

[0052] Acquire video ring back tone material, parse the video ring back tone material to obtain audio material and video frame material, and use the audio material and the video frame material as training samples of the audio and video dual-tower model; wherein the audio and video dual-tower model includes an audio material tower and a video material tower;

[0053] Input the audio material into the audio material tower to obtain the audio vector output by the audio material tower;

[0054] Input the video frame material into the video material tower to obtain a video frame vector output by the video material tower;

[0055] The cosine distance between the audio vector and the video frame vector is calculated, and the model parameters of the audio and video dual-tower model are optimized using the In-batch softmax objective function and returned to continue training until the training end condition is met to obtain a trained audio and video dual-tower model.

[0056] Specifically, the embodiment of the present invention adopts a dual-tower model, which is a model framework. The framework consists of two towers, a left tower and a right tower. The dual towers are two independent sub-networks, and the network parameters are not shared with each other. For example, the user tower and the item tower in the recommendation field, the query and document in the NLP field, and the image and text in the multimodal retrieval field can all be represented by the dual tower. The features of the two fields are encoded into a vector respectively, and then the cosine distance between the two vectors is calculated for model training, and finally similar recall is performed through vector similarity. When training the audio and video dual-tower model, the embodiment of the present invention selects a batch of high-quality content for each category through an online music platform or a full video ringtone library as video ringtone material, and decapsulates the video ringtone media through the FFmpeg tool, separates the video frame stream (extracts multiple pictures in the order of playback time) and the audio stream in the video ringtone, and outputs them to a separate data file to obtain audio material and video frame material, and uses the audio material and video frame material as training samples of the audio and video dual-tower model. The audio and video dual-tower model includes an audio material tower and a video material tower, and the video ringback tone media decapsulation table is shown in Table 1 below:

[0057] Table 1 Video ring back tone media decapsulation table

[0058]

[0059]

[0060] See also Figure 3 , Figure 3 It is a structural schematic diagram of an audio-video dual-tower model in a video ringback tone generation method provided by the present invention. In the embodiment of the present invention, the dual-tower model corresponds to a video material tower (video) and an audio material tower (audio), the video tower is used for inputting features related to video materials, for example: a video frame (picture) sequence, and the audio tower is used for inputting features related to audio materials, for example: an audio (sound) stream, and then inputting respective feature extraction networks (for example, a multi-layer DNN network) to obtain a video embedding video frame vector and an audio embedding audio vector. Then the cosine distance between the two embeddings, that is, the cosine distance between the audio vector and the video frame vector, can be calculated, and the In-batch softmax objective function (the video ringback tone video embedding is closer to its own audio embedding and farther from other video ringback tone audio embeddings) is used for model training, and the calculated loss is used to reversely update the parameters of the model until the training end condition is met, and a trained audio-video dual-tower model is obtained.

[0061] After the training of the audio and video dual-tower model is completed, the audio materials of all video ringtones can be obtained offline through the audio tower to obtain audio vectors, and the audio vectors can be imported into the audio Faiss index library for subsequent retrieval and matching of background sounds. When a new video is received, it can be first decapsulated through the FFmpeg tool to obtain the video frame sequence, and then its video vector is calculated through the video tower. Finally, the video frame vector is taken to the audio Faiss index library to calculate the similarity with the audio vector, and the topK (coarse screening) audios with the closest distance are returned as the retrieval result. When a new audio is received, the processing method is similar.

[0062] In another preferred embodiment, the method further comprises:

[0063] After parsing the video ring back tone material to obtain the audio material and the video frame material, the audio material is stored in an audio material library, and the video frame material is stored in the video material library.

[0064] Specifically, after parsing the video ringback tone material to obtain the audio material and the video frame material, the embodiment of the present invention stores the audio material in the audio material library and stores the video frame material in the video material library.

[0065] In another preferred embodiment, searching for multiple candidate audios corresponding to the video frame based on the video frame vector, and determining the target audio from the multiple candidate audios according to the personalized characteristics of the calling user, includes:

[0066] Calculate the similarity between the video frame vector and the audio vector stored in the audio Faiss index library, select the audio vector whose similarity is greater than a preset threshold, and obtain multiple candidate audio vectors; wherein the audio Faiss index library stores the audio vector predicted by the audio and video dual tower model;

[0067] Searching for corresponding audio from the audio material library according to the candidate audio vector to obtain a plurality of candidate audios;

[0068] Based on the trained user audio sorting model, multiple candidate audios are sorted according to the personalized characteristics of the calling user, and the target audio is determined from the multiple candidate audios.

[0069] Specifically, the embodiment of the present invention calculates the similarity between the video frame vector of the called user and the audio vector stored in the audio Faiss index library, selects the audio vector with a similarity greater than a preset threshold, and obtains multiple candidate audio vectors, namely, topK audio vectors. Among them, the audio Faiss index library stores audio vectors predicted by the audio and video dual-tower model. According to the multiple candidate audio vectors, the corresponding audio is searched one by one from the audio material library to obtain multiple candidate audios, namely, topK audios. Based on the trained user audio sorting model, the multiple candidate audios are sorted according to the personalized characteristics of the calling user, and the target audio is determined from the multiple candidate audios.

[0070] The embodiment of the present invention can obtain the vectors of video materials and audio materials through the trained audio and video dual tower model, and import the video frame vector into the video material library, and import the audio vector into the audio Faiss index library. When the user specifies the video material or audio material, the relevant topK audio material or video material can be quickly found by searching the Faiss index, and the user can select the corresponding audio material or video material to synthesize the video ringtone, which also provides a feasible solution for converting a large number of audio songs on the music platform into video ringtones.

[0071] In another preferred embodiment, the training method of the user audio ranking model includes:

[0072] Constructing a training sample of a binary classification model based on the historical listening data of the calling user, and performing feature extraction; wherein the user-side features include user ID, gender, age, interest tags, and user listening ID sequence; the item-side features include song ID and song audio modality features;

[0073] Vectorizing the user-side features and horizontally splicing them to generate a user input vector;

[0074] Vectorizing the song ID and horizontally concatenating it with the audio modal feature to generate an item input vector;

[0075] The user input vector and the item input vector are simultaneously input into a multi-layer DNN network for feature interaction, and after obtaining the output vector of the DNN network, binary classification is performed through softmax;

[0076] The cross entropy loss function is used to calculate the loss of the binary classification model, and the model parameters are updated in reverse until the training end condition is met to obtain a trained user audio sorting model.

[0077] For details, please refer to Figure 4 , Figure 4The embodiment of the present invention constructs a training sample of a binary classification model based on the historical listening data accumulated by the calling user on the online music platform when training the user audio sorting model.<user,item,label> , and perform feature extraction. Among them, the user-side features include user ID, gender, age, interest tags, and user song ID sequence; the item-side features include song ID and song audio modality features.<user,item,1> Indicates the user's playback behavior.<user,item,0> Indicates exposure without playback. The embodiment of the present invention uses a multi-layer DNN network to train a user audio sorting model. The model structure is as follows Figure 4 As shown. The original features on the user side are vectorized through the shared embedding layer and horizontally spliced ​​to generate the user input vector. The song ID features on the item side are vectorized through the embedding layer, and the audio modal features are horizontally spliced ​​to generate the item input vector. The user input vector and the item input vector are simultaneously input into the multi-layer DNN network for feature interaction. The DNN network output vector is binary classified through softmax, and the cross entropy loss function is used to calculate the model loss, and the model parameters are updated in reverse until the training end conditions are met to obtain the trained user audio sorting model. After the user audio sorting model is trained, for any given calling user and a batch of audio materials, the audio materials that the calling user is most interested in can be accurately predicted (finely screened).

[0078] The embodiment of the present invention preliminarily screens the audio candidate set based on the video material itself, and then accurately screens the matching audio material through the user audio sorting model in combination with the user interest portrait, that is, the embodiment of the present invention accurately screens the audio material through a two-step screening method, and then synthesizes the video ringback tone. For the same video material, the video ringback tone is synthesized for different people in different ways, with better adaptability. For the video material shared by the called user, different audios related to the video material are screened for different calling users, so as to synthesize a video ringback tone that is closely related to both the calling user and the called user, and the ringback tone and vibration are integrated, so that the calling user can perceive the video material shared by the called user, and the called user can perceive the audio preferred by the calling user, thereby reducing the gap, enhancing the sense of closeness between users, and communicating more smoothly.

[0079] Correspondingly, the present invention also provides a video ring back tone generation device, which can implement all the processes of the video ring back tone generation method in the above embodiment.

[0080] See also Figure 5 , Figure 5 The structure diagram of a preferred embodiment of a video ring back tone generation device provided by the present invention is shown in FIG. The video ring back tone generation device is applied to a video ring back tone platform and comprises:

[0081] The target video search module 501 is used to search for corresponding video frames and video frame vectors according to the information of the called user when the calling user initiates a call request to the called user;

[0082] A target audio determination module 502, configured to search for a plurality of candidate audios corresponding to the video frame based on the video frame vector, and determine a target audio from the plurality of candidate audios according to the personalized characteristics of the calling user;

[0083] The video ring back tone generation module 503 is used to generate a video ring back tone according to the video frame and the target audio, and send the video ring back tone to the calling user for playing.

[0084] Preferably, the video frame and the video frame vector are stored in a video material library, the video frame is obtained by parsing the video uploaded by the called user, and the video frame vector is predicted by a trained audio and video dual-tower model.

[0085] Preferably, the training method of the audio and video dual-tower model includes:

[0086] Acquire video ring back tone material, parse the video ring back tone material to obtain audio material and video frame material, and use the audio material and the video frame material as training samples of the audio and video dual-tower model; wherein the audio and video dual-tower model includes an audio material tower and a video material tower;

[0087] Input the audio material into the audio material tower to obtain the audio vector output by the audio material tower;

[0088] Input the video frame material into the video material tower to obtain a video frame vector output by the video material tower;

[0089] The cosine distance between the audio vector and the video frame vector is calculated, and the model parameters of the audio and video dual-tower model are optimized using the In-batch softmax objective function and returned to continue training until the training end condition is met to obtain a trained audio and video dual-tower model.

[0090] Preferably, the device is also used for:

[0091] After parsing the video ring back tone material to obtain the audio material and the video frame material, the audio material is stored in an audio material library, and the video frame material is stored in the video material library.

[0092] Preferably, searching for a plurality of candidate audios corresponding to the video frame based on the video frame vector, and determining the target audio from the plurality of candidate audios according to the personalized characteristics of the calling user, comprises:

[0093] Calculate the similarity between the video frame vector and the audio vector stored in the audio Faiss index library, select the audio vector whose similarity is greater than a preset threshold, and obtain multiple candidate audio vectors; wherein the audio Faiss index library stores the audio vector predicted by the audio and video dual tower model;

[0094] Searching for corresponding audio from the audio material library according to the candidate audio vector to obtain a plurality of candidate audios;

[0095] Based on the trained user audio sorting model, multiple candidate audios are sorted according to the personalized characteristics of the calling user, and the target audio is determined from the multiple candidate audios.

[0096] Preferably, the training method of the user audio sorting model includes:

[0097] Constructing a training sample of a binary classification model based on the historical listening data of the calling user, and performing feature extraction; wherein the user-side features include user ID, gender, age, interest tags, and user listening ID sequence; the item-side features include song ID and song audio modality features;

[0098] Vectorizing the user-side features and horizontally splicing them to generate a user input vector;

[0099] Vectorizing the song ID and horizontally concatenating it with the audio modal feature to generate an item input vector;

[0100] The user input vector and the item input vector are simultaneously input into a multi-layer DNN network for feature interaction, and after obtaining the output vector of the DNN network, binary classification is performed through softmax;

[0101] The cross entropy loss function is used to calculate the loss of the binary classification model, and the model parameters are updated in reverse until the training end condition is met to obtain a trained user audio sorting model.

[0102] In the specific implementation, the working principle, control process and technical effect achieved by the video ring back tone generation device provided in the embodiment of the present invention are the same as those of the video ring back tone generation method in the above embodiment, and will not be repeated here.

[0103] See also Figure 6 , Figure 6 6 is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 601, a memory 602, and a computer program stored in the memory 602 and configured to be executed by the processor 601. When the processor 601 executes the computer program, the video ringback tone generation method described in any of the above embodiments is implemented.

[0104] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 602 and executed by the processor 601 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0105] The processor 601 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor, or the processor 601 can be any conventional processor. The processor 601 is the control center of the terminal device, and various parts of the terminal device are connected using various interfaces and lines.

[0106] The memory 602 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 602 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 602 can also be other volatile solid-state storage devices.

[0107] It should be noted that the above terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 6 The structural diagram is only an example of the above-mentioned terminal device and does not constitute a limitation of the above-mentioned terminal device. It may include more or less components than shown in the figure, or combine certain components, or different components.

[0108] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the video ringback tone generation method described in any of the above embodiments.

[0109] An embodiment of the present invention further provides a computer program product, which includes a computer program or a computer instruction. When the computer program or the computer instruction is executed by a processor, the method for generating a video ringback tone described in any of the above embodiments is implemented.

[0110] The embodiments of the present invention provide a method, apparatus, device, medium and computer program product for generating a video ringback tone. When a calling user initiates a call request to a called user, the video ringback tone platform searches for the corresponding video frame and video frame vector according to the information of the called user; searches for multiple candidate audios corresponding to the video frame based on the video frame vector, and determines the target audio from the multiple candidate audios according to the personalized characteristics of the calling user; generates a video ringback tone according to the video frame and the target audio, and sends the video ringback tone to the calling user for playback. The embodiments of the present invention can generate a corresponding video ringback tone according to the personalized characteristics of the calling user for the video material shared by the called user, so as to achieve the effect of playing different audios for different calling users for the video material shared by the called user.

[0111] It should be noted that the system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the system embodiment provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0112] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating a video ring back tone, characterized in that: Applied to video ringback tone platform, including: When the calling user initiates a call request to the called user, the corresponding video frame and video frame vector are searched according to the called user's information; Searching for multiple candidate audios corresponding to the video frame based on the video frame vector, and determining a target audio from the multiple candidate audios according to the personalized characteristics of the calling user; A video ring back tone is generated according to the video frame and the target audio, and the video ring back tone is sent to the calling user for playing.

2. The video ringback tone generation method according to claim 1, characterized in that: The video frame and the video frame vector are stored in a video material library. The video frame is obtained by parsing the video uploaded by the called user, and the video frame vector is predicted by a trained audio and video dual-tower model.

3. The video ringback tone generation method according to claim 2, characterized in that: The training method of the audio and video dual tower model includes: Acquire video ring back tone material, parse the video ring back tone material to obtain audio material and video frame material, and use the audio material and the video frame material as training samples of the audio and video dual-tower model; wherein the audio and video dual-tower model includes an audio material tower and a video material tower; Input the audio material into the audio material tower to obtain the audio vector output by the audio material tower; Input the video frame material into the video material tower to obtain a video frame vector output by the video material tower; The cosine distance between the audio vector and the video frame vector is calculated, and the model parameters of the audio and video dual-tower model are optimized using the In-batch softmax objective function and returned to continue training until the training end condition is met to obtain a trained audio and video dual-tower model.

4. The video ringback tone generation method according to claim 3, characterized in that: The method further comprises: After parsing the video ring back tone material to obtain the audio material and the video frame material, the audio material is stored in an audio material library, and the video frame material is stored in the video material library.

5. The video ringback tone generation method according to claim 4, characterized in that: The searching for a plurality of candidate audios corresponding to the video frame based on the video frame vector, and determining a target audio from the plurality of candidate audios according to the personalized characteristics of the calling user, comprises: Calculate the similarity between the video frame vector and the audio vector stored in the audio Faiss index library, select the audio vector whose similarity is greater than a preset threshold, and obtain multiple candidate audio vectors; wherein the audio Faiss index library stores the audio vector predicted by the audio and video dual tower model; Searching for corresponding audio from the audio material library according to the candidate audio vector to obtain a plurality of candidate audios; Based on the trained user audio sorting model, multiple candidate audios are sorted according to the personalized characteristics of the calling user, and the target audio is determined from the multiple candidate audios.

6. The video ringback tone generation method according to claim 5, characterized in that: The training method of the user audio sorting model includes: Constructing a training sample of a binary classification model based on the historical listening data of the calling user, and performing feature extraction; wherein the user-side features include user ID, gender, age, interest tags, and user listening ID sequence; the item-side features include song ID and song audio modality features; Vectorizing the user-side features and horizontally splicing them to generate a user input vector; Vectorizing the song ID and horizontally concatenating it with the audio modal feature to generate an item input vector; The user input vector and the item input vector are simultaneously input into a multi-layer DNN network for feature interaction, and after obtaining the output vector of the DNN network, binary classification is performed through softmax; The cross entropy loss function is used to calculate the loss of the binary classification model, and the model parameters are updated in reverse until the training end condition is met to obtain a trained user audio sorting model.

7. A video ring back tone generating device, characterized in that: Applied to video ringback tone platform, including: A target video search module, used to search for corresponding video frames and video frame vectors according to the called user's information when the calling user initiates a call request to the called user; A target audio determination module, configured to search for a plurality of candidate audios corresponding to the video frame based on the video frame vector, and determine a target audio from the plurality of candidate audios according to the personalized characteristics of the calling user; The video ring back tone generation module is used to generate a video ring back tone according to the video frame and the target audio, and send the video ring back tone to the calling user for playing.

8. A terminal device, characterized in that: The device comprises a processor and a memory, wherein a computer program is stored in the memory and the computer program is configured to be executed by the processor, and when the processor executes the computer program, the video ringback tone generation method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the video color ring back tone generation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product includes a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, the video ringback tone generation method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Video polyphonic ringtone processing method and device, and computer readable storage medium

    CN112134999A

  • Video polyphonic ringtone synthesis method, device and equipment and computer readable storage medium

    CN113656635A

  • Video polyphonic ringtone setting method, system and device and storage medium

    CN114218429A

  • Method, system and device for identifying customer based on video polyphonic ringtone, and storage medium

    CN114283851A

  • Video polyphonic ringtone generation method and device, equipment and storage medium

    CN115065654A

Cited By

  • Multidimensional collaborative scheduling polyphonic ringtone playing method, system, device and medium

    CN120378532A