Multimedia file generation method and device, equipment and storage medium

Abstract

The invention relates to a multimedia file generation method and device, equipment and a storage medium. The multimedia file generation method comprises the steps that a target video file and a target subtitle file of the target video file are acquired, and the target subtitle file comprises at least one row of subtitle texts; performing sentiment analysis on the at least one row of subtitle text to generate a sentiment analysis result; according to the sentiment analysis result and preference information of a target user who is about to watch the target video file, the at least one line of subtitle text is converted into a target audio file, and the preference information is audio parameter information used for describing an audio demand of the target user; and merging the target audio file and the target video file to generate a target multimedia file. According to the method provided by the invention, the generated multimedia file can meet the personalized requirements of the user to a certain extent, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a multimedia file generation method, device, equipment and storage medium. Background Art

[0002] Nowadays, users consume video content in a variety of ways. In addition to visual subtitles, auxiliary methods such as voice reading / audio playback have become important means to improve user experience. At present, most videos are encapsulated with proprietary audio, which cannot fully meet the personalized needs of users. For example, for users in different language environments or visually impaired users with personalized needs, the proprietary audio configured in the video is very likely to not conform to the user's language environment, resulting in a poor user experience. Therefore, it is urgent to provide a multimedia file generation method that can meet the personalized needs of users. Summary of the invention

[0003] In order to solve the above technical problems, the embodiments of the present disclosure provide a method, apparatus, device and storage medium for generating a multimedia file. The generated multimedia file can meet the personalized needs of users to a certain extent, thereby improving the user experience.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for generating a multimedia file, comprising:

[0005] Acquire a target video file and a target subtitle file of the target video file, wherein the target subtitle file includes at least one line of subtitle text;

[0006] Performing sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result;

[0007] Converting the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file, wherein the preference information is audio parameter information for describing the audio requirements of the target user;

[0008] The target audio file and the target video file are combined and processed to generate a target multimedia file.

[0009] Optionally, converting the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file includes:

[0010] Acquire first preference information of a target user who is to watch the target video file, and / or acquire historically viewed multimedia files of the target user, and determine second preference information of the target user based on the historically viewed multimedia files;

[0011] According to the sentiment analysis result and the first preference information and / or the second preference information, the at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

[0012] Optionally, converting the at least one line of subtitle text into a target audio file that meets the audio requirement of the target user and the emotional tendency of the at least one line of subtitle text according to the emotion analysis result and the first preference information and / or the second preference information includes:

[0013] Constructing an audio parameter set according to at least one audio parameter information respectively included in the first preference information and the second preference information;

[0014] For the same audio parameter in the audio parameter set, comparing a first set priority of the first preference information and a second set priority of the second preference information, and determining the one with a higher priority as the target audio parameter;

[0015] generating target preference information according to the target audio parameter and the remaining audio parameters in the audio parameter set except the same audio parameter;

[0016] According to the emotion analysis result and the target preference information, the at least one line of subtitle text is converted into target audio that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

[0017] Optionally, before converting the at least one line of subtitle text into target audio that meets the audio requirement of the target user and the emotional tendency of the at least one line of subtitle text, the method further includes:

[0018] Adjust the text content of the at least one line of subtitle text according to the sentiment analysis result and the preference information to generate a target subtitle text that conforms to the language style of the target user;

[0019] The converting the at least one line of subtitle text into a target audio file comprises:

[0020] The target subtitle text is converted into a target audio file.

[0021] Optionally, performing sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result includes:

[0022] Based on the target subtitle file and / or video frame file, sentiment analysis is performed on each line of subtitle text to generate a sentiment analysis result, wherein the video frame file refers to at least one frame image sequence corresponding to each line of subtitle text intercepted from the target video file, and the sentiment analysis result includes the emotional tendency expressed by each line of subtitle text.

[0023] Optionally, the target subtitle file further includes first time information corresponding to each line of subtitle text in the target video file, and the merging of the target audio file and the target video file to generate the target multimedia file includes:

[0024] Determine the second time information corresponding to each line of subtitle text in the target audio file;

[0025] Based on the correspondence between the first time information and the second time information for each line of subtitle text, the target audio file and the target video file are merged and processed, and the original audio file of the target video file is muted to generate a target multimedia file.

[0026] Optionally, after converting the at least one line of subtitle text into a target audio file, the method further includes:

[0027] After playing the target audio file, adjusting the preference information, and / or regenerating a sentiment analysis result according to the at least one line of subtitle text;

[0028] The target audio file is updated based on the adjusted preference information and / or the regenerated sentiment analysis result.

[0029] In a second aspect, an embodiment of the present disclosure provides a multimedia file generating device, including:

[0030] An acquisition unit, configured to acquire a target video file and a target subtitle file of the target video file, wherein the target subtitle file includes at least one line of subtitle text;

[0031] A sentiment analysis unit, used to perform sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result;

[0032] a text conversion unit, configured to convert the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file, wherein the preference information is audio parameter information for describing the audio requirements of the target user;

[0033] The merging unit is used to merge the target audio file and the target video file to generate a target multimedia file.

[0034] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0035] Memory;

[0036] Processor; and

[0037] Computer programs;

[0038] The computer program is stored in the memory and is configured to be executed by the processor to implement the multimedia file generation method as described above.

[0039] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the multimedia file generation method as described above.

[0040] The multimedia file generation method provided by the present disclosure includes: obtaining a target video file and a target subtitle file of the target video file, wherein the target subtitle file includes at least one line of subtitle text; performing sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result; converting the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file, wherein the preference information is audio parameter information for describing the audio requirements of the target user; merging the target audio file and the target video file to generate a multimedia file, so as to ensure that the target audio file and the target video file can be played synchronously, thereby meeting the personalized requirements of the user and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of a multimedia file generation method provided by an embodiment of the present disclosure;

[0044] Figure 2 A schematic diagram of the structure of a multimedia file generation system provided by an embodiment of the present disclosure;

[0045] Figure 3 A schematic diagram of the structure of a multimedia file generating device provided in an embodiment of the present disclosure;

[0046] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0049] Specifically, for users with special needs, such as visually impaired users, language learning users, users in different language environments, or users who want to immerse themselves in different language environments, existing speech generation systems cannot fully meet the personalized needs.

[0050] In view of the above technical problems, the disclosed embodiments provide a multimedia file generation method, which uses natural language processing technology to perform sentiment analysis on the text content of the subtitle file, identify the emotions expressed by the text content, obtain sentiment analysis results, and analyze the user's preferred language style based on the user's historical viewing of multimedia files to obtain preference information. The subtitle file is converted into an audio / voice file based on the user's preference information and sentiment analysis results using text-to-speech technology. Finally, the voice file and the video file are merged for synchronous playback to form a new viewing experience, which meets the user's personalized viewing needs. For example, for visually impaired users, by setting the accent, volume, and speech speed to provide a suitable voice reading of video subtitles, the user is ensured to have a better audio and video experience. This is described in detail through one or more of the following embodiments.

[0051] The multimedia file generation method provided by the embodiment of the present disclosure can be applied to the scenario of configuring personalized audio for video to achieve optimal audio and video playback. The method can be executed by a multimedia file generation device, which can be implemented by software and / or hardware, and the device can be integrated in an electronic device. Among them, the electronic device can include but is not limited to mobile terminals such as smart phones, laptops, digital broadcast receivers, personal digital assistants (Personal Digital Assistant, PDA), tablet computers (Tablet Personal Computer, Tablet PC), PMP (portable multimedia player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), wearable devices, etc., and fixed terminals such as digital televisions, desktop computers, smart home devices, etc.

[0052] Specifically, the multimedia file generation method can be executed by a terminal or a server. Specifically, the terminal or the server can convert the subtitle file into a personalized voice file to achieve synchronous playback of the video file and the personalized voice file.

[0053] Figure 1 A flowchart of a multimedia file generation method provided by an embodiment of the present disclosure specifically includes the following steps: Figure 1 Steps shown:

[0054] S101, obtaining a target video file and a target subtitle file of the target video file.

[0055] The target subtitle file includes at least one line of subtitle text.

[0056] Understandably, the target video file is obtained, and the target video file refers to the relevant file of the target video to be watched by the target user. The target video file includes the target video and the original voice, or the target video file may not include the original voice, that is, the target video file may be a pure video without voice, and the target video file may be uploaded through the user interface or the target video may be directly selected and the relevant files may be obtained. Subsequently, the target subtitle file of the target video file is obtained, and the target subtitle file may be the original subtitle file of the target video file, and the original subtitle file may be uploaded through the user interface or the subtitle file associated with the selected target video may be directly obtained, wherein the original subtitle file may be a subtitle file in text format (SubRip Text, SRT, or Web Video Text Tracks, VTT), and other possible formats are not limited here. Alternatively, the target subtitle file may also be an original subtitle file associated with other videos, that is, an audio file generated based on any original subtitle file may be configured for the target video, that is, the relevance between the target subtitle file and the target video file is not limited, and the source of the subtitle file is not limited, and it can be selected according to the personalized needs of the user. For example, the original subtitle file of video A (other video) is used as the target subtitle file of video B (target video), a target audio file is generated based on the original subtitle file of video A, and the target video file and the target video file are merged to obtain a multimedia file that meets the user's personalized needs.

[0057] A possible application scenario is that for users with language learning needs, target audio (also called "personalized voice") read in the language to be learned is generated based on the original subtitle file associated with the target video. The personalized voice can also be compared with the original subtitle file, effectively improving the user's learning experience.

[0058] Optionally, after obtaining the target subtitle file, determine the text format of the target subtitle file and the content to be parsed, wherein the content to be parsed includes a timestamp and text content; perform parsing processing corresponding to the text format on the target subtitle file, extract the timestamp and text content of each line of subtitles, and obtain a processed subtitle file, wherein the processed subtitle file includes at least one line of subtitle text and the first time information corresponding to each line of subtitle text in the target video file.

[0059] It is understandable that the target subtitle file includes multiple lines of subtitles, and the text content of each line of subtitles is parsed to obtain the subtitle text, and the timestamp of the line of subtitle text in the target video is determined to obtain the first time information. The first time information represents the correspondence between the subtitle text and the target video, that is, it indicates the time period when the subtitle text appears in the target video. For example, the text content of subtitle text 1 is "AAAAA", and the time period when subtitle text 1 appears in the target video is 00:01:23 to 00:01:30. It is understandable that the processed subtitle file (hereinafter referred to as the subtitle file) can be in the form of a subtitle object list, and each subtitle object in the list represents a subtitle entry, and the subtitle entry contains the start time, end time and text content (subtitle text) of the subtitle.

[0060] S102: Perform sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result.

[0061] It is understandable that, based on the above S101, sentiment analysis is performed on each line of subtitle text and / or multiple lines of subtitle text to generate sentiment analysis results, wherein sentiment analysis refers to analyzing the emotional tone expressed by the text content in the subtitle text, for example, emotional types / emotional types such as happy, angry, sad and bland. In one possible situation, a single line of subtitle text can fully and clearly express a certain emotion, or multiple lines of subtitle text can fully and clearly express a certain emotion after context analysis, that is, when performing sentiment analysis, a certain emotion type to be expressed can be determined by a single line of subtitle text, or a certain emotion type to be expressed can be determined by multiple lines of subtitle text. The number of subtitle text lines targeted by the specific sentiment analysis is not limited and can be determined according to the specific text content. Among them, the sentiment analysis results include the emotion type corresponding to the single line of subtitle text and / or the emotion type corresponding to the multiple lines of subtitle text. Specifically, a pre-trained sentiment analysis model can be used to perform sentiment classification on the subtitle text to extract the emotional information of at least one paragraph / line of subtitle text.

[0062] Optionally, performing sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result may be specifically implemented through the following steps:

[0063] Based on the target subtitle file and / or video frame file, sentiment analysis is performed on each line of subtitle text to generate a sentiment analysis result, wherein the video frame file refers to at least one frame image sequence corresponding to each line of subtitle text intercepted from the target video file, and the sentiment analysis result includes the emotional tendency expressed by each line of subtitle text.

[0064] It is understandable that the first emotional tendency can be determined by performing sentiment analysis / classification based on the text content of the single-line subtitle text and / or multiple-line subtitle text included in the target subtitle file, and the second emotional tendency can be determined by performing sentiment analysis based on the video frame file corresponding to the single-line subtitle text and / or multiple-line subtitle text in the target video, wherein the video frame file refers to at least one frame image sequence corresponding to each line of subtitle text intercepted from the target video file. It is understandable that each line of subtitle text has at least one corresponding frame image, and at least one frame image constitutes a video frame file. Based on the above example, all video frames involved in the time period in which the subtitle text 1 appears in the target video are video frame files. The video frame file can be a video composed of a series of continuous image sequences, or it can be a multi-frame image. By performing sentiment analysis on the multi-frame image or a video, the emotional tendency expressed by the subtitle text 1 is determined, wherein the emotional tendency refers to the emotion or attitude expressed in the text or language, reflecting the text's feelings about a certain subject or object, and the emotional tendency can be positive, negative or neutral. Subsequently, the first emotional tendency and the second emotional tendency can be comprehensively judged to determine the emotional tendency finally expressed by the subtitle text. For example, the same emotional tendency is determined as the final emotional tendency, or the final emotional tendency is determined according to the priority of each method. The comprehensive judgment method is not limited.

[0065] S103: Convert the at least one line of subtitle text into a target audio file according to the sentiment analysis result and the preference information of the target user who is to watch the target video file.

[0066] The preference information is audio parameter information used to describe the audio requirements of the target user.

[0067] The preference information includes at least one audio parameter information of accent, speaking speed, intonation, sound quality, pitch and loudness.

[0068] It is understandable that, based on the above S102, the preference information of the target user of the target video file to be watched is obtained. The preference information refers to the audio parameter information that can describe the user's audio needs. The audio parameter information can be information such as accent, speaking speed, intonation, sound quality, pitch and loudness. That is to say, the preference information represents the user's current audio needs or the current favorite audio style. The preference information reflects the user's personalized needs for audio. For example, the audio demand is normal speaking speed, etc., and the corresponding preference information includes the speaking speed set to 1. Subsequently, according to the emotional analysis results and preference information, at least one line of subtitle text is converted into a personalized voice file that meets the current personalized needs (current audio needs) of the target user, that is, the subtitle text is converted into a target audio file with an audio style and emotional tendency that meets the target user's preferences. This kind of audio synthesis parameters are customized according to the user's historical viewing records and preferences, so as to generate audio that meets the user's personal preferences. This personalized customization greatly improves the user experience, enabling it to obtain an audio and video viewing experience that is closer to personal needs and preferences.

[0069] Optionally, the converting of the at least one line of subtitle text into a target audio file according to the sentiment analysis result and the preference information of the target user to watch the target video file may be specifically implemented by the following steps:

[0070] Acquire first preference information of a target user who is to watch the target video file, and / or acquire historically watched multimedia files of the target user, and determine second preference information of the target user based on the historically watched multimedia files; and convert the at least one line of subtitle text into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text according to the sentiment analysis result and the first preference information and / or the second preference information.

[0071] It is understandable that a database for storing user preference information is pre-built. If the pre-built database stores the preference information of the target user, the first preference information can be directly obtained from the database, wherein the preference information stored in the database is marked by a user identifier for quick location and acquisition, for example, the user identifier is user_id. It is understandable that the user's preference information can be updated according to the historical viewing record or the method set by the user. Alternatively, the target user's historical viewing multimedia files can also be obtained, and the historical viewing multimedia files can be analyzed by natural language processing technology to extract the target user's language style and preference information, etc., to obtain the second preference information. Among them, the historical viewing multimedia files include historical viewing videos, voice files and subtitle files of historical viewing videos, and the voice files can be the original voice files and / or personalized voice files of historical viewing videos. Specifically, the historical viewing multimedia files can be segmented and pre-processed to extract the user's favorite accent, speech speed, language habits and other preference information, and other methods for extracting preference information from historical viewing multimedia files are not limited.

[0072] It can be understood that after determining the first preference information and / or the second preference information, at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user based on the sentiment analysis results and the first preference information and / or the second preference information, that is, at least one preference information can be selected from the first preference information and the second preference information for generating the target audio file, or multiple audio files can be generated based on the combination of the first preference information and the second preference information, and the final target audio file can be determined among the multiple audio files.

[0073] Optionally, the converting of the at least one line of subtitle text into a target audio file having an audio requirement and an emotional tendency of the at least one line of subtitle text according to the emotional analysis result and the first preference information and / or the second preference information can be specifically implemented by the following steps:

[0074] An audio parameter set is constructed based on at least one audio parameter information respectively included in the first preference information and the second preference information; for the same audio parameters in the audio parameter set, a first set priority of the first preference information and a second set priority of the second preference information are compared, and the one with a higher priority is determined as the target audio parameter; target preference information is generated based on the target audio parameter and the remaining audio parameters in the audio parameter set except the same audio parameter; and based on the emotion analysis result and the target preference information, the at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

[0075] It can be understood that when the first preference information and the second preference information are selected for generating the target audio file, an audio parameter set is constructed according to at least one audio parameter information respectively included in the first preference information and the second preference information, that is, the audio parameter information included in all the preference information is integrated (for example, taking the union) to obtain the audio parameter set; then, the same audio parameters with the same specific parameter information and the remaining audio parameters with different specific parameter information in the audio parameter set are counted, and each audio parameter has specific parameter information, and the parameter information may be a parameter value, a parameter type, or a parameter identifier, etc. For example, the specific parameter information for the audio parameter of accent in the first preference information and the second preference information is both "AA", so the accent can be determined as the same audio parameter. For another example, the speaking rate in the first preference information is "1.5" and the speaking rate in the second preference information is "1.0", and the speaking rate can be determined as the remaining audio parameters. For the same audio parameters, the first set priority of the first preference information and the second set priority of the second preference information are obtained, and the specific parameter information of the same audio parameters included in the preference information with higher priority is used as the target audio parameter. The priority of the preference information obtained in different ways (for example, directly obtained from the database, or obtained based on historical multimedia files) can be determined according to user needs and is not limited here. After determining the target audio parameters, the target preference information is generated based on the target audio parameters and the remaining audio parameters. Subsequently, based on the sentiment analysis results and the target preference information, the subtitle text is converted into a personalized voice file / target audio file that meets the audio needs of the target user.

[0076] Optionally, before converting the at least one line of subtitle text into a target audio file, the method further includes:

[0077] The text content of the at least one line of subtitle text is adjusted according to the sentiment analysis result and the preference information to generate a target subtitle text that conforms to the language style of the target user.

[0078] It is understandable that before converting the subtitle text to the target audio, the subtitle text extracted from the subtitle file is further optimized, and the language style and text content of the subtitle text are adjusted to match the personalized audio needs of the user. Specifically, the subtitle text is reconstructed according to the user's preference information and the sentiment analysis results, and a personalized subtitle text (also called a target subtitle text) that conforms to the user's language style is generated. For example, the text content of subtitle text 2 is "BBBB", the sentiment tendency of subtitle text 2 is happy, and the accent in the preference information is C accent. Based on the above-mentioned limiting conditions, the text content of subtitle text 2 is adjusted to "ABBBBC", where "A" can be a word emphasizing happiness, and "C" can be an interjection, etc., and the subtitle text is optimized to facilitate users to better and more intuitively experience the video content based on voice. A possible implementation method is to use a pre-trained language model to reconstruct the subtitle text according to the user's preference information and sentiment analysis results, and the input prompt of the language model can be set to "interpret the subtitle text with the sentiment analysis results", and the model output results are adjusted according to the preference information to generate a personalized text that conforms to the sentiment expressed by the subtitle and the personalized audio needs of the user. Other methods for optimizing subtitle text are not limited. This method of optimizing and reconstructing subtitle text, that is, generating more natural, fluent and user-expected text content based on the user's language preference information and sentiment analysis results, greatly improves the quality and personalization of the subtitle text compared to the original subtitles, making the generated audio more natural and realistic.

[0079] It is understandable that the language style can be quantitatively analyzed to a certain extent. Specifically, the historical sentences of the target users are obtained, and the language features such as vocabulary features, syntactic structures, and grammatical features in the historical sentences are counted. The language features are analyzed through a pre-built language style model to determine a more comprehensive and accurate language style of the target users. Other methods of determining the language style are not limited here.

[0080] Optionally, the above-mentioned converting the at least one line of subtitle text into a target audio file can be specifically implemented by the following steps:

[0081] The target subtitle text is converted into a target audio file.

[0082] It is understandable that after completing the generation of personalized subtitle content, the personalized subtitle text is converted into a personalized voice file that meets the personalized audio needs of the target user, that is, based on the optimized subtitle text, a suitable text-to-speech engine and conversion parameters are selected to generate personalized voice audio with emotional expression. Specifically, the subtitle text can be converted into a voice file using a speech synthesis service.

[0083] Optionally, after converting the at least one line of subtitle text into a target audio file, the method further includes:

[0084] After playing the target audio file, the preference information is adjusted, and / or the sentiment analysis result is regenerated according to the at least one line of subtitle text; and the target audio file is updated based on the adjusted preference information and / or the regenerated sentiment analysis result.

[0085] Understandably, after generating a personalized voice file, the personalized voice is played, or the audition voice of the personalized voice file is played to obtain adjustment information for the relevant parameters of the personalized voice file. The user can adjust the preference information such as voice type, accent and speaking speed, for example, adjust the voice type of a young boy to the voice type of a middle-aged boy, etc., and can also adjust the emotional tendency of each subtitle text included in the emotional analysis result, for example, adjust the emotional tendency of subtitle text 3 from happy to normal, that is, the emotional tendency of a single-line subtitle, or adjust the text content of the target subtitle text, etc. Other adjustment methods for the emotional analysis results, preference information and target subtitle text are not limited. After obtaining any of the above adjustment information, the subtitle text is converted into a personalized voice file again according to the adjustment information.

[0086] S104: Merge the target audio file and the target video file to generate a target multimedia file.

[0087] It is understandable that, based on the above S103, the generated target audio file and target video file are merged and processed to generate a target multimedia file to ensure that the personalized voice and target video can be played synchronously. Specifically, the audio file and the video file can be merged through a multimedia processing tool, and a multimedia player can be used to achieve smooth real-time playback of audio and video. Among them, a multimedia file refers to a file containing two or more different types of media content, which usually integrates elements such as audio, video, images, and text to form a richer and more interactive experience. This embodiment uses natural language processing and sentiment analysis technology to perform sentiment classification on the subtitle text, and integrates the sentiment information into the audio synthesis process. The generated audio not only meets the user's preferences, but also accurately expresses the original emotional tendency of the video, such as anger, joy, and sadness. This kind of emotionally rich audio and video synthesis enables users to better understand and feel the emotional level of the video content.

[0088] The subtitle file further includes first time information corresponding to each line of subtitle text in the target video file.

[0089] Optionally, the above-mentioned merging of the target audio file and the target video file to generate a target multimedia file can be specifically implemented through the following steps:

[0090] Determine the second time information corresponding to each line of subtitle text in the target audio file; based on the correspondence between the first time information and the second time information for each line of subtitle text, merge the target audio file and the target video file, and mute the original audio file of the target video file.

[0091] Understandably, determine the second time information corresponding to the subtitle text in the target audio file, the second time information represents the specific time of the audio corresponding to the subtitle text in the entire target audio, that is, the audio time in the target audio file has a corresponding relationship with the subtitle text. There is a temporal correspondence between the subtitle text and the target video, and there is also a temporal correspondence between the subtitle text and the target audio. According to the temporal correspondence included in the first time information and the second time information, the target audio and the target video are merged and processed to ensure that the obtained audio and video (multimedia file) are played synchronously. For example, subtitle 1 appears at 0.5s in the target video, and subtitle 1 appears at 0.3s in the target audio. Taking subtitle 1 as a benchmark, 0.5s in the video and 0.3s in the audio are corresponding, and the video and audio are directly merged according to the temporal correspondence. In the merging process, if there is original voice in the target video, the original voice is muted to ensure that the target video and the target audio are played synchronously.

[0092] It is understandable that the synchronous playback of the target video and the target audio means that the target audio is played while the target video is played. The subtitles displayed in the target video and the subtitles read by the target audio may correspond one to one or may not correspond. A possible application scenario is that the target video displays the text content of subtitle 1, and the target audio reads the text content of subtitle 2 at the same time, wherein the text content of subtitle 1 and subtitle 2 are different, and subtitle 1 is displayed before subtitle 2, that is, the subtitles displayed in the target video and the subtitles read by the target audio are not synchronized, that is, there may be a delay between the subtitles displayed in the target video and the subtitles read in the target audio. For example, for the personalized needs of language learning, the text content of subtitle 2 can be read first in the target audio, and the text content of subtitle 2 can be displayed in the target audio after a certain period of time to increase the user's interest in learning.

[0093] A multimedia file generation method provided by an embodiment of the present disclosure performs sentiment analysis on acquired subtitle texts to determine the sentiment tendency represented by each line of subtitle texts, so as to facilitate the subsequent generation of personalized texts and personalized voices with accurate sentiment expression; at the same time, based on the user's preference information and sentiment analysis results, the subtitle texts are optimized to better match the user's preference information such as accent and language style; subsequently, the optimized subtitle texts are converted into personalized voices with a speech speed, sound type and emotion type that meet the user's requirements, and the personalized voices and videos are played synchronously. This audio and video generation method can fully meet the diverse needs of users, and the personalized voices dynamically generated according to the preference information such as accent or language style selected by the user, or based on the user's viewing history and usage habits, effectively improve the user's audio-visual experience, thereby increasing the interest in watching videos.

[0094] A possible embodiment is applied to a multimedia file generation device, which is suitable for personalized multimedia file playback scenarios. Specifically, the target video selected by the user and the target subtitles uploaded by the user are obtained, where the target video refers to a pure video without voice, and the target subtitles are the original subtitles of the target video; sentiment analysis is performed on the text content in the target subtitles to identify the emotional tone of the text content, such as happiness, anger, sadness, etc.; the user's preferred language style, such as speaking speed, timbre, accent, etc., is analyzed based on the user's viewing history and the use record of the audio and video player; the text-to-speech technology is used to convert the user's preferred language style, voice, etc. into speech. The target subtitles are converted into target audio based on the set voice style and sentiment analysis results of the subtitles. In the language learning scenario, the target audio is represented in the form of the language to be learned (language B) and is in line with the user's current learning situation, such as slow playback in standard tone, etc.; finally, the target audio after the language style conversion is played together with the target video to form a new viewing experience. When the user watches the audio and video, the target video displays the original subtitles (language A) and the target audio plays language B to achieve the comparison of audio and subtitles, thereby completing the generation of user-personalized audio and video, fully meeting the user's language learning needs, and also improving the user's learning interest.

[0095] Based on the above embodiments, Figure 2The present invention is a structural schematic diagram of a multimedia file generation system provided in an embodiment of the present invention. The multimedia file generation system includes a subtitle reading module, a user preference and historical data storage module, a historical analysis and sentiment analysis module, a natural language processing and optimization module, a personalized speech synthesis module and a real-time output module, wherein the subtitle reading module is used to read the timestamp and text content of each line of subtitles in the subtitle file, the user preference and historical data storage module is used to store the user's preference information and historical viewing video files, the historical analysis and sentiment analysis module is used to extract preference information from historical viewing video files and perform sentiment analysis on each line of subtitle text, the natural language processing and optimization module is used to further optimize the extracted subtitle text according to the sentiment analysis results and preference information, the personalized speech synthesis module is used to generate personalized speech with emotional expression and meeting personalized needs based on the optimized subtitle text, and the real-time output module is used to play the generated personalized speech and video synchronously.

[0096] It is understandable that the specific implementation process of each module involved in the multimedia file generation system can be found in the above embodiment and will not be described in detail here.

[0097] Figure 3 The structure diagram of the multimedia file generation device provided by the embodiment of the present disclosure is as follows. The multimedia file generation device provided by the embodiment of the present disclosure can execute the processing flow provided by the multimedia file generation method embodiment, such as Figure 3 As shown, the multimedia file generating device 300 includes an acquiring unit 301, a sentiment analyzing unit 302, a text converting unit 303 and a merging unit 304, wherein:

[0098] An acquisition unit 301 is used to acquire a target video file and a target subtitle file of the target video file, wherein the target subtitle file includes at least one line of subtitle text;

[0099] The sentiment analysis unit 302 is used to perform sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result;

[0100] A text conversion unit 303 is used to convert the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file, wherein the preference information is audio parameter information for describing the audio requirement of the target user;

[0101] The merging unit 304 is used to merge the target audio file and the target video file to generate a target multimedia file.

[0102] Optionally, the text conversion unit 303 is used to:

[0103] Acquire first preference information of a target user who is to watch the target video file, and / or acquire historically viewed multimedia files of the target user, and determine second preference information of the target user based on the historically viewed multimedia files;

[0104] According to the sentiment analysis result and the first preference information and / or the second preference information, the at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

[0105] Optionally, the text conversion unit 303 is used to:

[0106] Constructing an audio parameter set according to at least one audio parameter information respectively included in the first preference information and the second preference information;

[0107] For the same audio parameter in the audio parameter set, comparing a first set priority of the first preference information and a second set priority of the second preference information, and determining the one with a higher priority as the target audio parameter;

[0108] generating target preference information according to the target audio parameter and the remaining audio parameters in the audio parameter set except the same audio parameter;

[0109] According to the emotion analysis result and the target preference information, the at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

[0110] Optionally, the multimedia file generating device 300 is further used for:

[0111] The text content of the at least one line of subtitle text is adjusted according to the sentiment analysis result and the preference information of the target user who is to watch the target video, so as to generate a target subtitle text that conforms to the language style of the target user.

[0112] Optionally, the text conversion unit 303 is used to:

[0113] The target subtitle text is converted into a target audio file.

[0114] Optionally, the sentiment analysis unit 302 is used to:

[0115] Based on the target subtitle file and / or video frame file, sentiment analysis is performed on each line of subtitle text to generate a sentiment analysis result, wherein the video frame file refers to at least one frame image sequence corresponding to each line of subtitle text intercepted from the target video file, and the sentiment analysis result includes the emotional tendency expressed by each line of subtitle text.

[0116] Optionally, the merging unit 304 is used for:

[0117] Determine the second time information corresponding to each line of subtitle text in the target audio file;

[0118] Based on the correspondence between the first time information and the second time information for each line of subtitle text, the target audio file and the target video file are merged and processed, and the original audio file of the target video file is muted to generate a target multimedia file.

[0119] Optionally, the multimedia file generating device 300 is further used for:

[0120] After playing the target audio file, adjusting the preference information, and / or regenerating a sentiment analysis result according to the at least one line of subtitle text;

[0121] The target audio file is updated based on the adjusted preference information and / or the regenerated sentiment analysis result.

[0122] Figure 3 The audio and video generation device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0123] Figure 4 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 4 , which shows a schematic diagram of the structure of an electronic device 400 suitable for implementing the embodiment of the present disclosure. The electronic device 400 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle terminals (such as vehicle navigation terminals), wearable electronic devices, etc., and fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 4 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Figure 4As shown, the electronic device 400 may include a processing device 401 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403 to implement the multimedia file generation method of the embodiment described in the present disclosure. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0125] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0126] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart, thereby implementing the multimedia file generation method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0127] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0128] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0129] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0130] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0131] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0132] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0133] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0134] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0135] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or gateway that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or gateway. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or gateway that includes the elements.

[0137] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a multimedia file, characterized in that: include: Acquire a target video file and a target subtitle file of the target video file, wherein the target subtitle file includes at least one line of subtitle text; Performing sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result; Converting the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file, wherein the preference information is audio parameter information for describing the audio requirements of the target user; The target audio file and the target video file are combined and processed to generate a target multimedia file.

2. The method according to claim 1, characterized in that The converting the at least one line of subtitle text into a target audio file according to the sentiment analysis result and the preference information of the target user to watch the target video file comprises: Acquire first preference information of a target user who is to watch the target video file, and / or acquire historically viewed multimedia files of the target user, and determine second preference information of the target user based on the historically viewed multimedia files; According to the sentiment analysis result and the first preference information and / or the second preference information, the at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

3. The method according to claim 2, characterized in that The converting, according to the emotion analysis result and the first preference information and / or the second preference information, the at least one line of subtitle text into a target audio file that meets the audio requirement of the target user and the emotional tendency of the at least one line of subtitle text comprises: Constructing an audio parameter set according to at least one audio parameter information respectively included in the first preference information and the second preference information; For the same audio parameter in the audio parameter set, comparing a first set priority of the first preference information and a second set priority of the second preference information, and determining the one with a higher priority as the target audio parameter; generating target preference information according to the target audio parameter and the remaining audio parameters in the audio parameter set except the same audio parameter; According to the emotion analysis result and the target preference information, the at least one line of subtitle text is converted into a target audio file that meets the audio requirements of the target user and the emotional tendency of the at least one line of subtitle text.

4. The method according to claim 1, characterized in that Before converting the at least one line of subtitle text into a target audio file, the method further comprises: Adjust the text content of the at least one line of subtitle text according to the sentiment analysis result and the preference information to generate a target subtitle text that conforms to the language style of the target user; The converting the at least one line of subtitle text into a target audio file comprises: The target subtitle text is converted into a target audio file.

5. The method according to claim 1, characterized in that The performing sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result includes: Based on the target subtitle file and / or video frame file, sentiment analysis is performed on each line of subtitle text to generate a sentiment analysis result, wherein the video frame file refers to at least one frame image sequence corresponding to each line of subtitle text intercepted from the target video file, and the sentiment analysis result includes the emotional tendency expressed by each line of subtitle text.

6. The method according to claim 1, characterized in that The target subtitle file also includes first time information corresponding to each line of subtitle text in the target video file, and the merging of the target audio file and the target video file to generate a target multimedia file includes: Determine the second time information corresponding to each line of subtitle text in the target audio file; Based on the correspondence between the first time information and the second time information for each line of subtitle text, the target audio file and the target video file are merged and processed, and the original audio file of the target video file is muted to generate a target multimedia file.

7. The method according to claim 1, characterized in that After converting the at least one line of subtitle text into a target audio file, the method further includes: After playing the target audio file, adjusting the preference information, and / or regenerating a sentiment analysis result according to the at least one line of subtitle text; The target audio file is updated based on the adjusted preference information and / or the regenerated sentiment analysis result.

8. A multimedia file generating device, characterized in that: include: An acquisition unit, configured to acquire a target video file and a target subtitle file of the target video file, wherein the target subtitle file includes at least one line of subtitle text; A sentiment analysis unit, used to perform sentiment analysis on the at least one line of subtitle text to generate a sentiment analysis result; A text conversion unit, configured to convert the at least one line of subtitle text into a target audio file according to the sentiment analysis result and preference information of a target user to watch the target video file, wherein the preference information is audio parameter information for describing the audio requirements of the target user; The merging unit is used to merge the target audio file and the target video file to generate a target multimedia file.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the multimedia file generation method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimedia file generation method according to any one of claims 1 to 7 are implemented.

Images