Subtitle processing method, device, electronic device and storage medium for multimedia files

Abstract

The present application provides a method, device, electronic device, computer-readable storage medium and computer program product for processing subtitles of multimedia files; the method comprises: in response to a play trigger operation, playing a multimedia file, wherein the multimedia file is associated with multiple subtitles, and the types of the multimedia files include video files and audio files, and in the process of playing the multimedia file, sequentially displaying the multiple subtitles in a human-computer interaction interface, wherein the styles of the multiple subtitle applications are related to the content of the multimedia file. Through the present application, the expression form of subtitles can be enriched, and the information dissemination effect and the user's viewing experience can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a method, device, electronic device, computer-readable storage medium and computer program product for processing subtitles of multimedia files. Background Art

[0002] With the development of Internet technology, especially the development of Internet applications such as online video, online music, and online education, multimedia files (such as various types of video files or audio files) as information carriers are widely used, making information more quickly and conveniently presented to people. Among them, subtitles play an indispensable role. In addition to prompting the dialogue of characters, subtitles can also describe, summarize or conclude the content of video files or audio files. For example, when users watch some foreign video files, they need subtitles to help them understand the content of the video files.

[0003] However, in the solutions provided by the related art, during the playback of video files or audio files, the display style of subtitles is single, which cannot adapt to the needs of personalized presentation of information on the Internet, affecting the information dissemination effect and user experience. Summary of the invention

[0004] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for processing subtitles of multimedia files, which can enrich the presentation of subtitles, improve the information dissemination effect and the user's viewing experience.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a method for processing subtitles of a multimedia file, including:

[0007] In response to a play trigger operation, a multimedia file is played, wherein the multimedia file is associated with a plurality of subtitles, and the types of the multimedia file include video files and audio files, and

[0008] During the playing of the multimedia file, the multiple subtitles are displayed in sequence in the human-computer interaction interface, wherein the styles of the multiple subtitles are related to the content of the multimedia file.

[0009] In the above scheme, the display module is also used to display the multiple subtitles, all of which have the style applied, in sequence in the human-computer interaction interface; wherein the style is adapted to the content features of at least one dimension of the multimedia file, and the content features of at least one dimension include: style, object, scene, plot, and tone.

[0010] In the above scheme, the device also includes an acquisition module for acquiring content features of at least one dimension of the multimedia file; the device also includes a conversion module for performing style conversion processing on multiple original subtitles associated with the multimedia file based on the content features of the at least one dimension to obtain multiple new subtitles; wherein the multiple new subtitles are used as the multiple subtitles to be displayed in the human-computer interaction interface.

[0011] In the above scheme, the conversion module is also used to call the subtitle model based on the value corresponding to the content feature of the at least one dimension and the multiple original subtitles associated with the multimedia file to obtain multiple new subtitles; wherein the subtitle model is used as a generative model and is trained by forming a generative adversarial network with the discriminative model.

[0012] In the above scheme, the multimedia file includes multiple fragments, and the types of the fragments include at least one of the following: object fragments, scene fragments, and plot fragments; the display module is also used to perform the following processing during the playback of each of the fragments of the multimedia file: based on a style adapted to the content characteristics of at least one dimension of the fragment, at least one subtitle associated with the fragment is displayed in sequence in the human-computer interaction interface.

[0013] In the above scheme, the acquisition module is also used to acquire the content features of the static dimension of the segment, wherein the content features of the static dimension of the object segment include at least one of the following object attributes of the sound-emitting object in the object segment: role type, gender, age; the features of the static dimension of the scene segment include the scene type of the scene segment; the features of the static dimension of the plot segment include the plot progress of the plot segment; the display module is also used to synchronously display at least one subtitle associated with the segment in the human-computer interaction interface based on a style adapted to the content features of the static dimension of the segment, wherein the style remains unchanged during the playback of the segment.

[0014] In the above scheme, the segment includes multiple sub-segments, and the multiple sub-segments have content characteristics of the static dimension of the segment and content characteristics of the dynamic dimension of the segment, and different sub-segments have different content characteristics of the dynamic dimension; the display module is also used to perform the following processing during the playback of each sub-segment of the segment: based on a style adapted to the content characteristics of the static dimension and the content characteristics of the dynamic dimension of the sub-segment, display at least one subtitle associated with the sub-segment.

[0015] In the above scheme, the content features of the static dimension of the object segment include at least one of the following object attributes: the role type, gender, and age of the sound-making object in the object segment; the content features of the dynamic dimension of the object segment include the following object attributes: the emotion of the sound-making object in the object segment; the content features of the static dimension of the plot segment include the plot type of the plot segment, and the content features of the dynamic dimension of the plot segment include at least one of the following: the scene types of different scenes in which the plot segment appears, and the object attributes of different sound-making objects that appear in the plot segment; the content features of the static dimension of the scene segment include: the type of scene involved in the scene segment; the content features of the dynamic dimension of the scene segment include at least one of the following: the object attributes of different sound-making objects that appear in the scene segment, and the types of different plots that appear in the scene segment.

[0016] In the above scheme, when the at least one dimension is multiple dimensions, the device also includes a fusion module, which is used to fuse the content features of multiple dimensions of the fragment to obtain fused content features; the conversion module is also used to perform style conversion processing on at least one original subtitle associated with the fragment based on the fused content features to obtain at least one new subtitle; wherein the at least one new subtitle is used as the at least one subtitle to be displayed in the human-computer interaction interface.

[0017] In the above scheme, the acquisition module is also used to call the content feature recognition model to perform content feature recognition processing on the content of the multimedia file to obtain content features of at least one dimension of the multimedia file; wherein the content feature recognition model is trained based on sample multimedia files and labels annotated for the content of the sample multimedia files.

[0018] In the above scheme, when the multimedia file is the video file, the acquisition module is also used to perform the following processing on the target object appearing in the video file: preprocessing the target video frame where the target object is located; extracting features from the preprocessed target video frame to obtain image features corresponding to the target video frame; performing dimensionality reduction processing on the image features, and classifying the image features after dimensionality reduction processing through a trained classifier to obtain object attributes of the target object.

[0019] In the above scheme, when the multimedia file is the video file, the acquisition module is also used to perform the following processing on the target object appearing in the video file: extract the local binary pattern features corresponding to the target video frame where the target object is located, and perform dimensionality reduction processing on the local binary pattern features; extract the directional gradient histogram features corresponding to the target video frame, and perform dimensionality reduction processing on the directional gradient histogram features; perform canonical correlation analysis processing on the local binary pattern features and the directional gradient histogram image after dimensionality reduction processing to obtain analysis results; perform regression processing on the analysis results to obtain object attributes of the target object.

[0020] In the above scheme, when the multimedia file is the video file, the acquisition module is also used to perform the following processing on the target object appearing in the video file: normalize the target video frame where the target object is located, and partition the target video frame after normalization to obtain multiple sub-regions; extract the local binary pattern features corresponding to each of the sub-regions, and perform statistical processing on the multiple local binary pattern features to obtain the local histogram statistical features corresponding to the target video frame; perform local sparse reconstruction representation of the local histogram statistical features through the local feature library of the training set, and perform local reconstruction residual weighted recognition processing on the local sparse reconstruction representation result to obtain the object attributes of the target object.

[0021] In the above scheme, when multiple objects appear in the video file, the device also includes a determination module, which is used to determine the target object from the multiple objects in any one of the following ways: determining the object that appears the longest time in the video file as the target object; determining the object that meets the user's preference in the video file as the target object; determining the object related to user interaction in the video file as the target object.

[0022] The present application provides a subtitle processing device for multimedia files, including:

[0023] A playing module, used for playing a multimedia file in response to a play trigger operation, wherein the multimedia file is associated with a plurality of subtitles, and the types of the multimedia file include video files and audio files;

[0024] The display module is used to display the multiple subtitles in sequence in a human-computer interaction interface during the playback of the multimedia file, wherein the styles of the multiple subtitles are related to the content of the multimedia file.

[0025] An embodiment of the present application provides an electronic device, including:

[0026] A memory for storing executable instructions;

[0027] The processor is used to implement the subtitle processing method of the multimedia file provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0028] The embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the subtitle processing method for multimedia files provided by the embodiment of the present application.

[0029] The embodiment of the present application provides a computer program product, including a computer program or instructions, which is used to cause a processor to execute and implement the subtitle processing method of a multimedia file provided by the embodiment of the present application.

[0030] The embodiments of the present application have the following beneficial effects:

[0031] During the playback of multimedia files, subtitles of a style related to the content of the multimedia files are displayed in the human-computer interaction interface. By enriching the expression form of the subtitles, diversified display effects of information related to the multimedia files are achieved. This can be applicable to diversified subtitle display needs in different application scenarios of multimedia files, thereby improving the efficiency and effect of information display and also improving the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 1 is a schematic diagram of the architecture of a subtitle processing system 100 for multimedia files provided in an embodiment of the present application;

[0033] Figure 2 is a schematic diagram of the structure of a terminal device 400 provided in an embodiment of the present application;

[0034] Figure 3 It is a flowchart of a subtitle processing method for multimedia files provided by an embodiment of the present application;

[0035] Figure 4A It is a schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application;

[0036] Figure 4B It is a schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application;

[0037] Figure 4C It is a schematic diagram of the principle of dividing the fragments provided in the embodiment of the present application;

[0038] FIG. 4D to FIG. 4F It is a schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application;

[0039] Figure 5AIt is a flowchart of a subtitle processing method for multimedia files provided by an embodiment of the present application;

[0040] Figure 5B It is a flowchart of a subtitle processing method for multimedia files provided by an embodiment of the present application;

[0041] FIG. 6A to FIG. 6C It is a schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application;

[0042] Figure 7 It is a schematic diagram of video content dimensions provided by an embodiment of the present application;

[0043] Figure 8 This is a schematic diagram of the principle of person gender recognition provided by an embodiment of the present application;

[0044] Fig. 9 This is a schematic diagram of the principle of character age recognition provided by an embodiment of the present application;

[0045] Fig.10 It is a schematic diagram of the principle of character emotion recognition provided by an embodiment of the present application;

[0046] Fig.11 It is a schematic diagram of the video style recognition principle provided by an embodiment of the present application;

[0047] Fig.12 It is a schematic diagram of the training process of the generative adversarial network model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0049] It is understandable that in the embodiments of the present application, related data such as user information is involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0052] 1) Subtitles refer to texts used for various purposes in videos such as movies and television, and audios such as dramas and songs, such as copyright logos, title subtitles, cast lists, caption subtitles (used to introduce the content of multimedia files, such as displaying relevant information about characters or scenery in multimedia files in text form), lyrics subtitles, and dialogue subtitles (synchronized with the sound object, used to display the voice content of the sound object in text form to help users understand the content of video files or audio files (such as audio novels)).

[0053] 2) Multimedia files, in terms of data form, include streaming media files (streaming media technology refers to a technology that compresses a series of multimedia data and transmits them in segments in a streaming manner on the network to achieve real-time transmission of audio and video for playback on the network, corresponding to the network playback scenario) and non-streaming media files (the entire multimedia file needs to be downloaded before playback, corresponding to the local playback scenario); in terms of the content carried, they include video files and audio files.

[0054] 3) Content features, including content features of static dimensions and content features of dynamic dimensions. The content features of static dimensions remain unchanged during the playback of multimedia files, such as the gender and age of the subject; the content features of dynamic dimensions change during the playback of multimedia files, such as the emotions of the subject.

[0055] 4) Style, also known as subtitle style, is a visual attribute related to the subtitle. Through different transformations of the same attribute and the combination of different attributes, multiple styles can be formed. For example, attributes may include: font, color, font size, character spacing, bold, italic, underline, strikethrough, shadow offset and color, alignment, vertical margin, etc.

[0056] During the implementation of the embodiments of the present application, the applicant discovered that in the solutions provided by the related art, during the playback of multimedia files (such as video files), the display method of video subtitles is the same and fixed. For example, during the playback of video files, the subtitles are always displayed in a fixed font. In other words, the display style of subtitles in the related art is relatively single and cannot adapt to the needs of personalized presentation of information on the Internet, which affects the information dissemination effect and the user's viewing experience.

[0057] In view of this, the embodiments of the present application provide a subtitle processing method, device, electronic device, computer-readable storage medium and computer program product for multimedia files, which can enrich the expression form of subtitles, improve the information dissemination effect and the user's viewing experience. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (for example, mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and vehicle-mounted terminals, and can also be implemented collaboratively by servers and terminal devices. Below, an exemplary application of the electronic device when it is implemented as a terminal device will be described.

[0058] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the subtitle processing system 100 for multimedia files provided in an embodiment of the present application. In order to realize an application supporting a rich subtitle expression form, the terminal device 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0059] The server 200 is a background server of the client 410 running on the terminal device 400. For example, when the client 410 is a browser, the server 200 may be a background server of a video website or an audio website. After receiving a network request sent by the terminal device 400, the server 200 sends the requested multimedia file (such as a streaming media file) to the terminal device 400 through the network 300, wherein the multimedia file is associated with multiple subtitles.

[0060] The client 410 running on the terminal device 400 can be various types of clients, such as a video playback client, an audio playback client, a browser, and an instant messaging client, etc. When the client 410 receives a playback trigger operation (for example, receiving a user's click operation on a play button displayed in the human-computer interaction interface), it plays the multimedia file received in real time from the server 200, and in the process of playing the multimedia file, displays multiple subtitles in sequence in the human-computer interaction interface, wherein the styles of the multiple subtitles applied are related to the content of the multimedia file (which will be explained in detail below).

[0061] In some embodiments, the subtitle processing method for multimedia files provided in the embodiments of the present application can also be implemented by the terminal device alone. For example, a downloaded multimedia file (multiple subtitles are associated with the multimedia file) is pre-stored locally on the terminal device 400. When the client 410 receives a play trigger operation, it plays the multimedia file stored locally on the terminal device 400, and in the process of playing the multimedia file, it displays multiple subtitles in sequence in the human-computer interaction interface, wherein the styles of the multiple subtitles applied are related to the content of the multimedia file.

[0062] In other embodiments, the embodiments of the present application can also be implemented with the aid of cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing.

[0063] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources.

[0064] For example, Figure 1 The server 200 shown in the figure can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal device 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal device 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0065] In some embodiments, the terminal device 400 can also implement the subtitle processing method of multimedia files provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run (that is, the above-mentioned client 410), such as a video playback client, an audio playback client, a browser, etc.; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0066] Next, continue Figure 1 The structure of the terminal device 400 shown in FIG. Figure 2 , Figure 2 It is a schematic diagram of the structure of the terminal device 400 provided in an embodiment of the present application. Figure 2 The terminal device 400 shown includes: at least one processor 420, a memory 460, at least one network interface 430 and a user interface 440. The various components in the terminal device 400 are coupled together via a bus system 450. It is understood that the bus system 450 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 450 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 450 is not shown in FIG. Figure 2 Various buses are labeled as bus system 450 .

[0067] The processor 420 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0068] The memory 460 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 460 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 460 optionally includes one or more storage devices that are physically far away from the processor 420.

[0069] In some embodiments, memory 460 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0070] Operating system 461, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0071] A network communication module 462, for reaching other computing devices via one or more (wired or wireless) network interfaces 430, exemplary network interfaces 430 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0072] a presentation module 463 for enabling presentation of information via one or more output devices 441 (e.g., display screen, speaker, etc.) associated with the user interface 440 (e.g., a user interface for operating peripherals and displaying content and information);

[0073] The input processing module 464 is used to detect one or more user inputs or interactions from one of the one or more input devices 442 and translate the detected inputs or interactions.

[0074] In some embodiments, the subtitle processing device for multimedia files provided in the embodiments of the present application can be implemented in software. The subtitle processing device for multimedia files provided in the embodiments of the present application can be provided as various software embodiments, including various forms including application programs, software, software modules, scripts or codes.

[0075] Figure 2 The subtitle processing device 465 of the multimedia file stored in the memory 460 is shown, which can be software in the form of a program and a plug-in, and includes a series of modules, including a playback module 4651, a display module 4652, an acquisition module 4653, a conversion module 4654, a fusion module 4655 and a determination module 4656. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. It should be pointed out that in Figure 2 For the sake of convenience, all the above modules are shown at once, but it should not be considered that the subtitle processing device 465 for multimedia files excludes the implementation that can only include the playback module 4651 and the display module 4652. The functions of each module will be explained below.

[0076] In other embodiments, the subtitle processing device for multimedia files provided in the embodiments of the present application can be implemented in hardware. As an example, the subtitle processing device for multimedia files provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the subtitle processing method for multimedia files provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (ASICs), digital signal processing (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic components.

[0077] As mentioned above, the subtitle processing method for multimedia files provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 3 , Figure 3 is a flowchart of a subtitle processing method for multimedia files provided by an embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0078] It should be noted that Figure 3 The method shown can be executed by various forms of computer programs running on the terminal device 400, and is not limited to the client 410 running on the above-mentioned terminal device 400. For example, it can also be the operating system 461, software modules, scripts and applets mentioned above. Therefore, the example of the client below should not be regarded as a limitation on the embodiments of the present application.

[0079] In step S101, in response to a play triggering operation, a multimedia file is played.

[0080] Here, the multimedia file is associated with multiple subtitles, and each subtitle corresponds to a playback time period on the playback time axis of the multimedia file. A subtitle is the basic unit of subtitle display, which can be one or more lines of text, including multilingual texts such as character languages, plots, character introductions, etc. Each subtitle is set with a corresponding display time period, including a start display time and an end display time. For example, for a subtitle A, the corresponding display time period can be 10:00-10:05, that is, a corresponding subtitle can be displayed according to the playback time period of the real-time playback progress of the multimedia file, and the subtitle is applied with a style adapted to the content features of at least one dimension of the multimedia file, that is, for subtitles associated with different multimedia files, the corresponding display styles are different, thereby achieving a diversified display effect of information related to the multimedia file, improving the information dissemination effect and the user's viewing experience.

[0081] In addition, the data form of multimedia files may include streaming media files (corresponding to network playback scenarios, such as the client responding to a playback trigger operation and requesting streaming media files from the server in real time and playing them) and non-streaming media files (corresponding to local playback scenarios, such as the client responding to a playback trigger operation and playing non-streaming media files pre-stored locally on the terminal device), and the type of multimedia files (i.e., the content they carry) may include video files and audio files.

[0082] For example, taking the multimedia file as a video file, assuming that the subtitle "How do you know?" corresponds to a playback period of 10:00 to 10:05 on the playback timeline of the video file, then during the playback of the video file from 10:00 to 10:05, the subtitle "How do you know?" is displayed, and the subtitle "How do you know?" is applied with a corresponding style, for example, a style adapted to the attributes of the sound-emitting object in the object segment with a playback period of 10:00 to 10:05 (such as age, gender, emotion, etc.).

[0083] In some embodiments, the format of the subtitle file may include image format and text format, wherein the subtitle file in image format is composed of idx and sub files, idx is equivalent to an index file, which includes the time code of the subtitle appearance (i.e., the above-mentioned play period) and the attributes of the subtitle display (i.e., the above-mentioned style), and the sub file is the subtitle data itself. Since it is in image format, it occupies a relatively large space, so it can be compressed to save space. The extension of the subtitle file in text format is usually ass, srt, sml, ssa or sub (the same as the suffix of the subtitle in the above-mentioned image format, but the data format is different). Because it is in text format, it occupies less space.

[0084] It should be noted that, for subtitles in text format, subtitle styles (including original styles and new styles (i.e., styles adapted to content features of at least one dimension of multimedia files)) can be recorded in files such as ass, srt, sml, ssa or sub.

[0085] In step S102, during the process of playing the multimedia file, a plurality of subtitles are sequentially displayed in the human-computer interaction interface.

[0086] Here, the styles applied to the multiple subtitles (ie, the multiple subtitles associated with the multimedia file, which can be obtained by reading from the above-mentioned subtitle file, for example) are related to the content of the multimedia file.

[0087] In some embodiments, the styles of multiple subtitle applications may be the same (i.e., the subtitle style remains unchanged during the entire playback process of the multimedia file), and the above-mentioned sequential display of multiple subtitles in the human-computer interaction interface may be achieved in the following manner: multiple subtitles applying the same style are sequentially displayed in the human-computer interaction interface; wherein the same style of the subtitle application is adapted to the content features of at least one dimension of the multimedia file.

[0088] For example, taking the multimedia file as a video file, in the process of playing the video file, multiple subtitles using a style adapted to the content feature of at least one dimension of the video file (for example, the style of the video file) (for example, when the style of the video file is a comedy, the corresponding style may be Chinese colorful clouds, size four, and blue) are displayed in sequence in the human-computer interaction interface. That is, in the process of playing the entire video file, the subtitles are displayed in the style of Chinese colorful clouds, size four, and blue. In other words, when the style of the video file is a comedy, the corresponding subtitle style is also cartoon-like and funny, which fits the content of the video file more closely, thereby enhancing the user's sense of immersion in the process of watching the video file.

[0089] It should be noted that, in actual applications, subtitles can be displayed at a fixed position in the human-computer interaction interface (for example, displaying subtitles in the lower middle part of the human-computer interaction interface). Of course, the position of the subtitle display can also change dynamically. For example, for video files, subtitles can be displayed in the human-computer interaction interface in a position that avoids objects that appear in the video screen, or they can be displayed in a way that they are superimposed on the video screen. The embodiment of the present application does not specifically limit the position of the subtitle display.

[0090] In other embodiments, the styles of multiple subtitle applications may also be different (i.e., the subtitle style may change during the playback of the entire multimedia file, for example, multiple subtitles may respectively apply a style adapted to the content features of at least one dimension of the fragment to which they belong), and the above-mentioned sequential display of multiple subtitles in the human-computer interaction interface may be achieved in the following manner: the multimedia file is divided to obtain multiple fragments (the types of fragments may include: object fragments, scene fragments, plot fragments, etc.); the following processing is performed during the playback of each fragment of the multimedia file: based on the style adapted to the content features of at least one dimension of the fragment, at least one subtitle associated with the fragment is sequentially displayed in the human-computer interaction interface.

[0091] For example, a multimedia file can be divided into multiple object segments according to the objects appearing in the multimedia file (such as people, animals, etc.), wherein each object segment includes an object (for example, the object included in object segment A is object A, and the object included in object segment B is object B, wherein object A and object B are two different objects with different object attributes, such as object A is male and object B is female, or object A is a young person and object B is an elderly person), and then the following processing is performed during the playback of each object segment of the multimedia file: based on a style adapted to the content features of at least one dimension of the object segment (for example, object segment A) (for example, a style adapted to the object attributes of object A included in object segment A (for example, identifying that object A is a male) and assuming that the style adapted to males is bold and size 5, that is, the style of the subtitle style is masculine), at least one subtitle associated with object segment A is displayed in sequence in the human-computer interaction interface (that is, at least one subtitle associated with object segment A is displayed in bold and size 5).

[0092] For example, a multimedia file can be divided into multiple scene segments according to different scenes (for example, for various types of history or geographical documentaries, the documentaries can be divided into multiple different scene segments according to the scenes), wherein each scene segment includes a scene (the scenes included in different scene segments may be different, for example, the scene included in scene segment A is a campus, and the scene included in scene segment B is a park), and then the following processing is performed during the playback of each scene segment of the multimedia file: based on a style adapted to the content features of at least one dimension of the scene segment (for example, scene segment B) (for example, a style adapted to the scene included in scene segment B, assuming that the scene included in scene segment B is the seaside, then the style adapted to the seaside may be regular script and blue, that is, the style of the subtitle style is adapted to the seaside), at least one subtitle associated with scene segment B is displayed in sequence in the human-computer interaction interface (that is, at least one subtitle associated with scene segment B is displayed in a style with regular script font and blue color).

[0093] For example, a multimedia file can be divided into multiple plot segments according to the content of the multimedia file (for example, for a video file, it can be divided into multiple plot segments such as the occurrence, development, climax and ending of the story), wherein each plot segment corresponds to a plot (the plots corresponding to different plot segments may be different, for example, plot segment A corresponds to the development stage of the story, and plot segment B corresponds to the climax stage of the story), and then the following processing is performed during the playback of each plot segment of the multimedia file: based on a style adapted to the content features of at least one dimension of the plot segment (for example, plot segment C) (for example, assuming that plot segment C is a climax segment, the style adapted to the climax segment may be Chinese Amber, No. 3, with a larger font size and a more serious font style, which is adapted to the climax segment), at least one subtitle associated with plot segment C is displayed in sequence in the human-computer interaction interface (that is, at least one subtitle associated with plot segment C is displayed in the style of Chinese Amber font and No. 3 font size).

[0094] It should be noted that the above process of dividing multimedia files is only a logical identification and division of multimedia files, and the data format of the multimedia files does not change, that is, there is no need to split the multimedia files, but only to add corresponding marks on the playback timeline of the multimedia files to logically divide the multimedia files into different segments.

[0095] In addition, it should be noted that in actual applications, a single type of division can be performed, that is, multiple fragments of one type are identified (for example, a multimedia file can be divided into multiple object fragments based solely on the objects appearing in the multimedia file); a composite type of division can also be performed, that is, multiple different types of fragments are identified (for example, a multimedia file can be divided based on both the objects and scenes appearing in the multimedia file. In this way, the multiple fragments obtained by division can include object fragments and scene fragments at the same time. Subsequently, the object fragments and scene fragments obtained by division are merged and deduplicated (for example, when object fragment A (assuming the corresponding time period is 10:00-12:00) and scene fragment B (assuming the corresponding time period is also 10:00-12:00) overlap, only one is retained), thereby obtaining the final division result). The embodiment of the present application does not specifically limit the division method.

[0096] In other embodiments, the style of at least one subtitle application associated with the same segment may be the same (i.e., the subtitle style will not change during the playback of the same segment), and the above-mentioned style based on adaptation to the content features of at least one dimension of the segment can be implemented in the following way, and at least one subtitle associated with the segment is displayed in sequence in the human-computer interaction interface: the content features of the static dimension of the segment are obtained, and at least one subtitle associated with the segment is synchronously displayed in the human-computer interaction interface, wherein the style of subtitle application remains unchanged during the playback of the segment.

[0097] For example, taking the type of the clip as an object clip, the content features of the static dimension of the object clip may include at least one of the following object attributes of the sound-emitting object in the object clip: role type (including positive and negative roles), gender, and age; for example, for object clip A, first obtain the object attributes of the sound-emitting object (such as object A) in object clip A (such as the gender of object A, assuming that the gender of object A is identified as female), and then synchronously display at least one subtitle associated with object clip A in the human-computer interaction interface, wherein the style of the subtitle application is adapted to females (for example, the style may be thin round and pink, that is, the subtitle style is more feminine), and remains unchanged during the playback of object clip A (that is, during the playback of object clip A, the subtitles are always displayed in a style with a thin round font and a pink color).

[0098] For example, taking the multimedia file as a video file, see Figure 4A , Figure 4A Schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application. Figure 4A As shown, for a certain object segment (the corresponding playback period is 40:30 to 40:40), when the gender of the sound-emitting object 401 in the object segment is identified as female, at least one subtitle associated with the object segment is synchronously displayed in the human-computer interaction interface, for example, subtitle 402 is displayed at 40:30 ("I'm so happy. I bought new clothes"), and subtitle 403 is displayed at 40:40 ("But I'll have to eat dirt next month"), and subtitles 402 and subtitles 403 are applied with a style adapted to females (for example, the font style of subtitles 402 and subtitles 403 is cute), and the style remains unchanged during the playback of the object segment (that is, the styles applied to subtitles 402 and subtitles 403 are the same).

[0099] It should be noted that, for plot clips, the content features of the static dimension of the plot clips may include the plot progress of the plot clips, and for the same plot clip, during the playback of the plot clip, the styles applied to all subtitles associated with the plot clip may be the same (for example, a style adapted to the plot progress is applied to all subtitles associated with the plot clip); and for scene clips, the content features of the static dimension of the scene clips may include the scene type of the scene clip, and for the same scene clip, during the playback of the scene clip, the styles applied to all subtitles associated with the scene clip may be the same (for example, a style adapted to the scene type is applied to all subtitles associated with the scene clip).

[0100] In addition, it should be noted that the styles of subtitles applied to different segments can be different. For example, taking the object segment as an example, see Figure 4B , Figure 4B Schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application. Figure 4B As shown, sound object 404 and sound object 406 belong to sound objects included in different object segments, and the subtitles 405 ("What shall we eat tonight") associated with the object segment where sound object 404 is located and the subtitles 407 ("How is the barbecue") associated with the object segment where sound object 406 is located are applied in different styles (for example, the font of subtitle 405 is Fang Zheng Shuti, while the font of subtitle 407 is Hua Wen Cai Yun). In this way, different styles are applied to the subtitles corresponding to different sound objects (for example, the style applied to subtitle 405 is Fang Zheng Shuti (the font style is relatively soft) adapted for females, while the style applied to subtitle 407 is Hua Wen Cai Yun (the font style is relatively masculine) adapted for males), thereby facilitating users to distinguish different objects appearing in the video file.

[0101] In some embodiments, after the multimedia file is divided into multiple segments, any one or more of the multiple segments may be further divided into multiple sub-segments. Figure 4C , Figure 4C is a schematic diagram of the principle of segmentation provided by the embodiment of the present application, such as Figure 4CAs shown, taking the scene segment 1 and the plot segment 2 (scene segment 1 and plot segment 2 may be two adjacent segments, i.e., after the scene segment 1 is played, the plot segment 2 continues to be played) among the multiple segments obtained by dividing the multimedia file as an example, for the scene segment 1, the scene segment 1 may be further divided into three different character sub-segments according to the characters appearing in the scene segment 1 (for example, including the character sub-segment 1, the character sub-segment 2 and the character sub-segment 3, wherein the characters included in different character sub-segments may be different, for example, the character sub-segment 1 includes the character A, the character sub-segment 2 includes the character B, and the character sub-segment 3 includes the character C); for the plot segment 2, the plot segment 2 may also be further divided into two different scene sub-segments according to the scenes appearing in the plot segment 2 (for example, including the scene sub-segment 1 and the scene sub-segment 2, wherein the scenes included in different scene sub-segments may be different, for example, the scene of the scene sub-segment 1 is a campus, and the scene of the scene sub-segment 2 is a park). Taking scene segment 1 as an example, after dividing into three character sub-segments, for each character sub-segment, during the playback of the character sub-segment, at least one subtitle associated with the character sub-segment can be displayed based on a style adapted to the content features of at least one dimension of the sub-segment, as described in detail below.

[0102] In some embodiments, when different styles are applied to multiple subtitles associated with the same segment, the above-mentioned style based on adaptation to the content features of at least one dimension of the segment can be implemented in the following manner, and at least one subtitle associated with the segment is displayed in sequence in the human-computer interaction interface: the segment is divided to obtain multiple sub-segments, wherein the multiple sub-segments have content features of a static dimension of the segment (the content features of the static dimension remain unchanged during the playback of the segment) and content features of a dynamic dimension of the segment (content features of different dynamic dimensions will change during the playback of the segment), and different sub-segments have different content features of the dynamic dimension; the following processing is performed during the playback of each sub-segment of the segment: based on a style adapted to the content features of the static dimension and the content features of the dynamic dimension of the sub-segment, at least one subtitle associated with the sub-segment is displayed.

[0103] For example, for an object segment, the content features of the static dimension of the object segment may include at least one of the following object attributes: the role type, gender, and age of the sound-emitting object in the object segment; the content features of the dynamic dimension of the object segment may include the following object attributes: the emotion of the sound-emitting object in the object segment; for example, taking the multimedia file as a video file, for a certain object segment in the video file (for example, object segment A), the object segment A is first divided into multiple sub-segments, and then the following processing is performed during the playback of each sub-segment of the object segment A: based on a style adapted to the content features of the static dimension (for example, the gender of the sound-emitting object in the object segment A) and the content features of the dynamic dimension (for example, the emotion of the sound-emitting object in the current sub-segment) of the sub-segment, at least one subtitle associated with the sub-segment is displayed (that is, the style applied to the at least one subtitle associated with the sub-segment is adapted to the gender of the sound-emitting object and the emotion of the sound-emitting object in the current sub-segment).

[0104] For example, taking the multimedia file as a video file, see Figure 4D , Figure 4D Schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application. Figure 4D As shown, sub-segment 408 and sub-segment 409 belong to different sub-segments of the same object segment. In sub-segment 408, the expression of the sound object 410 is sad, while in sub-segment 409, the expression of the sound object 410 changes from sadness to happiness. Correspondingly, the style applied to subtitle 411 ("Don't leave me") associated with sub-segment 408 and the style applied to subtitle 412 ("It's nice to see you again") associated with sub-segment 409 are different (for example, the font of subtitle 411 is Chinese Lishu, the font size is size 4, the font size is small, and the font style is serious, which is adapted to the sad emotion; while the font of subtitle 412 is Chinese Caiyun, the font size is size 4, the font size is large, and the font style is festive, which is adapted to the happy emotion). In this way, for the same object segment, the subtitle style will be adjusted accordingly as the emotion of the sound object changes, so that the subtitle style is more closely aligned with the video content, thereby enhancing the user's immersion in the video viewing process.

[0105] For example, for a plot segment, the content features of the static dimension of the plot segment may include the plot type of the plot segment, and the content features of the dynamic dimension of the plot segment may include at least one of the following: scene types of different scenes appearing in the plot segment, object properties of different sound-emitting objects appearing in the plot segment; for example, taking the multimedia file as a video file, for a certain plot segment in the video file (for example, plot segment B), first, plot segment B is divided into multiple sub-segments, and then the following processing is performed during the playback of each sub-segment of plot segment B: based on a style adapted to the content features of the static dimension of the sub-segment (for example, the plot type of plot segment B) and the content features of the dynamic dimension (for example, the scene type appearing in the current sub-segment), at least one subtitle associated with the sub-segment is displayed (that is, the style applied to the at least one subtitle associated with the sub-segment is adapted to the plot type of plot segment B and the scene type appearing in the current sub-segment).

[0106] For example, taking the multimedia file as a video file, see Figure 4E , Figure 4E Schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application. Figure 4E As shown, sub-segment 413 and sub-segment 414 are different sub-segments belonging to the same plot segment. The scene in sub-segment 413 is home, while the scene in sub-segment 414 switches from home to outdoors. Correspondingly, subtitles 415 ("Dad, shall we go hiking?") associated with sub-segment 413 and subtitles 416 ("Dad, wait for me") associated with sub-segment 414 have different styles (for example, the font of subtitle 415 is bold, while the font of subtitle 416 is Chinese amber). In this way, for different sub-segments of the same plot segment, the subtitle style will be adjusted accordingly as the content characteristics of the dynamic dimensions of different sub-segments change, so that users can understand the video content more easily according to the changes in the subtitle style.

[0107] For example, for a scene segment, the content features of the static dimension of the scene segment may include: the type of scene involved in the scene segment, and the content features of the dynamic dimension of the scene segment may include at least one of the following: object attributes of different sound-emitting objects appearing in the scene segment, and types of different plots appearing in the scene segment; for example, taking the multimedia file as a video file, for a certain scene segment in the video file (such as scene segment C), the scene segment C is first divided into multiple sub-segments, and then the following processing is performed during the playback of each sub-segment of the scene segment C: based on a style adapted to the content features of the static dimension of the sub-segment (such as the type of scene involved in the scene segment C) and the content features of the dynamic dimension (such as the type of plot appearing in the current sub-segment), at least one subtitle associated with the sub-segment is displayed (that is, the style applied to the at least one subtitle associated with the sub-segment is adapted to the scene type involved in the scene segment C and the type of plot appearing in the current sub-segment).

[0108] For example, taking the multimedia file as a video file, see Figure 4F , Figure 4F Schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application. Figure 4F As shown, sub-segment 417 and sub-segment 418 are different sub-segments belonging to the same scene segment. The type of plot in sub-segment 417 is the development stage, while the type of plot in sub-segment 418 enters the climax stage from the development stage. Correspondingly, subtitles 419 associated with sub-segment 417 ("Architecture in the Middle Ages was relatively simple") and subtitles 420 associated with sub-segment 418 ("Architecture in the Renaissance era is more modern") have different styles (for example, the font of subtitle 419 is Chinese regular script, while the font of subtitle 420 is Youyuan). In this way, for different sub-segments of the same scene segment, the subtitle style will be adjusted accordingly as the content characteristics of the dynamic dimensions of different sub-segments change, so that users can understand the video content more easily according to the changes in the subtitle style.

[0109] The following is an explanation of the subtitle style conversion process.

[0110] In some embodiments, for the case where the styles of multiple subtitles associated with a multimedia file are the same (that is, the subtitle style remains unchanged during the entire multimedia file playback process), when executing Figure 3 Before step S102 shown in FIG. Figure 5A Steps S103A and S104A shown in FIG. Figure 5A The steps shown are explained.

[0111] In step S103A, a content feature of at least one dimension of the multimedia file is obtained.

[0112] Here, the content features of at least one dimension of a multimedia file may include: style (for example, for video files, the corresponding style types may include comedy, horror, suspense, cartoon, etc.; for audio files, the corresponding style types may include pop, rock, etc.), objects (for example, characters and animals appearing in the multimedia file), scenes, plots, and tones.

[0113] In some embodiments, the above-mentioned step S103A can be implemented in the following manner: calling a content feature recognition model to perform content feature recognition processing on the content of the multimedia file to obtain content features of at least one dimension of the multimedia file, wherein the content feature recognition model is trained based on sample multimedia files and labels annotated for the content of the sample multimedia files.

[0114] For example, the content feature recognition model can be a separate style recognition model, scene recognition model, plot recognition model and tone recognition model, or it can be a combined model (for example, a model that can simultaneously recognize the style and scene of a multimedia file). The content feature recognition model can be a neural network model (for example, a convolutional neural network, a deep convolutional neural network, or a fully connected neural network, etc.), a decision tree model, a gradient boosting tree, a multi-layer perceptron, and a support vector machine, etc. The embodiments of the present application do not specifically limit the type of content feature recognition model.

[0115] In other embodiments, when the multimedia file is a video file, the above-mentioned step S103A can be implemented in the following manner: the following processing is performed on the target object appearing in the video file: the target video frame where the target object is located is preprocessed (for example, the size of the target video frame is cropped, and the target object in the target video frame is adjusted. In addition, when multiple target video frames are obtained, the clarity of the target object included in each target video frame can be determined, and the target video frame with the highest clarity is selected for subsequent processing); feature extraction is performed on the preprocessed target video frame (for example, wavelet features used to describe image texture information in the target video frame are extracted to obtain image features corresponding to the target video frame); dimensionality reduction is performed on the image features (for example, the principal component analysis method is used to extract the main feature components of the image features to achieve dimensionality reduction of the image features), and the image features after dimensionality reduction are classified by a trained classifier to obtain object attributes of the target object (for example, the gender of the target object).

[0116] In some embodiments, when the multimedia file is a video file, the above-mentioned step S103A can also be implemented in the following manner: the following processing is performed on the target object appearing in the video file: the local binary pattern features (an operator used to describe the local features of the image, having characteristics such as grayscale invariance and rotation invariance) corresponding to the target video frame where the target object is located are extracted, and the local binary pattern features are subjected to dimensionality reduction processing; the directional gradient histogram features (a feature descriptor used for object detection in computer vision and image processing) corresponding to the target video frame are extracted, and the directional gradient histogram features are subjected to dimensionality reduction processing; the local binary pattern features and the directional gradient histogram images after dimensionality reduction processing are subjected to canonical correlation analysis processing (used to mine the correlation between the local binary pattern features and the directional gradient histogram features) to obtain analysis results; the analysis results are subjected to regression processing to obtain the object attributes of the target object (for example, the age of the target object).

[0117] In other embodiments, when the multimedia file is a video file, the above-mentioned step S103A can also be implemented in the following manner: performing the following processing on the target object appearing in the video file: normalizing the target video frame where the target object is located, and partitioning the normalized target video frame to obtain multiple sub-regions; extracting the local binary pattern features corresponding to each sub-region, and performing statistical processing on the multiple local binary pattern features to obtain the local histogram statistical features corresponding to the target video frame; performing local sparse reconstruction representation of the local histogram statistical features through the local feature library of the training set, and performing local reconstruction residual weighted recognition processing on the local sparse reconstruction representation result to obtain the object attributes of the target object (such as the emotion of the target object).

[0118] In some embodiments, continuing from the above, when there are multiple objects in a video file, the target object can be determined from the multiple objects in any of the following ways: determining the object that appears the longest in the video file as the target object; determining the object in the video file that meets the user's preferences (for example, determining the user's portrait based on the user's historical viewing record, and determining the object with the highest similarity to the user's portrait as the object that meets the user's preferences) as the target object; determining the object in the video file that is related to the user's interaction (for example, an object that the user has liked or forwarded) as the target object.

[0119] In addition, it should be noted that when the multimedia file is an audio file, the object attributes (such as gender, age, emotion, etc.) of the target object appearing in the audio file can be identified in the following ways. For example, the gender of the target object can be determined based on the frequency of the sound (the frequency of female pronunciation is relatively high, while the frequency of male pronunciation is relatively low); the age of the target object can be identified based on the pitch (for example, children's vocal cords are usually tighter, so the pitch is higher, and as they age, the vocal cords become looser and the pitch gradually decreases); the emotion of the target object can be determined based on information such as the speaking speed and volume. For example, when the target object is angry, the corresponding volume will be relatively loud and the speaking speed will be relatively fast.

[0120] In step S104A, based on the content feature of at least one dimension of the multimedia file, style conversion processing is performed on a plurality of original subtitles associated with the multimedia file to obtain a plurality of new subtitles.

[0121] Here, multiple new subtitles (the styles of the multiple new subtitles are the same, for example, they are all obtained by performing style conversion processing on multiple original subtitles associated with the multimedia file based on the style of the identified multimedia file) are used as multiple subtitles to be displayed in the human-computer interaction interface (that is, as multiple subtitles displayed in sequence in the human-computer interaction interface in step S102).

[0122] In some embodiments, the above-mentioned step S104A can be implemented in the following manner: based on the value corresponding to the content feature of at least one dimension of the multimedia file and a plurality of original subtitles associated with the multimedia file, a subtitle model is called to obtain a plurality of new subtitles, wherein the subtitle model can be used as a generative model and trained by forming a generative adversarial network with a discriminative model.

[0123] For example, taking the multimedia file as a video file, after obtaining the content feature of at least one dimension of the video file (for example, the style of the video file, assuming it is comedy), the subtitle model can be called based on the value corresponding to the style of the video file and the multiple original subtitles associated with the multimedia file (assuming that the fonts of the multiple original subtitles are all in Kaiti), to obtain multiple new subtitles (for example, assuming that the fonts of the multiple new subtitles obtained after the style conversion process are all in Yiyuan, and the font style is adapted to comedy and is cartoon-like, that is, during the playback of the video file, the fonts of the multiple subtitles displayed in sequence in the human-computer interaction interface are all in Yiyuan).

[0124] In other embodiments, the subtitle model may also be obtained by training in other ways. For example, the subtitle model may be trained separately. The embodiments of the present application do not specifically limit the training method of the subtitle model.

[0125] In addition, it should be noted that the above-mentioned style conversion processing can be for subtitles in picture format. For example, a picture with the original font (such as a picture with the subtitle content in Kaiti font) can be converted into a picture with a font that matches the style of the video file (such as a picture with the subtitle content in Huawen Caiyun font); and for subtitles in text format, the subtitles in text format can be first converted into picture format, and then the above-mentioned style conversion processing can be performed.

[0126] As an alternative, style conversion processing can be performed directly on subtitles in text format. For example, various attributes of the original style subtitles (such as font, font size, etc.) can be first encoded to obtain the corresponding matrix vector, and then the matrix vector can be style converted (for example, the values ​​corresponding to the matrix vector and the style of the video file can be input into the subtitle model) to obtain a new matrix vector (i.e., the matrix vector corresponding to the new style subtitles), and then decoding processing is performed based on the new matrix vector to obtain new style subtitles (i.e., subtitles of a style adapted to the style of the video file). Finally, the new style subtitles are used to replace the original style subtitles in the subtitle file. Subtitles in text format are more conducive to storage and updating, such as correcting text errors.

[0127] In other embodiments, for the case where the styles of multiple subtitles associated with a multimedia file are different (that is, the subtitle style will change during the playback of the entire multimedia file), the subtitle style can be changed during the execution of Figure 3 Before step S102 shown in FIG. Figure 5B Steps S103B and S104B shown in FIG. Figure 5B The steps shown are explained.

[0128] In step S103B, a content feature of at least one dimension of each segment in the multimedia file is obtained.

[0129] In some embodiments, the multimedia file can be first divided (the specific division process can be referred to the description above, and the embodiments of the present application will not be repeated here) to obtain multiple segments, wherein each segment is associated with at least one original subtitle (for example, segment 1 is associated with original subtitles 1 to original subtitles 3, and segment 2 is associated with original subtitles 4 and original subtitles 5), and then the content features of at least one dimension of each segment are obtained respectively (the method for obtaining the content features of the segments is similar to the method for obtaining the content features of the multimedia files, and can be implemented by referring to the method for obtaining the content features of the multimedia files above, and the embodiments of the present application will not be repeated here).

[0130] In step S104B, the following processing is performed for each segment: based on the content feature of at least one dimension of the segment, style conversion processing is performed on at least one original subtitle associated with the segment to obtain at least one new subtitle.

[0131] Here, after performing style conversion processing on at least one original subtitle associated with each segment, at least one new subtitle corresponding to each segment can be combined to obtain multiple new subtitles, wherein the order of the multiple new subtitles is the same as the order of the multiple original subtitles, and the multiple new subtitles serve as the multiple subtitles to be displayed in the human-computer interaction interface (i.e., the multiple subtitles displayed in sequence in the human-computer interaction interface in step S102).

[0132] In some embodiments, after obtaining the content features of at least one dimension of each fragment in the multimedia file, the following processing can be performed for each fragment: a subtitle model is called based on the value corresponding to the content feature of at least one dimension of the fragment (for example, fragment A) and at least one original subtitle associated with fragment A to obtain at least one new subtitle associated with fragment A, and then the at least one new subtitle associated with fragment A can be used to replace the at least one original subtitle associated with fragment A stored in the subtitle file. In this way, during the subsequent playback of the multimedia file, for example, when playing to fragment A, at least one new subtitle associated with fragment A can be read from the subtitle file and displayed in the human-computer interaction interface.

[0133] It should be noted that, taking the emotion of the target object appearing in the video file as an example of the content feature of the clip, the emotion of the target object may change during the playback of the video file, that is, emotion belongs to the content feature of the dynamic dimension. In different clips, the emotion of the target object may be different. Therefore, when performing style conversion processing based on the emotion of the target object, after the style conversion processing, the styles of the new subtitles obtained by converting the original subtitles associated with different clips may be different (for example, in Clip 1, the emotion of the target object is happy, and the font of the new subtitles obtained after the style conversion processing is Youyuan; in Clip 2, the emotion of the target object is sad, and the font of the new subtitles obtained after the style conversion processing is Huawen Caiyun). In other words, during the playback of the video file, the subtitle style will be adjusted accordingly as the emotion of the target object changes.

[0134] In addition, it should be noted that, in the case where the styles of multiple subtitles associated with the same segment (for example, segment A) are different, segment A can be divided again to obtain multiple sub-segments, and then the content features of at least one dimension of each sub-segment in segment A are obtained, and then the following processing is performed on each sub-segment: the subtitle model is called based on the value corresponding to the content feature of at least one dimension of the sub-segment and at least one original subtitle associated with the sub-segment to obtain at least one new subtitle associated with the sub-segment. In this way, when the content features of at least one dimension corresponding to different sub-segments are different, the styles of at least one new subtitle associated with different sub-segments are also different, thereby achieving that the subtitle style will also change during the playback of the same segment.

[0135] In some embodiments, the style of subtitle application can also be adapted to the fused content features obtained after fusing the content features of multiple dimensions of the fragment. The above-mentioned style based on adaptation to the content features of at least one dimension of the fragment can be implemented in the following way, and at least one subtitle associated with the fragment is displayed in sequence in the human-computer interaction interface: the content features of multiple dimensions of the fragment are fused to obtain the fused content features; based on the fused content features, at least one original subtitle associated with the fragment is style converted to obtain at least one new subtitle; wherein the at least one new subtitle is used as the at least one subtitle to be displayed in the human-computer interaction interface.

[0136] For example, taking the multimedia file as a video file, first obtain the content features of the video file in multiple dimensions (such as the style of the video file, the color tone of the video file, etc.), then fuse the style of the video file and the color tone of the video file (for example, sum the value corresponding to the style of the video file and the value corresponding to the color tone of the video file) to obtain the fused content feature, and then call the subtitle model based on the value corresponding to the fused content feature and at least one original subtitle associated with the fragment to obtain at least one new subtitle (the style of the new subtitle is adapted to the style of the video file and the color tone of the video file at the same time). In this way, by comprehensively considering the content features of the video file in multiple dimensions, the subtitles finally presented can be more in line with the video content, further improving the user's viewing experience.

[0137] In other embodiments, the style of subtitle application may also be related to the content of the multimedia file and the user's portrait at the same time. For example, the user's (i.e., viewer's) emotion towards the multimedia file may be quantified, the user's portrait may be determined based on the user's historical viewing record, and then the user's preference for the current multimedia file may be calculated. Finally, the style of the subtitle may be determined based on the preference and the content characteristics of at least one dimension of the multimedia file (for example, the value corresponding to the preference and the value of the content characteristics of at least one dimension of the multimedia file may be fused (for example, the two values ​​may be added), and a subtitle model may be called based on the value obtained by the fusion process and multiple original subtitles associated with the multimedia file to obtain multiple new subtitles, that is, the styles of the multiple new subtitles are adapted to the content of the multimedia file and the user's portrait at the same time, that is, for the same multimedia file, the subtitles displayed on different users' user terminals may also be different). In this way, by comprehensively considering the user's own factors and the content characteristics of the multimedia file, the subtitles finally presented can better meet the user's personalized needs, further improving the user's viewing experience.

[0138] The subtitle processing method for multimedia files provided in the embodiment of the present application displays subtitles of a style related to the content of the multimedia file in a human-computer interaction interface during the playback of the multimedia file, and achieves diversified display effects of information related to the multimedia file by enriching the expression form of the subtitles. It can be adapted to the diversified subtitle display requirements of different application scenarios of the multimedia file, thereby improving the efficiency and effect of information display and also improving the user's viewing experience.

[0139] The following is an explanation of an exemplary application of the embodiment of the present application in an actual video file playback scenario.

[0140] The embodiment of the present application provides a subtitle processing method for multimedia files, which can understand the content of video files (for example, mining the character attributes of characters appearing in the video files, the overall style of the video files, etc.) to generate subtitles of relevant styles in real time, thereby improving the user's viewing experience.

[0141] The subtitle processing method for multimedia files provided in the embodiment of the present application can be applied to the subtitle generation of major video websites. It can generate subtitles in real time in a style related to the content of the identified video file based on the content of the video file (including style recognition of the video file and character attribute recognition of the characters appearing in the video file, such as recognition of the character's age, gender, and emotions, etc.).

[0142] For example, see FIG. 6A to FIG. 6C , FIG. 6A to FIG. 6C is a schematic diagram of an application scenario of the subtitle processing method for multimedia files provided in an embodiment of the present application, wherein: Fig. 6A The style of the video 601 shown in FIG. 6 is an animation, and the overall style is cute and cartoonish. Therefore, the subtitles 602 associated with the video 601 are also in this style. In addition, the color of the subtitles 602 can also be adapted to the main color of the background (for example, when the background is the sky, the color of the subtitles 602 can be blue); Figure 6B The style of the video 603 shown in FIG. 6 is a comedy, and the overall style is funny. Therefore, the subtitles 604 associated with the video 603 are also cartoon-like, which is adapted to the style of the video 603. Figure 6C The style of the video 605 shown in the figure belongs to a heroic film, and the overall style is relatively serious. Therefore, the font style of the subtitle 606 associated with the video 605 is also more serious and formal. In other words, the subtitles corresponding to videos of different styles are different in style, and are highly consistent with the style of the video, so that users have a better immersive experience when watching videos.

[0143] The subtitle processing method for multimedia files provided in the embodiment of the present application mainly involves two parts: understanding the content of the video file, and generating video subtitles of relevant styles in real time based on the understanding results of the video content. The following first describes the process of understanding the video content.

[0144] For example, see Figure 7 , Figure 7 is a schematic diagram of video content dimensions provided by an embodiment of the present application, such as Figure 7 As shown, the understanding of the video content in the embodiments of the present application mainly involves the following dimensions: character attributes (including the gender, age, emotions, etc. of the characters appearing in the video) and the overall style of the video (style types include cartoons, comedies, horror, suspense, etc.). The following first describes the process of identifying character attributes.

[0145] (1)Character attributes:

[0146] The recognition of character attributes includes the recognition of character gender, character age and character emotion.

[0147] For example, the gender recognition of a person can be performed by (but not limited to) a face gender classification algorithm based on Adaboost (an iterative algorithm whose core idea is to train different classifiers (weak classifiers) for the same training set, and then combine these weak classifiers to form a stronger final classifier (strong classifier)) and support vector machine (SVM, Support Vector Machine), such as Figure 8 As shown in FIG. 1 , the face gender classification algorithm based on Adaboost+SVM is mainly divided into two stages: (a) training stage: by preprocessing the sample image and extracting the Gabor (a feature that can be used to describe the image texture information) wavelet features of the preprocessed sample image, the feature dimension reduction is performed through the Adaboost classifier, and finally the SVM classifier is trained; (b) testing stage: by preprocessing the test image, extracting the Gabor wavelet features of the preprocessed test image, and the feature dimension reduction is performed through the Adaboost classifier, and finally the trained SVM classifier is used for recognition, and the recognition result (i.e., the gender of the person) is output.

[0148] For example, the age estimation of a person can adopt (but is not limited to) a face age estimation algorithm that integrates local binary patterns (LBP, Local Binary Patterns, an operator used to describe local texture features of an image, with significant advantages such as rotation invariance and grayscale invariance) and directional gradient histogram (HOG, Histogram of Oriented Gradient, a feature descriptor used for object detection in computer vision and image processing, HOG features are constructed by calculating and counting the gradient directional histogram of the local area of ​​the image) features, such as Fig. 9 As shown in the figure, the algorithm extracts local statistical features of the face that are closely related to age changes (i.e., LBP features and HOG features), then performs dimensionality reduction processing on the extracted features (for example, using the principal component analysis (PCA) method to reduce the dimensionality of the LBP features and the HOG features), and uses the canonical correlation analysis (CCA) method to fuse the two reduced-dimensional features, and finally trains and tests the face library through the support vector machine regression (SVR) method.

[0149] For example, the recognition of human emotions can adopt (but is not limited to) a facial expression recognition algorithm that combines LBP features and local sparse representation, such as Fig.10 As shown in FIG, the steps of the algorithm are as follows: first, the normalized training set face images are feature partitioned, the LBP features of each face partition are calculated, and the feature vectors of the region are integrated using the histogram statistics method to form a training set local feature library composed of local features of a specific face; second, for the test face, the face image is normalized, the face is partitioned, the local LBP features are calculated, and the local histogram statistics are performed; finally, for the local histogram statistical features of the test face, the local feature library of the training set is used for local sparse reconstruction representation, and the local sparse reconstruction residual weighting method is used to obtain the final facial expression classification and recognition results.

[0150] (2) Video style

[0151] The video style can be identified using a convolutional neural network (CNN) model, where the training data can come from video files provided by video websites and style classification labels (generally identified by operators), such as Fig.11As shown in the figure, L consecutive (L is a positive integer greater than 1) frame images in the video are input into the trained convolutional neural network model. After the feature map corresponding to each frame image is obtained through convolution and pooling processing, the Gram matrix is ​​used to calculate the correlation between the two feature maps to represent the style information of the video. Finally, it is connected to the regression function (such as the Softmax function) to output the probabilities corresponding to different styles, and the style corresponding to the maximum probability is determined as the style of the video.

[0152] The subtitle generation process is described below.

[0153] The generation of subtitles can be achieved using the Generative Adversarial Networks (GAN) model, in which GAN contains two models: a generative model and a discriminative model. The final generation result is achieved by the confrontation between the generative model and the discriminative model.

[0154] For example, see Fig.12 , Fig.12 This is a schematic diagram of the training process of the generative adversarial network model provided in the embodiment of the present application. The specific algorithm process is as follows:

[0155] (1) The original font image x and the converted target domain c (the target domain c corresponds to the dimension understood from the video content) are combined and input into the generative model G to generate a fake font image x ′ (i.e., based on the original font image and the corresponding value of the target domain c, a font image adapted to the understood video content dimension is generated);

[0156] (2) x the fake font image ′ and the original font image x are input into the discriminant model D respectively. The discriminant model D needs to determine whether the input font image is real and which domain the font image comes from;

[0157] (3) Generate a fake font image x ′ The domain information corresponding to the original font image x (i.e., source domain c ′ ) are combined and input into the generation model G, which is required to be able to reconstruct the original font image x.

[0158] It should be noted that if the original subtitles are in text format, such as subtitle files of types such as srt, ass, etc., the subtitles in text format can be first converted into image format, and then the above processing can be performed.

[0159] The solution for generating subtitles in real time according to video content provided by the embodiment of the present application has the following beneficial effects:

[0160] (1) The subtitle style is highly consistent with the video content and is not abrupt;

[0161] (2) The subtitle style is more consistent with the video content or the role characteristics of the characters appearing in the video, which is more immersive;

[0162] (3) The subtitle style is automatically generated by a machine (such as a terminal device or server), and there is no need to purchase the copyright of the subtitle library, thus saving copyright costs.

[0163] The following is a description of an exemplary structure of a multimedia file subtitle processing device 465 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the subtitle processing device 465 for multimedia files stored in the memory 460 may include: a playing module 4651 and a display module 4652 .

[0164] The playback module 4651 is used to play a multimedia file in response to a playback trigger operation, wherein the multimedia file is associated with multiple subtitles, and the types of multimedia files include video files and audio files; the display module 4652 is used to display multiple subtitles in sequence in the human-computer interaction interface during the playback of the multimedia file, wherein the styles of the multiple subtitles applied are related to the content of the multimedia file.

[0165] In some embodiments, the display module 4652 is also used to sequentially display multiple subtitles, all of which have styles applied, in the human-computer interaction interface; wherein the style is adapted to the content features of at least one dimension of the multimedia file, and the content features of at least one dimension include: style, object, scene, plot, and tone.

[0166] In some embodiments, the subtitle processing device 465 for multimedia files also includes an acquisition module 4653 for acquiring content features of at least one dimension of the multimedia file; the subtitle processing device 465 for multimedia files also includes a conversion module 4654 for performing style conversion processing on multiple original subtitles associated with the multimedia file based on the content features of at least one dimension to obtain multiple new subtitles; wherein the multiple new subtitles are used as multiple subtitles to be displayed in the human-computer interaction interface.

[0167] In some embodiments, the conversion module 4654 is also used to call the subtitle model based on the value corresponding to the content feature of at least one dimension and multiple original subtitles associated with the multimedia file to obtain multiple new subtitles; wherein the subtitle model is used as a generative model and is trained by forming a generative adversarial network with the discriminative model.

[0168] In some embodiments, the multimedia file includes multiple segments, and the types of the segments include at least one of the following: object segments, scene segments, and plot segments; the display module 4652 is also used to perform the following processing during the playback of each segment of the multimedia file: based on a style adapted to the content characteristics of at least one dimension of the segment, display at least one subtitle associated with the segment in the human-computer interaction interface in sequence.

[0169] In some embodiments, the acquisition module 4653 is also used to acquire content features of the static dimension of the fragment, wherein the content features of the static dimension of the object fragment include at least one of the following object attributes of the sound-emitting object in the object fragment: role type, gender, age; the features of the static dimension of the scene fragment include the scene type of the scene fragment; the features of the static dimension of the plot fragment include the plot progress of the plot fragment; the display module 4652 is also used to synchronously display at least one subtitle associated with the fragment in the human-computer interaction interface based on a style adapted to the content features of the static dimension of the fragment, wherein the style remains unchanged during the playback of the fragment.

[0170] In some embodiments, a segment includes multiple sub-segments, and the multiple sub-segments have content characteristics of a static dimension of the segment and content characteristics of a dynamic dimension of the segment, and different sub-segments have different content characteristics of the dynamic dimension; the display module 4652 is also used to perform the following processing during the playback of each sub-segment of the segment: based on a style adapted to the content characteristics of the static dimension and the content characteristics of the dynamic dimension of the sub-segment, display at least one subtitle associated with the sub-segment.

[0171] In some embodiments, the content features of the static dimension of the object segment include at least one of the following object attributes: the role type, gender, and age of the sound-making object in the object segment; the content features of the dynamic dimension of the object segment include the following object attributes: the emotion of the sound-making object in the object segment; the content features of the static dimension of the plot segment include the plot type of the plot segment, and the content features of the dynamic dimension of the plot segment include at least one of the following: the scene types of different scenes in which the plot segment appears, and the object attributes of different sound-making objects that appear in the plot segment; the content features of the static dimension of the scene segment include: the type of scene involved in the scene segment; the content features of the dynamic dimension of the scene segment include at least one of the following: the object attributes of different sound-making objects that appear in the scene segment, and the types of different plots that appear in the scene segment.

[0172] In some embodiments, when at least one dimension is multiple dimensions, the subtitle processing device 465 of the multimedia file also includes a fusion module 4655, which is used to fuse the content features of multiple dimensions of the fragment to obtain fused content features; the conversion module 4654 is also used to perform style conversion processing on at least one original subtitle associated with the fragment based on the fused content features to obtain at least one new subtitle; wherein the at least one new subtitle is used as at least one subtitle to be displayed in the human-computer interaction interface.

[0173] In some embodiments, the acquisition module 4653 is also used to call a content feature recognition model to perform content feature recognition processing on the content of a multimedia file to obtain content features of at least one dimension of the multimedia file; wherein the content feature recognition model is trained based on sample multimedia files and labels annotated for the content of the sample multimedia files.

[0174] In some embodiments, when the multimedia file is a video file, the acquisition module 4653 is also used to perform the following processing on the target object appearing in the video file: preprocessing the target video frame where the target object is located; extracting features from the preprocessed target video frame to obtain image features corresponding to the target video frame; performing dimensionality reduction processing on the image features, and classifying the image features after dimensionality reduction processing through a trained classifier to obtain object attributes of the target object.

[0175] In some embodiments, when the multimedia file is a video file, the acquisition module 4653 is also used to perform the following processing on the target object appearing in the video file: extract the local binary pattern features corresponding to the target video frame where the target object is located, and perform dimensionality reduction processing on the local binary pattern features; extract the directional gradient histogram features corresponding to the target video frame, and perform dimensionality reduction processing on the directional gradient histogram features; perform canonical correlation analysis processing on the local binary pattern features and directional gradient histogram images after dimensionality reduction processing to obtain analysis results; perform regression processing on the analysis results to obtain object properties of the target object.

[0176] In some embodiments, when the multimedia file is a video file, the acquisition module 4653 is also used to perform the following processing on the target object appearing in the video file: normalize the target video frame where the target object is located, and partition the normalized target video frame to obtain multiple sub-regions; extract the local binary pattern features corresponding to each sub-region, and perform statistical processing on the multiple local binary pattern features to obtain the local histogram statistical features corresponding to the target video frame; perform local sparse reconstruction representation of the local histogram statistical features through the local feature library of the training set, and perform local reconstruction residual weighted recognition processing on the local sparse reconstruction representation result to obtain the object attributes of the target object.

[0177] In some embodiments, when multiple objects appear in a video file, the subtitle processing device 465 for multimedia files also includes a determination module 4656 for determining a target object from multiple objects in any of the following ways: determining the object that appears the longest in the video file as the target object; determining the object that meets the user's preference in the video file as the target object; determining the object in the video file that is related to user interaction as the target object.

[0178] It should be noted that the description of the device in the embodiment of the present application is similar to the implementation of the subtitle processing method for multimedia files in the above text, and has similar beneficial effects, so it is not repeated here. Figure 3 , Figure 5A ,or Figure 5B The present invention should be understood by referring to the description of any one of the accompanying drawings.

[0179] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the subtitle processing method of the multimedia file described in the embodiment of the present application.

[0180] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the subtitle processing method of the multimedia file provided by the present application embodiment, for example, Figure 3 , Figure 5A ,or Figure 5B A subtitle processing method for a multimedia file is shown.

[0181] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0182] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0183] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0184] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0185] To sum up, in the process of playing multimedia files, the embodiments of the present application display subtitles of a style related to the content of the multimedia file in the human-computer interaction interface, and achieve a diversified display effect of information related to the multimedia file by enriching the expression form of the subtitles. It can be suitable for the diversified subtitle display needs of different application scenarios of multimedia files, and at the same time improves the effect of information dissemination and the user's viewing experience.

[0186] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for processing subtitles of multimedia files, characterized in that: The method comprises: In response to a play trigger operation, playing a multimedia file, wherein the multimedia file is associated with a plurality of original subtitles, and the types of the multimedia file include video files and audio files; Dividing the multimedia file to obtain a plurality of segments, wherein the types of the segments include at least one of the following: object segments, scene segments, and plot segments; Acquire content features of at least one dimension of each of the segments in the multimedia file, wherein the content features include content features of a static dimension and content features of a dynamic dimension; Re-dividing any one or more of the multiple segments to obtain multiple sub-segments, wherein the multiple sub-segments have content features of a static dimension of the segment and content features of a dynamic dimension of the segment, and different sub-segments have different content features of the dynamic dimension; For each of the sub-segments, a content feature of a static dimension and a content feature of a dynamic dimension of the sub-segment are fused to obtain a fused content feature, and a style conversion process is performed on at least one of the original subtitles associated with the sub-segment based on the fused content feature to obtain at least one new subtitle; During the playing of each sub-segment, at least one new subtitle associated with the sub-segment is sequentially displayed in the human-computer interaction interface.

2. The method according to claim 1, characterized in that The performing style conversion processing on at least one original subtitle associated with the sub-segment based on the fused content feature to obtain at least one new subtitle includes: Invoking a subtitle model based on a value corresponding to the fused content feature and at least one original subtitle associated with the sub-segment to obtain at least one new subtitle; The subtitle model is trained as a generative model and is combined with a discriminative model to form a generative adversarial network.

3. The method according to claim 1, It is characterized in that in, The content features of the static dimension of the object segment include at least one of the following object attributes: the role type, gender, and age of the object making the sound in the object segment; the content features of the dynamic dimension of the object segment include the following object attributes: the emotion of the object making the sound in the object segment; The content features of the static dimension of the plot segment include the plot type of the plot segment, and the content features of the dynamic dimension of the plot segment include at least one of the following: scene types of different scenes in which the plot segment appears, object attributes of different sound-making objects in which the plot segment appears; The static dimension content features of the scene clip include: the type of scene involved in the scene clip; The content features of the dynamic dimension of the scene segment include at least one of the following: object attributes of different sound-emitting objects appearing in the scene segment, and types of different plots appearing in the scene segment.

4. The method according to claim 1, characterized in that: The obtaining of at least one dimension of content features of each of the segments in the multimedia file comprises: Calling a content feature recognition model to perform content feature recognition processing on the content of each of the segments in the multimedia file to obtain content features of at least one dimension of each of the segments in the multimedia file; The content feature recognition model is trained based on sample multimedia files and labels annotated with respect to the contents of the sample multimedia files.

5. The method according to claim 1, characterized in that When the multimedia file is the video file, obtaining the content feature of at least one dimension of each of the segments in the multimedia file includes: The following processing is performed for the target object appearing in each segment of the video file: Preprocessing the target video frame where the target object is located; Performing feature extraction on the preprocessed target video frame to obtain image features corresponding to the target video frame; The image features are subjected to dimensionality reduction processing, and the image features subjected to dimensionality reduction processing are classified by a trained classifier to obtain object attributes of the target object.

6. The method according to claim 1, characterized in that When the multimedia file is the video file, obtaining the content feature of at least one dimension of each of the segments in the multimedia file includes: The following processing is performed for the target object appearing in each segment of the video file: Extracting local binary pattern features corresponding to the target video frame where the target object is located, and performing dimensionality reduction processing on the local binary pattern features; Extracting a histogram of oriented gradients corresponding to the target video frame, and performing dimensionality reduction processing on the histogram of oriented gradients; Performing canonical correlation analysis on the local binary pattern features and the directional gradient histogram image after the dimension reduction process to obtain analysis results; Regression processing is performed on the analysis results to obtain object attributes of the target object.

7. The method according to claim 1, characterized in that When the multimedia file is the video file, obtaining the content feature of at least one dimension of each of the segments in the multimedia file includes: The following processing is performed for the target object appearing in each segment of the video file: Normalizing the target video frame where the target object is located, and partitioning the target video frame after the normalization process to obtain a plurality of sub-regions; Extracting a local binary pattern feature corresponding to each of the sub-regions, and performing statistical processing on a plurality of the local binary pattern features to obtain a local histogram statistical feature corresponding to the target video frame; The local histogram statistical features are locally sparsely reconstructed and represented through the local feature library of the training set, and the local sparse reconstruction representation result is subjected to local reconstruction residual weighted recognition processing to obtain the object attributes of the target object.

8. The method according to any one of claims 5 to 7, characterized in that: When multiple objects appear in the video file, the target object is determined from the multiple objects by any of the following methods: Determine the object that appears the longest in the video file as the target object; Determine the object in the video file that meets the user's preference as the target object; An object in the video file that is related to the user interaction is determined as the target object.

9. A subtitle processing device for multimedia files, characterized in that: The device comprises: A playing module is used to play a multimedia file in response to a play trigger operation, wherein the multimedia file is associated with a plurality of original subtitles, and the types of the multimedia file include video files and audio files; divide the multimedia file into a plurality of segments, wherein the types of the segments include at least one of the following: object segments, scene segments, and plot segments; An acquisition module, configured to acquire content features of at least one dimension of each of the segments in the multimedia file, wherein the content features include content features of a static dimension and content features of a dynamic dimension; The playback module is further configured to divide any one or more of the multiple segments again to obtain multiple sub-segments, wherein the multiple sub-segments have content features of a static dimension of the segment and content features of a dynamic dimension of the segment, and different sub-segments have different content features of the dynamic dimension; a conversion module, configured to, for each of the sub-segments, fuse the content features of the static dimension and the content features of the dynamic dimension of the sub-segment to obtain a fused content feature, and perform style conversion processing on at least one of the original subtitles associated with the sub-segment based on the fused content feature to obtain at least one new subtitle; The display module is used to sequentially display the at least one new subtitle associated with the sub-segment in the human-computer interaction interface during the playing of each sub-segment.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; The processor is configured to implement the subtitle processing method for multimedia files according to any one of claims 1 to 8 when executing the executable instructions stored in the memory.

11. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the subtitle processing method for multimedia files according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the subtitle processing method for multimedia files according to any one of claims 1 to 8 is implemented.

Images