Video processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a video processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring a multimedia file in response to a video playing request; the video playing request carries a subtitle display style; extracting a video stream and a subtitle stream from the multimedia file; according to the playing time of the video stream and the playing time of the subtitle stream, aligning subtitle frames in the subtitle stream with image frames in the video stream; generating a caption image with the caption display style according to the caption frame, and fusing the caption image with the aligned image frame to obtain a fused image frame; and rendering the fused image frame to obtain a target video picture. By adopting the method, the user can adjust the subtitle style, and the synchronism of the picture and the subtitle is considered at the same time.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a video processing method, apparatus, computer device, storage medium, and computer program product. Background Art

[0002] Subtitles are text content that matches the video content in the video frame. Through subtitles, users can better understand the video content. There are two traditional subtitle display methods: "hard subtitles" and "soft subtitles". Hard subtitles refer to permanently embedding subtitles into the video frame as part of the video frame, while soft subtitles are to temporarily superimpose subtitles as an independent file on the video frame.

[0003] However, although the hard subtitle method can keep the subtitles and the picture in sync, it lacks flexibility, and users cannot switch languages or adjust styles. Although soft subtitles can allow users to switch languages or adjust styles, the player needs to first decode the video and then separately process the subtitles, which is prone to problems such as delay and frame freezing, resulting in the problem of out-of-sync subtitles and pictures. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a video processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can keep the subtitles and the picture in sync and can flexibly adjust the subtitle style.

[0005] On the one hand, this application provides a video processing method. The method includes:

[0006] In response to a video playback request, obtain a multimedia file; the video playback request carries a subtitle display style;

[0007] Extract a video stream and a subtitle stream from the multimedia file;

[0008] Align subtitle frames in the subtitle stream with image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream;

[0009] Generate a subtitle image with the subtitle display style according to the subtitle frames, and fuse the subtitle image with the aligned image frames to obtain a fused image frame;

[0010] Render the fused image frame to obtain a target video frame.

[0011] On the other hand, this application also provides a video processing apparatus. The apparatus includes:

[0012] An obtaining module, configured to obtain a multimedia file in response to a video playback request; the video playback request carries a subtitle display style;

[0013] An extraction module for extracting a video stream and a subtitle stream from the multimedia file;

[0014] An alignment module for aligning subtitle frames in the subtitle stream and image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream;

[0015] A fusion module for generating a subtitle image with the subtitle display style according to the subtitle frame, and fusing the subtitle image with the aligned image frame to obtain a fused image frame;

[0016] A rendering module for rendering the fused image frame to obtain a target video picture.

[0017] In one embodiment, the video stream includes image frames in a first data format; the fusion module is further configured to generate a subtitle image with the subtitle display style according to the subtitle frame, the subtitle image is an image in a second data format; convert the format of the subtitle image in the second data format to obtain a subtitle image in the first data format; fuse the subtitle image in the first data format and the aligned image frame to obtain a fused image frame.

[0018] In one embodiment, the fusion module is further configured to obtain a preset offset ratio between the subtitle image in the first data format and the aligned image frame; determine the size of the aligned image frame, and determine the subtitle addition position in the aligned image frame according to the size of the aligned image frame and the preset offset ratio; fuse the subtitle image in the first data format to the subtitle addition position in the aligned image frame to obtain a fused image frame.

[0019] In one embodiment, the fusion module is further configured to determine a first luminance value of the subtitle image in the luminance channel and a second chrominance value in the chrominance channel; determine a second luminance value of the subtitle addition position in the luminance channel and a second chrominance value in the chrominance channel; fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused image frame.

[0020] In one embodiment, the subtitle image includes a character outline image; the fusion module is further configured to perform outline drawing on each character in the subtitle frame to obtain a character outline image corresponding to the targeted character; adjust the style of the character outline image according to the subtitle display style to obtain a character outline image with the subtitle display style.

[0021] In one embodiment, the fusion module is further configured to identify characters in the subtitle frame and determine multiple key points for each character; for each character in the subtitle frame, connect the multiple key points of the targeted character to obtain the corresponding character contour image of the targeted character, and obtain the corresponding character contour image of the targeted character.

[0022] In one embodiment, the multiple key points include multiple first key points and multiple second key points; the fusion module is further configured to, for each character in the subtitle frame, connect the multiple first key points of the targeted character to obtain the first character contour of the targeted character; connect the multiple second key points of the targeted character to obtain the second character contour of the targeted character; fill the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character.

[0023] In one embodiment, the subtitle display style includes the subtitle language type; the extraction module is further configured to extract a video file and a subtitle file from the multimedia file; decode the video file to obtain a video stream; extract the subtitle stream belonging to the subtitle language type from the subtitle file.

[0024] In one embodiment, the extraction module is further configured to, when there is no subtitle file in the multimedia file, extract an audio file from the multimedia file; perform speech recognition and text conversion on the audio file to obtain a subtitle file corresponding to the audio file.

[0025] In one embodiment, the acquisition module is further configured to, in response to a video playback request, detect the ambient light brightness in the current playback environment;

[0026] The rendering module is further configured to render the fusion image frame to obtain a candidate video picture; adjust the brightness of the candidate video picture according to the ambient light brightness to obtain a target video picture.

[0027] In one embodiment, the device further includes:

[0028] A jump module, configured to, in response to a trigger operation on the subtitle in the target video picture, detect a video picture matching the subtitle; jump from the target video picture to the video picture matching the subtitle and play it.

[0029] On the other hand, the present application also provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] In response to a video playback request, obtain a multimedia file; the video playback request carries a subtitle display style;

[0031] Extract a video stream and a subtitle stream from the multimedia file;

[0032] Align subtitle frames in the subtitle stream and image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream;

[0033] Generate a subtitle image with the subtitle display style according to the subtitle frames, and fuse the subtitle image with the aligned image frames to obtain a fused image frame;

[0034] Render the fused image frame to obtain a target video picture.

[0035] On the other hand, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] In response to a video playback request, obtain a multimedia file; the video playback request carries a subtitle display style;

[0037] Extract a video stream and a subtitle stream from the multimedia file;

[0038] Align subtitle frames in the subtitle stream and image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream;

[0039] Generate a subtitle image with the subtitle display style according to the subtitle frames, and fuse the subtitle image with the aligned image frames to obtain a fused image frame;

[0040] Render the fused image frame to obtain a target video picture.

[0041] On the other hand, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0042] In response to a video playback request, obtain a multimedia file; the video playback request carries a subtitle display style;

[0043] Extract a video stream and a subtitle stream from the multimedia file;

[0044] Align subtitle frames in the subtitle stream and image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream;

[0045] Generate a subtitle image with the subtitle display style according to the subtitle frame, and fuse the subtitle image with the aligned image frame to obtain a fused image frame;

[0046] Render the fused image frame to obtain a target video frame.

[0047] The above video processing method, device, computer device, storage medium, and computer program product, by responding to a video playback request, obtain a multimedia file indicated by the video playback request, and extract an independent video stream and an independent subtitle stream from the multimedia file. According to the playback time of the video stream and the playback time of the subtitle stream, align the subtitle frames in the subtitle stream with the image frames in the video stream, which can maintain the synchronization of the subtitle frames in the subtitle stream and the image frames in the video stream in terms of playback time. The subtitle display style is carried in the video playback request, so that a subtitle image with the user-specified subtitle display style can be generated according to the subtitle frame. Fusing the subtitle image with the aligned image frame can embed the subtitle image into the aligned image frame as a part of the image frame. Rendering the fused image frame to obtain a target video frame can effectively avoid the problem of out-of-sync subtitles and pictures. Moreover, the user can still adjust the subtitle display style in real time, achieving flexible adjustment of subtitles and compatibility of synchronous playback of subtitles and video frames. Description of the Drawings

[0048] Figure 1 It is an application environment diagram of the video processing method in an embodiment;

[0049] Figure 2 It is a schematic flowchart of the video processing method in an embodiment;

[0050] Figure 3 It is a schematic diagram of aligning video frames and subtitle frames in time in an embodiment;

[0051] Figure 4 It is a schematic diagram of the principle of fusing the subtitle image with the aligned image frame by means of channel fusion in another embodiment;

[0052] Figure 5 It is a schematic diagram of the principle of calculating the subtitle adding position according to the size of the aligned image frame and a preset offset ratio in an embodiment;

[0053] Figure 6 It is a schematic diagram of drawing the outer contour and inner contour of a character in an embodiment;

[0054] Figure 7 It is a schematic diagram of magnifying the character contour away from the inside of the character to obtain the outer contour in an embodiment;

[0055] Figure 8 Schematic diagram of reducing a character outline in a direction close to the inside of the character to obtain an inner outline in an embodiment;

[0056] Figure 9 Schematic diagram of determining multiple key points of a character in an embodiment;

[0057] Figure 10 Schematic diagram of the architecture of video processing in an embodiment;

[0058] Figure 11 Block diagram of the structure of a video processing device in an embodiment;

[0059] Figure 12 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0060] In order to make the objectives, technical solutions, and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] The video processing method provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the terminal 102 communicates with the server 104 through a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed in the cloud or other devices. Both the terminal 102 and the server 104 can independently execute the video processing method provided in the embodiments of the present application. The terminal 102 and the server 104 can also cooperate to execute the video processing method provided in the embodiments of the present application. When the terminal 102 and the server 104 cooperate to execute the video processing method provided in the embodiments of the present application, the terminal 102 responds to a video playback request, sends the video playback request to the server 104, and the video playback request carries a subtitle display style. The server 104 receives the video playback request and obtains the corresponding multimedia file. The server 104 extracts a video stream and a subtitle stream from the multimedia file, aligns the subtitle frames in the subtitle stream and the image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream. The server 104 generates a subtitle image with the subtitle display style according to the subtitle frames, and fuses the subtitle image with the aligned image frames to obtain a fused image frame. The server 104 renders the fused image frame to obtain a target video picture. The server 104 feeds back the target video picture to the terminal 102, and the target video picture is played on the terminal 102.

[0062] Among them, the terminal 102 can be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0063] In one embodiment, as Figure 2 shown, a video processing method is provided. Taking this method applied to Figure 1 the computer device in Figure 1 such as the terminal or server shown in

[0064] as an example, the method includes the following steps:

[0064] Step S202, in response to a video playback request, obtain a multimedia file; the video playback request carries a subtitle display style.

[0065] Among them, the video playback request is a request triggered by the user, used to indicate playing a video. The video playback request carries a subtitle display style. The subtitle display style refers to the style in which subtitles are displayed in the video frame. The subtitle display style includes at least one of the following: subtitle language type, subtitle size, subtitle color, subtitle position, font type.

[0066] The multimedia file is a file related to the target video. The multimedia file may include a video file, and may also include a subtitle file, an audio file, etc.

[0067] Specifically, when the user wants to play a video, the user can select the target video to be played on the video playback interface of the playback application. The computer device generates a video playback request in response to the user's selection operation, determines the target video indicated by the video playback request, and thus obtains the multimedia file corresponding to the target video.

[0068] Among them, the video playback interface is an interface for playing the target video, and this playback interface can be a pop-up window, a floating layer, or an independent interface.

[0069] The target video can be any type of video. For example, the target video can be a complete video, or a video segment in a complete video, or a video synthesized by mixing multiple videos.

[0070] Specifically, a playback application with video playback function can be run on the terminal, and enter the video playback interface of the playback application to play the video.

[0071] A playback application refers to an application with video playback functionality, such as specifically a long-video application, a short-video application, a music application, a live broadcast application, etc., but not limited to these. The playback application can be presented to the user in the form of an application program, and the user can play videos through the application program. The application program can refer to a client installed in the terminal. The application program can also refer to a non-installable application program, that is, an application program that can be used without downloading and installation. Such an application program can also be called a mini-program. It usually runs as a subprogram in the parent application, and this subprogram is also called a sub-application.

[0072] In one embodiment, the playback application can also be a cloud application. A cloud application refers to an application that runs in the cloud. A cloud application refers to an application that interacts between the terminal and the cloud. The running mode of the cloud application is to encode the running process into an audio-video stream through the powerful computing power of the cloud emulator and transmit it to the terminal through the network to achieve interaction with the user.

[0073] The cloud refers to the cloud server. A cloud server is based on a large-scale distributed computing system, and integrates computer resources through virtualization technology to provide services for Internet infrastructure. The network that provides resources is called the "cloud". The resources in the "cloud" seem to the user to be infinitely expandable, and can be obtained at any time, used on demand, expanded at any time, and paid according to usage. Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, enabling various application systems to obtain computing power, storage space, and information services according to needs.

[0074] In one embodiment, the user can select the subtitle display style on the video playback interface and select the video to be played. The computer device generates a video playback request carrying the subtitle display style in response to the user's selection operation.

[0075] In this embodiment, when the user does not select the subtitle display style, the computer device can obtain the default display style as the subtitle display style.

[0076] In one embodiment, the video playback request can be a request triggered by the user at any time. For example, it can be triggered before the target video is played or at any time during the playback of the target video.

[0077] Step S204, extract the video stream and subtitle stream from the multimedia file.

[0078] Specifically, the computer device parses the multimedia file to separate the video stream and subtitle stream.

[0079] In this embodiment, the computer device can parse the multimedia file to separate the video file and the subtitle file. The video stream is extracted from the video file, and the subtitle stream is extracted from the subtitle file.

[0080] Step S206: Align the subtitle frames in the subtitle stream with the image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream.

[0081] Specifically, the computer device can obtain the playback time of the video stream and the playback time of the subtitle stream, and the playback time of the video stream and the playback time of the subtitle stream are kept synchronized. The computer device can align the subtitle frames in the subtitle stream with the image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream, and obtain the image frames aligned with each subtitle frame.

[0082] As Figure 3 shown, the playback time of subtitle frame 1 is the same as the playback time of image frame 1, the playback time of subtitle frame 2 is the same as the playback time of image frame 2, and the playback time of subtitle frame 3 is the same as the playback time of image frame 3. Align subtitle frames 1, 2, and 3 with image frames 1, 2, and 3 respectively according to the playback time.

[0083] In one embodiment, a preset number of subtitle frames and image frames can be aligned each time. For example, if there are 100 frames in both the subtitle stream and the audio stream, 10 subtitle frames and image frames can be aligned each time.

[0084] Step S208: Generate a subtitle image with a subtitle display style according to the subtitle frame, and fuse the subtitle image with the aligned image frame to obtain a fused image frame.

[0085] Specifically, the computer device generates a subtitle image with a subtitle display style according to the subtitle frames in the subtitle stream, and fuses the subtitle image with the aligned image frame to obtain a fused image frame.

[0086] In one embodiment, the computer device performs text rendering on the subtitle frames based on the subtitle display style to obtain a subtitle image with the subtitle display style. In other embodiments, the subtitle frames can also be first text-rendered, and the image obtained by the text rendering is adjusted to the subtitle display style to obtain a subtitle image with the subtitle display style.

[0087] In one embodiment, fusing the subtitle image with the aligned image frame to obtain a fused image frame includes:

[0088] Determine the subtitle addition position in the image frame aligned with the subtitle image; fuse the subtitle image to the subtitle addition position in the aligned image frame to obtain a fused image frame.

[0089] In one embodiment, the subtitle image can be fused to the subtitle addition position in a pixel fusion manner to obtain a fused image frame. For example, the pixels of the subtitle image and the pixels at the subtitle addition position in the aligned image frame can be fused, so as to add the subtitle image to the subtitle addition position and obtain a fused image frame.

[0090] In one embodiment, the subtitle image can be fused to the subtitle addition position in a channel fusion manner to obtain a fused image frame. As Figure 4 shown, determine the first luminance value of the subtitle image on the luminance channel and the first chrominance value on the chrominance channel, and determine the second luminance value of the subtitle addition position on the luminance channel and the second chrominance value on the chrominance channel. Fuse the first luminance value and the second luminance value to obtain a fused luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused chrominance value, whereby a fused image frame can be obtained.

[0091] In one embodiment, the subtitle addition position can be specified by the user or calculated according to the size of the aligned image frame and a preset offset ratio. When the user specifies the subtitle addition position, the subtitle image is fused to the subtitle addition position specified by the user. When the user does not specify the subtitle addition position, the subtitle addition position can be calculated according to the size of the aligned image frame and the preset offset ratio.

[0092] In one embodiment, fusing the subtitle image with the aligned image frame to obtain a fused image frame includes: obtaining a preset offset ratio between the subtitle image and the aligned image frame; determining the size of the aligned image frame, and determining the subtitle addition position in the aligned image frame according to the size of the aligned image frame and the preset offset ratio; fusing the subtitle image to the subtitle addition position in the aligned image frame to obtain a fused image frame.

[0093] Step S210, rendering the fused image frame to obtain a target video picture.

[0094] Specifically, the computer device renders the fused image frame to obtain a target video picture. The computer device can play the target video picture to complete the response to the user's video play request.

[0095] In one embodiment, a video stream, a subtitle stream, and an audio stream are extracted from the multimedia file. According to the playback time of the video stream, the playback time of the subtitle stream, and the playback time of the audio stream, subtitle frames in the subtitle stream, image frames in the video stream, and audio frames in the audio stream are aligned. The target video picture and the aligned audio frames are played simultaneously so that the video picture, subtitles, and audio are synchronized.

[0096] In this embodiment, in response to a video playback request, a multimedia file indicated by the video playback request is obtained, and an independent video stream and an independent subtitle stream are extracted from the multimedia file. According to the playback time of the video stream and the playback time of the subtitle stream, subtitle frames in the subtitle stream are aligned with image frames in the video stream, so as to maintain the synchronization of subtitle frames in the subtitle stream and image frames in the video stream in terms of playback time. The subtitle display style is carried in the video playback request, so that subtitle images in the specified subtitle display style can be generated according to the subtitle frames. The subtitle images are fused with the aligned image frames, so that the subtitle images can be embedded into the aligned image frames as a part of the image frames. The fused image frames are rendered to obtain a target video picture, which can effectively avoid the problem of out-of-sync subtitles and pictures. Moreover, the user can still adjust the display style of the subtitles in real time, realizing flexible adjustment of the subtitles and compatibility of synchronous playback of the subtitles and the video picture.

[0097] The video processing method in this embodiment is a processing method that can synchronously fuse and render subtitles and image frames into a single-frame video picture in real time, which can take into account the advantages of maintaining subtitle and picture synchronization in hard subtitles and the advantages of users being able to adjust subtitle styles at any time in soft subtitles.

[0098] In one embodiment, the video stream includes image frames in a first data format; generating subtitle images with a subtitle display style according to the subtitle frames, and fusing the subtitle images with the aligned image frames to obtain fused image frames, includes:

[0099] Generating subtitle images with a subtitle display style according to the subtitle frames, where the subtitle images are images in a second data format; converting the format of the subtitle images in the second data format to obtain subtitle images in the first data format; fusing the subtitle images in the first data format and the aligned image frames to obtain fused image frames.

[0100] Specifically, the computer device can decode the video file to obtain a video stream in the first data format. The video stream includes multiple frames of image frames in the first data format.

[0101] Subtitle images with a subtitle display style can be generated according to the subtitle frames, and the generated subtitle images are images in the second data format. The computer device can obtain the conversion relationship between the first data format and the second data format, and based on this conversion relationship, convert the format of the subtitle images in the second data format to obtain subtitle images in the first data format. The computer device fuses the subtitle images in the first data format and the aligned image frames to obtain fused image frames.

[0102] In this embodiment, the computer device can obtain each pixel in the subtitle image in the second data format, convert each pixel in the subtitle image according to the conversion relationship, and obtain each pixel in the first data format. Each pixel in the first data format constitutes the subtitle image in the first data format.

[0103] In one embodiment, the first data format may be the YUV format, and the second data format may be the RGB format. The image frame in the video stream is a YUV image, and the subtitle image in the second data format is an RGB image. The computer device converts the subtitle image from the RGB format to the YUV format according to the conversion relationship between the YUV format and the RGB format, and then fuses the subtitle image in the YUV format and the aligned YUV-format image frame to obtain a fused image frame.

[0104] In this embodiment, the video stream includes image frames in the first data format. A subtitle image with a subtitle display style is generated according to the subtitle frame. The subtitle image is an image in the second data format. The subtitle image in the second data format is subjected to format conversion to obtain a subtitle image in the first data format, so that the data formats of the subtitle image and the image frames in the video stream are consistent. Fusing the subtitle image in the first data format and the aligned image frames to obtain a fused image frame can achieve lossless fusion of the subtitle and the image frame, effectively improve the compatibility of subtitle display and the consistency of picture quality, while maintaining the visual quality of the original image frame, and ensuring the accuracy and efficiency of the subtitle overlay process.

[0105] In one embodiment, fusing the subtitle image in the first data format and the aligned image frames to obtain a fused image frame includes:

[0106] Obtaining a preset offset ratio between the subtitle image in the first data format and the aligned image frames; determining the size of the aligned image frames, and determining the subtitle addition position in the aligned image frames according to the size of the aligned image frames and the preset offset ratio; fusing the subtitle image in the first data format to the subtitle addition position in the aligned image frames to obtain a fused image frame.

[0107] Wherein, the preset offset ratio is the offset ratio of the subtitle relative to the target position in the image frame set in advance. The preset offset ratio includes a first offset ratio in the first direction and a second offset ratio in the second direction. The first direction and the second direction are perpendicular to each other.

[0108] The size of the image frame includes a first size in the first direction and a second size in the second direction. For example, the first size in the first direction may be the width of the image frame, and the second size in the second direction may be the height of the image frame.

[0109] Specifically, the computer device obtains a preset offset ratio between the subtitle image in the first data format and the aligned image frame. The computer device determines the size of the aligned image frame, and determines the subtitle addition position in the aligned image frame according to the size of the aligned image frame and the preset offset ratio. The subtitle image in the first data format is fused to the subtitle addition position in the aligned image frame to obtain a fused image frame.

[0110] Further, as Figure 5 shown, the computer device determines a first addition position according to the first size and the first offset ratio of the aligned image frame. A second addition position is determined according to the second size and the second offset ratio of the aligned image frame. The position corresponding to both the first addition position and the second addition position in the aligned image frame is used as the subtitle addition position.

[0111] For example, the coordinate system of the image frame usually takes the upper left corner as the origin coordinate (0, 0), and the origin is the target position. The lower right corner coordinate is (width, height), which represents the width and height of the image frame. The subtitle addition position is represented by the (x, y) coordinates, and the unit is pixels. The calculation method of the subtitle addition position is as follows:

[0112] x = width of the image frame × first offset ratio;

[0113] y = height of the image frame × second offset ratio;

[0114] For example, x = width of the image frame × 0.8, y = height of the image frame × 0.9, indicating that the subtitle will be displayed in the lower right corner of the image frame.

[0115] In this embodiment, by obtaining the preset offset ratio between the subtitle image in the first data format and the aligned image frame, determining the size of the aligned image frame, determining the subtitle addition position in the aligned image frame according to the size of the aligned image frame and the preset offset ratio, and fusing the subtitle image in the first data format to the subtitle addition position in the aligned image frame to obtain a fused image frame, the precise addition position of the subtitle in the image frame is dynamically calculated through the preset offset ratio, ensuring the spatial layout coordination between the subtitle and the picture content. At the same time, the subtitle position is adaptively adjusted based on the size of the target image frame, which not only avoids blocking key picture areas but also improves the flexibility and accuracy of subtitle overlay, ultimately achieving seamless fusion of high-quality and position-controllable subtitles and video pictures.

[0116] In one of the embodiments, fusing the subtitle image in the first data format to the subtitle addition position in the aligned image frame to obtain a fused image frame includes:

[0117] Determine the first luminance value of the subtitle image in the first data format on the luminance channel and the first chrominance value on the chrominance channel; determine the second luminance value of the subtitle addition position on the luminance channel and the second chrominance value on the chrominance channel; fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused image frame.

[0118] Specifically, the computer device determines the first luminance value of the subtitle image in the first data format on the luminance channel and determines the first chrominance value of the subtitle image in the first data format on the chrominance channel. The computer device determines the second luminance value of the subtitle addition position in the image frame on the luminance channel and the second chrominance value of the subtitle addition position in the image frame on the chrominance channel. Fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused image frame.

[0119] In one embodiment, there are two chrominance channels. The computer device determines the first chrominance value of the subtitle image in the first data format on each chrominance channel and determines the second chrominance value of the subtitle addition position in the image frame on each chrominance channel. For each chrominance channel, fuse the first chrominance value and the second chrominance value of the targeted chrominance channel.

[0120] For example, luminance (Y) channel superposition:

[0121] Y_new = Y_video × (1 - alpha) + Y_subtitle × alpha;

[0122] Where alpha represents the subtitle transparency, 0 means completely transparent, and 1 means completely opaque. Y_subtitle is the first luminance value of the subtitle image. Y_video is the second luminance value of the subtitle addition position in the image frame. Y_new is the fused luminance value obtained by fusing the first luminance value and the second luminance value.

[0123] Chrominance (U, V) channel superposition:

[0124] U_new = U_video × (1 - alpha) + U_subtitle × alpha;

[0125] V_new = V_video × (1 - alpha) + V_subtitle × alpha;

[0126] Among them, U and V are two different chrominance channels respectively. U_subtitle is the first chrominance value of the subtitle image on the chrominance channel U. U_video is the second chrominance value of the subtitle addition position in the image frame on the chrominance channel U. U_new is the chrominance value obtained by fusing the first chrominance value and the second chrominance value on the chrominance channel U.

[0127] V_subtitle is the first chrominance value of the subtitle image on the chrominance channel V. V_video is the second chrominance value of the subtitle addition position in the image frame on the chrominance channel V. V_new is the chrominance value obtained by fusing the first chrominance value and the second chrominance value on the chrominance channel V.

[0128] In this embodiment, the first luminance value of the subtitle image in the first data format on the luminance channel and the second chrominance value on the chrominance channel are determined, the second luminance value of the subtitle addition position on the luminance channel and the second chrominance value on the chrominance channel are determined, the first luminance value and the second luminance value are fused, and the first chrominance value and the second chrominance value are fused to obtain a fused image frame. By calculating the numerical differences between the subtitle image and the target image on the luminance channel and the chrominance channel respectively and performing precise fusion on them, the clear readability of the subtitle is retained, and at the same time, the color and luminance of the subtitle and the background picture are ensured to have a natural transition, so as to avoid the subtitle being obtrusive while achieving visual consistency with the original picture and improving the overall visual experience.

[0129] In one of the embodiments, the subtitle image includes a character outline image; generating a subtitle image with a subtitle display style according to the subtitle frame includes:

[0130] For each character in the subtitle frame, contour drawing is performed on the targeted character to obtain a character outline image corresponding to the targeted character; the character outline image is adjusted in style according to the subtitle display style to obtain a character outline image with the subtitle display style.

[0131] Among them, the character outline image can specifically be at least one of a vector font image, a raster font image, or an outline font image. A vector font, also known as a vector-based font, is a font representation method based on geometric graphics.

[0132] In a vector font, each character is composed of a series of straight lines, curves, and other geometric graphics. Since it is vector-based, the font can maintain clarity and smoothness regardless of the font size change.

[0133] A raster font, also known as a bitmap font, is a method of representing text in pixels. Each character is composed of a series of pixels, and different strokes are represented by different pixel values.

[0134] A contour font is a representation method that combines vector fonts and raster fonts. In a contour font, each character is represented as a closed geometric contour, which can be filled or stroked. Contour fonts are compatible with the advantages of vector fonts, such as scale invariance, and the fast processing speed of raster fonts.

[0135] Specifically, the computer device can recognize each character in the subtitle frame. For each character in the subtitle frame, perform contour drawing on the targeted character to obtain a character contour image corresponding to the targeted character. Adjust the style of the character contour image according to the subtitle display style to obtain a character contour image with the subtitle display style.

[0136] In one embodiment, as Figure 6 shown, for each character in the subtitle frame, the computer device can perform outer contour drawing and inner contour drawing on the targeted character respectively to obtain a character contour image corresponding to the targeted character. The character contour image includes the outer contour and inner contour of the character, as well as the closed area between the outer contour and the inner contour.

[0137] In one embodiment, the computer device can perform contour drawing on the targeted character respectively to obtain a character contour; perform a scaling operation on the character contour to obtain a scaled character contour, and the scaling operation includes at least one of the following operations: a shrinking operation in the direction towards the inside of the targeted character, or an enlarging operation in the direction away from the inside of the targeted character.

[0138] In this embodiment, the scaling operation can be a shrinking operation on the character contour or an enlarging operation on the character contour.

[0139] As Figure 7 shown, enlarge the character contour 1 in the direction away from the inside of the character to obtain the outer contour 2. The area between the character contour 1 and the character contour 2 is the closed area.

[0140] Or, as Figure 8 shown, shrink the character contour 1 in the direction towards the inside of the character to obtain the inner contour 3. The area between the character contour 1 and the character contour 3 is the closed area.

[0141] Or, take the character contour 1, the character contour 3, and the closed area between the character contour 1 and the character contour 3 together as the character contour image.

[0142] In this embodiment, through character-by-character contour drawing and style adjustment, the vector characteristics are maintained, ensuring that the characters have clear edges at any resolution, avoiding scaling distortion, and achieving a high-precision and customizable subtitle rendering effect. Moreover, it can flexibly adjust attributes such as color, stroke, and shadow according to the display style specified by the user to meet diverse subtitle display requirements. Additionally, the contour-based drawing method reduces the bitmap processing overhead, is especially suitable for dynamic subtitles or complex scenes, and is compatible with different devices and playback environments.

[0143] In one embodiment, for each character in the subtitle frame, contour drawing is performed on the targeted character to obtain a character contour image corresponding to the targeted character, including:

[0144] Identify the characters in the subtitle frame and determine multiple key points for each character; for each character in the subtitle frame, connect the multiple key points of the targeted character to obtain a character contour image corresponding to the targeted character.

[0145] Specifically, the computer device can identify each character in the subtitle frame. For each character, determine multiple key points of the targeted character. Among them, the multiple key points can characterize the contour of the character, as Figure 9 shown.

[0146] For each character in the subtitle frame, the computer device can use a curve to connect the multiple key points of the targeted character to obtain a character contour image corresponding to the targeted character.

[0147] In one embodiment, the computer device can use a closed curve to connect the multiple key points of the targeted character and fill the closed area formed after connecting the multiple key points to obtain a character contour image corresponding to the targeted character.

[0148] In this embodiment, identify the characters in the subtitle frame and determine multiple key points for each character. For each character in the subtitle frame, use a curve to connect the multiple key points of the targeted character and fill the closed area formed after the curve connects the multiple key points. It can accurately generate the character contour through key point recognition, curve connection, and closed filling, accurately restore the glyph structure, accurately capture the geometric features of the character through key point positioning and curve fitting, and ensure that the contour is consistent with the original design. Moreover, the vector-based contour generation method supports arbitrary scaling and deformation and adapts to different resolution scenarios.

[0149] In one embodiment, the multiple key points include multiple first key points and multiple second key points; for each character in the subtitle frame, using a curve to connect the multiple key points of the targeted character to obtain a character contour image corresponding to the targeted character, including:

[0150] For each character in the subtitle frame, connect multiple first key points of the targeted character to obtain the first character contour of the targeted character; connect multiple second key points of the targeted character to obtain the second character contour of the targeted character; fill the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character.

[0151] Specifically, for each character, the computer device determines multiple first key points of the targeted character and determines multiple second key points of the targeted character. Use a curve to connect multiple first feature points to obtain the first character contour of the targeted character. Use a curve to connect multiple second feature points to obtain the second character contour of the targeted character. A closed area is formed between the first character contour and the second character contour of the targeted character, and this closed area is filled to obtain the corresponding character contour image of the targeted character.

[0152] In one embodiment, identifying the characters in the subtitle frame and determining multiple key points for each character includes:

[0153] Identify the characters in the subtitle frame and determine multiple first key points for each character; for each character, move the multiple first key points in a direction towards the inside of the targeted character by a first preset distance to obtain multiple second key points of the targeted character. Among them, the first character contour obtained by connecting each first key point is the outer contour of the targeted character. The second character contour obtained by connecting each first key point is the inner contour of the targeted character.

[0154] In one embodiment, identifying the characters in the subtitle frame and determining multiple key points for each character includes:

[0155] Identify the characters in the subtitle frame and determine multiple first key points for each character; for each character, move the multiple first key points in a direction away from the inside of the targeted character by a second preset distance to obtain multiple second key points of the targeted character. Among them, the first character contour obtained by connecting each first key point is the inner contour of the targeted character. The second character contour obtained by connecting each first key point is the outer contour of the targeted character.

[0156] In one embodiment, for each character in the subtitle frame, performing contour drawing on the targeted character to obtain the corresponding character contour image of the targeted character includes:

[0157] In one embodiment, multiple key points include multiple first key points and multiple second key points; for each character in the subtitle frame, using a curve to connect multiple key points of the targeted character to obtain the corresponding character contour image of the targeted character includes:

[0158] For each character in the subtitle frame, connect multiple first key points of the targeted character to obtain the first character contour of the targeted character; connect multiple second key points of the targeted character to obtain the second character contour of the targeted character; based on the first character contour and the second character contour, generate a character contour image corresponding to the targeted character.

[0159] In this embodiment, the multiple key points include multiple first key points and multiple second key points. For each character in the subtitle frame, connect the multiple first key points of the targeted character to obtain the first character contour of the targeted character, and connect the multiple second key points of the targeted character to obtain the second character contour of the targeted character, so as to accurately generate the character contour through key point positioning. Fill the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character, which can accurately restore the glyph structure through key point recognition, curve connection and closed filling, and the vector-based contour generation method supports arbitrary scaling and deformation, which can effectively avoid subtitle distortion and can adapt to different resolution scenarios.

[0160] In one embodiment, the subtitle display style includes the subtitle language type; extracting the video stream and subtitle stream from the multimedia file includes:

[0161] Extract the video file and subtitle file from the multimedia file; decode the video file to obtain the video stream; extract the subtitle stream belonging to the subtitle language type from the subtitle file.

[0162] Among them, the subtitle display style includes the subtitle language type, and the subtitle language type refers to the language type of the subtitle, such as displaying Chinese subtitles, displaying English subtitles, etc., but not limited to this.

[0163] Specifically, the computer device unpacks the multimedia file to obtain the video file and subtitle file. The computer device performs decoding processing on the video file to obtain the video stream. The video stream includes multiple image frames.

[0164] When the subtitle display style includes the subtitle language type, it means that the user has selected the subtitle language to be displayed, then the computer device can extract the subtitle stream belonging to the subtitle language type from the subtitle file. The computer device generates a subtitle image with the subtitle language type according to the subtitle frames in the subtitle stream.

[0165] In one embodiment, the multimedia file further includes an audio file, and the audio file may include audio of at least one language type. The computer device extracts the audio stream belonging to the subtitle language type from the audio file, so that when the target video is played later, the video picture, the displayed subtitle and the audio can be synchronized.

[0166] In one embodiment, the subtitle file may include subtitle streams of at least one language type. When the subtitle display style includes the subtitle language type, the computer device may determine whether there is a subtitle stream belonging to the subtitle language type in the subtitle file. If so, it extracts the subtitle stream belonging to the subtitle language type from the subtitle file.

[0167] In one embodiment, when there is no subtitle stream belonging to the subtitle language type in the subtitle file, the computer device extracts the audio file from the multimedia file. Perform speech recognition on the audio file to identify whether there is audio belonging to the subtitle language type in the audio file. If so, perform text conversion on the audio belonging to the subtitle language type to obtain a subtitle stream belonging to the subtitle language type.

[0168] In this embodiment, the subtitle display style includes the subtitle language type selected by the user. By separating and extracting the video file and the subtitle file from the multimedia file, and selectively decoding the corresponding subtitle stream for the subtitle language type specified by the user, efficient and accurate subtitle extraction and matching are achieved. This can reduce redundant data processing, only decode the target language subtitles, reduce system resource consumption, and improve processing efficiency. Moreover, it can ensure the synchronous output of the video stream and the specified language subtitle stream, enhancing the user experience in multi-language scenarios.

[0169] In one embodiment, the method further includes:

[0170] When there is no subtitle file in the multimedia file, extract the audio file from the multimedia file; perform speech-to-text processing on the audio file to obtain a subtitle file corresponding to the audio file.

[0171] Specifically, for some videos without subtitles, when it is detected that there is no subtitle file in the multimedia file, the audio file can be extracted from the multimedia file. The computer device can perform speech recognition and text conversion on the audio file to obtain a subtitle file corresponding to the audio file.

[0172] In one embodiment, the audio file includes audio and the playback time of the audio. After the computer device converts the audio file into a subtitle file, the subtitle file includes a subtitle stream and the playback time of the subtitle stream. The playback time of the subtitle stream is the playback time of the audio.

[0173] In this embodiment, in the case where the multimedia file lacks a subtitle file, a subtitle file can be automatically generated by extracting the audio file and performing speech recognition, enabling the automatic generation of usable subtitles even when the original file has no subtitles, achieving intelligent subtitle supplementation and enhancement, and improving compatibility. Moreover, it realizes automated subtitle generation, reduces the cost of manual subtitle production, and is suitable for batch processing of a large number of media files.

[0174] In one embodiment, the method further includes: in response to a video playback request, detecting the ambient light brightness in the current playback environment;

[0175] Rendering the fused image frame to obtain a target video picture, including: rendering the fused image frame to obtain a candidate video picture; adjusting the brightness of the candidate video picture according to the ambient light brightness to obtain the target video picture.

[0176] Specifically, the computer device, in response to a video playback request, obtains a multimedia file and detects the ambient light brightness in the current playback environment. After obtaining the fused image frame, the computer device renders the fused image frame to obtain a candidate video picture, and the computer device adjusts the brightness of the candidate video picture according to the ambient light brightness to obtain the target video picture.

[0177] In this embodiment, in response to a video playback request, the ambient light brightness in the current playback environment is automatically detected, the fused image frame is rendered to obtain a candidate video picture, and the brightness of the candidate video picture is adjusted according to the ambient light brightness, so that the brightness of the rendered video picture can be automatically adjusted according to the ambient light brightness, making the target video picture suitable for the brightness in the current playback environment and improving the user's viewing experience.

[0178] In one embodiment, the method further includes:

[0179] In response to a trigger operation on the subtitle in the target video picture, detecting a video picture matching the subtitle; jumping from the target video picture to the video picture matching the subtitle and playing it.

[0180] Specifically, when playing the target video picture, the user can trigger the subtitle in the target video picture to jump to the video picture that the user wants to watch and play it.

[0181] When the computer device detects a trigger operation on the subtitle in the target video picture, it detects a video picture matching the subtitle. When a video picture matching the subtitle is detected, it jumps from the target video picture to the video picture matching the subtitle and plays it.

[0182] For example, in a teaching video, when the user wants to jump from the currently played video chapter picture to the video chapter picture mentioned in the subtitle, the user can directly click on the subtitle, and the computer device can detect the video chapter picture matching the subtitle from all the video chapter pictures, so as to jump to the matching video chapter picture.

[0183] In this embodiment, in response to a trigger operation on the subtitles in the target video frame, a video frame matching the subtitles is detected, and the video jumps from the target video frame to the video frame matching the subtitles and plays. This enables, even when the subtitles are embedded in the image frames, intelligent detection of the subtitle content triggered by the user, automatic positioning and jumping to the associated video frame, achieving precise linkage between the subtitles and the video content, significantly enhancing the convenience of video interaction and the viewing experience, allowing the user to quickly locate key information while maintaining the smoothness and coherence of the playback process.

[0184] In one embodiment, a video processing method is provided, which is applied to a computer device and includes:

[0185] In response to a video playback request, a multimedia file is obtained, and the ambient light brightness in the current playback environment is detected; the subtitle display style is carried in the video playback request, and the subtitle display style includes the subtitle language type;

[0186] The video file and the subtitle file are extracted from the multimedia file;

[0187] The video file is decoded to obtain a video stream; the video stream includes image frames in a first data format;

[0188] The subtitle stream belonging to the subtitle language type is extracted from the subtitle file;

[0189] According to the playback time of the video stream and the playback time of the subtitle stream, the subtitle frames in the subtitle stream and the image frames in the video stream are aligned;

[0190] The characters in the subtitle frames are recognized, and multiple first key points and multiple second key points of each character are determined;

[0191] For each character in the subtitle frame, the multiple first key points of the targeted character are connected to obtain the first character contour of the targeted character;

[0192] The multiple second key points of the targeted character are connected to obtain the second character contour of the targeted character;

[0193] The area between the first character contour and the second character contour is filled to obtain the corresponding character contour image of the targeted character, and the character contour image is an image in a second data format;

[0194] The character contour image is adjusted in style according to the subtitle display style to obtain a character contour image with the subtitle display style;

[0195] The character contour image in the second data format is converted in format to obtain a character contour image in the first data format;

[0196] Obtain the preset offset ratio between the character contour image in the first data format and the aligned image frames;

[0197] Determine the size of the aligned image frames, and based on the size of the aligned image frames and the preset offset ratio, determine the subtitle addition positions in the aligned image frames;

[0198] Determine the first luminance value of the character contour image in the first data format on the luminance channel, and the second chrominance value on the chrominance channel;

[0199] Determine the second luminance value of the subtitle addition positions on the luminance channel, and the second chrominance value on the chrominance channel;

[0200] Fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused image frame.

[0201] Render the fused image frame to obtain a candidate video frame; adjust the luminance of the candidate video frame according to the ambient light luminance to obtain the target video frame.

[0202] In response to a trigger operation on the subtitle in the target video frame, detect the video frame matching the subtitle; jump from the target video frame to the video frame matching the subtitle and play it.

[0203] In this embodiment, the subtitle display style includes the subtitle language type selected by the user. By separating and extracting the video file and the subtitle file from the multimedia file, and selectively decoding the corresponding subtitle stream for the subtitle language type specified by the user, efficient and accurate subtitle extraction and matching are achieved. This can reduce redundant data processing, only decode the subtitles in the target language, reduce system resource consumption, and improve processing efficiency.

[0204] For each character in the subtitle frame of the subtitle stream, connect multiple first key points of the targeted character to obtain the first character contour of the targeted character, and connect multiple second key points of the targeted character to obtain the second character contour of the targeted character, thereby achieving accurate generation of the character contour through key point positioning. Fill the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character, which can accurately restore the glyph structure through key point recognition, curve connection, and closed filling, and the vector-based contour generation method supports arbitrary scaling and deformation, which can effectively avoid subtitle distortion and can adapt to different resolution scenarios.

[0205] Determine the first luminance value of the subtitle image in the first data format on the luminance channel and the second chrominance value on the chrominance channel. Determine the second luminance value of the subtitle addition position on the luminance channel and the second chrominance value on the chrominance channel. Fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused image frame. By calculating the numerical differences between the subtitle image and the target image on the luminance channel and the chrominance channel respectively and performing precise fusion on them, not only the clear readability of the subtitle is retained, but also the natural transition of the color and luminance between the subtitle and the background picture is ensured, thus avoiding the abruptness of the subtitle while achieving visual consistency with the original picture and enhancing the overall visual experience.

[0206] In response to a video playback request, automatically detect the ambient light luminance in the current playback environment, and automatically adjust the luminance of the rendered video picture according to the ambient light luminance, so that the target video picture can be suitable for the luminance in the current playback environment, which can improve the user's viewing experience.

[0207] During the playback of the target video picture, in response to a trigger operation on the subtitle in the target video picture, detect the video picture matching the subtitle, and jump from the target video picture to the video picture matching the subtitle and play it, so that in the case of embedding the subtitle in the image frame, it is still possible to automatically detect the subtitle content triggered by the user through intelligent detection, automatically locate and jump to the associated video picture, realizing the precise linkage between the subtitle and the video content, significantly improving the convenience of video interaction and the viewing experience, enabling the user to quickly locate key information, and at the same time maintaining the fluency and coherence of the playback process.

[0208] In one embodiment, the application scenario of a video processing method is improved. The schematic diagram in this application scenario is as Figure 10 shown. Obtain a multimedia file, which includes a video file, an audio file, and a subtitle file. The subtitle file is a multilingual subtitle file, including subtitle files in multiple languages. Demultiplex the multimedia file to separate the video file, the audio file, and the subtitle file. In the video decoder, perform decoding processing on the video file to obtain a video stream, and the video stream includes an image sequence. Extract the subtitle stream from the subtitle file through a subtitle synchronization controller, and align the image frames in the video stream and the subtitle frames in the subtitle stream according to the playback time. At the same time, the subtitle synchronization controller also converts the subtitle image into a subtitle image with a subtitle display style. The subtitle overlay engine overlays the subtitle image onto the aligned image frame and outputs a fused image frame. Render the fused image frame to obtain a target video picture. Output the target video picture to a playback application for playback.

[0209] In this embodiment, a solution for synchronously fusing subtitles and video frames into a single-frame picture in real time is provided, which can be compatible with the advantages of traditional hard subtitles in maintaining subtitle and picture synchronization, as well as the advantage that users can adjust subtitles at any time in traditional soft subtitles. That is: when the user needs to play a video, subtitles and pictures are directly merged according to time during the video file decoding process to generate a complete single-frame picture of "video frame + subtitles". In this way, only single-frame pictures need to be processed, effectively ensuring the synchronization of the video picture and the displayed subtitles to avoid the problem of out-of-sync subtitles and pictures. Moreover, users can adjust the subtitles at any time, enhancing the interactivity between the user and the subtitles during the video viewing process and effectively improving the user's viewing experience.

[0210] In this application scenario, the subtitle synchronization controller is a key module, which is responsible for dynamically aligning the playback time axis of multi-language subtitles with the playback time axis of image frames, and managing the priority of subtitles to ensure the precise synchronization of subtitles and image frames and support the dynamic switching of multi-language subtitles.

[0211] The inputs of the subtitle synchronization controller are the image sequence and multi-language subtitle frames. The image sequence is obtained by video decoding, and the image sequence carries timestamps (PTS, in seconds or milliseconds). The multi-language subtitle frames contain the playback time (such as 00:01:15 - 00:01:20) and subtitles in text format (such as SRT, ASS format).

[0212] The subtitle synchronization controller outputs synchronization instructions to instruct the subtitle overlay engine to display specific subtitles on the image frame at a specific time. The synchronization instruction is like "display 'Hello' on the image frame at the 5th second".

[0213] Timeline alignment algorithm: Compare the timestamps of the image frames with the timestamps of the subtitle frames to match the image frames and subtitle frames. For example: the timestamps corresponding to image frame 1, image frame 2, and image frame 3 are 1000, 1500, and 1800, while the timestamp of subtitle frame 1 is 1000 - 2000, indicating that subtitle frame 1 is aligned with the timestamps of these three image frames, namely image frame 1, image frame 2, and image frame 3. It can be understood that one subtitle frame can be aligned with multiple image frames, and one image frame can also be aligned with multiple subtitle frames.

[0214] Priority management of multi-language subtitles:

[0215] Priority rule: The priority of the language manually selected by the user > the priority of the default language > the priority of other languages. Other languages can be arranged in encoding order or alphabetical order.

[0216] The subtitle overlay engine is responsible for converting the synchronized subtitles into graphic objects (i.e., subtitle images), synthesizing them with the image frames, controlling the display effects and hierarchical relationships of the subtitles, ensuring the natural fusion of the subtitles and the video picture, and supporting multiple subtitle styles and dynamic effects.

[0217] Specifically, when the user wants to play a video, the user can select the target video to be played on the video playback interface of the playback application. Moreover, the user can also select the subtitle display style on the video playback interface, such as selecting the desired language subtitle, the added position of the subtitle (i.e., the subtitle display position), the display size of the subtitle, the display color, the display font, etc., but not limited thereto.

[0218] The terminal generates a video playback request in response to the user's selection operation, and the video playback request carries the subtitle display style. The terminal can send the video playback request to the server, and the server determines the target video indicated by the video playback request, and thus feeds back the multimedia file corresponding to the target video.

[0219] The terminal unpacks the multimedia file and separates the video file, the audio file, and the subtitle file. The video file and the subtitle file are input into a video decoder, and the video decoder includes a subtitle synchronization controller and a subtitle overlay engine.

[0220] The video decoder decodes the video file to obtain a video stream, and the video stream includes an image sequence, that is, a plurality of image frames that are temporally continuous. The image frames in the video stream are images in YUV format.

[0221] The subtitle synchronization controller extracts a subtitle stream from the subtitle file, and the subtitle stream includes subtitle frames that are temporally continuous. The subtitle synchronization controller aligns the image frames in the video stream and the subtitle frames in the subtitle stream according to the playback time. When the subtitle display style includes the subtitle language type selected by the user, the subtitle synchronization controller extracts the subtitle stream belonging to the subtitle language type from the subtitle file. When the user does not select the subtitle language type, the default language type is used as the subtitle language type in the subtitle display style, and the subtitle stream belonging to the default language type is extracted from the subtitle file.

[0222] At the same time, the subtitle synchronization controller also generates a subtitle image in RGB format according to the characters in the subtitle frame, and adjusts it to the subtitle image with the color, size, and font specified by the user. Specifically, the characters in the subtitle frame are drawn as character outline images through text rendering technology. The text rendering can specifically be a vector font processing method. The character outline images generated by the vector font processing support arbitrary scaling without distortion. Moreover, the font edges can be smoothed through pixel sampling (such as MSAA). A dotted or solid border can also be added around the text to achieve a stroking effect.

[0223] During the fusion process of the subtitle image and the aligned image frame, the subtitle overlay engine first maintains a drawing order stack (such as [video layer, subtitle layer 1, subtitle layer 2]) to ensure that the subtitle layer covers the image frame layer. That is, set the subtitle layer to cover the image frame layer, and set the subtitle to be displayed only in the transparent area of the image frame to avoid blocking the content in the image frame. The user can customize the transparency of the subtitle.

[0224] The subtitle display style can include the subtitle addition position specified by the user. The coordinate system of the image frame in the subtitle overlay engine usually has the upper left corner as the origin coordinate (0, 0), and the origin is the target position. The lower right corner coordinate is (width, height), which represents the width and height of the image frame. The subtitle addition position is represented by the (x, y) coordinates, and the unit is pixels. If the user specifies the subtitle addition position, the coordinates of the subtitle addition position in the image frame are:

[0225] x = the fixed pixel value specified by the user (such as 100px);

[0226] y = the fixed pixel value specified by the user (such as 200px).

[0227] If the user does not specify the position, it can be automatically calculated according to the width and height of the image frame. That is, the calculation method of the subtitle addition position is as follows:

[0228] x = the width of the image frame × the first offset ratio;

[0229] y = the height of the image frame × the second offset ratio;

[0230] A first offset ratio of 0.1 means the subtitle is left-aligned, and a first offset ratio of 0.5 means the subtitle is centered.

[0231] A second offset ratio of 0.9 means the subtitle is bottom-aligned.

[0232] For example, x = width × 0.8, y = height × 0.9, which means the subtitle is displayed in the lower right corner of the image frame.

[0233] After determining the subtitle addition position, the subtitle overlay engine can convert the subtitle image in RGB format to YUV format, and overlay the YUV format subtitle image into the aligned YUV format image frame to obtain a fused image frame. This fused image frame has the YUV format. Here, it is mainly necessary to modify the luminance (Y) and chrominance (U, V) channels.

[0234] For example, the luminance (Y) channel overlay:

[0235] Y_new = Y_video × (1 - alpha) + Y_subtitle × alpha;

[0236] Among them, alpha represents the subtitle transparency, 0 means completely transparent, and 1 means completely opaque. Y_subtitle is the first luminance value of the subtitle image. Y_video is the second luminance value at the subtitle addition position in the image frame. Y_new is the fused luminance value obtained by fusing the first luminance value and the second luminance value.

[0237] Chrominance (U, V) channel superposition:

[0238] U_new = U_video × (1 - alpha) + U_subtitle × alpha;

[0239] V_new = V_video × (1 - alpha) + V_subtitle × alpha;

[0240] Among them, U and V are two different chrominance channels respectively. U_subtitle is the first chrominance value of the subtitle image on the chrominance channel U. U_video is the second chrominance value at the subtitle addition position in the image frame on the chrominance channel U. U_new is the chrominance value obtained by fusing the first chrominance value and the second chrominance value on the chrominance channel U.

[0241] V_subtitle is the first chrominance value of the subtitle image on the chrominance channel V. V_video is the second chrominance value at the subtitle addition position in the image frame on the chrominance channel V. V_new is the chrominance value obtained by fusing the first chrominance value and the second chrominance value on the chrominance channel V.

[0242] Among them, the resolution of the chrominance channel is usually lower than that of the luminance channel (such as YUV420), which can be achieved by downsampling.

[0243] Boundary processing: Ensure that the subtitle overlay area does not exceed the boundary of the image frame. Anti-aliasing can be performed on the subtitle edges to avoid jagged display and make the subtitles smooth.

[0244] After obtaining the fused image frame in YUV format, the fused image frame in YUV format can be converted into a fused image frame in RGB format, and the fused image frame can be rendered to obtain the target video picture. Play the target video picture on the video playback interface.

[0245] During the playback process, the ambient light luminance in the current playback environment can be detected, and the luminance of the played video picture can be dynamically adjusted based on the ambient light luminance so that the luminance of the played video picture matches the current ambient light luminance.

[0246] Moreover, the user can click on the subtitles in the target video frame to detect other video frames that match the subtitles, and thus jump to the video frame with subtitle matching and continue playing.

[0247] During the playback process, the user can adjust the display style of the subtitles at any time. When the terminal detects the user's operation of adjusting the subtitle display style, it can re-execute the processes of aligning the subtitle frames in the subtitle stream and the image frames in the video stream, generating subtitle images with the subtitle display style, fusing the subtitle images with the aligned image frames, and rendering the fused image frames to obtain the target video frame, etc., according to the above various embodiments, so as to ensure both subtitle and video synchronization and allow the user to adjust the subtitle style at any time.

[0248] In this embodiment, through the pipelining processing method, the subtitle overlay and decoding processes are executed in parallel, avoiding the serial delay in the traditional solution that needs to wait for complete decoding before processing, and can significantly reduce the processing delay. When the end-to-end delay is reduced by 30%-50%, it is especially suitable for high-timeliness scenarios such as live streaming and video conferencing.

[0249] Moreover, directly operating on the memory of the decoded YUV frames without additional memory copying or format conversion (such as converting YUV format to RGB respectively) can reduce the CPU / GPU load by 20%-30% and achieve efficient utilization of computing resources.

[0250] Moreover, modifying the pixel data in the subtitle coverage area to achieve local update optimization, avoiding full-frame traversal, and saving computing resources. It is also possible to directly operate on the pixels in the YUV space to flexibly implement dynamic effects of complex subtitles such as semi-transparency, shadows, and gradients. Operating directly in the YUV space can also avoid color distortion caused by RGB conversion and achieve the effect of lossless picture quality.

[0251] The video processing method in this embodiment can be applied to various application scenarios such as real-time streaming media, offline video editing, and multi-screen terminals (such as set-top boxes and smart TVs). Moreover, by docking different decoder interfaces (FFmpeg / hardware API) through the adaptation layer, multi-platform deployment on Windows / Linux / Android / iOS, etc., is achieved, realizing cross-platform compatibility. It supports video processing from low resolution (480p) to ultra-high definition resolution (4K / 8K), and realizes resolution adaptive adjustment.

[0252] In addition, by dynamically matching the subtitles with the video frames through the PTS timestamps, accurate synchronization between the subtitles and the image frames is achieved, avoiding the problem of out-of-sync audio and video.

[0253] In other embodiments, the application scenarios of the video processing method include, but are not limited to, live broadcast scenarios, video conferencing scenarios, subtitle assistance scenarios for the hearing-impaired, and in-vehicle AR navigation scenarios. When applied to a live broadcast scenario, it enables the product in the live broadcast to be synchronized with the explanatory subtitles. When applied to a video conferencing scenario, it can achieve zero delay between the speaker's voice and the translated subtitles. When applied to the subtitle assistance scenario for the hearing-impaired, it can achieve the synchronous overlay of sign language animations and text. When applied to in-vehicle AR navigation, it makes the road name indication more consistent with the actual road, and the navigation more accurate.

[0254] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least some of the steps or stages in other steps or other steps.

[0255] Based on the same inventive concept, the embodiments of the present application also provide a video processing device for implementing the above-mentioned video processing method. The solution provided by this device to solve problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the video processing device provided below can refer to the limitations on the video processing method in the above text, and will not be repeated here.

[0256] In one embodiment, as Figure 11 shown, a video processing device 1100 is provided, including:

[0257] An acquisition module 1102, configured to acquire a multimedia file in response to a video playback request; the video playback request carries a subtitle display style.

[0258] An extraction module 1104, configured to extract a video stream and a subtitle stream from the multimedia file.

[0259] An alignment module 1106, configured to align the subtitle frames in the subtitle stream with the image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream.

[0260] A fusion module 1108, configured to generate a subtitle image with a subtitle display style according to the subtitle frames, and fuse the subtitle image with the aligned image frames to obtain a fused image frame.

[0261] A rendering module 1110, configured to render the fused image frames to obtain a target video picture.

[0262] In this embodiment, in response to a video playback request, a multimedia file indicated by the video playback request is obtained, and an independent video stream and an independent subtitle stream are extracted from the multimedia file. According to the playback time of the video stream and the playback time of the subtitle stream, the subtitle frames in the subtitle stream and the image frames in the video stream are aligned, so as to maintain the synchronization of the subtitle frames in the subtitle stream and the image frames in the video stream in terms of playback time. The subtitle display style is carried in the video playback request, so that subtitle images in the user-specified subtitle display style can be generated according to the subtitle frames. The subtitle images are fused with the aligned image frames, so that the subtitle images can be embedded in the aligned image frames as a part of the image frames. The fused image frames are rendered to obtain a target video picture, which can effectively avoid the problem of out-of-sync subtitles and pictures. Moreover, the user can still adjust the display style of the subtitles in real time, realizing flexible adjustment of the subtitles and the compatibility of synchronous playback of the subtitles and the video picture.

[0263] The video processing device in this embodiment is a processing device that can synchronously fuse and render subtitles and image frames into a single-frame video picture in real time, and can take into account the advantages of maintaining the synchronization of subtitles and pictures in hard subtitles and the advantages of the user being able to adjust the subtitle style at any time in soft subtitles.

[0264] In one embodiment, the video stream includes image frames in a first data format; the fusion module 1108 is further configured to generate subtitle images with a subtitle display style according to the subtitle frames, the subtitle images being images in a second data format; convert the format of the subtitle images in the second data format to obtain subtitle images in the first data format; and fuse the subtitle images in the first data format with the aligned image frames to obtain fused image frames.

[0265] In this embodiment, the video stream includes image frames in a first data format. Subtitle images with a subtitle display style are generated according to the subtitle frames, the subtitle images being images in a second data format. The format of the subtitle images in the second data format is converted to obtain subtitle images in the first data format, so that the data formats of the subtitle images and the image frames in the video stream are consistent. The subtitle images in the first data format are fused with the aligned image frames to obtain fused image frames, which can achieve lossless fusion of subtitles and image frames, effectively improve the compatibility of subtitle display and the consistency of picture quality, while maintaining the visual quality of the original image frames, and ensure the accuracy and efficiency of the subtitle overlay process.

[0266] In one embodiment, the fusion module 1108 is further configured to obtain a preset offset ratio between the subtitle image in the first data format and the aligned image frames; determine the size of the aligned image frames, and determine the subtitle addition positions in the aligned image frames according to the size of the aligned image frames and the preset offset ratio; fuse the subtitle image in the first data format to the subtitle addition positions in the aligned image frames to obtain fused image frames.

[0267] In this embodiment, by obtaining the preset offset ratio between the subtitle image in the first data format and the aligned image frames, determining the size of the aligned image frames, determining the subtitle addition positions in the aligned image frames according to the size of the aligned image frames and the preset offset ratio, and fusing the subtitle image in the first data format to the subtitle addition positions in the aligned image frames to obtain fused image frames, the accurate addition positions of the subtitles in the image frames are dynamically calculated through the preset offset ratio, ensuring the coordination of the spatial layout between the subtitles and the picture content. At the same time, the subtitle positions are adaptively adjusted based on the size of the target image frames, avoiding the occlusion of key picture areas and improving the flexibility and accuracy of subtitle overlay, and finally achieving seamless fusion of high-quality and position-controllable subtitles and video pictures.

[0268] In one embodiment, the fusion module 1108 is further configured to determine a first luminance value of the subtitle image in the first data format on the luminance channel and a second chrominance value on the chrominance channel; determine a second luminance value of the subtitle addition position on the luminance channel and a second chrominance value on the chrominance channel; fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain fused image frames.

[0269] In this embodiment, by determining the first luminance value of the subtitle image in the first data format on the luminance channel and the second chrominance value on the chrominance channel, determining the second luminance value of the subtitle addition position on the luminance channel and the second chrominance value on the chrominance channel, fusing the first luminance value and the second luminance value, and fusing the first chrominance value and the second chrominance value to obtain fused image frames, the numerical differences between the subtitle image and the target image on the luminance channel and the chrominance channel are calculated respectively and precisely fused, not only retaining the clear readability of the subtitles, but also ensuring the natural transition of the color and luminance with the background picture, thus avoiding the abruptness of the subtitles and achieving visual consistency with the original picture, and improving the overall visual experience.

[0270] In one embodiment, the subtitle image includes a character outline image; the fusion module 1108 is further configured to perform outline drawing on each character in the subtitle frame to obtain a character outline image corresponding to the targeted character; adjust the style of the character outline image according to the subtitle display style to obtain a character outline image with the subtitle display style.

[0271] In this embodiment, through character-by-character contour drawing and style adjustment, the vector characteristics are maintained, ensuring that the characters have clear edges at any resolution, avoiding scaling distortion, and achieving a high-precision and customizable subtitle rendering effect. Moreover, it can flexibly adjust attributes such as color, stroke, and shadow according to the display style specified by the user to meet the diverse subtitle display requirements. Also, the contour-based drawing method reduces the bitmap processing overhead, is especially suitable for dynamic subtitles or complex scenes, and is compatible with different devices and playback environments.

[0272] In one of the embodiments, the fusion module 1108 is further configured to identify the characters in the subtitle frame and determine multiple key points for each character; for each character in the subtitle frame, connect the multiple key points of the targeted character to obtain the corresponding character contour image of the targeted character.

[0273] In this embodiment, identify the characters in the subtitle frame and determine multiple key points for each character. For each character in the subtitle frame, use a curve to connect the multiple key points of the targeted character and fill the closed area formed after the curve connects the multiple key points. It can accurately generate the character contour through key point recognition, curve connection, and closed filling, accurately restore the glyph structure, accurately capture the geometric features of the character through key point positioning and curve fitting, and ensure that the contour is consistent with the original design. Moreover, the vector-based contour generation method supports arbitrary scaling and deformation and adapts to different resolution scenarios.

[0274] In one of the embodiments, the multiple key points include multiple first key points and multiple second key points; the fusion module 1108 is further configured to, for each character in the subtitle frame, connect the multiple first key points of the targeted character to obtain the first character contour of the targeted character; connect the multiple second key points of the targeted character to obtain the second character contour of the targeted character; fill the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character.

[0275] In this embodiment, the multiple key points include multiple first key points and multiple second key points. For each character in the subtitle frame, connect the multiple first key points of the targeted character to obtain the first character contour of the targeted character, and connect the multiple second key points of the targeted character to obtain the second character contour of the targeted character, thereby accurately generating the character contour through key point positioning. Fill the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character. It can accurately restore the glyph structure through key point recognition, curve connection, and closed filling. Moreover, the vector-based contour generation method supports arbitrary scaling and deformation, can effectively avoid subtitle distortion, and can adapt to different resolution scenarios.

[0276] In one embodiment, the subtitle display style includes the subtitle language type; the extraction module 1104 is further configured to extract a video file and a subtitle file from the multimedia file; decode the video file to obtain a video stream; and extract a subtitle stream belonging to the subtitle language type from the subtitle file.

[0277] In this embodiment, the subtitle display style includes the subtitle language type selected by the user. By separating and extracting the video file and the subtitle file from the multimedia file, and selectively decoding the corresponding subtitle stream for the subtitle language type specified by the user, efficient and accurate subtitle extraction and matching are achieved. This can reduce redundant data processing, only decode the target language subtitles, reduce system resource consumption, and improve processing efficiency. Moreover, it can ensure the synchronous output of the video stream and the specified language subtitle stream, enhancing the user experience in multi-language scenarios.

[0278] In one embodiment, the extraction module 1104 is further configured to, when there is no subtitle file in the multimedia file, extract an audio file from the multimedia file; perform speech recognition and text conversion on the audio file to obtain a subtitle file corresponding to the audio file.

[0279] In this embodiment, in the case where the multimedia file lacks a subtitle file, a subtitle file can be automatically generated by extracting the audio file and performing speech recognition, enabling the automatic generation of usable subtitles even when the original file has no subtitles, achieving intelligent subtitle supplementation and enhancement, and improving compatibility. Moreover, it realizes automated subtitle generation, reduces the cost of manual subtitle production, and is suitable for batch processing of a large number of media files.

[0280] In one embodiment, the acquisition module 1102 is further configured to detect the ambient light brightness in the current playback environment in response to a video playback request;

[0281] The rendering module 1110 is further configured to render the fused image frame to obtain a candidate video picture; and adjust the brightness of the candidate video picture according to the ambient light brightness to obtain a target video picture.

[0282] In this embodiment, in response to a video playback request, the ambient light brightness in the current playback environment is automatically detected, the fused image frame is rendered to obtain a candidate video picture, and the brightness of the candidate video picture is adjusted according to the ambient light brightness. Thus, the brightness of the rendered video picture can be automatically adjusted according to the ambient light brightness, enabling the target video picture to be suitable for the brightness in the current playback environment and improving the user's viewing experience.

[0283] In one embodiment, the device further includes:

[0284] A jump module, configured to detect a video frame matching the subtitle in response to a trigger operation on the subtitle in the target video frame; and jump from the target video frame to the video frame matching the subtitle and play it.

[0285] In this embodiment, in response to a trigger operation on the subtitle in the target video frame, detecting a video frame matching the subtitle, and jumping from the target video frame to the video frame matching the subtitle and playing it, so that in the case where the subtitle is embedded in the image frame, it is still possible to intelligently detect the subtitle content triggered by the user, automatically locate and jump to the associated video frame, realizing the precise linkage between the subtitle and the video content, significantly improving the convenience of video interaction and the viewing experience, enabling the user to quickly locate key information, and at the same time maintaining the smoothness and coherence of the playback process.

[0286] Each module in the above video playback device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0287] In one embodiment, a computer device is provided, and the computer device can be a terminal or a server. Taking the server as an example, its internal structure diagram can be as Figure 12 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video processing data. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through a network connection. When the computer program is executed by the processor, it implements a video processing method.

[0288] In one embodiment, a computer device is provided, and the computer device may be a terminal. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a video processing method. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device may be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0289] Those skilled in the art can understand that Figure 12 the structure shown in

[0290] merely represents a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0291] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the above method embodiments.

[0292] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, it implements the steps in the above method embodiments.

[0293] It should be noted that the user information involved in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data that have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0294] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in this application can be general-purpose processors, central processors, graphics processors, digital signal processors, programmable logic devices, etc., and are not limited thereto.

[0295] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0296] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all fall within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A video processing method, characterized in that, The method includes: In response to a video playback request, obtain a multimedia file; the video playback request carries a subtitle display style. Extract a video stream and a subtitle stream from the multimedia file. Align subtitle frames in the subtitle stream and image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream. Generate a subtitle image with the subtitle display style according to the subtitle frames, and fuse the subtitle image with the aligned image frames to obtain a fused image frame. Render the fused image frame to obtain a target video picture.

2. The method according to claim 1, wherein The video stream includes image frames in a first data format. The step of generating a subtitle image with the subtitle display style according to the subtitle frames, and fusing the subtitle image with the aligned image frames to obtain a fused image frame includes: Generate a subtitle image with the subtitle display style according to the subtitle frames, where the subtitle image is an image in a second data format. Convert the format of the subtitle image in the second data format to obtain a subtitle image in the first data format. Fuse the subtitle image in the first data format and the aligned image frames to obtain a fused image frame.

3. The method according to claim 2, characterized in that, The step of fusing the subtitle image in the first data format and the aligned image frames to obtain a fused image frame includes: Obtain a preset offset ratio between the subtitle image in the first data format and the aligned image frames. Determine the size of the aligned image frames, and determine the subtitle addition position in the aligned image frames according to the size of the aligned image frames and the preset offset ratio. Fuse the subtitle image in the first data format to the subtitle addition position in the aligned image frames to obtain a fused image frame.

4. The method according to claim 3, wherein The step of fusing the subtitle image in the first data format to the subtitle addition position in the aligned image frames to obtain a fused image frame includes: Determine a first luminance value of the subtitle image in the first data format on the luminance channel and a first chrominance value on the chrominance channel. Determine a second luminance value of the subtitle addition position on the luminance channel and a second chrominance value on the chrominance channel. Fuse the first luminance value and the second luminance value, and fuse the first chrominance value and the second chrominance value to obtain a fused image frame.

5. The method according to claim 1, wherein The subtitle image includes a character outline image; the step of generating a subtitle image with the subtitle display style according to the subtitle frames includes: For each character in the subtitle frames, perform outline drawing on the targeted character to obtain a character outline image corresponding to the targeted character. Adjust the style of the character outline image according to the subtitle display style to obtain a character outline image with the subtitle display style.

6. The method according to claim 5, characterized in that, The step of, for each character in the subtitle frames, performing outline drawing on the targeted character to obtain a character outline image corresponding to the targeted character includes: Identify the characters in the subtitle frames and determine multiple key points of each character. For each character in the subtitle frames, connect the multiple key points of the targeted character to obtain a character outline image corresponding to the targeted character.

7. The method according to claim 6, wherein The multiple key points include multiple first key points and multiple second key points; for each character in the subtitle frame, connecting the multiple key points of the targeted character by a curve to obtain the corresponding character contour image of the targeted character includes: For each character in the subtitle frame, connecting the multiple first key points of the targeted character to obtain the first character contour of the targeted character; Connecting the multiple second key points of the targeted character to obtain the second character contour of the targeted character; Filling the area between the first character contour and the second character contour to obtain the corresponding character contour image of the targeted character.

8. The method according to claim 1, wherein The subtitle display style includes the subtitle language type; extracting the video stream and subtitle stream from the multimedia file includes: Extracting the video file and subtitle file from the multimedia file; Decoding the video file to obtain the video stream; Extracting the subtitle stream belonging to the subtitle language type from the subtitle file.

9. The method according to claim 8, characterized in that The method further includes: When there is no subtitle file in the multimedia file, extracting the audio file from the multimedia file; Performing speech-to-text processing on the audio file to obtain the subtitle file corresponding to the audio file.

10. The method according to claim 1, wherein The method further includes: In response to a video playback request, detecting the ambient light brightness in the current playback environment; Rendering the fused image frame to obtain the target video picture includes: Rendering the fused image frame to obtain a candidate video picture; Adjusting the brightness of the candidate video picture according to the ambient light brightness to obtain the target video picture.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: In response to a trigger operation on the subtitle in the target video picture, detecting the video picture matching the subtitle; Jumping from the target video picture to the video picture matching the subtitle and playing it.

12. A video processing device, characterized in that, The apparatus includes: An acquisition module, configured to acquire a multimedia file in response to a video playback request; the subtitle display style is carried in the video playback request; An extraction module, configured to extract the video stream and subtitle stream from the multimedia file; An alignment module, configured to align the subtitle frames in the subtitle stream and the image frames in the video stream according to the playback time of the video stream and the playback time of the subtitle stream; A fusion module, configured to generate a subtitle image with the subtitle display style according to the subtitle frame, and fuse the subtitle image with the aligned image frame to obtain a fused image frame; A rendering module, configured to render the fused image frame to obtain the target video picture.

13. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images