Subtitle generation method, display method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of text generation, and provides a subtitle generation method, a display method, electronic equipment and a storage medium. The method comprises the following steps: determining a video frame comprising embedded subtitles by using a detection model; determining a plurality of subtitle display time periods of the multimedia resource, wherein each video frame in each subtitle display time period comprises an embedded subtitle; determining a to-be-displayed subtitle text corresponding to each subtitle display time period by using a first recognition model; a subtitle file is obtained according to the to-be-displayed subtitle text, when the multimedia resources are played in the elder exclusive subtitle mode, the subtitle file is used for generating to-be-displayed subtitles displayed in an externally hung subtitle mode, and the word size of the to-be-displayed subtitles is larger than that of the embedded subtitles. According to the subtitle generation method disclosed by the invention, the subtitle file can be automatically generated according to the video frame of the multimedia resource, the subtitle file can generate the subtitles to be displayed which are displayed in an external hanging form, and the word size of the subtitles to be displayed is larger than that of the embedded subtitles, so that the watching experience of old users is optimized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of text generation, and in particular to a subtitle generation method, a display method, an electronic device, and a storage medium. Background Art

[0002] Most multimedia resources provide subtitles. Subtitles may be in external form or embedded in the video frame of the multimedia resource. Subtitles displayed in external form can be closed and hidden individually, and their font, font size and other attributes can also be adjusted. However, embedded subtitles cannot be closed and hidden individually when the multimedia resource is playing, and their font, font size and other attributes cannot be adjusted.

[0003] Elderly users have reduced vision and are less able to recognize embedded subtitles. If embedded subtitles are still displayed when playing multimedia resources, the viewing experience of elderly users may be reduced. Therefore, how to improve the viewing experience of elderly users for multimedia resources including embedded subtitles has become a research hotspot in this field. Summary of the invention

[0004] In view of this, the present disclosure proposes a subtitle generation method, a display method, an electronic device and a storage medium. The subtitle generation method disclosed in the present disclosure can automatically generate a subtitle file according to a video frame of a multimedia resource, and the subtitle file can generate subtitles to be displayed in a plug-in form, and make the font size of the subtitles to be displayed larger than the embedded subtitles, thereby optimizing the viewing experience of elderly users.

[0005] According to one aspect of the present disclosure, a subtitle generation method is provided, the method being used to generate a subtitle file for a multimedia resource supporting an exclusive subtitle mode for elders, the method comprising: using a detection model to detect video frames of the multimedia resource to determine video frames including embedded subtitles; determining multiple subtitle display time periods of the multimedia resource according to timestamps of the video frames including embedded subtitles, wherein each video frame within each subtitle display time period includes embedded subtitles; using a first recognition model to recognize video frames within each subtitle display time period, and determining subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result; obtaining a subtitle file of the multimedia resource according to the subtitle text to be displayed corresponding to the multiple subtitle display time periods, wherein when the multimedia resource is played in the exclusive subtitle mode for elders, the subtitle file is used to generate subtitles to be displayed in the form of external subtitles, wherein the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles.

[0006] In a possible implementation, the detection model is used to detect the video frames of the multimedia resource to determine the video frames including embedded subtitles, including: using the detection model to detect images located in a text box detection area in the video frames of the multimedia resource to determine the video frames including embedded subtitles, and determining at least one text box for each video frame including embedded subtitles; determining a reference distance between a center point of a valid text box and a first reference edge of the video frame based on a maximum distance and a minimum distance between the embedded subtitle display area of ​​the multimedia resource and the first reference edge of the video frame, as well as a reference height of the valid text box; for each determined text box, calculating a first difference between an actual distance between the center point of the text box and the first reference edge and the reference distance, as well as a second difference between the reference height and the height of the text box based on the coordinates of the text box; when the absolute value of the first difference corresponding to any text box is greater than a first threshold or the second difference corresponding to any text box is greater than a second threshold, the text box is removed.

[0007] In one possible implementation, the first recognition model supports a frame-by-frame recognition mode, the first recognition model includes multiple sub-models with different parameters, and in the frame-by-frame recognition mode, the first recognition model is used to recognize the video frames in each subtitle display time period, including: for each video frame in each subtitle display time period, using multiple sub-models to recognize at least one text box corresponding to the video frame, and the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; for each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame, and the recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0008] In a possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition results includes: selecting a submodel from the multiple submodels; when the similarity of the recognition results of the selected submodel for any two video frames in any subtitle display time period is less than a third threshold, dividing the subtitle display time period into multiple subtitle display time periods, so that the similarity of the recognition results of the selected submodel for every two video frames in each subtitle display time period is greater than or equal to the third threshold; when the similarity of the recognition results of the selected submodel for every two video frames in any subtitle display time period is greater than or equal to the third threshold, finding the recognition result with the largest number of occurrences from the intersection of all recognition results of each submodel for the video frames in the subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0009] In a possible implementation, the first recognition model supports a sampling recognition mode, the first recognition model includes multiple sub-models with different parameters, and when each video frame in each subtitle display time period of the multimedia resource includes the same embedded subtitles, in the sampling recognition mode, the first recognition model is used to recognize the video frames in each subtitle display time period, including: selecting at least one video frame from each subtitle display time period, for each video frame selected from any subtitle display time period, using multiple sub-models to recognize the image of at least one text box corresponding to the video frame, the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; for each sub-model, the recognition result of the sub-model for the text box corresponding to any video frame is merged as the recognition result of the sub-model for the video frame, and the recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0010] In one possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition results includes: finding the recognition result with the largest number of occurrences from the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0011] In one possible implementation, the method further includes: when all recognition results of each sub-model for video frames within any subtitle display time period have no intersection, expanding the text box detection area, and re-executing the steps of determining at least one text box and subsequent steps for each video frame including embedded subtitles.

[0012] In a possible implementation, the method further includes: when all recognition results of the sub-models for the video frames within any subtitle display time period have no intersection, using a second recognition model to perform speech recognition on the audio of the multimedia resource in the subtitle display time period to obtain a speech recognition subtitle text corresponding to the subtitle display time period; when the speech recognition subtitle text has an intersection with all recognition results of any sub-model for the video frames within the subtitle display time period, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the speech recognition subtitle text.

[0013] In one possible implementation, for each sub-model, the recognition result of the text box corresponding to any video frame by the sub-model is merged as the recognition result of the sub-model for the video frame, including: for any video frame corresponding to multiple text boxes, when merging the recognition results of multiple text boxes corresponding to the video frame by any sub-model, adding spaces between the recognition results corresponding to adjacent text boxes.

[0014] In a possible implementation, the recognition result of the sub-model for the video frame also includes a space, and the method further includes: when the recognition result of any sub-model for any video frame includes a space, judging whether there is a space at the same position in the subtitle text to be displayed corresponding to the subtitle display time period to which the video frame belongs, based on the position of the space; when no space exists, adding a space at the same position.

[0015] In a possible implementation, the method further includes: when the subtitle texts to be displayed corresponding to adjacent subtitle display time periods are the same and the time interval does not exceed a fourth threshold, the adjacent subtitle display time periods are merged into one subtitle display time period, and the subtitle texts to be displayed corresponding to the adjacent subtitle display time periods are used as the subtitle texts to be displayed corresponding to the merged subtitle display time period; when the subtitles to be displayed are displayed in the form of external subtitles, the subtitle texts to be displayed corresponding to each subtitle display time period are displayed within the subtitle display time period.

[0016] According to another aspect of the present disclosure, a subtitle generation device is provided, the device is used to generate a subtitle file for a multimedia resource that supports a senior-exclusive subtitle mode, the device comprising: a detection module, used to detect video frames of the multimedia resource using a detection model, and determine video frames including embedded subtitles; a first determination module, used to determine multiple subtitle display time periods of the multimedia resource according to timestamps of video frames including embedded subtitles, each video frame in each subtitle display time period including embedded subtitles; a first recognition module, used to recognize video frames in each subtitle display time period using a first recognition model, and determine subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result; a second determination module, used to obtain a subtitle file of the multimedia resource according to the subtitle text to be displayed corresponding to the multiple subtitle display time periods, when the multimedia resource is played in the senior-exclusive subtitle mode, the subtitle file is used to generate subtitles to be displayed in the form of external subtitles, and the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles.

[0017] In a possible implementation, the detection model is used to detect the video frames of the multimedia resource to determine the video frames including embedded subtitles, including: using the detection model to detect images located in a text box detection area in the video frames of the multimedia resource to determine the video frames including embedded subtitles, and determining at least one text box for each video frame including embedded subtitles; determining a reference distance between a center point of a valid text box and a first reference edge of the video frame based on a maximum distance and a minimum distance between the embedded subtitle display area of ​​the multimedia resource and the first reference edge of the video frame, as well as a reference height of the valid text box; for each determined text box, calculating a first difference between an actual distance between the center point of the text box and the first reference edge and the reference distance, as well as a second difference between the reference height and the height of the text box based on the coordinates of the text box; when the absolute value of the first difference corresponding to any text box is greater than a first threshold or the second difference corresponding to any text box is greater than a second threshold, the text box is removed.

[0018] In one possible implementation, the first recognition model supports a frame-by-frame recognition mode, the first recognition model includes multiple sub-models with different parameters, and in the frame-by-frame recognition mode, the first recognition model is used to recognize the video frames in each subtitle display time period, including: for each video frame in each subtitle display time period, using multiple sub-models to recognize at least one text box corresponding to the video frame, and the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; for each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame, and the recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0019] In a possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition results includes: selecting a submodel from the multiple submodels; when the similarity of the recognition results of the selected submodel for any two video frames in any subtitle display time period is less than a third threshold, dividing the subtitle display time period into multiple subtitle display time periods, so that the similarity of the recognition results of the selected submodel for every two video frames in each subtitle display time period is greater than or equal to the third threshold; when the similarity of the recognition results of the selected submodel for every two video frames in any subtitle display time period is greater than or equal to the third threshold, finding the recognition result with the largest number of occurrences from the intersection of all recognition results of each submodel for the video frames in the subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0020] In a possible implementation, the first recognition model supports a sampling recognition mode, the first recognition model includes multiple sub-models with different parameters, and when each video frame in each subtitle display time period of the multimedia resource includes the same embedded subtitles, in the sampling recognition mode, the first recognition model is used to recognize the video frames in each subtitle display time period, including: selecting at least one video frame from each subtitle display time period, for each video frame selected from any subtitle display time period, using multiple sub-models to recognize the image of at least one text box corresponding to the video frame, the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; for each sub-model, the recognition result of the sub-model for the text box corresponding to any video frame is merged as the recognition result of the sub-model for the video frame, and the recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0021] In one possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition results includes: finding the recognition result with the largest number of occurrences from the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0022] In one possible implementation, the device also includes: an expansion module, which is used to expand the text box detection area when all recognition results of each sub-model for video frames within any subtitle display time period have no intersection, and cause each module included in the device to re-execute the steps of determining at least one text box and subsequent steps for each video frame including embedded subtitles.

[0023] In a possible implementation, the device also includes: a second recognition module, which is used to use the second recognition model to perform speech recognition on the audio of the multimedia resource in the subtitle display time period when all recognition results of each submodel for the video frames in any subtitle display time period have no intersection, so as to obtain the speech recognition subtitle text corresponding to the subtitle display time period; and a third determination module, which is used to determine the subtitle text to be displayed corresponding to the subtitle display time period according to the speech recognition subtitle text when the speech recognition subtitle text has an intersection with all recognition results of any submodel for the video frames in the subtitle display time period.

[0024] In one possible implementation, for each sub-model, the recognition result of the text box corresponding to any video frame by the sub-model is merged as the recognition result of the sub-model for the video frame, including: for any video frame corresponding to multiple text boxes, when merging the recognition results of multiple text boxes corresponding to the video frame by any sub-model, adding spaces between the recognition results corresponding to adjacent text boxes.

[0025] In one possible implementation, the recognition result of the sub-model for the video frame also includes a space, and the device also includes: a judgment module, which is used to judge whether there is a space at the same position in the subtitle text to be displayed corresponding to the subtitle display time period to which the video frame belongs, according to the position of the space when the recognition result of any sub-model for any video frame includes a space; and an adding module, which is used to add a space at the same position when no space exists.

[0026] In a possible implementation, the device also includes: a merging module, which is used to merge the adjacent subtitle display time periods into one subtitle display time period when the subtitle texts to be displayed corresponding to adjacent subtitle display time periods are the same and the time interval does not exceed a fourth threshold, and use the subtitle texts to be displayed corresponding to the adjacent subtitle display time periods as the subtitle texts to be displayed corresponding to the merged subtitle display time period; when displaying the subtitles to be displayed in the form of plug-in subtitles, the subtitle texts to be displayed corresponding to each subtitle display time period are displayed within the subtitle display time period.

[0027] According to another aspect of the present disclosure, a subtitle display method is provided, the method comprising: in response to an elder-only subtitle mode being triggered, obtaining a subtitle file of a currently played multimedia resource including embedded subtitles; rendering the subtitles to be displayed of the multimedia resource according to the subtitle file, the font size of the subtitles to be displayed being larger than the font size of the embedded subtitles of the multimedia resource; masking or blurring the overlapping area with the subtitles to be displayed in each video frame of the multimedia resource to obtain the video frame to be displayed; and displaying the video frame to be displayed and the subtitles to be displayed to a user, the subtitles to be displayed being displayed in the form of external subtitles.

[0028] In a possible implementation manner, the subtitle file is generated according to the subtitle generation method described above.

[0029] According to another aspect of the present disclosure, a subtitle display device is provided, the device comprising: an acquisition module, for acquiring a subtitle file of a currently played multimedia resource including embedded subtitles in response to the elder-only subtitle mode being triggered; a rendering module, for rendering according to the subtitle file to obtain subtitles to be displayed of the multimedia resource, the font size of the subtitles to be displayed being larger than the font size of the embedded subtitles of the multimedia resource; a processing module, for masking or blurring an overlapping area with the subtitles to be displayed in each video frame of the multimedia resource, to obtain a video frame to be displayed; and a display module, for displaying the video frame to be displayed and the subtitles to be displayed to a user, the subtitles to be displayed being displayed in the form of external subtitles.

[0030] In a possible implementation manner, the subtitle file is generated by the subtitle generating device described above.

[0031] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0032] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0033] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0034] According to the subtitle generation method of the embodiment of the present disclosure, for multimedia resources that support the exclusive subtitle mode for elders, by using the detection model to detect the video frames of the multimedia resources, the video frames including embedded subtitles can be determined; according to the timestamps of the video frames including embedded subtitles, multiple subtitle display time periods of the multimedia resources can be determined, and each video frame in each subtitle display time period includes embedded subtitles; the video frames in each subtitle display time period are identified using the first recognition model, and the subtitle text to be displayed corresponding to the subtitle display time period can be determined according to the recognition result; according to the subtitle text to be displayed corresponding to the multiple subtitle display time periods, the subtitle file of the multimedia resource can be obtained, and when the multimedia resource is played in the exclusive subtitle mode for elders, the subtitle file is used to generate the subtitles to be displayed in the form of plug-in subtitles, and the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles. In summary, the subtitle generation method of the present disclosure can automatically generate a subtitle file according to the video frames of the multimedia resource, and the subtitle file can generate the subtitles to be displayed in the form of plug-in, and make the font size of the subtitles to be displayed larger than the font size of the embedded subtitles, thereby optimizing the viewing experience of elderly users.

[0035] According to the subtitle display method of the embodiment of the present disclosure, in response to the elder-exclusive subtitle mode being triggered, a subtitle file of the currently played multimedia resource including embedded subtitles is obtained, and the subtitles to be displayed of the multimedia resource are rendered according to the subtitle file. The font size of the subtitles to be displayed is larger than the font size of the embedded subtitles of the multimedia resource, so that the subtitles to be displayed are more adapted to the needs of elderly users; the overlapping area with the subtitles to be displayed in each video frame of the multimedia resource is masked or blurred to obtain the video frame to be displayed, and the video frame to be displayed and the subtitles to be displayed are displayed to the user. The subtitles to be displayed are displayed in the form of plug-in subtitles, so that the background of the subtitles to be displayed is an image after masking or blurring, thereby improving the display effect of the subtitles to be displayed.

[0036] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0038] Figure 1 Exemplary application scenarios of the subtitle generation method and subtitle display method according to embodiments of the present disclosure are shown.

[0039] Figure 2 A schematic diagram showing the process of a subtitle generation method according to an embodiment of the present disclosure.

[0040] Figure 3An example of a subtitle presentation period according to an embodiment of the present disclosure is shown.

[0041] Figure 4 A schematic diagram showing an area detected by a detection model according to an embodiment of the present disclosure.

[0042] Figure 5 An example of recognition results of the sub-model for video frame 1 and video frame 2 according to an embodiment of the present disclosure is shown.

[0043] Figure 6 An example of a video frame in which spaces appear in embedded subtitles according to an embodiment of the present disclosure is shown.

[0044] Figure 7 A schematic diagram showing the process of a subtitle display method according to an embodiment of the present disclosure.

[0045] Figure 8 A schematic diagram showing the structure of a second device according to an embodiment of the present disclosure.

[0046] Fig. 9 A schematic diagram showing the structure of a subtitle generating device according to an embodiment of the present disclosure.

[0047] Fig.10 A schematic diagram showing the structure of a subtitle display device according to an embodiment of the present disclosure.

[0048] Fig.11 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0049] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0050] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0051] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0052] Most multimedia resources provide subtitles. Subtitles may be in external form or embedded in the video frame of the multimedia resource. Subtitles displayed in external form can be closed and hidden individually, and their font, font size and other attributes can also be adjusted. However, embedded subtitles cannot be closed and hidden individually when the multimedia resource is playing, and their font, font size and other attributes cannot be adjusted.

[0053] Elderly users have reduced vision and are less able to recognize embedded subtitles. If embedded subtitles are still displayed when playing multimedia resources, the viewing experience of elderly users may be reduced. Therefore, how to improve the viewing experience of elderly users for multimedia resources including embedded subtitles has become a research hotspot in this field.

[0054] In view of this, the present disclosure proposes a subtitle generation method, a display method, an electronic device and a storage medium. The subtitle generation method disclosed in the present disclosure can automatically generate a subtitle file according to a video frame of a multimedia resource, and the subtitle file can generate subtitles to be displayed in a plug-in form, and make the font size of the subtitles to be displayed larger than the embedded subtitles, thereby optimizing the viewing experience of elderly users.

[0055] Optical Character Recognition (OCR) is a text generation technology used to automatically identify and extract characters from images, scanned documents or photos. This technology is widely used in electronic medical records, logistics tracking, digital documents, license plate recognition, intelligent search, quality inspection, bill document processing and other fields. The use of optical character recognition technology can significantly reduce the time and cost of manual data input, improve data accuracy and processing efficiency, and play an important supporting role in the digital transformation of various industries. The subtitle generation method of the disclosed embodiment can use OCR technology to identify video frames, determine the subtitle text to be displayed based on the recognition results, and generate a subtitle file based on the subtitle text to be displayed, and further improve the recognition method and the processing method of the recognition results, thereby improving the accuracy of the subtitle file.

[0056] Automatic Speech Recognition (ASR) is a text generation technology used to convert audio content into text content. When the accuracy of the recognition result obtained by using OCR technology is difficult to determine, the subtitle generation method of the embodiment of the present disclosure also proposes that the audio of the multimedia resource can be automatically recognized, and the recognition result is proofread according to the recognized subtitle text to obtain the subtitle text to be displayed. When the recognition result cannot be proofread by the recognized subtitle text, the subtitle text to be displayed is manually determined, thereby ensuring the accuracy of the subtitle file while reducing labor costs.

[0057] The subtitle display method of the disclosed embodiment supports an elder-exclusive subtitle mode designed for elderly users. In this mode, the subtitle file is rendered to obtain and display the subtitles in an external form, and the subtitles are displayed in a font size larger than the embedded subtitles, which is convenient for elderly users to watch.

[0058] Figure 1 Exemplary application scenarios of the subtitle generation method and subtitle display method according to embodiments of the present disclosure are shown.

[0059] like Figure 1 As shown, the subtitle generation method of the embodiment of the present disclosure can be executed by a first device (such as a server), and the subtitle display method can be executed by a second device (such as a terminal device used by a user). The subtitle generation method can also be executed by the second device.

[0060] In an exemplary application scenario, multiple multimedia resources may be stored on the first device. When any multimedia resource supports the exclusive subtitle mode for elders, the first device executes the subtitle generation method and can generate a subtitle file for the multimedia resource. At least one font file may also be stored on the first device. Among them, the exclusive subtitle mode for elders may also be called the exclusive subtitle mode for the elderly, etc., and the specific name of the mode may be set as needed. The mode may be presented as a control or option in the application, which may be triggered by the user to enter the mode, or the mode may be automatically triggered when the application is started or the video is played according to the identity information such as the user's age. This mode makes the video playback mode more adapted to the needs of the elderly, for example, making the subtitles in the video more suitable for the elderly to read.

[0061] For example, when a user uses the second device and clicks on a multimedia resource, the second device can download the multimedia resource from the first device and play it.

[0062] The first device downloads the subtitle file of the multimedia resource. Optionally, if the second device does not store the font file, the second device may also download the font file from the first device.

[0063] The second device may render the subtitle file to obtain the subtitles to be displayed. The font of the subtitles to be displayed may be determined according to the font file. The second device may adjust the properties of the subtitles to be displayed, such as font size, color, etc., according to application scenario requirements.

[0064] When playing the multimedia resource, the second device can display the subtitles to be displayed to the user, and the subtitles to be displayed are displayed in the form of plug-in subtitles.

[0065] The above is only an exemplary description, and the embodiments of the present disclosure do not limit the specific division of the structures of the server and the terminal device and the functions implemented. For the sake of clarity, the following takes the first device executing the subtitle generation method and the second device executing the subtitle display method as an example.

[0066] Figure 2 A schematic diagram showing the process of a subtitle generation method according to an embodiment of the present disclosure.

[0067] like Figure 2 As shown, in a possible implementation, the embodiment of the present disclosure proposes a subtitle generation method, the method is used to generate a subtitle file for a multimedia resource supporting an elder-specific subtitle mode, the method comprising:

[0068] Step S21, using a detection model to detect video frames of multimedia resources, and determining video frames including embedded subtitles;

[0069] Step S22, determining multiple subtitle display time periods of the multimedia resource according to the timestamps of the video frames including the embedded subtitles, wherein each video frame in each subtitle display time period includes the embedded subtitles;

[0070] Step S23, using the first recognition model to recognize the video frames in each subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result;

[0071] Step S24, obtaining the subtitle file of the multimedia resource according to the subtitle texts to be displayed corresponding to the multiple subtitle display time periods. When the multimedia resource is played in the elder-exclusive subtitle mode, the subtitle file is used to generate the subtitles to be displayed in the form of external subtitles. The font size of the subtitles to be displayed is larger than the font size of the embedded subtitles.

[0072] For example, the detection model and the first recognition model used in the subtitle generation method of the embodiment of the present disclosure are both used to parse and process video frames. Therefore, before executing the subtitle generation method of the embodiment of the present disclosure, the multimedia resources can be separated into audio and video to obtain video frames and audio. The sampling rate of the audio can be greater than 16K.

[0073] Multimedia resources can support multiple definitions. Definition affects recognition accuracy, and the number of pixels affects recognition speed. Therefore, before executing the subtitle generation method of the embodiment of the present disclosure, the definition of the video frame can also be adjusted, for example, to 540P (e.g., 960×540, 864×486), so as to take into account both recognition speed and recognition accuracy.

[0074] Next, the subtitle generation method of the embodiment of the present disclosure can be started. First, step S21 is executed to detect the video frames of the multimedia resource using the detection model to determine the video frames including the embedded subtitles. The detection model can be a neural network model of the prior art or any other type of detection model, which can be used to identify whether there is text in the image. The details of the working method of the detection model are given later.

[0075] Not all video frames in the multimedia resource include embedded subtitles. Therefore, after determining all video frames in the multimedia resource that include embedded subtitles, step S22 can be executed to determine multiple subtitle display time periods of the multimedia resource based on the timestamps of the video frames that include embedded subtitles, and each video frame in each subtitle display time period includes embedded subtitles.

[0076] Figure 3 An example of a subtitle presentation period according to an embodiment of the present disclosure is shown.

[0077] like Figure 3 As shown, the multimedia resource includes video frames 1 to 10, and the timestamps of video frames 1 to 10 are t1 to t10 respectively. If video frame 3 does not include embedded subtitles, the multimedia resource may include 2 subtitle display time periods, subtitle display time period 1 includes video frames 1 and 2, and subtitle display time period 2 includes video frames 4 to 10. It can be seen that the video frames in each subtitle display time period are continuous and include embedded subtitles.

[0078] Furthermore, the minimum duration of a single subtitle display period can be set, for example, 240ms. In this case, a single subtitle display period includes at least 6 video frames (each frame is displayed for 40ms). Figure 3 The multimedia resource shown has only one subtitle display time period, including video frame 4 to video frame 10. The embodiment of the present disclosure does not limit whether to set the minimum duration of a single subtitle display time period and the specific value of the minimum duration.

[0079] After determining the subtitle display time period of the multimedia resource, step S23 can be executed to use the first recognition model to recognize the video frames within each subtitle display time period. The first recognition model can be a neural network model that can realize optical character recognition in the prior art, and the recognition result can include the specific content of the text in the image. The details of the working method of the first recognition model are given later. Those skilled in the art should understand that the first recognition model can also be other types of character recognition models in the prior art, and the embodiments of the present disclosure do not limit the specific recognition method of the first recognition model.

[0080] According to the recognition results of the video frames in any subtitle display time period, the subtitle text to be displayed corresponding to the subtitle display time period can be determined. The recognition results can be directly used as the subtitle text to be displayed, or the recognition results can be optimized by denoising and other optimizations before being used as the subtitle text to be displayed. The exemplary method can be found in the following description of the subtitle text to be displayed.

[0081] After obtaining the subtitle texts to be displayed corresponding to each subtitle display time period, step S24 can be performed to obtain the subtitle file of the multimedia resource according to the subtitle texts to be displayed corresponding to the multiple subtitle display time periods. When playing the multimedia resource in the elder-exclusive subtitle mode, the subtitle file is used to generate the subtitles to be displayed in the form of plug-in subtitles.

[0082] The subtitle file can record the subtitle display time period and the corresponding subtitle text to be displayed. When displaying the subtitles to be displayed generated by the subtitle file, the subtitles to be displayed displayed in each subtitle display time period can be the subtitle text to be displayed corresponding to the subtitle display time period. The subtitles to be displayed are generated and displayed when multimedia resources are played in the elder-exclusive subtitle mode, and the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles. In this case, when the font size of the embedded subtitles of the multimedia resources is small and difficult for elderly users to recognize, the second device can display external subtitles with a larger font size to the user for easy recognition by elderly users.

[0083] Those skilled in the art should understand that, in order to facilitate identification by elderly users, the font, color and other attributes of the generated subtitles to be displayed may be further preset, and the embodiments of the present disclosure do not limit the display effect of the subtitles to be displayed.

[0084] According to the subtitle generation method of the embodiment of the present disclosure, for multimedia resources that support the exclusive subtitle mode for elders, by using the detection model to detect the video frames of the multimedia resources, the video frames including embedded subtitles can be determined; according to the timestamps of the video frames including the embedded subtitles, multiple subtitle display time periods of the multimedia resources can be determined, and each video frame in each subtitle display time period includes embedded subtitles; the video frames in each subtitle display time period are identified using the first recognition model, and the subtitle text to be displayed corresponding to the subtitle display time period can be determined according to the recognition result; according to the subtitle text to be displayed corresponding to the multiple subtitle display time periods, the subtitle file of the multimedia resource can be obtained, and when the multimedia resource is played in the exclusive subtitle mode for elders, the subtitle file is used to generate the subtitles to be displayed in the form of plug-in subtitles, and the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles. In summary, the subtitle generation method of the embodiment of the present disclosure can automatically generate a subtitle file according to the video frames of the multimedia resource, and the subtitle file can generate the subtitles to be displayed in the form of plug-in, and make the font size of the subtitles to be displayed larger than the font size of the embedded subtitles, thereby optimizing the viewing experience of elderly users.

[0085] The following describes an exemplary working method of the detection model of the embodiment of the present disclosure.

[0086] In a possible implementation, using a detection model to detect video frames of a multimedia resource to determine video frames including embedded subtitles includes:

[0087] Using the detection model, detect an image in a text box detection area in a video frame of the multimedia resource, determine a video frame including embedded subtitles, and determine at least one text box for each video frame including embedded subtitles;

[0088] Determine a reference distance between a center point of a valid text box and the first reference edge, and a reference height of the valid text box according to a maximum distance and a minimum distance between an embedded subtitle display area of ​​the multimedia resource and a first reference edge of the video frame;

[0089] For each determined text box, calculate, according to the coordinates of the text box, a first difference between an actual distance between a center point of the text box and a first reference side and a reference distance, and a second difference between a reference height and a height of the text box;

[0090] When the absolute value of the first difference corresponding to any text box is greater than the first threshold or the second difference corresponding to any text box is greater than the second threshold, the text box is removed.

[0091] For example, the embedded position of subtitles of a single multimedia resource is usually fixed, such as in the middle of the bottom of the video frame. However, other areas of the video frame may have background text, special patterns similar to the shape of text, etc., which may interfere with the detection and recognition of text. Therefore, the detection model does not need to detect the entire frame of the image, but only needs to detect the area where the embedded subtitles may appear. Figure 4 A schematic diagram showing an area detected by a detection model according to an embodiment of the present disclosure.

[0092] For example, a staff member may first select N video frames (N is a positive integer) including embedded subtitles, and then determine the first reference edge of the video frame. The selected N video frames are used to determine the area in the video frame where the embedded subtitles may appear. The N video frames including embedded subtitles may be randomly selected, or N video frames with a large number of subtitles may be selected, so that the determined area covers all subtitle content as much as possible. The embodiment of the present disclosure does not limit the selection method of the N video frames. Figure 4 As shown, the video frame may include four edges, and the first reference edge may be any edge parallel to the reading direction of the embedded subtitles ( Figure 4In the example of the top edge of the video frame). The detection model may be used to determine the minimum value y1 and the maximum value y2 of the vertical coordinate of the embedded subtitles in the selected video frame. Exemplarily, the detection model may first determine all pixels covered by the embedded subtitles, and determine the minimum value y1 and the maximum value y2 of the vertical coordinate of the embedded subtitles according to the vertical coordinates of each covered pixel. The minimum value y1 of the vertical coordinate of the embedded subtitles may be the minimum value of the vertical coordinate of each pixel covered by the embedded subtitles, and the maximum value y2 of the vertical coordinate of the embedded subtitles may be the maximum value of the vertical coordinate of each pixel covered by the embedded subtitles.

[0093] Since N video frames including embedded subtitles are selected, there are N y1 and y2 respectively. The N y1 can be counted to find the y1 with the largest number of occurrences, and the y1 with the largest number of occurrences can be subtracted from the ordinate of the first reference edge to obtain the minimum distance TOP_Y between the embedded subtitle display area and the first reference edge. The N y2 can be counted to find the y2 with the largest number of occurrences, and the ordinate with the first reference edge can be subtracted to obtain the maximum distance BOT_Y between the embedded subtitle display area and the first reference edge.

[0094] Those skilled in the art should understand that it is also possible to statistically analyze the N y1s to find the y1 with the smallest value and make a difference between it and the ordinate of the first reference side, or to make a difference between the average value of the N y1s and the ordinate of the first reference side, and to analyze the N y2s to find the y2 with the largest value and make a difference between it and the ordinate of the first reference side, or to make a difference between the average value of the N y2s and the ordinate of the first reference side. The embodiments of the present disclosure do not limit the specific size of the embedded subtitle display area in the ordinate direction, as long as the area includes the embedded subtitles.

[0095] Those skilled in the art should understand that a second reference edge (not shown) can also be determined, and the second reference edge can be any edge perpendicular to the reading direction of the embedded subtitles. The detection model is then used to further determine the minimum and maximum values ​​of the horizontal coordinates of the embedded subtitles in the selected video frame, and based on this, the maximum distance RIGHT_X and the minimum distance LEFT_X between the embedded subtitle display area and the second reference edge are determined, so that the determined embedded subtitle display area is further reduced in the horizontal coordinate direction. The disclosed embodiment does not limit the specific size of the embedded subtitle display area in the horizontal coordinate direction, as long as the area includes embedded subtitles.

[0096] In actual applications, the first device may also automatically select N video frames from the video frames of the multimedia resource, and use the detection model to detect the selected video frames. Those skilled in the art should understand that if there are video frames that do not include embedded subtitles among the selected video frames, other video frames may continue to be selected until the number of selected video frames that include embedded subtitles reaches N. The embodiments of the present disclosure do not limit the specific method of selecting video frames that include embedded subtitles.

[0097] The embedded subtitle display area is determined by the above distances TOP_Y, BOT_Y, RIGHT_X, and LEFT_X, that is, subtitles may exist in the area. After the first device determines the embedded subtitle display area, the staff can also confirm and adjust the embedded subtitle display area to improve the completeness of the subtitles in the embedded subtitle display area. For example, when the embedded subtitle display area is too small and does not contain all the subtitles, the embedded subtitle display area is expanded. The adjustment method will not be described in detail here.

[0098] The detection model also has the function of detecting text boxes. A text box can be a rectangular box containing embedded subtitles. The embedded subtitle display area can be directly determined as the text box detection area (that is, the detection area of ​​the detection model). The embedded subtitle display area can be directly used to detect text boxes, but since the detection efficiency of the detection model for images containing only text is lower than that for images containing text and background, Figure 4 As shown, the embedded subtitle display area can also be enlarged to obtain a text box detection area.

[0099] For example, a threshold value h1 and a threshold value h2 may be preset so that the maximum distance between the embedded subtitle display area and the first reference edge of the video frame increases by h1, and the minimum distance between the embedded subtitle display area and the first reference edge of the video frame decreases by h2, thereby obtaining a text box detection area. In this case, the minimum distance between the text box detection area and the first reference edge is TOP_Y-h1, and the maximum distance between the text box detection area and the first reference edge is BOT_Y+h2.

[0100] The threshold h1 and the threshold h2 can be positive numbers set according to the application scenario requirements. In one example, h1 = h2 = BOT_Y - TOP_Y. The embodiment of the present disclosure does not limit the specific values ​​of the threshold h1 and the threshold h2.

[0101] After determining the text box detection area, the detection model is used to detect the image in the text box detection area in each video frame of the multimedia resource to determine all video frames including embedded subtitles. For each video frame including embedded subtitles, at least one text box is also determined.

[0102] The embedded subtitles of a single video frame may be densely arranged embedded subtitles without spaces, or may be non-densely arranged embedded subtitles including spaces. Ideally, the text box can be the smallest bounding box that includes complete, densely arranged embedded subtitles. For non-densely arranged embedded subtitles, spaces are used as the boundaries of the text box. The text box is detected and determined from the text box detection area, so the text box detection area is larger than the text box.

[0103] Whether there are spaces in the embedded subtitles is identified by the detection model. Since there may be errors in the identification, the embedded subtitles including spaces may also be mistakenly identified as densely arranged embedded subtitles, so that only one text box is determined. That is, if there are no spaces in the embedded subtitles of a single video frame, one text box is determined. If there are spaces in the embedded subtitles of a single video frame, one text box may be determined for the single video frame, and the embedded subtitles in the text box include spaces; or multiple text boxes may be determined, and spaces are used as the boundaries of the text boxes.

[0104] For example, the detection model can first distinguish between space areas, densely arranged embedded subtitle areas, and non-subtitle areas. The detection model can determine whether there are pixels covered by embedded subtitles in each column of pixels in the text box detection area. If there are pixels covered by embedded subtitles in any two columns of pixels and the distance in the horizontal axis direction is less than the preset space length, the area between the two columns of pixels is considered to be a densely arranged embedded subtitle area. If there are pixels covered by embedded subtitles in any two columns of pixels and the distance in the horizontal axis direction is greater than or equal to the preset space length, and the area between the two columns of pixels does not include pixels covered by embedded subtitles, the area between the two columns of pixels is considered to be a space area. If there are pixels covered by embedded subtitles in any column of pixels and the area with a horizontal coordinate less than that of the column of pixels does not include pixels covered by embedded subtitles, or the area with a horizontal coordinate greater than that of the column of pixels does not include pixels covered by embedded subtitles, the area is considered to be a non-subtitle area.

[0105] The detection model can continue to detect the vertical coordinates of the pixels covered by the embedded subtitles in the densely arranged embedded subtitle area, and use the minimum vertical coordinate as the minimum vertical coordinate of the text box, and use the maximum vertical coordinate as the maximum vertical coordinate of the text box. The minimum horizontal coordinate and maximum horizontal coordinate of the densely arranged embedded subtitle area can be directly used as the minimum horizontal coordinate and maximum horizontal coordinate of the text box.

[0106] Those skilled in the art should understand that the detection model for determining a text box should be more than the above example, as long as the text box is within the text box detection area and includes embedded subtitles. The embodiments of the present disclosure do not limit the specific method for determining the text box.

[0107] The coordinates of the detected text box can be stored in a matrix form first. When the first recognition model performs recognition later, the image included in the text box can be found and recognized according to the coordinates of the text box.

[0108] Optionally, the multimedia resource may include a title and an end credit, which generally do not include embedded subtitles. In particular, the end credit may display a cast list, which may interfere with the detection and recognition of text. The time period corresponding to the title and the end credit may be recorded in the attribute information of the multimedia resource. The detection model may also find other video frames other than the title and the end credit for detection based on the attribute information of the multimedia resource to improve the detection efficiency and detection accuracy.

[0109] It should be understood that background text and special patterns close to the shape of text may still appear in the text box detection area, and these text and patterns may also determine the text box, but since the text box does not include embedded subtitles, this text box is actually an invalid text box. If the determined text boxes are all recognized by the first recognition model, the content that is not embedded subtitles may be recognized as subtitle text. Therefore, it is necessary to screen out invalid text boxes and remove them.

[0110] The reference distance HOR_Y between the center point of the effective text box and the first reference edge, and the reference height SUB_H of the effective text box can be determined according to the maximum distance BOT_Y and the minimum distance TOP_Y between the embedded subtitle display area of ​​the multimedia resource and the first reference edge of the video frame. In one example, HOR_Y = (BOT_Y + TOP_Y) / 2, SUB_H = BOT_Y - TOP_Y. The embodiments of the present disclosure do not limit the specific setting method of the reference distance between the center point of the effective text box and the first reference edge and the reference height of the effective text box.

[0111] The above-mentioned reference distance and reference height can represent the position range where a valid text box should appear. It can be considered that when the actual distance CENTER_Y between the center point of the text box and the first reference edge is close to the reference distance HOR_Y, and the height of the text box is also close to the reference height SUB_H, the text box is valid. In this regard, a first threshold TOL1 and a second threshold TOL2 can be set in advance. The first threshold TOL1 is the maximum deviation of the distance between the center point of the text box and the first reference edge, and the second threshold TOL2 is the maximum deviation of the height of the text box. In an example, the first threshold TOL1 can be equal to the second threshold TOL2, and equal to SUB_H×0.25. The embodiments of the present disclosure do not limit the specific values ​​of the first threshold TOL1 and the second threshold TOL2.

[0112] For each text box determined by the detection model, the actual distance CENTER_Y between the center point of the text box and the first reference edge can be calculated according to the coordinates of the text box, and the first difference between the actual distance CENTER_Y and the reference distance HOR_Y, as well as the second difference between the reference height SUB_H and the height of the text box can be calculated. When the absolute value of the first difference is less than or equal to the first threshold value TOL1 and the second difference is less than or equal to the second threshold value TOL2, it means that the distance and height between the center point of the text box and the first reference edge do not exceed the allowable deviation, and the text box is valid. When the first difference is greater than the first threshold value TOL1 or the second difference is greater than the second threshold value TOL2, the text box is considered invalid and removed.

[0113] In this case, the absolute value of the difference between the actual distance CENTER_Y between the center point of any retained text box and the first reference edge and the reference distance HOR_Y is less than or equal to the first threshold TOL1, that is, -TOL1≤CENTER_Y-HOR_Y≤TOL1. The height of any retained text box is greater than or equal to SUB_H-TOL2.

[0114] In this way, the cost of the detection model for detecting text boxes can be reduced and the accuracy of text box detection can be improved.

[0115] Those skilled in the art should understand that the detection of the detection model may be time-consuming. To avoid data loss, the detection result may be stored in the form of a file on the first device.

[0116] There are two possible subtitle embedding methods for multimedia resources including embedded subtitles. One is the non-interval subtitle embedding method, and the other is the intermittent subtitle embedding method. In the period of time when the lines are dense, when the non-interval subtitle embedding method is used, there is no need to separate the subtitle-free video between two video frames embedded with different subtitles; when the intermittent subtitle embedding method is used, there needs to be an intermittent subtitle video frame between two video frames embedded with different subtitles.

[0117] Each video frame displays an embedded subtitle. When the multimedia resource adopts the non-interval embedded subtitle method, multiple different embedded subtitles may be displayed in a subtitle display time period of step S22. Since it is impossible to determine which video frames include the same embedded subtitles based on the detection results of the detection model, each video frame in a subtitle display time period needs to be identified. At this time, the first recognition model needs to adopt a frame-by-frame recognition mode. When the multimedia resource adopts the intermittent subtitle embedding method, only one embedded subtitle will be displayed in a subtitle display time period of step S22 (all video frames in the same subtitle display time period display the same embedded subtitle). Therefore, each video frame can be identified in a subtitle display time period, or only a small number of video frames can be sampled for identification. At this time, the first recognition model can adopt a frame-by-frame recognition mode or a sampling recognition mode.

[0118] The subtitle embedding method used by the multimedia resource can be manually determined by the staff and recorded in the attribute information of the multimedia resource. The first device can determine which recognition mode the first recognition model uses to identify the video frame of the multimedia resource based on the attribute information of the multimedia resource. Among them, when the subtitle embedding method is a non-interval subtitle embedding method, a frame-by-frame recognition mode can be used. When the subtitle embedding method is an interval subtitle embedding method, any one of the sampling recognition mode and the frame-by-frame recognition mode can be selected according to the application scenario requirements.

[0119] The following first introduces an exemplary working method of the first recognition model in the frame-by-frame recognition mode.

[0120] In a possible implementation, the first recognition model supports a frame-by-frame recognition mode. The first recognition model includes a plurality of sub-models with different parameters. In the frame-by-frame recognition mode,

[0121] The first recognition model is used to recognize the video frames in each subtitle display time period, including:

[0122] For each video frame in each subtitle display time period, use multiple sub-models to recognize at least one text box corresponding to the video frame, and the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box;

[0123] For each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame. The recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0124] For example, to improve the recognition accuracy, the first recognition model may include multiple sub-models with different parameters. The sub-models with different parameters can be trained using different training sets. For example, the sub-model trained using a training set including more rare Chinese characters has a better recognition effect on rare Chinese characters, and the sub-model trained using a training set including embedded subtitles in a specific font has a better recognition effect on the embedded subtitles in this font. The embodiments of the present disclosure do not limit which specific sub-models the first recognition model includes.

[0125] Taking Figure 3 the first subtitle display time period shown as an example, for video frame 1 and video frame 2 included therein, multiple sub-models can be used to separately recognize each text box corresponding to each video frame. The recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box.

[0126] Figure 5 An example showing the recognition results of the sub-model for video frame 1 and video frame 2 according to the embodiments of the present disclosure.

[0127] As Figure 5 shown, the embedded subtitle of video frame 1 can be "one, two, three, four, five, six". The embedded subtitle of video frame 2 can be "seven, eight, nine, ten". The detection model can determine that video frame 1 corresponds to text box 1 and text box 2, text box 1 includes "one, two, three", and text box 2 includes "four, five, six". The detection model can determine that video frame 2 corresponds to text box 3, and text box 3 includes "seven, eight, nine, ten".

[0128] The first recognition model may include sub-model 1 and sub-model 2. Sub-model 1 is used to recognize text boxes 1, 2, and 3, and sub-model 2 is used to recognize text boxes 1, 2, and 3. The recognition results of sub-model 1 and sub-model 2 for text box 1 are both "one, two, three", and the recognition results for text box 2 are both "four, five, six". The recognition result of sub-model 1 for text box 3 can be "seven, eight, nine, ten", and the recognition result of sub-model 2 for text box 3 can be "seven, eight, nine, ten".

[0129] For each sub-model, the recognition results of the sub-model for the text boxes corresponding to any one video frame can be merged as the recognition result of the sub-model for this video frame. The recognition result corresponding to this video frame includes the recognition results of each sub-model for this video frame.

[0130] In a possible implementation manner, for each sub-model, merging the recognition results of the sub-model for the text boxes corresponding to any one video frame as the recognition result of the sub-model for this video frame includes:

[0131] When merging the recognition results of any one sub - model for multiple text boxes corresponding to any video frame, add a space between the recognition results corresponding to adjacent text boxes.

[0132] Take Figure 5 as an example. When merging the recognition results of sub - model 1 for the text boxes corresponding to video frame 1, the text boxes 1 and 2 corresponding to video frame 1 are adjacent. Add a space between the recognition result "one two three" corresponding to text box 1 and the recognition result "four five six" corresponding to text box 2, and we can get "one two three four five six", which is used as the recognition result of sub - model 1 for video frame 1. Similarly, when merging the recognition results of sub - model 2 for the text boxes corresponding to video frame 1, we also get "one two three four five six", which is used as the recognition result of sub - model 2 for video frame 1.

[0133] In this way, the recognition result of any one sub - model for the video frame is closer to the embedded subtitle of the video frame.

[0134] For a video frame corresponding to only one text box, there is no need to merge the recognition results of the corresponding text box. Take Figure 5 as an example. Video frame 2 corresponds to only one text box. Therefore, there is no need to merge the recognition results of the text box corresponding to video frame 2. The recognition result "seven eight nine ten" of sub - model 1 for text box 3 can be directly used as the recognition result of sub - model 1 for video frame 2. The recognition result "seven eight wan ten" of sub - model 2 for text box 3 can be directly used as the recognition result of sub - model 2 for video frame 2.

[0135] The recognition results corresponding to video frame 1 can include the recognition results of each sub - model for video frame 1, that is, include "one two three four five six" (sub - model 1), "one two three four five six" (sub - model 2). Similarly, the recognition results corresponding to video frame 2 can include "seven eight nine ten" (sub - model 1), "seven eight wan ten" (sub - model 2).

[0136] In this way, the first recognition model can obtain more recognition results for each video frame, which is convenient for obtaining more accurate recognition results based on multiple results.

[0137] In a possible implementation manner, determining the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result includes:

[0138] Select one sub - model from multiple sub - models;

[0139] When the similarity of the recognition results of the selected sub-model for any two video frames in any subtitle display time period is less than a third threshold, the subtitle display time period is divided into a plurality of subtitle display time periods, so that the similarity of the recognition results of the selected sub-model for every two video frames in each subtitle display time period is greater than or equal to the third threshold;

[0140] When the similarities of the recognition results of the selected sub-model for every two video frames within any subtitle display time period are greater than or equal to the third threshold, the recognition result with the largest number of occurrences is found from the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, and the subtitle text to be displayed corresponding to the subtitle display time period is determined based on the recognition result with the largest number of occurrences.

[0141] For example, since multimedia resources use a non-interval-free subtitle embedding method, multiple different embedded subtitles may be displayed within a subtitle display time period, and the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box. Therefore, the sub-model's recognition results for text boxes corresponding to video frames including different embedded subtitles are also different, that is, the similarity of the sub-model's recognition results for every two video frames with different embedded subtitles within any subtitle display time period is relatively small.

[0142] Based on this, a third threshold can be set in advance, and a sub-model can be selected from multiple sub-models. When the similarity of the recognition results of the selected sub-model for any two video frames in any subtitle display time period is less than the third threshold, it is considered that the embedded subtitles of at least two video frames in the subtitle display time period are different. The subtitle display time period can be divided into multiple subtitle display time periods, so that the similarity of the recognition results of the selected sub-model for every two video frames in each subtitle display time period is greater than or equal to the third threshold. In this case, it can be considered that the embedded subtitles included in each video frame in each subtitle display time period after division are the same.

[0143] The selected sub-model may be any one of the multiple sub-models, or a sub-model with a faster recognition speed may be selected to improve the overall recognition efficiency of the first recognition model. The embodiment of the present disclosure does not limit the method of selecting the sub-model.

[0144] When multimedia resources adopt the intermittent subtitle embedding method, each subtitle display time period has satisfied the condition that the similarity of the recognition results of any two video frames included in the subtitle display time period by the selected sub-model is less than the third threshold, and no further segmentation is required.

[0145] After that, the recognition result with the largest number of occurrences can be found from the intersection of all recognition results of each sub-model for the video frames in each subtitle display time period. The recognition result with the largest number of occurrences in the intersection has a higher accuracy than other recognition results in the intersection. Therefore, the recognition result with the largest number of occurrences in the intersection is the recognition result finally determined by the first recognition model and corresponding to the subtitle display time period.

[0146] For example, assume that the embedded subtitles of all video frames in subtitle display time period 1 are "Throw it away, I don't want it anymore". Subtitle display time period 1 includes video frame 11, video frame 12, and video frame 13. The first recognition model includes sub-model 1 and sub-model 2.

[0147] The recognition result of sub-model 1 for video frame 11 may be "throw it away, I don't want it anymore"; the recognition result for video frame 12 may be "throw it away, I don't want it anymore"; and the recognition result for video frame 13 may be "throw it away, I don't want it anymore".

[0148] The recognition result of sub-model 2 for video frame 11 may be "throw it away, I don't want it anymore"; the recognition result for video frame 12 may be "throw it away, I don't want it anymore"; and the recognition result for video frame 13 may be "throw it away, I don't want it anymore".

[0149] The intersection of all recognition results of sub-model 1 and sub-model 2 for video frames in subtitle display time period 1 includes "throw it away, I don't want it anymore" and "throw it away, I don't want it anymore", that is, sub-model 1 and sub-model 2 both obtained these two recognition results.

[0150] Statistics show that among all the recognition results of sub-model 1 and sub-model 2 for video frames in subtitle display time period 1, the recognition result "throw it away, I don't want it anymore" appears 3 times, and the recognition result "throw it away, I don't want it anymore" appears 2 times. The recognition result "throw it away, I don't want it anymore" with the largest number of occurrences can be the recognition result finally determined by the first recognition model and corresponding to subtitle display time period 1.

[0151] If the intersection includes only one recognition result, then the recognition result must be the recognition result with the largest number of occurrences in the intersection, and the recognition result is the recognition result finally determined by the first recognition model and corresponding to the subtitle display time period.

[0152] The subtitle text to be displayed corresponding to the subtitle display time period can be determined based on the recognition result with the largest number of occurrences. In one example, the recognition result with the largest number of occurrences can be directly determined as the subtitle text to be displayed corresponding to the subtitle display time period to improve the generation efficiency of the subtitle file. In another example, the recognition result with the largest number of occurrences can also be optimized by denoising. For example, the recognition result with the largest number of occurrences may include some special symbols that rarely appear in some dialogues, such as %, &, etc. It can be considered that only Chinese, English, numbers, blank characters, and book title marks are valid data, and other content is noise data. Therefore, the noise data in the recognition result with the largest number of occurrences can be removed and then determined as the subtitle text to be displayed corresponding to the subtitle display time period. In this way, the accuracy of the subtitle text to be displayed can be improved.

[0153] Those skilled in the art should understand that there are more ways to optimize the recognition result with the largest number of occurrences, as long as the accuracy of the subtitle text to be displayed can be improved. The embodiments of the present disclosure do not limit the specific optimization methods.

[0154] In this way, the first recognition model can complete the recognition of video frames within each subtitle display time period in the frame-by-frame recognition mode. And this recognition mode is applicable to both the subtitle embedding method without intervals and the subtitle embedding method with intervals, and has a wider scope of application. Under different background pictures, the text box corresponding to the same embedded subtitle determined by the detection model may be different. Even for two text boxes including the same embedded subtitles, the recognition results of the recognition model may be different. In the frame-by-frame recognition mode, more video frames are recognized, and the detection error and recognition error of a single video frame have less impact on the final recognition effect. Therefore, the frame-by-frame recognition mode has a higher fault tolerance rate.

[0155] Those skilled in the art should understand that in the frame-by-frame recognition mode, detection and recognition can be performed in parallel. For example, after the detection model determines a video frame including embedded subtitles, the first recognition model can be used to recognize the video frame. After the recognition results of all video frames in a subtitle display time period are determined, the subtitle text to be displayed corresponding to the subtitle display time period can be determined. In this way, the recognition efficiency in the frame-by-frame recognition mode can be improved.

[0156] In a possible implementation, the method further includes:

[0157] When all recognition results of the sub-models for the video frames within any subtitle display time period have no intersection, the text box detection area is expanded, and the steps of determining at least one text box and subsequent steps are re-executed for each video frame including embedded subtitles.

[0158] For example, each sub-model may have a certain recognition error, and therefore, all recognition results of each sub-model for video frames within each subtitle display time period may not have an intersection.

[0159] For example, assume that the embedded subtitles of all video frames in subtitle display time period 1 are "Throw it away, I don't want it anymore". Subtitle display time period 1 includes video frame 11, video frame 12, and video frame 13. The first recognition model includes sub-model 1 and sub-model 2.

[0160] The recognition result of sub-model 1 for video frame 11 may be "throw it away, I don't want it anymore"; the recognition result for video frame 12 may be "throw it away, I don't want it anymore"; and the recognition result for video frame 13 may be "throw it away, I don't want it anymore".

[0161] The recognition result of sub-model 2 for video frame 11 can be "It's thrown away and I won't take it anymore"; the recognition result for video frame 12 can be "It's thrown away and I won't take it anymore"; the recognition result for video frame 13 can be "It's thrown away and I won't take it anymore".

[0162] It can be seen that sub-model 1 and sub-model 2 have no intersection in all recognition results for video frames in subtitle display time period 1. The reason for this recognition result may be that the text box detection area is too small. Therefore, the text box detection area can be enlarged, and the steps of determining at least one text box and the subsequent steps for each video frame including embedded subtitles can be re-executed.

[0163] The extent of the expansion of the text box detection area can be set according to the application scenario requirements, for example, set to a length of 1 pixel. If after the text box detection area is expanded, all recognition results of the sub-models for the video frames in each subtitle display time period still have no intersection, the text box detection area can be further expanded, and the steps of determining at least one text box and subsequent steps for each video frame including embedded subtitles can be performed again.

[0164] Furthermore, the upper limit of the expansion degree of the text box detection area can be set according to the application scenario requirements (for example, 6 pixels, with no limit on the value). When the expansion degree of the text box detection area reaches the upper limit, if all the recognition results of the sub-models for the video frames in each subtitle display time period still have no intersection, it can be considered that further expansion of the text box detection area is unlikely to work, and other methods can be considered to determine the accurate subtitle text, such as manual determination or with the help of speech recognition. An exemplary method for determining the accurate subtitle text with the help of speech recognition is given later.

[0165] In this way, the capability of the first recognition model can be improved.

[0166] When all recognition results of the sub-models for the video frames in each subtitle display time period have no intersection, the text box detection area may not be expanded, and the accurate subtitle text may be determined directly by means of speech recognition.

[0167] In a possible implementation, the method further includes:

[0168] When all recognition results of the sub-models for the video frames in any subtitle display time period do not intersect, the second recognition model is used to perform speech recognition on the audio of the multimedia resource in the subtitle display time period to obtain the speech recognition subtitle text corresponding to the subtitle display time period;

[0169] When the speech recognition subtitle text intersects with all recognition results of any sub-model for the video frames within the subtitle display time period, the subtitle text to be displayed corresponding to the subtitle display time period is determined according to the speech recognition subtitle text.

[0170] For example, when all recognition results of the sub-models for the video frames in any subtitle display time period do not intersect, the second recognition model can also be used to perform speech recognition on the audio of the multimedia resource in the subtitle display time period to obtain the speech recognition subtitle text corresponding to the subtitle display time period. The second recognition model can be implemented based on existing technology, and speech recognition can be completed using ASR technology. The embodiments of the present disclosure do not limit the specific implementation method of the second recognition model and the specific method of the second recognition model completing speech recognition.

[0171] It should be understood that if the speech recognition subtitle text has an intersection with all the recognition results of any sub-model for the video frames in the subtitle display time period, it means that the speech recognition subtitle text is a more accurate subtitle text. The subtitle text to be displayed corresponding to the subtitle display time period can be determined based on the speech recognition subtitle text.

[0172] For example, assume that the embedded subtitles of all video frames in subtitle display time period 1 are "Throw it away, I don't want it anymore". Subtitle display time period 1 includes video frame 11, video frame 12, and video frame 13. The first recognition model includes sub-model 1 and sub-model 2.

[0173] The recognition result of sub-model 1 for video frame 11 may be "throw it away, I don't want it anymore"; the recognition result for video frame 12 may be "throw it away, I don't want it anymore"; and the recognition result for video frame 13 may be "throw it away, I don't want it anymore".

[0174] The recognition result of sub-model 2 for video frame 11 can be "It's thrown away and I won't take it anymore"; the recognition result for video frame 12 can be "It's thrown away and I won't take it anymore"; the recognition result for video frame 13 can be "It's thrown away and I won't take it anymore".

[0175] It can be seen that there is no intersection between sub-model 1 and sub-model 2 in all recognition results of video frames in subtitle display time period 1.

[0176] If the speech recognition subtitle text is "throw it away, I don't want it anymore", then the speech recognition subtitle text and all recognition results of sub-model 1 for video frames in subtitle display time period 1 have an intersection, and the intersection is the speech recognition subtitle text "throw it away, I don't want it anymore". The subtitle text to be displayed can be determined based on the speech recognition subtitle text "throw it away, I don't want it anymore".

[0177] Furthermore, the speech recognition subtitle text may also be optimized by denoising, etc. The denoising method has been described above and will not be repeated here.

[0178] Those skilled in the art should understand that the embodiments of the present disclosure do not limit whether the speech recognition means must be used after the text box detection area is expanded.

[0179] If the speech recognition subtitle text has no intersection with all recognition results of any sub-model for the video frames within the subtitle display time period, the subtitle text to be displayed corresponding to the subtitle display time period can be manually determined.

[0180] In this way, the credibility of the subtitle text to be displayed can be improved.

[0181] The following describes an exemplary working method of the first recognition model in the sampling recognition mode.

[0182] In a possible implementation, the first recognition model supports a sampling recognition mode, the first recognition model includes a plurality of sub-models with different parameters, and when each video frame in each subtitle display time period of the multimedia resource includes the same embedded subtitle, in the sampling recognition mode,

[0183] The first recognition model is used to recognize the video frames in each subtitle display time period, including:

[0184] Selecting at least one video frame from each subtitle display time period, and for each video frame selected from any subtitle display time period, using a plurality of sub-models to recognize an image of at least one text box corresponding to the video frame, wherein the recognition result of each sub-model for each text box includes an initial subtitle text corresponding to the text box;

[0185] For each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame. The recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0186] The sampling recognition model is only applicable to the subtitle embedding method with intervals. Therefore, in the sampling recognition mode, each video frame in each subtitle display time period includes the same embedded subtitles. Therefore, at least one video frame can be selected from each subtitle display time period for recognition, and the recognition result corresponding to the subtitle display time period obtained based on the recognition result of the selected video frame is also a recognition result with higher accuracy.

[0187] That is, at least one video frame is selected from each subtitle display time period, and for each video frame selected from any subtitle display time period, multiple sub-models are used to recognize the image of at least one text box corresponding to the video frame. The video frame can be randomly selected from the subtitle display time period. The way in which the sub-model recognizes the selected video frame and the way in which the sub-model determines the recognition result of the video frame are the same as in the frame-by-frame recognition mode, and will not be repeated here.

[0188] In this case, the number of video frames that the first recognition model needs to recognize is greatly reduced, which can improve the recognition efficiency of the first recognition model.

[0189] In a possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result includes:

[0190] From the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, find the recognition result with the largest number of occurrences, and determine the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0191] In the sampling recognition mode, each video frame in each subtitle display time period includes the same embedded subtitles, and there is no need to segment the subtitle display time period. The recognition result with the largest number of occurrences can be directly found from the intersection of all recognition results of each sub-model for the video frames in the subtitle display time period, and the subtitle text to be displayed corresponding to the subtitle display time period can be determined based on the recognition result. The implementation method of finding the recognition result with the largest number of occurrences and the implementation method of determining the subtitle text to be displayed based on the recognition result are the same as those in the frame-by-frame recognition mode.

[0192] For example, assume that the embedded subtitles of all video frames in subtitle display time period 1 are "Throw it away, I don't want it anymore". Subtitle display time period 1 includes video frame 11, video frame 12, and video frame 13. The first recognition model includes sub-model 1 and sub-model 2.

[0193] The recognition result of sub-model 1 for video frame 11 may be "throw it away, I don't want it anymore"; the recognition result for video frame 12 may be "throw it away, I don't want it anymore"; and the recognition result for video frame 13 may be "throw it away, I don't want it anymore".

[0194] The recognition result of sub-model 2 for video frame 11 may be "throw it away, I don't want it anymore"; the recognition result for video frame 12 may be "throw it away, I don't want it anymore"; and the recognition result for video frame 13 may be "throw it away, I don't want it anymore".

[0195] The intersection of all recognition results of sub-model 1 and sub-model 2 for video frames in subtitle display time period 1 includes “throw it away, I don’t want it anymore” and “throw it away, I don’t want it anymore”.

[0196] Statistics show that among all the recognition results of sub-model 1 and sub-model 2 for video frames in subtitle display time period 1, the recognition result "throw it away, I don't want it anymore" appears 3 times, and the recognition result "throw it away, I don't want it anymore" appears 2 times. The recognition result "throw it away, I don't want it anymore" with the largest number of occurrences can be the recognition result finally determined by the first recognition model and corresponding to subtitle display time period 1.

[0197] If the intersection includes only one recognition result, then the recognition result must be the recognition result with the largest number of occurrences in the intersection, and the recognition result is the recognition result finally determined by the first recognition model and corresponding to the subtitle display time period.

[0198] The subtitle text to be displayed corresponding to the subtitle display time period can be determined based on the recognition result with the largest number of occurrences. In one example, the recognition result with the largest number of occurrences can be directly determined as the subtitle text to be displayed corresponding to the subtitle display time period to improve the generation efficiency of the subtitle file. In another example, the recognition result with the largest number of occurrences can also be optimized by denoising. For example, the recognition result with the largest number of occurrences may include some special symbols that rarely appear in some dialogues, such as %, &, etc. It can be considered that only Chinese, English, numbers, blank characters, and book title marks are valid data, and other content is noise data. Therefore, the noise data in the recognition result with the largest number of occurrences can be removed and then determined as the subtitle text to be displayed corresponding to the subtitle display time period. In this way, the accuracy of the subtitle text to be displayed can be improved.

[0199] Similarly, in the sampling recognition mode, when all the recognition results of the sub-models for the video frames within the subtitle display time period do not intersect, the text box detection area can be expanded to re-recognize the recognized text, or the accurate subtitle text can be determined with the help of speech recognition. When neither expanding the text box detection area nor speech recognition can obtain accurate subtitle text, the subtitle text to be displayed is manually determined. The specific implementation methods of expanding the text box detection area for re-recognition and speech recognition have been described in the relevant description of the frame-by-frame recognition mode and will not be repeated here.

[0200] In this way, the recognition accuracy of the first recognition model in the sampling recognition mode can be guaranteed, and the accuracy of the subtitle text to be displayed that is finally determined can be guaranteed.

[0201] The disclosed embodiment also supports post-processing of the subtitle text to be displayed, so as to further improve the display effect of the text to be displayed.

[0202] For example, subtitles displayed in an external form may have a visual subtitle delay when displayed, that is, the time when a certain external subtitle starts to be displayed may be earlier or later than the start time of the subtitle display time period corresponding to the external subtitle, and accordingly, the time when the external subtitle ends to be displayed may be later or earlier than the end time of the subtitle display time period corresponding to the external subtitle. Therefore, when generating a subtitle file, the subtitle display time period corresponding to each subtitle text to be displayed recorded in the subtitle file can be adjusted. For example, without affecting the display of other subtitle texts to be displayed, the start time of the subtitle display time period corresponding to the current subtitle text to be displayed is advanced and the end time is postponed. The degree of advance and postponement can be set according to the application scenario requirements, and the embodiments of the present disclosure are not limited to this.

[0203] Those skilled in the art should understand that the first device may not adjust the subtitle display time period when generating a subtitle file, and only needs to adjust the display time of the subtitles to be displayed when the second device displays the subtitles to be displayed.

[0204] For example, due to the limitations of the recognition capabilities of each sub-model, when an embedded subtitle in a text box includes multiple spaces, a single sub-model may not be able to recognize all of the spaces, resulting in a deviation between the recognition results of the video frame and the actual position and number of spaces in the embedded subtitles. The position and number of spaces in the subtitle text to be displayed determined based on the recognition results of the video frame are also inaccurate.

[0205] In this regard, in a possible implementation, the recognition result of the sub-model for the video frame also includes a space, and the method further includes:

[0206] When the recognition result of any sub-model for any video frame includes a space, judging whether there is a space at the same position in the subtitle text to be displayed corresponding to the subtitle display time period to which the video frame belongs according to the position of the space;

[0207] If there is no space, add a space at the same position.

[0208] Figure 6 An example of a video frame in which spaces appear in embedded subtitles according to an embodiment of the present disclosure is shown.

[0209] like Figure 6As shown, the embedded subtitle of the video frame includes two spaces. The first space is between "thrown away" and "fallen down", and the second space is between "fallen down" and "I don't want it anymore". If the detection model only determines the text box as shown in Figure 6 for this video frame, the recognition results of each sub-model for this video frame may also include spaces.

[0210] For example, the recognition result of sub-model 1 for this video frame can be "thrown away fallen down I don't want it anymore", the first space is successfully recognized, and the second space is not recognized. The recognition result of sub-model 2 for this video frame can be "thrown away fallen down I don't want it anymore", the second space is successfully recognized, and the first space is not recognized.

[0211] In this case, it is possible to judge whether there is a space at the same position in the subtitle text to be displayed corresponding to the subtitle display time period of this video frame according to the position of the space in the recognition result of the sub-model for this video frame; when there is no space, add a space at the same position.

[0212] Suppose the subtitle text to be displayed corresponding to the subtitle display time period of this video frame is "thrown away fallen down I don't want it anymore". Then, according to the recognition result of sub-model 1 for this video frame, it can be determined that there is 1 space between "thrown away" and "fallen down". Since there is no space at the same position of this space in the subtitle text to be displayed corresponding to the subtitle display time period of this video frame, a space can be added at the same position, and the subtitle text to be displayed is updated to "thrown away fallen down I don't want it anymore".

[0213] According to the recognition result of sub-model 2 for this video frame, it can be determined that there is a space between "fallen down" and "I don't want it anymore". After judgment, there is already a space at the same position of this space in the subtitle text to be displayed, so the subtitle text to be displayed does not need to be updated.

[0214] In this way, the similarity between the subtitle text to be displayed and the embedded subtitle can be improved, and the subtitle display effect can be improved.

[0215] Again, for example, in some scenarios, the accuracy of the detection model in recognizing whether a video frame includes an embedded subtitle may not reach 100%. In fact, for a relatively long subtitle display time period, if the detection model misidentifies a certain video frame in the middle as a video frame without an embedded subtitle, according to the detection result of the detection model, this subtitle display time period will be misidentified as two subtitle display time periods. The generated subtitle file is also recorded according to two subtitle display time periods, which not only causes redundancy in the subtitle file, but also results in the lack of subtitles for the video frames within the interval between the two subtitle display time periods, reducing the subtitle display effect.

[0216] In response to this, in one possible implementation, the method further includes:

[0217] When the subtitle texts to be displayed corresponding to adjacent subtitle display time periods are the same and the time interval does not exceed the fourth threshold, the adjacent subtitle display time periods are merged into one subtitle display time period, and the subtitle texts to be displayed corresponding to the adjacent subtitle display time periods are used as the subtitle texts to be displayed corresponding to the merged subtitle display time period;

[0218] When the subtitles to be displayed are displayed in the form of plug-in subtitles, the subtitle text to be displayed corresponding to each subtitle display time period is displayed within the subtitle display time period.

[0219] The fourth threshold can be pre-set according to application scenario requirements, for example, set to 1s. The embodiment of the present disclosure does not limit the specific value of the fourth threshold.

[0220] When the subtitle texts to be displayed corresponding to adjacent subtitle display time periods are the same and the time interval does not exceed the fourth threshold, it can be considered that the two subtitle display time periods actually belong to the same subtitle display time period that is mistakenly identified as two subtitle display time periods, and the two subtitle display time periods can be merged into one subtitle display time period. That is, the earliest time point covered by the two subtitle display time periods is used as the earliest time point of the merged subtitle display time period, and the latest time point covered by the two subtitle display time periods is used as the latest time point of the merged subtitle display time period.

[0221] by Figure 3 For example, subtitle display time period 1 and subtitle display time period 2 are adjacent subtitle display time periods, the earliest time point covered by subtitle display time period 1 is t1 and the latest time point is t2, and the earliest time point covered by subtitle display time period 2 is t4 and the latest time point is t10. Assuming that the subtitle texts to be displayed corresponding to subtitle display time period 1 and subtitle display time period 2 are the same, and the time interval (t4-t2) between subtitle display time period 1 and subtitle display time period 2 does not exceed the fourth threshold, then subtitle display time period 1 and subtitle display time period 2 can be merged into one subtitle display time period, and the merged subtitle display time period includes video frame 1-video frame 10, and the earliest time point covered by the merged subtitle display time period is t1 and the latest time point is t10.

[0222] Accordingly, the subtitle texts to be displayed corresponding to the adjacent subtitle display time periods are also used as the subtitle texts to be displayed corresponding to the merged subtitle display time periods. In this case, the two subtitle display time periods originally to be recorded in the subtitle file can be replaced by one subtitle display time period, reducing the redundancy of the subtitle file.

[0223] When displaying the subtitles to be displayed in the form of plug-in subtitles, the subtitle text to be displayed corresponding to each subtitle display time period is displayed within the subtitle display time period, that is, displayed on each video frame within the subtitle display time period. Since the missing subtitles have been supplemented, the subtitle display effect can be improved.

[0224] The disclosed embodiment also provides a subtitle display method. Figure 7 A schematic diagram showing the process of the subtitle display method according to an embodiment of the present disclosure is shown. The process takes the "elders exclusive subtitle mode" as an exemplary application scenario.

[0225] like Figure 7 As shown, in a possible implementation, the method includes:

[0226] Step S71, in response to the elder-specific subtitle mode being triggered, obtaining a subtitle file of a currently played multimedia resource including embedded subtitles;

[0227] Step S72, rendering the subtitle file to obtain the subtitles to be displayed of the multimedia resource, wherein the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles of the multimedia resource;

[0228] Step S73, masking or blurring the overlapping area of ​​each video frame of the multimedia resource with the subtitles to be displayed, to obtain the video frame to be displayed;

[0229] Step S74, displaying the video frames and subtitles to be displayed to the user, wherein the subtitles to be displayed are displayed in the form of plug-in subtitles.

[0230] The subtitle display method may be executed by a second device. Figure 8 A schematic diagram showing the structure of a second device according to an embodiment of the present disclosure.

[0231] like Figure 8 As shown, the second device may include a download module, a cache pool, a subtitle engine, a graphics processor GPU, etc. When the user uses the second device, if a multimedia resource is clicked, the second device can download the multimedia resource from the first device and play it. When the user selects the exclusive subtitle mode for elders, or when the application automatically enters the exclusive subtitle mode for elders, the download module can be used to download the subtitle file of the multimedia resource from the first device. Optionally, if the second device does not store the font file, the second device can also use the download module to download the font file from the first device. When downloading subtitle files (and font files), the download module can turn on the anti-hijacking function to prevent the downloaded files from being damaged. The download module can store the downloaded subtitle files (and font files) in the cache pool through the cache management function. The download module is an existing module in the prior art, and the anti-hijacking function and the cache management function are existing functions in the prior art. The specific download process and the cache process will not be repeated here.

[0232] The second device can use the subtitle engine to obtain subtitle-related files (subtitle files and font files) from the cache pool, and the subtitle engine uses the font file to render the subtitle file. The subtitle engine can have a control function, a thread management function, an attribute management function, a driver management function, and a rendering management function. Among them, the subtitle engine can set the rendering steps through the control function, start and manage the threads corresponding to each step through the thread management function, adjust the characters, fonts, size (font size), color, stroke, position and other attributes of the subtitles to be displayed through the attribute management function, drive the hardware that implements the rendering function through the driver management function, and control the hardware through the rendering management function to complete the rendering to obtain the subtitles to be displayed. Since the exclusive subtitle mode for the elderly is triggered, the second device can determine that the current user is an elderly user, and when adjusting the size (font size) of the subtitles to be displayed, the font size of the subtitles to be displayed can be made larger than the font size of the embedded subtitles. The font size of the embedded subtitles can be determined by the staff based on experience, or determined based on the relevant information of the video, or determined based on the height of the text box and the number of text lines, etc., and the embodiment of the present disclosure does not limit this. The above functions of the subtitle engine can be implemented based on the existing technology, and the specific implementation methods of each function will not be repeated here.

[0233] The display position of the subtitles to be displayed in the video frame can be determined according to the position of the embedded subtitles. For example, in a video frame, the display position of the subtitles to be displayed can coincide with the outer boundaries of all text boxes corresponding to the subtitles to be displayed in the video frame, or be slightly larger than the outer boundaries, to ensure that the original embedded subtitles can be covered after the subtitles are displayed. Alternatively, the embedded subtitle display area or text box detection area mentioned above can also be used as the display position of the subtitles to be displayed in the video frame. When the multimedia resource clicked by the user includes embedded subtitles, the second device can also use the graphics processor GPU to mask (for example, replace or cover with a solid color block) or blur (for example, Gaussian blur) the overlapping area with the subtitles to be displayed in the video frame of the multimedia resource to obtain the video frame to be displayed. The second device can display the video frame to be displayed and the subtitles to be displayed to the user, and the subtitles to be displayed are displayed in the form of plug-in subtitles. Figure 8 Taking the use of Gaussian algorithm for blurring and making the overlapping area as the Gaussian background of the external subtitles as an example, those skilled in the art should understand that the embodiments of the present disclosure do not limit the algorithm used to implement the masking or blurring process, as long as the clarity of the overlapping area with the subtitles to be displayed in the video frame of the multimedia resource can be reduced, or the original content of the area can be blocked. In this way, the effect presented is that the area where the original embedded subtitles are located is blurred or blocked, and the area corresponds to the subtitles to be displayed in the form of external plug-ins, and the font size is enlarged.

[0234] In addition to increasing the font size, the subtitles to be displayed can also be made bolder relative to the embedded subtitles, changed to a more eye-catching or less visually stimulating color, etc., to make it easier for the elderly to read.

[0235] Those skilled in the art should understand that the post-processing of the subtitle text to be displayed described above can also be completed by the second device. The embodiments of the present disclosure do not limit whether to post-process the subtitle text to be displayed and the specific object for completing the post-processing of the subtitle text to be displayed.

[0236] According to the subtitle display method of the embodiment of the present disclosure, in response to the elder-exclusive subtitle mode being triggered, a subtitle file of the currently played multimedia resource including embedded subtitles is obtained, and the subtitles to be displayed of the multimedia resource are rendered according to the subtitle file. The font size of the subtitles to be displayed is larger than the font size of the embedded subtitles of the multimedia resource, so that the subtitles to be displayed are more adapted to the needs of elderly users; the overlapping area with the subtitles to be displayed in each video frame of the multimedia resource is masked or blurred to obtain the video frame to be displayed, and the video frame to be displayed and the subtitles to be displayed are displayed to the user. The subtitles to be displayed are displayed in the form of plug-in subtitles, so that the background of the subtitles to be displayed is an image after masking or blurring, thereby improving the display effect of the subtitles to be displayed.

[0237] Furthermore, when multimedia resources are played at double speed, the subtitle display time period corresponding to the subtitles to be displayed also changes accordingly, so that the subtitles to be displayed can be synchronized with the audio and video frames at the frame level. At any playback speed, the subtitles to be displayed can be accurately matched with the audio and video frames, improving the viewing experience. On the premise of ensuring frame-level synchronization, the display time of the subtitles to be displayed can also be increased as much as possible to facilitate users to read the subtitles.

[0238] In a possible implementation, the subtitle file is generated according to the subtitle generation method described above. The exemplary generation method has been described above and will not be repeated here.

[0239] The subtitle file may also be a subtitle file obtained by other methods of the prior art, as long as the subtitle text and the corresponding subtitle display time period are recorded. The embodiments of the present disclosure do not limit the specific method of obtaining the subtitle file.

[0240] The present disclosure also provides a subtitle generating device. Fig. 9 A schematic diagram showing the structure of a subtitle generating device according to an embodiment of the present disclosure.

[0241] like Fig. 9 As shown, in a possible implementation, the device is used to generate a subtitle file for a multimedia resource supporting an elder-specific subtitle mode, and the device includes:

[0242] A detection module 91, configured to detect the video frames of the multimedia resource using a detection model, and determine the video frames including embedded subtitles;

[0243] A first determining module 92, configured to determine a plurality of subtitle display time periods of the multimedia resource according to timestamps of video frames including embedded subtitles, wherein each video frame in each subtitle display time period includes embedded subtitles;

[0244] A first recognition module 93 is used to recognize the video frames in each subtitle display time period using a first recognition model, and determine the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result;

[0245] The second determination module 94 is used to obtain the subtitle file of the multimedia resource according to the subtitle texts to be displayed corresponding to the multiple subtitle display time periods. When the multimedia resource is played in the elder-exclusive subtitle mode, the subtitle file is used to generate subtitles to be displayed in the form of external subtitles, and the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles.

[0246] In a possible implementation, the detection model is used to detect the video frames of the multimedia resource to determine the video frames including embedded subtitles, including: using the detection model to detect images located in a text box detection area in the video frames of the multimedia resource to determine the video frames including embedded subtitles, and determining at least one text box for each video frame including embedded subtitles; determining a reference distance between a center point of a valid text box and a first reference edge of the video frame based on a maximum distance and a minimum distance between the embedded subtitle display area of ​​the multimedia resource and the first reference edge of the video frame, as well as a reference height of the valid text box; for each determined text box, calculating a first difference between an actual distance between the center point of the text box and the first reference edge and the reference distance, as well as a second difference between the reference height and the height of the text box based on the coordinates of the text box; when the absolute value of the first difference corresponding to any text box is greater than a first threshold or the second difference corresponding to any text box is greater than a second threshold, the text box is removed.

[0247] In one possible implementation, the first recognition model supports a frame-by-frame recognition mode, the first recognition model includes multiple sub-models with different parameters, and in the frame-by-frame recognition mode, the first recognition model is used to recognize the video frames in each subtitle display time period, including: for each video frame in each subtitle display time period, using multiple sub-models to recognize at least one text box corresponding to the video frame, and the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; for each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame, and the recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0248] In a possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition results includes: selecting a submodel from the multiple submodels; when the similarity of the recognition results of the selected submodel for any two video frames in any subtitle display time period is less than a third threshold, dividing the subtitle display time period into multiple subtitle display time periods, so that the similarity of the recognition results of the selected submodel for every two video frames in each subtitle display time period is greater than or equal to the third threshold; when the similarity of the recognition results of the selected submodel for every two video frames in any subtitle display time period is greater than or equal to the third threshold, finding the recognition result with the largest number of occurrences from the intersection of all recognition results of each submodel for the video frames in the subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0249] In a possible implementation, the first recognition model supports a sampling recognition mode, the first recognition model includes multiple sub-models with different parameters, and when each video frame in each subtitle display time period of the multimedia resource includes the same embedded subtitles, in the sampling recognition mode, the first recognition model is used to recognize the video frames in each subtitle display time period, including: selecting at least one video frame from each subtitle display time period, for each video frame selected from any subtitle display time period, using multiple sub-models to recognize the image of at least one text box corresponding to the video frame, the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; for each sub-model, the recognition result of the sub-model for the text box corresponding to any video frame is merged as the recognition result of the sub-model for the video frame, and the recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

[0250] In one possible implementation, determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition results includes: finding the recognition result with the largest number of occurrences from the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

[0251] In one possible implementation, the device also includes: an expansion module, which is used to expand the text box detection area when all recognition results of each sub-model for video frames within any subtitle display time period have no intersection, and cause each module included in the device to re-execute the steps of determining at least one text box and subsequent steps for each video frame including embedded subtitles.

[0252] In a possible implementation, the device also includes: a second recognition module, which is used to use the second recognition model to perform speech recognition on the audio of the multimedia resource in the subtitle display time period when all recognition results of each submodel for the video frames in any subtitle display time period have no intersection, so as to obtain the speech recognition subtitle text corresponding to the subtitle display time period; and a third determination module, which is used to determine the subtitle text to be displayed corresponding to the subtitle display time period according to the speech recognition subtitle text when the speech recognition subtitle text has an intersection with all recognition results of any submodel for the video frames in the subtitle display time period.

[0253] In one possible implementation, for each sub-model, the recognition result of the text box corresponding to any video frame by the sub-model is merged as the recognition result of the sub-model for the video frame, including: for any video frame corresponding to multiple text boxes, when merging the recognition results of multiple text boxes corresponding to the video frame by any sub-model, adding spaces between the recognition results corresponding to adjacent text boxes.

[0254] In one possible implementation, the recognition result of the sub-model for the video frame also includes a space, and the device also includes: a judgment module, which is used to judge whether there is a space at the same position in the subtitle text to be displayed corresponding to the subtitle display time period to which the video frame belongs, according to the position of the space when the recognition result of any sub-model for any video frame includes a space; and an adding module, which is used to add a space at the same position when no space exists.

[0255] In a possible implementation, the device also includes: a merging module, which is used to merge the adjacent subtitle display time periods into one subtitle display time period when the subtitle texts to be displayed corresponding to adjacent subtitle display time periods are the same and the time interval does not exceed a fourth threshold, and use the subtitle texts to be displayed corresponding to the adjacent subtitle display time periods as the subtitle texts to be displayed corresponding to the merged subtitle display time period; when displaying the subtitles to be displayed in the form of plug-in subtitles, the subtitle texts to be displayed corresponding to each subtitle display time period are displayed within the subtitle display time period.

[0256] The disclosed embodiment also provides a subtitle display device, Fig.10 A schematic diagram showing the structure of a subtitle display device according to an embodiment of the present disclosure.

[0257] like Fig.10 As shown, in a possible implementation, the device includes:

[0258] The acquisition module 101 is used for acquiring the subtitle file of the currently played multimedia resource including the embedded subtitles in response to the elder-specific subtitle mode being triggered;

[0259] A rendering module 102, configured to render the subtitles to be displayed of the multimedia resource according to the subtitle file, wherein the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles of the multimedia resource;

[0260] The processing module 103 is used to mask or blur the overlapping area of ​​each video frame of the multimedia resource with the subtitles to be displayed, so as to obtain the video frame to be displayed;

[0261] The display module 104 is used to display the video frame to be displayed and the subtitles to be displayed to the user, and the subtitles to be displayed are displayed in the form of plug-in subtitles.

[0262] The acquisition module may include the download module described above, the rendering module may include the subtitle engine described above, and the processing module may include the graphics processor described above.

[0263] In a possible implementation, the subtitle file is generated by the subtitle generating device described above. The exemplary generation method has been described above and will not be repeated here.

[0264] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0265] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0266] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0267] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0268] Fig.11 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Fig.11 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0269] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0270] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0271] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0272] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0273] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0274] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0275] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0276] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0277] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0278] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0279] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A subtitle generation method, characterized in that: The method is used to generate a subtitle file for a multimedia resource supporting an elder-specific subtitle mode, and the method comprises: Using the detection model to detect the video frames of the multimedia resource, and determine the video frames including the embedded subtitles; Determining, according to the timestamp of the video frame including the embedded subtitles, a plurality of subtitle display time periods of the multimedia resource, wherein each video frame in each subtitle display time period includes the embedded subtitles; Using the first recognition model to recognize the video frames in each subtitle display time period, and determining the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result; A subtitle file of the multimedia resource is obtained according to the subtitle texts to be displayed corresponding to the multiple subtitle display time periods. When the multimedia resource is played in the elder-exclusive subtitle mode, the subtitle file is used to generate subtitles to be displayed in the form of external subtitles, and the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles.

2. The method according to claim 1, characterized in that The detecting the video frames of the multimedia resource using the detection model to determine the video frames including the embedded subtitles includes: Using the detection model, detect an image in a text box detection area in a video frame of the multimedia resource, determine a video frame including embedded subtitles, and determine at least one text box for each video frame including embedded subtitles; Determine a reference distance between a center point of a valid text box and the first reference edge, and a reference height of the valid text box according to a maximum distance and a minimum distance between an embedded subtitle display area of ​​the multimedia resource and a first reference edge of a video frame; For each determined text box, calculate, according to the coordinates of the text box, a first difference between an actual distance between the center point of the text box and the first reference side and the reference distance, and a second difference between the reference height and the height of the text box; When the absolute value of the first difference corresponding to any text box is greater than the first threshold or the second difference corresponding to any text box is greater than the second threshold, the text box is removed.

3. The method according to claim 2, characterized in that The first recognition model supports a frame-by-frame recognition mode. The first recognition model includes a plurality of sub-models with different parameters. In the frame-by-frame recognition mode, The step of using the first recognition model to recognize the video frames within each subtitle display time period includes: For each video frame in each subtitle display time period, use multiple sub-models to recognize at least one text box corresponding to the video frame, and the recognition result of each sub-model for each text box includes the initial subtitle text corresponding to the text box; For each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame. The recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

4. The method according to claim 3, characterized in that The step of determining the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result includes: selecting a sub-model from the plurality of sub-models; When the similarity of the recognition results of the selected sub-model for any two video frames in any subtitle display time period is less than a third threshold, the subtitle display time period is divided into a plurality of subtitle display time periods, so that the similarity of the recognition results of the selected sub-model for every two video frames in each subtitle display time period is greater than or equal to the third threshold; When the similarities of the recognition results of the selected sub-model for every two video frames within any subtitle display time period are greater than or equal to the third threshold, the recognition result with the largest number of occurrences is found from the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, and the subtitle text to be displayed corresponding to the subtitle display time period is determined based on the recognition result with the largest number of occurrences.

5. The method according to claim 2, characterized in that: The first recognition model supports a sampling recognition mode, wherein the first recognition model includes a plurality of sub-models with different parameters. When each video frame in each subtitle display time period of the multimedia resource includes the same embedded subtitle, in the sampling recognition mode, The step of using the first recognition model to recognize the video frames within each subtitle display time period includes: Selecting at least one video frame from each subtitle display time period, and for each video frame selected from any subtitle display time period, using a plurality of sub-models to recognize an image of at least one text box corresponding to the video frame, wherein the recognition result of each sub-model for each text box includes an initial subtitle text corresponding to the text box; For each sub-model, the recognition result of the text box corresponding to any video frame of the sub-model is merged as the recognition result of the sub-model for the video frame. The recognition result corresponding to the video frame includes the recognition result of each sub-model for the video frame.

6. The method according to claim 5, characterized in that The step of determining the subtitle text to be displayed corresponding to the subtitle display time period according to the recognition result includes: From the intersection of all recognition results of each sub-model for the video frames within the subtitle display time period, find the recognition result with the largest number of occurrences, and determine the subtitle text to be displayed corresponding to the subtitle display time period based on the recognition result with the largest number of occurrences.

7. The method according to claim 4 or 6, characterized in that: The method further comprises: When all recognition results of the sub-models for video frames within any subtitle display time period have no intersection, the text box detection area is expanded, and the steps of determining at least one text box and subsequent steps are re-executed for each video frame including embedded subtitles.

8. The method according to claim 4 or 6, characterized in that: The method further comprises: When all recognition results of the sub-models for the video frames in any subtitle display time period do not intersect, the second recognition model is used to perform speech recognition on the audio of the multimedia resource in the subtitle display time period to obtain the speech recognition subtitle text corresponding to the subtitle display time period; When the speech recognition subtitle text and all recognition results of any sub-model for the video frames within the subtitle display time period intersect, the subtitle text to be displayed corresponding to the subtitle display time period is determined according to the speech recognition subtitle text.

9. The method according to claim 3 or 5, characterized in that: For each sub-model, merging the recognition result of the text box corresponding to any video frame by the sub-model as the recognition result of the sub-model for the video frame includes: For any video frame corresponding to multiple text boxes, when merging the recognition results of the multiple text boxes corresponding to the video frame by any sub-model, a space is added between the recognition results corresponding to adjacent text boxes.

10. The method according to claim 9, characterized in that The recognition result of the sub-model for the video frame also includes a space, and the method further includes: When the recognition result of any sub-model for any video frame includes a space, judging whether there is a space at the same position in the subtitle text to be displayed corresponding to the subtitle display time period to which the video frame belongs according to the position of the space; If there is no space, add a space at the same position.

11. The method according to claim 10, characterized in that The method further comprises: When the subtitle texts to be displayed corresponding to adjacent subtitle display time periods are the same and the time interval does not exceed the fourth threshold, the adjacent subtitle display time periods are merged into one subtitle display time period, and the subtitle texts to be displayed corresponding to the adjacent subtitle display time periods are used as the subtitle texts to be displayed corresponding to the merged subtitle display time period; When the subtitles to be displayed are displayed in the form of plug-in subtitles, the subtitle text to be displayed corresponding to each subtitle display time period is displayed within the subtitle display time period.

12. A subtitle display method, characterized in that: The method comprises: In response to the elder-only subtitle mode being triggered, obtaining a subtitle file of a currently played multimedia resource containing embedded subtitles; Rendering the subtitle file to obtain the subtitles to be displayed of the multimedia resource, wherein the font size of the subtitles to be displayed is larger than the font size of the embedded subtitles of the multimedia resource; Masking or blurring the overlapping area of ​​each video frame of the multimedia resource with the subtitles to be displayed to obtain the video frame to be displayed; The video frame to be displayed and the subtitle to be displayed are displayed to the user, and the subtitle to be displayed is displayed in the form of plug-in subtitles.

13. The method according to claim 12, characterized in that The subtitle file is generated according to the subtitle generation method according to any one of claims 1-11.

14. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 11, or the method described in claim 12 or 13 when executing the instructions stored in the memory.

15. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method described in any one of claims 1 to 11 is implemented, or the method described in claim 12 or 13 is implemented.

Images