Video subtitle erasure, model training, interactive method, device and storage medium

Abstract

The embodiment of the present application provides a video subtitle erasing, model training, interactive method, device and storage medium. The method includes: using a trained subtitle erasing model to erase the subtitles in the video to obtain a video after the subtitles are erased; the subtitle erasing model is used to: determine the mask information for perceiving the subtitle position according to multiple frame feature maps of the video; determine the first relationship weights of the first pixel area and the multiple pixel areas respectively through the attention mechanism according to the multiple frame feature maps; the multiple frame feature maps include the features of the multiple pixel areas; the multiple pixel areas include the first pixel area; the first relationship weight is corrected according to the mask information to obtain the corrected relationship weight; the first fused feature of the first pixel area is determined according to the multiple frame feature maps and the corrected relationship weight; the video after the subtitles are erased is determined according to the first fused features of the multiple pixel areas. The technical solution provided by the embodiment of the present application can improve the erasing effect.

Description

Technical Field

[0001] The present application relates to the field of visual technology, and in particular to a video subtitle erasing, model training, interaction method, device and storage medium. Background Art

[0002] In recent years, with the popularization of Internet infrastructure, video has become one of the main ways of information dissemination and has reached all aspects of people's lives, with film and television works produced explosively.

[0003] In the production and editing of film and television works, the editing and modification of local pixels in the film and television works are often involved, such as erasing subtitles in a video to restore background pixels blocked by subtitles.

[0004] The automatic erasing solution in the prior art has the problem of poor erasing effect. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide a video subtitle erasing, model training, interaction method, device and storage medium that solve the above problems or at least partially solve the above problems.

[0006] Therefore, in one embodiment of the present application, a video subtitle erasing method is provided, which includes:

[0007] Confirm the video;

[0008] Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model;

[0009] Wherein, the subtitle erasure model is used for:

[0010] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video;

[0011] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0012] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0013] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0014] A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

[0015] In another embodiment of the present application, a model training method is provided, which further includes:

[0016] Determining a sample video and its corresponding expected video;

[0017] Erasing the subtitles in the sample video using a subtitle erasing model to obtain a sample video after subtitle erasure, wherein the subtitle erasing model is a machine learning model;

[0018] Optimizing the subtitle erasure model according to the sample video after subtitle erasure and the expected video;

[0019] Wherein, the subtitle erasure model is used for:

[0020] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the sample video;

[0021] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0022] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0023] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0024] A sample video after subtitle erasure is determined according to the first fused features of the plurality of pixel regions.

[0025] In another embodiment of the present application, an interface interaction method is provided, which includes:

[0026] In response to a video input operation of a user on an operation interface, displaying an initial video input by the user;

[0027] In response to a subtitle erasing operation triggered by a user on the initial video, displaying the initial video after the subtitles are erased;

[0028] The initial video after the subtitles are erased is determined based on the video after the subtitles are erased; the video after the subtitles are erased is obtained by erasing the subtitles in the video using a trained subtitle erasure model; the video is determined based on the initial video; the subtitle erasure model is a machine learning model;

[0029] Wherein, the subtitle erasure model is used for:

[0030] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video;

[0031] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0032] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0033] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0034] A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

[0035] In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor, wherein:

[0036] The memory is used to store programs;

[0037] The processor is coupled to the memory and is used to execute the program stored in the memory to implement the above-mentioned erasing or training method.

[0038] In another embodiment of the present application, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a computer, the erasing or training method described in any one of the above can be implemented.

[0039] In another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory, a processor, and a display, wherein:

[0040] The memory is used to store programs;

[0041] The processor is coupled to the memory and the display respectively, and is used to execute the program stored in the memory to implement the above-mentioned interface interaction method.

[0042] In another embodiment of the present application, a computer-readable storage medium storing a computer program is provided, and when the computer program is executed by a computer, it can implement any of the above-mentioned interface interaction methods.

[0043] In another embodiment of the present application, a head mounted display device is provided, comprising: a memory, a processor and a display, wherein:

[0044] The memory is used to store programs;

[0045] The processor is coupled to the memory and the display, respectively, and is used to execute the program stored in the memory to achieve:

[0046] Confirm the video;

[0047] Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model;

[0048] Controlling the display to play the video after the subtitles are erased;

[0049] Wherein, the subtitle erasure model is used for:

[0050] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video;

[0051] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0052] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0053] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0054] A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

[0055] In another embodiment of the present application, a computer-readable storage medium storing a computer program is provided, wherein the computer program, when executed by a computer, can achieve:

[0056] Confirm the video;

[0057] Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model;

[0058] Play the video after the subtitles are erased;

[0059] Wherein, the subtitle erasure model is used for:

[0060] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video;

[0061] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0062] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0063] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0064] A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

[0065] In the technical solution provided by the embodiment of the present application, multiple pixel areas are scattered in multiple frame feature maps, and the first relationship weights of the first pixel area and the multiple pixel areas are determined, that is, the first relationship weights of the first pixel area and other pixel areas in the frame feature map where it is located and the first relationship weights of the first pixel area and the pixel areas in other frame feature maps are determined. It can be seen that the technical solution provided by the embodiment of the present application utilizes the spatiotemporal joint attention mechanism for subtitle erasure, which helps to improve the subtitle erasure effect. In addition, the first relationship weight is corrected according to the mask information used to perceive the position of the subtitles, which helps to further improve the subtitle erasure effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1 A flowchart of a video subtitle erasing method provided by an embodiment of the present application;

[0068] Figure 2 A schematic diagram of an erasing process of a subtitle erasing model provided in an embodiment of the present application;

[0069] Figure 3 A flowchart of a model training method provided in one embodiment of the present application;

[0070] Figure 4 A schematic diagram of an erasing process of a subtitle erasing model provided in yet another embodiment of the present application;

[0071] Figure 5 A flowchart of an interface interaction method provided in one embodiment of the present application;

[0072] Figure 6 A flowchart of a video subtitle erasing method provided by another embodiment of the present application;

[0073] Figure 7 A schematic diagram of the processing flow within the SRM provided in one embodiment of the present application;

[0074] Figure 8 A structural block diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0075] At present, the existing subtitle erasing schemes are difficult to learn the inter-frame and spatial global information, which easily leads to pixel leakage and poor subtitle erasing effect. In addition, the existing subtitle erasing schemes require explicit text mask images for supervision, which leads to problems such as difficulty in building training sets and cumbersome training design.

[0076] In order to solve or partially solve the above technical problems, the embodiment of the present application provides a new video subtitle erasing method. In the technical solution provided by the embodiment of the present application, a spatiotemporal joint attention mechanism is used to erase subtitles, which helps to improve the subtitle erasing effect; and subtitle erasing is guided according to mask information used to perceive the position of subtitles, which helps to further improve the subtitle erasing effect and reduce the problem of pixel leakage erasure.

[0077] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below according to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0078] In addition, some of the processes described in the specification, claims and the above-mentioned figures of the present application include multiple operations that appear in a specific order, and these operations may not be executed in the order in which they appear in this article or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0079] Figure 1A flow chart of a video subtitle erasing method provided in an embodiment of the present application is shown. The executor of the method may be a client or a server. The client may be a hardware with an embedded program integrated in a terminal, or an application software installed in the terminal, or a tool software embedded in a terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal may be any terminal device including a mobile phone, a tablet computer, a vehicle-mounted terminal device, etc. The server may be a common server, a cloud or a virtual server, etc., which is not specifically limited in the embodiment of the present application. Figure 1 As shown, the method includes:

[0080] 101. Confirm the video.

[0081] 102. Use the trained subtitle erasure model to erase the subtitles in the video to obtain a subtitle-erased video.

[0082] The subtitle erasure model is a machine learning model. In one example, the subtitle erasure model is a model based on deep learning.

[0083] like Figure 2 As shown, the subtitle erasing model is used to perform the following steps:

[0084] 201. Determine mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video.

[0085] 202. According to the multiple frame feature maps, determine the first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism.

[0086] Among them, the multiple frame feature maps include features of the multiple pixel areas; the multiple pixel areas include the first pixel area.

[0087] 203. Correct the first relationship weight according to the mask information to obtain a corrected relationship weight.

[0088] 204. Determine a first fused feature of the first pixel area according to the multiple frame feature maps and the corrected relationship weight.

[0089] 205. Determine a subtitle-erased video according to the first fused features of the plurality of pixel regions.

[0090] In the above 101, the above video can be a complete video or a video clip in a complete video. The length of the video clip can be determined by a preset number of frames, which is not specifically limited in the present embodiment of the application. The preset number of frames can be designed according to actual needs, for example, it can be set to 6 frames by default.

[0091] In practical applications, considering that the length of a complete video is relatively long and the correlation between two video frames with a large time interval is weak or even non-existent, the complete video can be cut into multiple video segments, and the above-mentioned erasing process is performed on each video segment separately. This can effectively reduce the amount of calculation and the calculation cost.

[0092] In the above 102, the video may be input into a trained subtitle erasure model to obtain a subtitle-erased video. The training method of the subtitle erasure model will be described in detail later.

[0093] In the above 201, feature extraction may be performed on each of the multiple video frames in the video to obtain a frame feature map corresponding to each video frame. Feature extraction may include: encoding processing. The multiple frame feature maps correspond to the multiple video frames one by one, that is, one video frame corresponds to one frame feature map.

[0094] The above-mentioned multiple pixel regions include: multiple pixel regions of each frame feature map in the multiple frame feature maps. Each frame feature map is formed by splicing multiple pixel regions of the frame feature map.

[0095] For example, the frame feature map is represented by a H*W*C structure, where H*W is the size (length and width) of the frame feature map, and C is the number of channels of the frame feature map, that is, the frame feature map includes W*H pixels, and the feature corresponding to each pixel is C-dimensional. The frame feature map includes features of multiple pixel areas, and the sizes of these multiple pixel areas can be the same or different. The embodiment of the present application does not specifically limit this. For ease of processing, the sizes of multiple pixel areas can be the same, and the length and width are consistent. For example, the size of the above-mentioned multiple pixel areas can be n*n, where the value of n can be 1, 2, 3, and so on. When the size of the pixel area is n*n, the feature of the pixel area is n*n*C, which can be converted into a feature vector, and the dimension of the feature vector is (n*n*C).

[0096] In the above 201, the mask information is used to indicate whether each pixel region among the plurality of pixel regions corresponds to a subtitle position. In one example, the mask information may include a mask value corresponding to each pixel region among the plurality of pixel regions, and the mask value is used to indicate whether the corresponding pixel region corresponds to a subtitle position.

[0097] In one example, the mask may be a hard mask, for example, the mask value may be 0 or 1; a mask value of 0 indicates that the corresponding pixel region corresponds to a subtitle position; a mask value of 1 indicates that the corresponding pixel region corresponds to a background position.

[0098] In another example, the mask may be a soft mask, for example, the mask value is between 0 and 1, the larger the mask value is, the smaller the possibility that the corresponding pixel area corresponds to the subtitle position is; the smaller the mask value is, the greater the possibility that the corresponding pixel area corresponds to the subtitle position is.

[0099] In the above 202, the plurality of pixel regions include a first pixel region, and the first pixel region can be used to refer to any pixel region among the plurality of pixel regions. The first relationship weight between the first pixel region and any pixel region among the plurality of pixel regions can be determined based on the similarity between the first pixel region and the any pixel region, and the greater the similarity, the greater the corresponding first relationship weight. The greater the first relationship weight, the greater the correlation between the first pixel region and the corresponding pixel region, and therefore, when determining the first fused feature of the first pixel region, more attention should be paid to the corresponding pixel region.

[0100] Since not only the first relationship weight between the first pixel region and other pixel regions in the frame feature map where it is located is determined, but also the first relationship weight between the first pixel region and pixel regions in other frame feature maps is determined, the above attention mechanism can also be called a spatiotemporal joint attention mechanism.

[0101] In the above 203, according to the mask information, the first relationship weights between the first pixel area and the plurality of pixel areas are respectively corrected to obtain the corrected relationship weights. In one example, the plurality of pixel areas include a second pixel area, wherein the second pixel area refers to any one of the plurality of pixel areas, and the mask information includes: a mask value corresponding to the second pixel area, and the product of the first relationship weight between the first pixel area and the second pixel area and the mask value corresponding to the second pixel area can be used as the corrected relationship weight between the first pixel area and the second pixel area.

[0102] In an achievable solution, the mask information includes: mask values ​​of the plurality of pixel regions; when the mask value is a first value (e.g., 1), it indicates that the corresponding pixel region corresponds to a background position (non-subtitle positions in a video frame are all considered background positions); when the mask value is a second value (e.g., 0), it indicates that the corresponding pixel region corresponds to a subtitle position; the first value is greater than the second value. In the above 203, "according to the mask information, the first relationship weight is modified to obtain a modified relationship weight" includes:

[0103] 2031. Utilize the product of the first relationship weight between the first pixel region and the second pixel region and the mask value of the second pixel region as the modified relationship weight between the first pixel region and the second pixel region.

[0104] The plurality of pixel regions include the second pixel region. The second pixel region may refer to any one of the plurality of pixel regions, that is, the second pixel region may be the first pixel region, or other pixel regions except the first pixel region.

[0105] According to the mask information, the first relationship weight is modified, which is equivalent to suppressing the weight of the relevant subtitle area and activating the weight of the relevant background area, which plays a guiding role in the subsequent erasure and filling.

[0106] In the above 204, in one example, the second target features of multiple pixel areas can be determined based on multiple frame feature maps; the second target features of the multiple pixel areas are weighted summed according to the corrected relationship weights to obtain the first fused feature of the first pixel area.

[0107] Specifically, the features of each pixel region in the multiple pixel regions in the multiple frame feature maps can be directly used as the second target features of each pixel region. Alternatively, each frame feature map in the multiple frame feature maps can be subjected to dimensionality reduction processing to obtain the second target features of the multiple pixel regions, wherein the dimensionality reduction processing only reduces the number of channels of each frame feature map, but does not reduce the size of each frame feature map. In one example, the dimensionality reduction processing can specifically include: convolution processing.

[0108] The corrected relationship weights include corrected relationship weights between the first pixel region and the plurality of pixel regions respectively.

[0109] The following is an example to introduce the weighted summation process: multiple frame feature maps include frame feature map A and frame feature map B, frame feature map A includes: feature A1 of pixel area a1, feature A2 of pixel area a2, frame feature map B includes feature B1 of pixel area b1 and feature B1 of pixel area b2 (Note: Here, for the convenience of introduction, the number of pixel areas in each frame feature map is exemplarily set to 2), where the corrected relationship weights of a1 and a1, a2, b1, and b2 are: λ1, λ2, λ3, and λ4, respectively, and the first fused feature of the first pixel area a1 is R1, and its calculation formula is as follows:

[0110] R1=λ1*A1+λ2*A2+λ3*B1+λ4*B2 (1)

[0111] The first fused features of other pixel areas except the first pixel area in the above-mentioned multiple pixel areas can also be obtained by the above-mentioned method or by other methods. The specific settings can be made according to actual needs, and the embodiments of the present application do not make specific limitations on this.

[0112] In the above 205, in an achievable solution, multiple first fused frame feature maps can be obtained by splicing according to the first fused features of the multiple pixel regions. The subtitle-erased video is obtained according to the multiple first fused frame feature maps. In one example, encoding processing can be performed on each of the multiple first fused frame feature maps to obtain the subtitle-erased video. The encoding processing may include: deconvolution processing.

[0113] In the technical solution provided in the embodiment of the present application, multiple pixel areas are scattered in multiple frame feature maps, and the first relationship weights of the first pixel area and the multiple pixel areas are determined, that is, the first relationship weights of the first pixel area and other pixel areas in the frame feature map where it is located and the first relationship weights of the first pixel area and pixel areas in other frame feature maps are determined. It can be seen that the technical solution provided in the embodiment of the present application utilizes the spatiotemporal joint attention mechanism for subtitle erasing, which helps to improve the subtitle erasing effect. In addition, correcting the first relationship weight according to the mask information used to perceive the position of the subtitles helps to further improve the subtitle erasing effect.

[0114] In the embodiment of the present application, the subtitle erasing model completes the pixel filling of the subtitle area while erasing the subtitles, so that the filled subtitle area can be adapted to the background area, and the erasing effect is more stable and smoother.

[0115] In addition, in the present application, in the subtitle erasing model, the erasing process is reflected in the suppression of the weight corresponding to the subtitle position in the first relationship weight by the mask information, wherein the first relationship weight represents the contribution weight (also understood as similarity) of multiple pixel regions to the first pixel region. In this way, during the model training process, the learning of the mask information can be guided by the gradient back propagation of the modified relationship weight. In other words, there is no need to use an explicit subtitle mask for supervision during model training, which reduces the training cost and difficulty.

[0116] In one example, in order to improve the accuracy of the mask information, the mask information for perceiving the position of the subtitles can be determined based on the multiple frame feature maps by using a spatiotemporal attention mechanism. In one example, the multiple frame feature maps can be subjected to 3D spatiotemporal convolution processing to obtain post-convolution features; and the mask information for perceiving the position of the subtitles can be determined based on the post-convolution features.

[0117] In the above embodiment, the mask information used to perceive the position of the subtitles is determined by 3D spatiotemporal convolution processing, which is just a fusion of local spatial information and local temporal information in the video. In order to consider the global spatial information and global temporal information in the video in the process of determining the mask information, the above 201 "determine the mask information used to perceive the position of the subtitles according to the multiple frame feature maps of the video" can be implemented by the following steps:

[0118] 2011. Determine first target features of the multiple pixel areas based on the multiple frame feature maps.

[0119] 2012. Determine second relationship weights between the first pixel region and the multiple pixel regions respectively according to the multiple frame feature maps.

[0120] 2013. Perform weighted summation on the first target features of the multiple pixel regions according to the second relationship weight to obtain a second fused feature of the first pixel region.

[0121] 2014. Determine mask information for perceiving a subtitle position according to the second fused features of the plurality of pixel regions.

[0122] In the above 2011, in one example, the feature of each pixel region in the multiple pixel regions in the multiple frame feature maps can be directly used as the first target feature of each pixel region.

[0123] In another example, a plurality of frame feature maps may be subjected to dimensionality reduction processing to obtain first target features of a plurality of pixel regions. The dimensionality reduction processing only reduces the number of channels of each frame feature map, but does not reduce the size of each frame feature map. In one example, the dimensionality reduction processing may specifically include: convolution processing.

[0124] In the above 2012, the second relationship weight between the first pixel region and any pixel region among the plurality of pixel regions can be determined according to the similarity between the first pixel region and the any pixel region, and the greater the similarity, the greater the corresponding second relationship weight. The greater the second relationship weight, the greater the correlation between the second pixel region and the corresponding pixel region, and therefore, when determining the second fused feature of the first pixel region, the corresponding pixel region should be given more attention.

[0125] Since not only the second relationship weight between the first pixel region and other pixel regions in the frame feature map where it is located is determined, but also the second relationship weight between the first pixel region and pixel regions in other frame feature maps is determined, the above attention mechanism can also be called a spatiotemporal joint attention mechanism.

[0126] In the above 2013, the first target features of the plurality of pixel regions are weighted and summed according to the second relationship weight to obtain the second fused feature of the first pixel region. The specific operation process of the weighted summation can refer to the corresponding content in the above embodiment, which will not be described in detail here.

[0127] In the above 2014, based on the second fused features of each pixel area in the multiple pixel areas, the feature representative value corresponding to each pixel area is determined; the feature representative value may include the average value and / or maximum value of the second fused features of the corresponding pixel area. The second fused features of each pixel area include: multiple feature elements, the average value refers to the average value of the multiple feature elements, and the maximum value refers to the maximum value of the multiple feature elements.

[0128] According to the feature representative values ​​of the plurality of pixel regions, mask values ​​of the plurality of pixel regions are determined.

[0129] For ease of processing, Figure 7 As shown, in one example, the average values ​​corresponding to the multiple pixel regions can be spliced ​​according to the positions of the multiple pixel regions in the multiple frame feature maps to obtain the average value feature maps 71 corresponding to the multiple frame feature maps; the number of channels of the average feature map 71 is 1; the maximum values ​​corresponding to the multiple pixel regions can also be spliced ​​according to the positions of the multiple pixel regions in the multiple frame feature maps to obtain the maximum value feature maps 72 corresponding to the multiple frame feature maps; the number of channels of the maximum value feature map 72 is also 1; according to the average value feature map 71 and the maximum value feature map 72 corresponding to each frame feature map in the multiple frame feature maps, the mask map 74 corresponding to each frame feature map is determined. The mask map includes: the mask value of each pixel region in the corresponding frame feature map.

[0130] In one example, if Figure 7 As shown, for each frame feature map, its average feature map 71 and maximum feature map 72 can be spliced ​​in the channel dimension to obtain a spliced ​​feature map; the spliced ​​feature map is subjected to dimensionality reduction processing to obtain a feature map after dimensionality reduction; the feature map after dimensionality reduction is normalized to obtain a normalized feature map 73; and the corresponding mask map 74 is determined according to the normalized feature map 73. Specifically, the normalized feature map 73 can be directly used as the corresponding mask map 74; or, according to the preset parameter θ, the normalized feature map 73 is binarized to obtain the corresponding mask map 74. Among them, the preset parameter θ can be 0.5, and of course it can also be other values, which is not specifically limited in this application. The binarization process includes: modifying the pixel values ​​in the normalized feature map that are greater than the preset parameter to 1, and modifying the pixel values ​​that are less than or equal to the preset parameter to 0.

[0131] In an achievable solution, the subtitle erasing model may be a transformer-based model, that is, the subtitle erasing model may include a transformer network, and the transformer network performs the above steps 201-204. In a specific example, there may be multiple transformer networks, and multiple transformer networks may be connected in series, with the output of the previous transformer network serving as the input of the next transformer network.

[0132] In addition, in the subtitle erasing scenario, the pixel area around the subtitle in the video frame has a greater reference significance, while the pixel area far from the subtitle has a smaller reference significance or even no reference significance. Therefore, in order to reduce the amount of calculation, the above method may further include:

[0133] 103. Obtain an initial video.

[0134] 104. Determine a subtitle frame corresponding to the initial video.

[0135] 105. According to the subtitle frame, multiple video frames in the initial video are trimmed to obtain the video.

[0136] In the above 103, the above initial video may be a complete video or a video clip of a complete video.

[0137] In the above 104, the above subtitle frame can frame the subtitles in each video frame in the initial video.

[0138] The above-mentioned subtitle frame may be determined through manual interaction or through a subtitle detection model.

[0139] Specifically, the subtitle frame may be determined in one of the following ways:

[0140] Method 1: input the initial video into a subtitle detection model to obtain the subtitle frame.

[0141] Method 2: determining the subtitle frame according to the user's subtitle frame drawing operation on the initial video.

[0142] In the above-mentioned method 1, the subtitle detection model can specifically be a target detection model based on a neural network. The internal implementation thereof can be specifically referred to the corresponding content in the prior art, and will not be described in detail here.

[0143] In the second method described above, a subtitle frame drawing interface may be provided to the user, the initial video may be displayed on the subtitle frame drawing interface, and the subtitle frame drawn by the user for the initial video on the subtitle frame drawing interface may be determined based on the user's subtitle frame drawing operation for the initial video.

[0144] Accordingly, the above method may further include:

[0145] 106. Process the initial video according to the video after subtitles are erased and the subtitle frame to obtain the initial video after subtitles are erased.

[0146] The subtitle-erased video includes multiple subtitle-erased video frames, and the initial video includes multiple initial video frames. According to the positions of the subtitle frames, the multiple subtitle-erased video frames can be pasted back to the multiple initial video frames respectively, thereby obtaining the subtitle-erased initial video.

[0147] Figure 3 A flow chart of a model training method provided by another embodiment of the present application is shown. The executor of the method may be a client or a server. The client may be a hardware with an embedded program integrated on a terminal, or an application software installed in the terminal, or a tool software embedded in a terminal operating system, etc., which is not limited in the present application embodiment. The terminal may be any terminal device including a mobile phone, a tablet computer, a vehicle-mounted terminal device, etc. The server may be a common server, a cloud or a virtual server, etc., which is not specifically limited in the present application embodiment. Figure 3 As shown, the method includes:

[0148] 301. Determine a sample video and its corresponding expected video.

[0149] 302. Erasing the subtitles in the sample video using a subtitle erasing model to obtain a sample video after the subtitles are erased.

[0150] Wherein, the subtitle erasing model is a machine learning model.

[0151] 303. Optimize the subtitle erasure model according to the sample video after subtitle erasure and the expected video.

[0152] like Figure 4 As shown, the subtitle erasing model is used to perform the following steps:

[0153] 401. Determine mask information for perceiving a subtitle position according to a plurality of frame feature maps of the sample video.

[0154] 402. Determine, based on the multiple frame feature maps, first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism.

[0155] Among them, the multiple frame feature maps include features of the multiple pixel areas; the multiple pixel areas include the first pixel area.

[0156] 403. Correct the first relationship weight according to the mask information to obtain a corrected relationship weight.

[0157] 404. Determine a first fused feature of the first pixel area according to the multiple frame feature maps and the corrected relationship weight.

[0158] 405. Determine a sample video after subtitle erasure according to the first fused features of the multiple pixel regions.

[0159] In the above 301, subtitles exist in the sample video, but subtitles do not exist in the expected video.

[0160] In the above 303, the subtitle erasure model is optimized according to the difference between the sample video after subtitle erasure and the expected video.

[0161] Specifically, according to the difference, the network parameters in the subtitle erasing model are optimized by gradient back propagation. The optimization process includes: optimizing the network parameters used to determine the mask information in the model by gradient back propagation of the corrected relationship weights.

[0162] The specific implementation of the above steps 401 to 405 can refer to the corresponding content in the above embodiments, which will not be repeated here.

[0163] In the embodiment of the present application, explicit supervision information only requires the expected video, and no explicit mask information is required, which can effectively reduce the training cost and difficulty.

[0164] Figure 5 The flowchart of the interface interaction method provided by another embodiment of the present application is shown. The execution subject of the method can be a client or a server. The client can be a hardware with an embedded program integrated on the terminal, or an application software installed in the terminal, or a tool software embedded in the terminal operating system, etc., which is not limited in the embodiment of the present application. The terminal can be any terminal device including a mobile phone, a tablet computer, a vehicle-mounted terminal device, etc. Figure 5 As shown, the interface interaction method includes:

[0165] 501. In response to a video input operation of a user on an operation interface, an initial video input by the user is displayed.

[0166] 502. In response to a subtitle erasing operation triggered by a user on the initial video, display the initial video after the subtitles are erased.

[0167] Among them, the initial video after the subtitles are erased is determined based on the video after the subtitles are erased; the video after the subtitles are erased is obtained by erasing the subtitles in the video using a trained subtitle erasure model; the video is determined based on the initial video; and the subtitle erasure model is a machine learning model.

[0168] like Figure 2 As shown, the subtitle erasing model is used to perform the following steps:

[0169] 201. Determine mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video.

[0170] 202. According to the multiple frame feature maps, determine the first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism.

[0171] Among them, the multiple frame feature maps include features of the multiple pixel areas; the multiple pixel areas include the first pixel area.

[0172] 203. Correct the first relationship weight according to the mask information to obtain a corrected relationship weight.

[0173] 204. Determine a first fused feature of the first pixel area according to the multiple frame feature maps and the corrected relationship weight.

[0174] 205. Determine a subtitle-erased video according to the first fused features of the plurality of pixel regions.

[0175] In the above 501, the client may provide an operation interface to the user.

[0176] In the above 502, in one example, a subtitle erasing control may be displayed on the above operation interface; in response to a user triggering operation on the subtitle erasing control, such as a click operation, the initial video after the subtitles are erased is displayed.

[0177] In one example, after the user triggers a subtitle erasing operation on the initial video, the client may send the initial video to the server, so that the server performs the corresponding operation to obtain the initial video after the subtitles are erased.

[0178] In another example, after the user triggers a subtitle erasing operation on the initial video, the client may locally perform the above corresponding operation to obtain the initial video after the subtitles are erased.

[0179] It should be noted here that: for the contents of each step not fully described in detail in the method provided in the embodiment of the present application, please refer to the corresponding contents in the above embodiment, which will not be repeated here. In addition, in addition to the above steps, the method provided in the embodiment of the present application may also include other parts or all of the steps in the above embodiments, which can be specifically referred to the corresponding contents in the above embodiments, which will not be repeated here.

[0180] The technical solution provided by the embodiment of the present application can be applied to various application scenarios, such as: subtitle removal scenario, Chinese and English subtitle replacement scenario, etc. In the subtitle removal scenario, the technical solution provided by the embodiment of the present application is used to remove the subtitles in the video. In the Chinese and English subtitle replacement scenario, the technical solution provided by the embodiment of the present application can be used to first remove the original English subtitles in the video, and then add Chinese subtitles.

[0181] The following will be combined Figure 6 The technical solution provided in the embodiment of the present application is introduced as follows:

[0182] Step S1: The user inputs an initial video to be processed on a mobile phone.

[0183] Step S2: crop the initial video according to the subtitle frame to obtain a video.

[0184] Step S3: the encoding network 61 in the subtitle erasure model 6 extracts features from a plurality of video frames in the video to obtain a plurality of frame feature maps.

[0185] Step S4: the transformer network 62 in the subtitle erasing model 6 performs the following two-branch steps:

[0186] First branch:

[0187] Perform different convolution processing (i.e., dimensionality reduction processing) on ​​each of the multiple frame feature maps to obtain a query feature map Q1, a key feature map K1, and a value feature map V1 corresponding to each of the frame feature maps; segment the Q1 corresponding to each of the multiple frame feature maps to obtain a query feature q1 of multiple pixel regions (pacth); segment the key feature map K1 corresponding to each of the multiple frame feature maps to obtain a key feature k1 of multiple pixel regions; segment the value feature map V1 corresponding to each of the multiple frame feature maps to obtain a value feature v1 of multiple pixel regions; for a first pixel region in the multiple pixel regions, perform a normalization operation (softmax) on the similarity between the query feature q1 of the first pixel region and the key feature k1 of the multiple pixel regions, to obtain a second relationship weight between the first pixel region and the multiple pixel regions, Figure 6S1 in the example includes the second relationship weights between the first pixel region and the plurality of pixel regions. According to the second relationship weights between the first pixel region and the plurality of pixel regions, the value features v1 of the plurality of pixel regions are weighted summed to obtain the second fused features of the first pixel region; according to the second fused features of the plurality of pixel regions, the spatial domain response subnetwork 621 is used to obtain mask information M' for perceiving the position of the subtitles.

[0188] The spatial response subnetwork 621 includes: a spatial response module (SRM).

[0189] Second branch:

[0190] Perform different convolution processing (i.e., dimensionality reduction processing) on ​​each of the multiple frame feature maps to obtain a query feature map Q2, a key feature map K2, and a value feature map V2 corresponding to each of the frame feature maps; segment the Q2 corresponding to each of the multiple frame feature maps to obtain a query feature q2 of multiple pixel regions (pacth); segment the key feature map K1 corresponding to each of the multiple frame feature maps to obtain a key feature k2 of multiple pixel regions; segment the value feature map V1 corresponding to each of the multiple frame feature maps to obtain a value feature v2 of multiple pixel regions; for a first pixel region in the multiple pixel regions, perform a normalization operation (softmax) on the similarity between the query feature q2 of the first pixel region and the key feature k2 of the multiple pixel regions, respectively, to obtain a first relationship weight between the first pixel region and the multiple pixel regions, respectively. Figure 6 S2 in the figure includes first relationship weights between the first pixel region and the plurality of pixel regions respectively. The first relationship weights between the first pixel region and the plurality of pixel regions respectively are modified according to the mask information M' to obtain modified relationship weights. Figure 6 S3 includes: the corrected relationship weights between the first pixel region and the multiple pixel regions. According to the corrected relationship weights between the first pixel region and the multiple pixel regions, the value features v2 of the multiple pixel regions are weighted summed to obtain the first fused feature of the first pixel region.

[0191] Step S5: The decoding network 63 in the subtitle erasing model 6 performs a decoding operation on the first fused features of the plurality of pixel regions, and finally obtains a subtitle-erased video.

[0192] Step S6: Process the initial video according to the video after subtitles are erased and the subtitle frame to obtain the initial video after subtitles are erased.

[0193] The above subtitle erasure model includes two branches. The first branch can be understood as a mask perception branch, and the second branch can be understood as a subtitle erasure branch. The mask perception branch provides spatiotemporal features through a spatiotemporal attention mechanism, and activates the background area and suppresses the text area through a spatial response module (SRM), guiding the erasure branch to perform spatiotemporal search features on the suppressed area to complete the filling.

[0194] To sum up, the technical solution provided in the embodiment of the present application completes erasure by utilizing the spatiotemporal attention mechanism to simultaneously learn features from multiple frames in the time domain and the global spatial domain. The erasure effect is more stable and smooth, and it uses a text mask without explicit supervision to automatically sense the area where the text appears and complete the erasure and filling, which brings simplicity and convenience in training.

[0195] Figure 8 FIG. 1 is a schematic diagram showing the structure of an electronic device provided by an embodiment of the present application. Figure 8 As shown, the electronic device includes a memory 1101 and a processor 1102. The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0196] The memory 1101 is used to store programs;

[0197] The processor 1102 is coupled to the memory 1101 and is used to execute the program stored in the memory 1101 to implement the methods provided by the above-mentioned method embodiments.

[0198] Further, if Figure 8 As shown, the electronic device also includes: a communication component 1103, a display 1104, a power component 1105, an audio component 1106 and other components. Figure 8 Only some components are shown schematically, which does not mean that the electronic device only includes Figure 8 Components shown.

[0199] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the steps or functions of the methods provided in the above-mentioned method embodiments can be implemented.

[0200] Figure 8 FIG. 1 shows a schematic diagram of the structure of a head mounted display device provided by an embodiment of the present application. Figure 8 As shown, the head-mounted display device includes a memory 1101, a processor 1102, and a display 1104. The memory 1101 can be configured to store various other data to support operations on the head-mounted display device. Examples of these data include instructions for any application or method operating on the head-mounted display device. The memory 1101 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0201] The memory 1101 is used to store programs;

[0202] The processor 1102 is coupled to the memory 1101 and is configured to execute the program stored in the memory 1101 to implement

[0203] Confirm the video;

[0204] Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model;

[0205] Controlling the display to play the video after the subtitles are erased;

[0206] Wherein, the subtitle erasure model is used for:

[0207] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video;

[0208] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0209] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0210] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0211] A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

[0212] In addition to implementing the above steps, the head-mounted display device provided in the embodiment of the present application can also implement other steps in the above embodiments. For details, please refer to the corresponding contents of the above embodiments, which will not be repeated here.

[0213] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the following can be achieved:

[0214] Confirm the video;

[0215] Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model;

[0216] Play the video after the subtitles are erased;

[0217] Wherein, the subtitle erasure model is used for:

[0218] Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video;

[0219] According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region;

[0220] Modifying the first relationship weight according to the mask information to obtain a modified relationship weight;

[0221] Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight;

[0222] A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

[0223] In the embodiments of the present application, in addition to implementing the above steps, the computer program can also implement other steps in the above embodiments when executed by the computer. For details, please refer to the corresponding contents of the above embodiments, which will not be repeated here.

[0224] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0225] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0226] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video subtitle erasing method, wherein: include: Confirm the video; Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model; Wherein, the subtitle erasure model is used for: Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video; According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region; Modifying the first relationship weight according to the mask information to obtain a modified relationship weight; Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight; A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

2. The method according to claim 1, wherein: Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video includes: Determining first target features of the plurality of pixel regions according to the plurality of frame feature maps; Determining, according to the multiple frame feature maps, second relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; performing weighted summation on the first target features of the plurality of pixel regions according to the second relationship weight to obtain a second fused feature of the first pixel region; Mask information for perceiving a subtitle position is determined according to the second fused features of the plurality of pixel regions.

3. The method according to claim 2, wherein: Determining first target features of the plurality of pixel regions according to the plurality of frame feature maps includes: Perform dimensionality reduction processing on the multiple frame feature maps to obtain first target features of the multiple pixel regions.

4. The method according to any one of claims 1 to 3, wherein: The mask information includes: mask values ​​of the plurality of pixel regions; when the mask value is a first value, it indicates that the corresponding pixel region corresponds to a background position; when the mask value is a second value, it indicates that the corresponding pixel region corresponds to a subtitle position; the first value is greater than the second value; Modifying the first relationship weight according to the mask information to obtain a modified relationship weight includes: multiplying the first relationship weight between the first pixel region and the second pixel region by the mask value of the second pixel region as the modified relationship weight between the first pixel region and the second pixel region; Among them, the multiple pixel areas include the second pixel area.

5. The method according to any one of claims 1 to 3, wherein: Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weights includes: Determining second target features of the plurality of pixel regions according to the plurality of frame feature maps; According to the corrected relationship weight, a weighted sum is performed on the second target features of the plurality of pixel regions to obtain a first fused feature of the first pixel region.

6. The method according to any one of claims 1 to 3, wherein: Also includes: Get the initial video; Determining a subtitle frame corresponding to the initial video; According to the subtitle frame, a plurality of video frames in the initial video are trimmed to obtain the video.

7. The method according to claim 6, wherein: Determining a subtitle frame corresponding to the initial video includes: Inputting the initial video into a subtitle detection model to obtain the subtitle frame; or The subtitle frame is determined according to a subtitle frame drawing operation performed by a user on the initial video.

8. The method according to claim 6, wherein: Also includes: The initial video is processed according to the video after subtitle erasure and the subtitle frame to obtain the initial video after subtitle erasure.

9. A model training method, wherein: Also includes: Determining sample videos and their corresponding expected videos; Erasing the subtitles in the sample video using a subtitle erasing model to obtain a sample video after subtitle erasure, wherein the subtitle erasing model is a machine learning model; Optimizing the subtitle erasure model according to the sample video after subtitle erasure and the expected video; Wherein, the subtitle erasure model is used for: Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the sample video; According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region; Modifying the first relationship weight according to the mask information to obtain a modified relationship weight; Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight; A sample video after subtitle erasure is determined according to the first fused features of the plurality of pixel regions.

10. An interface interaction method, wherein: include: In response to a video input operation of a user on an operation interface, displaying an initial video input by the user; In response to a subtitle erasing operation triggered by a user on the initial video, displaying the initial video after the subtitles are erased; The initial video after the subtitles are erased is determined based on the video after the subtitles are erased; the video after the subtitles are erased is obtained by erasing the subtitles in the video using a trained subtitle erasure model; the video is determined based on the initial video; the subtitle erasure model is a machine learning model; Wherein, the subtitle erasure model is used for: Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video; According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region; Modifying the first relationship weight according to the mask information to obtain a modified relationship weight; Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight; A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

11. An electronic device, wherein: include: A memory and a processor, wherein The memory is used to store programs; The processor is coupled to the memory, and is configured to execute the program stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a computer, the method according to any one of claims 1 to 9 can be implemented.

13. A head mounted display device, wherein: include: A memory, a processor, and a display, wherein: The memory is used to store programs; The processor is coupled to the memory and the display, respectively, and is used to execute the program stored in the memory to achieve: Confirm the video; Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model; Controlling the display to play the video after the subtitles are erased; Wherein, the subtitle erasure model is used for: Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video; According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region; Modifying the first relationship weight according to the mask information to obtain a modified relationship weight; Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight; A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

14. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a computer, it can achieve: Confirm the video; Erasing the subtitles in the video using a trained subtitle erasure model to obtain a subtitle-erased video, wherein the subtitle erasure model is a machine learning model; Play the video after the subtitles are erased; Wherein, the subtitle erasure model is used for: Determining mask information for perceiving a subtitle position according to a plurality of frame feature maps of the video; According to the multiple frame feature maps, determining first relationship weights between the first pixel region and the multiple pixel regions respectively through an attention mechanism; the multiple frame feature maps include features of the multiple pixel regions; the multiple pixel regions include the first pixel region; Modifying the first relationship weight according to the mask information to obtain a modified relationship weight; Determining a first fused feature of the first pixel region according to the multiple frame feature maps and the corrected relationship weight; A subtitle-erased video is determined according to the first fused features of the plurality of pixel regions.

Images