Video processing method and apparatus, electronic device, storage medium, and program product

Abstract

The present disclosure relates to the technical field of video processing and artificial intelligence, and provides a video processing method and apparatus, an electronic device, a storage medium, and a program product. The specific implementation solution of the method comprises: determining a target frame from a two-dimensional video frame sequence to be processed, wherein the target frame comprises foreground text; performing depth estimation on said two-dimensional video frame sequence to obtain a target depth map corresponding to each video frame, wherein depth information of the target depth map represents a fusion result of depth information of the current video frame and depth information of adjacent video frames; processing the target depth maps respectively corresponding to adjacent video frames of the target frame in said two-dimensional video frame sequence and a mask image to generate a target image; and on the basis of a position and a predetermined parallax relationship, processing the target depth maps respectively corresponding to the adjacent video frames and the target image to generate a target viewpoint image comprising the foreground text, and outputting a three-dimensional video on the basis of the target viewpoint image.

Description

Video processing method and device, electronic device, storage medium and program product TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video processing and artificial intelligence, and relates to the technical field of computer vision and deep learning, and particularly relates to a video processing method and device, an electronic device, a storage medium and a program product. BACKGROUND

[0002] A 3D (3-dimension) video is a video synthesized by a three-dimensional software according to real needs, in which a virtual three-dimensional world in a computer is built according to the real size of an object to be represented, a scene is built and a model is established, a realistic material, light and camera motion effect are configured, so as to bring a more immersive visual experience to a user than a 2D (2-dimension) video.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art.

[0004] SUMMARY

[0005] The present disclosure provides a video processing method and device, an electronic device, a storage medium and a program product.

[0006] According to a first aspect, the present disclosure provides a video processing method, comprising: determining a target frame from a sequence of two-dimensional video frames to be processed; wherein the target frame comprises foreground text; performing depth estimation on the sequence of two-dimensional video frames to be processed to obtain a target depth map corresponding to each video frame respectively, wherein the depth information of the target depth map represents the fusion result of the depth information of the current video frame and the depth information of the adjacent video frame; processing the target depth map corresponding to each adjacent video frame of the target frame in the sequence of two-dimensional video frames to be processed and a mask image to generate a target image, wherein the target image comprises complete image information of the target frame; the mask image is obtained by masking the target frame based on the position of the foreground text in the target frame; and processing the target depth map corresponding to each adjacent video frame and the target image based on the position and a predetermined parallax relationship to generate a target view image comprising the foreground text, and outputting a three-dimensional video based on the target view image.

[0007] According to a second aspect, the present disclosure provides a video processing apparatus, comprising a determining module, a depth estimation module, a processing module and a generating module. The determining module is configured to determine a target frame from a sequence of two-dimensional video frames to be processed, wherein the target frame comprises foreground text. The depth estimation module is configured to perform depth estimation on the sequence of two-dimensional video frames to be processed to obtain a target depth map corresponding to each video frame, wherein the depth information of the target depth map represents a fusion result of the depth information of the current video frame and the depth information of the adjacent video frame. The processing module is configured to process the target depth map corresponding to the adjacent video frame of the target frame in the sequence of two-dimensional video frames to be processed and a mask image to generate a target image, wherein the target image comprises complete image information of the target frame, and the mask image is obtained by masking the target frame based on the position of the foreground text in the target frame. The generating module is configured to process the target depth map corresponding to the adjacent video frame and the target image based on the position and a predetermined parallax relationship to generate a target viewpoint image comprising the foreground text, and output a three-dimensional video based on the target viewpoint image.

[0008] According to a third aspect, the present disclosure provides an electronic device, comprising at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] According to a fourth aspect, the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to perform the method described above.

[0010] According to a fifth aspect, the present disclosure provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the method described above.

[0011] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0012] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0013] FIG. 1 is an exemplary architecture diagram of a video processing method and apparatus applied to the embodiments of the present disclosure;

[0014] FIG. 2 is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0015] FIG. 3A is a schematic diagram of a method for determining a target frame in a video processing method according to an embodiment of the present disclosure;

[0016] FIG. 3B is a schematic diagram of a method for determining a target frame in a video processing method according to another embodiment of the present disclosure;

[0017] FIG. 4 is a schematic diagram of a method for generating a target depth map in a video processing method according to an embodiment of the present disclosure;

[0018] FIG. 5 is a schematic diagram of a model training for generating an initial depth map in a video processing method according to an embodiment of the present disclosure;

[0019] FIG. 6 is an exemplary architecture diagram of a network for generating an initial depth map according to an embodiment of the present disclosure;

[0020] FIG. 7 is an exemplary architecture diagram of a network for generating a target depth map according to an embodiment of the present disclosure;

[0021] FIG. 8 is an exemplary architecture diagram of a network for generating a target image according to an embodiment of the present disclosure;

[0022] FIG. 9 is an exemplary architecture diagram of a network for generating a target view image according to an embodiment of the present disclosure;

[0023] FIG. 10 is a flowchart of a video processing method according to another embodiment of the present disclosure;

[0024] FIG. 11 is a block diagram of a video processing apparatus according to an embodiment of the present disclosure; and

[0025] FIG. 12 is a schematic block diagram of an electronic device for performing a video processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are a part of the embodiments of the present disclosure, rather than all. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without any creative work fall within the scope of protection of the present disclosure. It should be noted that throughout the drawings, the same elements are denoted by the same or similar reference numerals. In the following description, some specific embodiments are only for the purpose of description, and should not be understood as any limitation on the present disclosure, but only as examples of the embodiments of the present disclosure. When it may cause confusion to the understanding of the present disclosure, the conventional structure or configuration will be omitted. It should be noted that the shapes and sizes of the components in the drawings do not reflect the true size and ratio, but only illustrate the content of the embodiments of the present disclosure.

[0027] Unless otherwise defined, technical terms or scientific terms used in the present disclosure should be interpreted as is customary in the art by a person skilled in the art. The terms "first", "second", and similar terms used in the present disclosure do not denote any order, quantity, or importance, but are used to distinguish different constituent parts.

[0028] Due to the long shooting and production cycle of 3D videos, it is difficult for 3D video content to meet user needs. Related examples generate 3D videos based on 2D video conversion. Although this method can effectively alleviate the shortage of 3D video content, for videos including foreground text, there are still problems such as edge jitter, font deformation, and poor video scene continuity in the playback process of the converted 3D video, which reduces user experience.

[0029] Therefore, the present disclosure provides a video processing method. The depth information of adjacent video frames and the depth information of the current video frame are fused to obtain a target depth map corresponding to each video frame, which can effectively reduce the edge jitter of the converted 3D video in the video playback process. The depth information of adjacent video frames is used to fill the image content of the video frame containing foreground text to obtain a complete image, which further improves the continuity of the image content of the video scene. Based on the position of the foreground text and the predetermined parallax relationship, the target depth map and the target image corresponding to each adjacent video frame are used to generate a target viewpoint image including the foreground text, which further improves the depth continuity of the video scene and reduces the probability of font deformation.

[0030] FIG. 1 is an exemplary architecture diagram of a video processing method and device applied to the present disclosure.

[0031] As shown in FIG. 1, the exemplary architecture 100 can include a terminal device 101 and a server 102. A user can input a 2D video frame sequence 110 to the terminal device 101. Then, the terminal device 101 can send the 2D video frame sequence 110 to the server 102. The server 102 can generate an image sequence 120 of a target viewpoint by executing the video processing method of the present disclosure, and send the image sequence 120 of the target viewpoint to the terminal device 101, so that the terminal device 101 takes each video frame in the 2D video frame sequence 110 as a left viewpoint image, takes each image in the image sequence 120 of the target viewpoint as a right viewpoint image, and outputs a three-dimensional video 130 according to a predetermined format based on the 2D video frame sequence 110 and the image sequence 120 of the target viewpoint.

[0032] It should be noted that the video processing method provided by the embodiments of the present disclosure can also be executed by the terminal device 101. For example, the 2D video frame sequence 110 can be pre-stored in the terminal device 101 or input into the terminal device 101. The terminal device 101 can first generate the image sequence 120 of the target view by executing the video processing method of the embodiments of the present disclosure. Then, each video frame in the 2D video frame sequence 110 is taken as a left view image, each image in the image sequence 120 of the target view is taken as a right view image, and the three-dimensional video 130 is output according to a predetermined format based on the 2D video frame sequence 110 and the image sequence 120 of the target view.

[0033] FIG. 2 is a flowchart of a video processing method according to an embodiment of the present disclosure.

[0034] As shown in FIG. 2, the method 200 can include operations S210-S240.

[0035] In operation S210, a target frame is determined from a to-be-processed two-dimensional video frame sequence.

[0036] In operation S220, depth estimation is performed on the to-be-processed two-dimensional video frame sequence to obtain a target depth map corresponding to each video frame.

[0037] In operation S230, the target depth map corresponding to each adjacent video frame of the target frame in the to-be-processed two-dimensional video frame sequence and the mask image are processed to generate a target image.

[0038] In operation S240, the target depth map corresponding to each adjacent video frame and the target image are processed based on the position and the predetermined parallax relationship to generate a target view image including foreground text, and a three-dimensional video is output based on the target view image.

[0039] According to the embodiments of the present disclosure, the target frame includes foreground text. The foreground text can include at least one of the following: a subtitle for displaying video voice information, a barrage for displaying user comment information, a subtitle for displaying the number of times that a user operates a target object in a game video, and the like. When determining the target frame, each video frame including the foreground text can be taken as the target frame, or the target frame can be determined from a plurality of video frames corresponding to the same foreground text based on a predetermined frame extraction strategy. The predetermined frame extraction strategy can be determined according to the actual application scene requirement, and the embodiments of the present disclosure do not make specific limitations thereon. For example, the frame extraction strategy can be equal time interval frame extraction or scene change based frame extraction. Therefore, the number of target frames can be associated with the frame extraction strategy. For example, the number of target frames can be greater than or equal to the number of scene types in the plurality of video frames corresponding to the same foreground text.

[0040] For example, text recognition can be performed on each video frame in the sequence of two-dimensional video frames to be processed based on an OCR (Optical Character Recognition) technology to determine the target frame.

[0041] The depth information of an image refers to information of a distance from a viewpoint to a target object in the image, and the viewpoint can be a device used to capture the image. By performing depth estimation on each video frame, depth information of each video frame can be obtained. Since the distance between the viewpoint and the target object changes over time during the capture of the video, the depth information of each pixel point obtained based on the depth estimation of a single frame of image also changes over time.

[0042] However, the depth information obtained based on the depth estimation of a single frame of image breaks the continuity of the depth information over time between adjacent video frames, resulting in severe edge jitter of the target object. Therefore, by performing depth estimation on the sequence of two-dimensional video frames to be processed, the depth information of each video frame is fused with the depth information of adjacent video frames to obtain a target depth map that can represent the fusion result of the depth information of the current video frame and the depth information of adjacent video frames, and the temporal information of adjacent video frames is effectively combined to reduce the edge jitter of the target object in the video frame.

[0043] According to an embodiment of the present disclosure, the depth information fusion between adjacent video frames can be achieved by smoothing the depth information, for example, a plurality of regions can be determined from a plurality of video frames adjacent to each other, and the division of the regions can be according to the type of target object or according to the foreground and background, which is not specifically limited by the embodiment of the present disclosure. Based on the average depth of the pixel points of each region, a smooth curve of the average depth of the region changing over time can be constructed with the average depth as the vertical coordinate and the time corresponding to the video frame as the horizontal coordinate. The depth value after smoothing is taken as the depth of the region.

[0044] According to an embodiment of the present disclosure, the depth information fusion between adjacent video frames can also be achieved by using a trained neural network. For example, a time transformer network based on an attention mechanism.

[0045] According to an embodiment of the present disclosure, for a non-target frame that does not include foreground text, a target viewpoint image corresponding to the non-target frame can be generated based on a target depth map corresponding to the non-target frame and a target depth map of a video frame adjacent to the non-target frame.

[0046] For example, the target depth map of the non-target frame and the video frame adjacent to the non-target frame can be input into the multi-head attention mechanism time sequence network to output the target viewpoint image corresponding to the non-target frame, and the target viewpoint can be the right viewpoint. The multi-head attention mechanism time sequence network can be a transformer network. The multi-head attention mechanism time sequence network can be obtained by supervised training with the right viewpoint image corresponding to the sample image as a label.

[0047] For the target frame, since the edge of the foreground text is sharp and the depth difference with the target object in the background is large, when the multi-head attention mechanism time sequence network described in the foregoing is directly used to process the target frame, the font of the foreground text in the output target viewpoint image is easily deformed or the foreground text is easily blocked. Therefore, the embodiment of the present disclosure can first process the background image of the target frame to generate the target viewpoint image, and then embed the foreground text to generate the target viewpoint image including the foreground text.

[0048] However, the foreground text region in the target frame is occluded to the background image, and directly processing the background image of the target frame to generate the target viewpoint image in the foreground text region will have a background defect, which interrupts the continuity of the video scene.

[0049] Therefore, the embodiment of the present disclosure first masks the target frame based on the position of the foreground text in the target frame to obtain a mask image. Then, the target depth map corresponding to each adjacent video frame of the target frame in the to-be-processed two-dimensional video frame sequence and the mask image are processed, the foreground text region of the target frame is filled based on the continuity of the video scene between the adjacent video frames, and a target image of the target frame with complete image information is generated.

[0050] For example, the network for generating the target image can be obtained by supervised training with the same network architecture as the multi-head attention mechanism time sequence network described in the foregoing and a sample image as a label.

[0051] According to the embodiment of the present disclosure, the network for generating the target viewpoint image described in the foregoing can be used to process the target depth map corresponding to each adjacent video frame and the target image to generate the target viewpoint image not including the foreground text. Then, the foreground text is embedded into the target viewpoint image based on the position of the foreground text in the target frame and the predetermined parallax relationship to generate the target viewpoint image including the foreground text. The target frame is taken as the left viewpoint image, the target viewpoint image is taken as the right viewpoint image, and the three-dimensional video is output in a predetermined output format.

[0052] According to an embodiment of the present disclosure, the depth information of adjacent video frames is fused with the depth information of the current video frame to obtain a target depth map corresponding to each video frame, which can effectively reduce the edge jitter of the converted 3D video during video playback. The depth information of adjacent video frames is used to fill the image content of the video frame containing the foreground text to obtain a complete image, which further improves the continuity of the image content of the video scene. Based on the position of the foreground text and the predetermined parallax relationship, the target depth map and the target image corresponding to each adjacent video frame are used to generate a target viewpoint image including the foreground text, which further improves the depth continuity of the video scene and reduces the probability of font deformation.

[0053] According to an embodiment of the present disclosure, in a video including a voice caption, the position of the foreground text region where the voice caption is located is the same in a plurality of adjacent video frames corresponding to the same voice caption, and the content of the voice caption is also the same.

[0054] When performing foreground text recognition on a single image based on the OCR technology, it is difficult to verify the accuracy of text recognition. However, there is a time dimension continuity between adjacent video frames. For example, for the same voice caption, the text content in the foreground text region of the adjacent video frames within the display duration of the voice caption is the same.

[0055] Therefore, the embodiment of the present disclosure can utilize the time dimension continuity of the foreground text between adjacent video frames to further improve the recognition accuracy of the foreground text.

[0056] For example, determining a target frame from a sequence of two-dimensional video frames to be processed can include the following operations: first, performing foreground text recognition on the sequence of two-dimensional video frames to be processed to determine a foreground text region where the foreground text is located; and in response to the text content difference between the target region in the adjacent video frame of the ith video frame and the foreground text region in the ith video frame being less than a first predetermined threshold, determining the ith video frame as the target frame, wherein the position of the target region in the adjacent video frame is the same as the position of the foreground text region in the ith video frame.

[0057] According to an embodiment of the present disclosure, the adjacent video frames can be the ith video frame and the ith+1 video frame adjacent to the ith video frame. Alternatively, the adjacent video frames can be taken within a target duration before and after the ith video frame at equal intervals. The target duration can be the display duration of the text in the foreground text region. Since the display duration of the text in the foreground text region is proportional to the number of texts, the display speed parameter can be set according to the specific application scenario.

[0058] For example, in the voice caption scenario, the display speed parameter can be the average speech speed of a human body, 0.2 seconds per word, the number of words in the foreground text region in the ith video frame can be M, and the display duration of the words in the foreground text region can be determined as 0.2M seconds. Therefore, a plurality of adjacent video frames can be taken at equal intervals within 0.2M seconds before and after the ith video frame.

[0059] According to an embodiment of the present disclosure, the first predetermined threshold value can represent a threshold value of the number of difference words, and the number of difference words can be determined by comparing the word content in the target region with the word content in the foreground text region word by word in a predetermined direction, for example, from left to right. When the number of difference words is less than the first predetermined threshold value, the ith video frame can be determined as the target frame.

[0060] According to an embodiment of the present disclosure, the first predetermined threshold value can also represent a threshold value of the text distance, for example, the Euclidean distance, and the text distance can be calculated by comparing the text distance in the target region with the text distance in the foreground text region. When the text distance is less than the first predetermined threshold value, the ith video frame can be determined as the target frame.

[0061] According to an embodiment of the present disclosure, by comparing the word content of the foreground text in the same position region between adjacent video frames, the continuity of the foreground text in the time dimension between adjacent video frames is utilized to further improve the recognition accuracy of the foreground text.

[0062] In actual application scenarios, the background image in the video frame can also include words, for example, the background image can include a building, and the building can be provided with a word signboard. When the OCR technology is used for text recognition, the words on the word signboard can also be recognized as foreground text.

[0063] Therefore, the embodiments of the present disclosure can further improve the recognition accuracy of the foreground text region in combination with other feature information of the foreground text.

[0064] For example, the word direction of the foreground text region is usually presented in the target frame in a horizontal direction or a vertical direction. Therefore, the positions of a plurality of words in the foreground text region in the ith video frame can be obtained, the word direction of the plurality of words can be determined by fitting the positions of the plurality of words, and the ith video frame can be determined as the target frame in response to the word direction being a predetermined direction.

[0065] According to an embodiment of the present disclosure, the positions of the plurality of words can be the coordinates of the plurality of words in the ith video frame, and the word direction of the plurality of words can be determined by linearly fitting the coordinates of the plurality of words by using any linear fitting method. The linear fitting method includes but is not limited to the least square method and the robust fitting, and the present disclosure does not make a specific limitation on the linear fitting method.

[0066] According to an embodiment of the present disclosure, whether the character direction of the plurality of characters is the predetermined direction can be determined according to an included angle between a curve obtained by linear fitting and a straight line of the predetermined direction. For example, when the included angle is less than 5°, it can be determined that the character direction of the plurality of characters is the predetermined direction, so as to determine that the plurality of characters in the region is foreground text, and further determine that the i-th video frame is the target frame.

[0067] According to an embodiment of the present disclosure, the character direction in the foreground text region is used as an auxiliary feature for determining the foreground text region, which effectively reduces the interference of the characters in the background region on the foreground text region, and further improves the recognition accuracy of the foreground text region.

[0068] According to an embodiment of the present disclosure, the character attributes in the foreground text region, such as color, gradient, and saturation, are more obviously different from the character attributes in the background image. Therefore, the character attributes of the plurality of characters in the foreground text region in the i-th video frame can be detected, and in response to the character attributes being predetermined attributes, the i-th video frame is determined as the target frame.

[0069] For example, the main body of the characters in the foreground text region can be white, the edge can be black, and the characters in the character identification card of the building in the background image can be red. The character color of the plurality of characters in the foreground text region in the i-th video frame can be detected, and when the main body of the plurality of characters is white and the edge is black, it can be determined that the characters in the region are foreground text, and further determine that the i-th video frame is the target frame.

[0070] According to an embodiment of the present disclosure, the position of the foreground text region is related to the video scene, for example, the position of the speech subtitles in the video is usually located at the bottom of the picture, and the position of the bullet screen is usually located at the top of the picture. The chat subtitles in the game video are usually located at the bottom left of the picture, and the subtitles for recording the number of user operations are usually located above the object being operated. Therefore, the position of the foreground text can be used to assist in determining the target frame, for example, in response to the position of the foreground text region in the i-th video frame belonging to a predetermined position range, the i-th video frame is determined as the target frame.

[0071] According to an embodiment of the present disclosure, the character attributes and / or position in the foreground text region are used as auxiliary features for determining the foreground text region, which can improve the recognition accuracy of the foreground text region without increasing additional calculation processes, so as to reduce the demand for hardware resource processing performance.

[0072] With the development of speech recognition technology, the accuracy of speech recognition is continuously improved, and the speech recognition result can be used to further judge the character content in the foreground text region, so as to improve the recognition accuracy of the foreground text region.

[0073] According to an embodiment of the present disclosure, speech recognition can be performed on the audio corresponding to the i-th video frame to obtain a speech recognition result; and in response to the speech recognition result being different from the text content in the foreground text region in the i-th video frame by less than a first predetermined threshold, the i-th video frame is determined as a target frame.

[0074] For example, the speech recognition result of the audio corresponding to the i-th video frame can be "today is sunny". The text content in the foreground text region obtained through OCR recognition can be "XX Mansion". It can be determined that the speech recognition result is inconsistent with the text content in the foreground text region, indicating that the foreground text region recognized based on the OCR technology is incorrect, and therefore the i-th video frame can be determined as a non-target frame.

[0075] For example, the speech recognition result of the audio corresponding to the i-th video frame can be "today is sunny". The foreground text region obtained through OCR recognition can include region S1 and region S2, the text content in region S1 can be "today is sunny", and the text content in region S2 can be "XX Mansion". It can be determined that the speech recognition result is the same as the text content in region S1, and therefore region S1 can be determined as the foreground text region of the i-th video frame, and the i-th video frame can be determined as a target frame.

[0076] According to an embodiment of the present disclosure, the speech recognition result of the corresponding audio is used to further judge the text content in the foreground text region, which further improves the recognition accuracy of the foreground text region.

[0077] Since the edge of the foreground text is sharp and the depth difference with the background image is large, but when the foreground text is recognized based on the OCR technology, the spatial depth information of the text is often ignored. Therefore, the depth information in the target depth map can be used to determine whether the foreground text is included in the video frame.

[0078] According to an embodiment of the present disclosure, the foreground text region can be binarized based on the target depth map corresponding to the i-th video frame to obtain first depth information corresponding to the text and second depth information corresponding to the background; and in response to the difference between the first depth information and the second depth information being greater than a second predetermined threshold, the i-th video frame is determined as a target frame.

[0079] According to an embodiment of the present disclosure, the first depth information can represent the average depth of the pixel points in the text region. The second depth information can represent the average depth of the pixel points in the background region. The depth difference between the first depth information and the second depth information can be obtained based on any mathematical operation.

[0080] For example, the depth difference between the first depth information and the second depth information can be obtained based on formula (1).

[0081] wherein, △dep represents the depth difference; represents the first depth information; represents the second depth information.

[0082] According to the embodiments of the present disclosure, in the process of converting 2D video to generate 3D video, depth estimation for each video frame is a necessary step, therefore, by using the depth difference between the text area and the background area, without introducing additional information, the recognition accuracy of the foreground text area can be improved, thereby further improving the detection accuracy of the target frame.

[0083] According to the embodiments of the present disclosure, the above-described methods for determining the target frame can be combined in any manner based on the requirements of the application scenario, and the combination manner can be serial or parallel. The method for determining the target frame in the combined manner is further described below in combination with FIGS. 3A-3B.

[0084] FIG. 3A is a schematic diagram of a method for determining a target frame in a video processing method according to an embodiment of the present disclosure.

[0085] As shown in FIG. 3A, the method 300A for determining the target frame is to execute the above-described methods for determining the target frame in a serial combination manner. The method 300A can include operations S311-S319.

[0086] In operation S311, the foreground text area of the ith video frame is determined.

[0087] In operation S312a, it is determined whether the text difference of the area with the same position between adjacent video frames is less than a? If yes, operation S313a is performed; if no, operation S319 is performed.

[0088] In operation S313a, it is determined whether the text direction in the foreground text area is a predetermined direction? If yes, operation S314a is performed; if no, operation S319 is performed.

[0089] In operation S314a, it is determined whether the text attribute in the foreground text area is a predetermined attribute? If yes, operation S315a is performed; if no, operation S319 is performed.

[0090] In operation S315a, it is determined whether the position of the foreground text area is within a predetermined position range? If yes, operation S316a is performed; if no, operation S319 is performed.

[0091] In operation S316a, it is determined whether the text difference between the foreground text area and the speech recognition result of the audio is less than a? If yes, operation S317a is performed; if no, operation S319 is performed.

[0092] In operation S317a, it is judged whether the depth difference between the character in the foreground text region and the background is greater than β. If yes, operation S318 is performed; if no, operation S319 is performed.

[0093] In operation S318, the i-th video frame is determined as a target frame.

[0094] In operation S319, the i-th video frame is determined as a non-target frame.

[0095] According to the embodiment of the present disclosure, the current frame is judged from different dimensions in a serial combination manner to determine whether the current frame includes foreground text, which further improves the determination accuracy of the target frame.

[0096] According to the embodiment of the present disclosure, the above-described multiple methods for determining the target frame can also be executed in a parallel combination manner, for example: the weight of each target parameter can be determined; and according to the weight, the target frame is determined by weighting processing on the judgment result corresponding to each target parameter. The multiple target parameters include at least two of the following: character direction, character attribute, position of the foreground text region, character content difference, and depth difference between the character and the background.

[0097] FIG. 3B is a schematic diagram of a method for determining a target frame in a video processing method according to another embodiment of the present disclosure.

[0098] As shown in FIG. 3B, the method 300B for determining a target frame is to execute the above-described multiple methods for determining a target frame in a parallel combination manner. The method 300B can include operations S311-S318b.

[0099] In operation S311, the foreground text region of the i-th video frame is determined.

[0100] In operation S312b, the character difference of the regions with the same position between adjacent video frames is compared to obtain a comparison result A312.

[0101] In operation S313b, the character direction in the foreground text region is compared with the predetermined direction to obtain a comparison result B313.

[0102] In operation S314b, the character attribute in the foreground text region is compared with the predetermined character attribute to obtain a comparison result C314.

[0103] In operation S315b, the position in the foreground text region is compared with the predetermined position range to obtain a comparison result D315.

[0104] In operation S316b, the character difference between the foreground text region and the audio recognition result is compared to obtain a comparison result E316.

[0105] In operation S317b, a contrast result F316 is obtained by comparing the depth difference between the text in the foreground text region and the background.

[0106] According to an embodiment of the present disclosure, the weights corresponding to the contrast results A-F in sequence can be W1-W6, and the target frame can be determined according to the weighted sum result in operation S318b.

[0107] According to an embodiment of the present disclosure, the current frame is determined whether to include foreground text from different dimensions in a parallel combination manner, which not only improves the determination accuracy of the target frame, but also improves the data processing efficiency.

[0108] The method of fusing the depth information between adjacent video frames to generate a target depth by using the trained deep neural network described above will be described below in combination with FIGS. 4-7.

[0109] According to an embodiment of the present disclosure, the depth estimation of the sequence of two-dimensional video frames to be processed to generate the target depth map corresponding to each video frame can include the following operations: performing depth estimation on each video frame to obtain an initial depth map corresponding to each video frame; and based on the attention mechanism, processing the initial depth map corresponding to the i-th video frame, the initial depth map corresponding to the i-1-th video frame, and the initial depth map corresponding to the i+1-th video frame to generate a target depth map corresponding to the i-th video frame; wherein i is an integer greater than 1 and less than or equal to I.

[0110] FIG. 4 is a schematic diagram for generating a target depth map in a video processing method according to an embodiment of the present disclosure.

[0111] As shown in FIG. 4, in embodiment 400, video frame 421_1-video frame 421_3 are input into spatial depth estimation network 401 respectively, and initial depth map 422_1-initial depth map 422_3 corresponding to each are output. Then, initial depth map 422_1-initial depth map 422_3 are input into temporal depth estimation network 402, and target depth map 423 is output.

[0112] According to an embodiment of the present disclosure, the video frames adjacent to the i-th video frame can be the video frames adjacent to the i-th video frame in time sequence in the original video frame sequence, but this way requires a large amount of computing resources. Therefore, in actual application scenarios, the video frame sequence to be processed can be extracted from the original video frame sequence based on a frame extraction strategy, and the video frames adjacent to the i-th video frame in time sequence are used as adjacent video frames. The frame extraction strategy can be determined based on the requirements of the application scenario, for example: extracting frames at equal time intervals, extracting key frames, etc. The present disclosure does not make specific limitations on this.

[0113] According to an embodiment of the present disclosure, since the image resolution of the video frame is large, the direct input of the spatial domain depth estimation network has a high demand for hardware resources, therefore, each video frame can be compressed first to obtain an image with a predetermined resolution, for example, an image with a resolution of 448x448, and then the image features of the image with the predetermined resolution are processed by using the spatial domain depth estimation network to obtain an initial depth map.

[0114] According to an embodiment of the present disclosure, the spatial domain depth estimation network can be trained by taking the depth information of each pixel in the sample image as a label.

[0115] FIG. 5 is a schematic diagram of model training for generating an initial depth map in a video processing method according to an embodiment of the present disclosure.

[0116] As shown in FIG. 5, in the embodiment 500, the spatial domain depth estimation network 401 can include a Resnet 401_1 and an upscaling network Upscaling Network 401_2. The sample image 510 can be an image with a resolution of 448x448. The sample image 510 is input into the Resnet 401_1, and the image features 511 are output. The image features 511 are processed by using the Upscaling Network 401_2 to generate a sample depth map 512. The sample label 513 can represent the real depth map of the sample image, and the real depth map of the sample image includes the real depth information of each pixel point.

[0117] According to an embodiment of the present disclosure, the loss value 514 can be obtained based on a target loss function according to the sample depth map 512 and the sample label 513. The model parameters of the spatial domain depth estimation network 401 are adjusted by using the loss value 514 until a predetermined number of iterations of training is reached or the loss value converges, and the trained spatial domain depth estimation network 401 is obtained.

[0118] When the image is subjected to target detection, the pixel loss of the edge region has a small influence on the detection accuracy. However, when the video frame is subjected to depth estimation, the pixel loss of the edge region will cause a large fluctuation of the depth value, thereby causing the edge of the target object to shake. Therefore, the embodiment of the present disclosure increases a weight factor of the edge region on the basis of the pixel-level loss function, and the target loss function is as shown in formula (2):

[0119] wherein, α Laplacian represents a Laplacian weight coefficient; Loss_pixel represents a pixel-level loss function.

[0120] According to an embodiment of the present disclosure, the pixel-level loss function can select a function capable of calculating the inter-pixel loss, for example: a cross-entropy loss function, a Pixel-wise loss function, a Berhu loss function, etc.

[0121] For example, the pixel-level loss function can be as shown in formula (3):

[0122] wherein label (x,y) represents the real depth information corresponding to the pixel point (x, y) in the sample label; represents the predicted depth information corresponding to the pixel point (x, y) in the sample depth map.

[0123] According to an embodiment of the present disclosure, a Laplacian may be a normalized parameter obtained by formula (4):

[0124] wherein label*Laplacian (x，y) represents the result of performing a Laplacian edge detection operation with a convolution kernel of 3x3 on the sample depth map.

[0125] It should be noted that the weight factor in formula (2) represents that the weight of the edge region is about 2 times that of the non-edge region. The weight factor can also be adjusted according to the needs of the actual application scenario.

[0126] The process of using the spatial depth estimation network to perform depth estimation on a single frame image will be described below in conjunction with FIG. 6.

[0127] FIG. 6 is an exemplary architecture diagram of a network for generating an initial depth map according to an embodiment of the present disclosure.

[0128] As shown in FIG. 6, first, a predetermined resolution image 610 is input into a Resnet for feature extraction, and multi-scale image features are obtained, for example: S / 2 scale image feature 611a, S / 4 scale image feature 611b, S / 8 scale image feature 611c, and S / 16 scale image feature 611d.

[0129] Then, the image feature 611d of S / 16 scale is input into the ASPP (atrous spatial pyramid pooling) 601 for sampling, and the sampling result is input into the Conv block (convolution layer) 602 to output the depth map R5 of S / 16 scale. And the above sampling result is input into the Upconv block (up-convolution layer) 603 for feature difference value to obtain the sampling feature of S / 8 scale. And the image feature 611c of S / 8 scale is subtracted from the sampling feature of S / 8 scale to obtain the difference feature L4 612d of S / 8 scale.

[0130] Then, the depth map R5 of S / 16 scale is subjected to bilinear upsampling 604 to obtain the depth feature of S / 8 scale. After the depth feature of S / 8 scale, the difference feature 612d of S / 8 scale and the image feature 611c of S / 8 scale are spliced, convolution is performed, and the convolution result is added to the difference feature 612d of S / 8 scale, the depth map R4 of S / 8 scale is output. In this way, through layer-by-layer upsampling, the scale of the depth map is gradually expanded. The result of each upsampling needs to calculate the difference with the image feature of the same scale to obtain the difference feature L3 612c of S / 4 scale, the difference feature L2 612b of S / 2 scale and the difference feature L1 612a of the initial scale, respectively, and the depth estimation is performed after splicing the depth feature, the difference feature and the image feature of the same scale, so that the edges in the depth map are gradually clear, until the initial depth map D1 of the same scale as the predetermined resolution image 610 is obtained.

[0131] According to an embodiment of the present disclosure, a Laplacian weight coefficient is introduced in the target loss function, so that the spatial depth estimation obtained by training can accurately identify the depth information of the object edges in the video frame.

[0132] According to an embodiment of the present disclosure, based on the attention mechanism, the initial depth map corresponding to the i-th video frame, the initial depth map corresponding to the i-1-th video frame and the initial depth map corresponding to the i+1-th video frame are processed to generate the target depth map corresponding to the i-th video frame, which can include the following operations: respectively extracting features from the initial depth map corresponding to the i-th video frame, the initial depth map corresponding to the i-1-th video frame and the initial depth map corresponding to the i+1-th video frame to generate the respective first intermediate feature maps; and based on the attention mechanism, fusing the respective first intermediate feature maps to generate the target depth map corresponding to the i-th video frame.

[0133] FIG. 7 is an exemplary architecture diagram of a network for generating a target depth map according to an embodiment of the present disclosure.

[0134] As shown in FIG. 7, the time domain depth estimation module 402 can include 3 1x1 convs 402_1-402_3 and 1 3x3 conv 402_4. First, the initial depth map of each video frame needs to pass through 3 1x1 convs 402_1-402_3 to generate the respective corresponding query matrix Q 4221, key matrix K 4222 and value matrix V 4223.

[0135] For example: the initial depth map 422_2 of the i-th video frame passes through the 1x1 conv 402_1, and each element in the output query matrix Q i can be represented as q i As shown in formula (5):

[0136] wherein q i represents an element in the query matrix Q i ; W q represents the weight matrix corresponding to the query matrix Q i ; pos i represents the position encoding of the i-th video frame; represents the initial depth information of the i-th video frame.

[0137] For example: the initial depth map 422_2 of the i-th video frame passes through the 1x1 conv 402_2, and each element in the output query matrix K i can be represented as k i As shown in formula (6):

[0138] wherein k i represents an element in the query matrix K i ; W k represents the weight matrix corresponding to the query matrix K i ; pos i represents the position encoding of the i-th video frame; represents the initial depth information of the i-th video frame.

[0139] For example: the initial depth map 422_2 of the i-th video frame passes through the 1x1 conv 402_3, and each element in the output query matrix V i can be represented as v i As shown in formula (7):

[0140] wherein v i represents an element in the query matrix V i ; W v represents the weight matrix corresponding to the query matrix V i ; posi represents the position encoding of the i-th video frame; represents the initial depth information of the i-th video frame.

[0141] Then, based on the attention mechanism, the depth information of adjacent frames is fused, as shown in formula (8):

[0142] wherein b n represents the fused feature; d k represents the dimension of k.

[0143] Finally, the fused feature is input into the 3x3conv 402_4, and the target depth map of the i-th video frame is output.

[0144] According to an embodiment of the present disclosure, the temporal depth estimation network 402 can be trained to obtain the depth information of each pixel in the sample image at a predetermined resolution based on a similar training method as the spatial depth estimation network. Details are not described here.

[0145] According to an embodiment of the present disclosure, the trained temporal depth estimation network is used to fuse the initial depth information of adjacent video frames and the initial depth information of the current frame based on the attention mechanism, which effectively reduces the difference between the depth information of single-frame images, and further reduces the shaking of the object edges in the video due to the difference in depth information.

[0146] Because the occlusion of the foreground text of the target frame to the background image will cause the phenomenon that there are background defects between multiple characters of the foreground text in the target view image. Under normal circumstances, the background image content of adjacent video frames is the same, but the depth information may be different, so the background image of the foreground text region in the target frame can be filled using adjacent video frames.

[0147] For example: the target depth map and the mask image corresponding to each adjacent video frame can be respectively extracted to generate the second intermediate feature map corresponding to each; the second intermediate feature map corresponding to each is spliced to obtain the third intermediate feature map; based on the attention mechanism, the third intermediate feature map is processed to generate the first difference information between the target image and the mask image; and based on the first difference information, the mask image is processed to generate the target image.

[0148] According to an embodiment of the present disclosure, the intermediate feature map corresponding to the target depth map can include the depth features of adjacent video frames. The intermediate feature map corresponding to the mask image can include the image features of the mask image.

[0149] According to an embodiment of the present disclosure, the attention mechanism can be a multi-head attention mechanism. By performing attention calculation on the mask image features and the depth features in the third intermediate feature map, the first difference information between the target image and the mask image can be generated after convolution processing. Then, the mask image and the first difference information can be merged to generate the target image.

[0150] The method of generating the target image using the target depth map and the mask image corresponding to each of the adjacent video frames of the target frame described above will be described below in conjunction with FIG. 8.

[0151] FIG. 8 is an exemplary architecture diagram of a network for generating a target image according to an embodiment of the present disclosure.

[0152] As shown in FIG. 8, in the embodiment 800, first, the mask image 831_2 of the target frame is input into the 3x3conv 801_2 for feature extraction, and the image features of the mask image are output. The target depth map 831_1 and 831_3 of the adjacent video frames are input into the 3x3conv 801_1 and 3x3conv 801_3, respectively, and the corresponding depth features are output.

[0153] Then, the image features of the mask image and the corresponding depth features of the adjacent video frames are spliced to obtain the feature map 832.

[0154] Next, the feature map 832 (corresponding to the third intermediate feature map described above) can be divided to obtain a plurality of feature blocks 833. Based on the attention mechanism, the plurality of feature blocks are processed to generate a plurality of local features corresponding to each of the plurality of pixel blocks. The plurality of local features are spliced to obtain a global fusion feature, and the global fusion feature is processed to generate the first difference information.

[0155] For example: for Patch1, first, the 3 1x1conv 802_1-802_3 are input respectively, and the query matrix Q 834_1, the key matrix K 834_2 and the value matrix V 834_3 corresponding to Patch1 are output respectively. Based on the multi-head attention mechanism, the local feature corresponding to Patch1 is obtained. Similarly, the local features corresponding to Patch1-PatchN can be obtained. Then, the N local features can be rearranged according to the division manner of the feature map 832 and then spliced to generate the global fusion feature. Next, the global fusion feature is input into the 3x3conv 803, and the first difference information is output.

[0156] Finally, the first difference information and the mask image 831_2 of the target frame are merged to output the target image 836.

[0157] According to an embodiment of the present disclosure, the network shown in FIG. 8 can be trained based on a training method similar to the spatial depth estimation network, with sample images of a predetermined resolution being labeled. The difference lies in the target loss function, and other training processes are not described herein.

[0158] According to an embodiment of the present disclosure, the target loss function for training the network shown in FIG. 8 can be as shown in formula (9):

[0159] wherein α Gussian represents the coefficient of Gaussian smoothing of the foreground text region and the background region.

[0160] According to an embodiment of the present disclosure, since the first difference information fuses the image features and the depth features of the adjacent video frames, the complete image information of the target frame includes both the content of the background image and the depth of the background image, the natural filling of the foreground text region is achieved, the coherence and consistency of the target image and the visual scene of the adjacent video frames are ensured, the authenticity of the three-dimensional video is further improved, and the user experience is improved.

[0161] For the target frame including the foreground text, since there is a large difference between the depth of the text in the foreground text region and the depth of the background region, the foreground text is processed together with the background region in related examples, which is easy to cause the font deformation, resulting in the decrease of the readability and visibility of the foreground text.

[0162] Therefore, in the embodiments of the present disclosure, the target depth map and the target image corresponding to the adjacent video frames are processed based on the position and the predetermined parallax relationship to generate the target viewpoint image including the foreground text, which can include the following operations: processing the target depth map and the target image corresponding to the adjacent video frames to generate an initial viewpoint image; and filling the foreground text into the initial viewpoint image based on the position and the predetermined parallax relationship to generate the target viewpoint image.

[0163] According to an embodiment of the present disclosure, the transformer network trained can be used to process the target depth map and the target image corresponding to the adjacent video frames, so that the image features of the target image and the depth information of the adjacent video frames are fused to generate the initial viewpoint image not including the foreground text. The target viewpoint can be a right viewpoint, and the viewpoint direction of the initial viewpoint image and the target viewpoint image is the same.

[0164] According to an embodiment of the present disclosure, the transformer network can be trained based on a training method similar to the spatial depth estimation network, with the right viewpoint image of the sample image including the foreground text being labeled. The difference lies in the target loss function, and other training processes are not described herein.

[0165] According to an embodiment of the present disclosure, the loss function used to train the transformer network can be a pixel-level loss function described above, such as a cross-entropy loss function, etc.

[0166] According to an embodiment of the present disclosure, the predetermined disparity relationship can represent a disparity geometric relationship between the left-view image and the right-view image, and the target position of the foreground text in the right-view image can be determined according to the predetermined disparity relationship based on the position of the foreground text region in the i-th video frame. Then, the foreground text can be filled into the right-view image according to the target position.

[0167] According to an embodiment of the present disclosure, by separately processing the background image and the foreground text of the video frame, the probability of font deformation of the foreground text can be reduced, thereby improving the readability and visibility of the foreground text under different viewing angles.

[0168] According to an embodiment of the present disclosure, processing the target depth map and the target image corresponding to each of the adjacent video frames to generate the initial view image can include the following operations: performing feature extraction on the target depth map and the target image corresponding to each of the adjacent video frames respectively to generate a fourth intermediate feature map corresponding to each of the adjacent video frames; splicing the fourth intermediate feature map corresponding to each of the adjacent video frames to obtain a fifth intermediate feature map; processing the fifth intermediate feature map based on an attention mechanism to generate second difference information between the initial view image and the target image; and processing the target image based on the second difference information to generate the initial view image.

[0169] The method for generating a target view image will be described in detail below in conjunction with FIG. 9.

[0170] FIG. 9 is an exemplary architecture diagram of a network for generating a target view image according to an embodiment of the present disclosure.

[0171] As shown in FIG. 9, in the embodiment 900, first, the target image 941_2 is input into the 3x3conv 901_2 for feature extraction, and the image features of the target image are output. The target depth maps 941_1, 941_3 of the adjacent video frames are input into the 3x3conv 901_1, 3x3conv 901_3 respectively, and the depth features corresponding to each of the adjacent video frames are output.

[0172] Then, the image features of the target image and the depth features corresponding to each of the adjacent video frames are spliced to obtain the feature map 942.

[0173] Next, the feature map 942 is divided to obtain a plurality of feature blocks 943. For example, for Patch1, first, 3 1x1 convs 902_1-902_3 are respectively input, and a query matrix Q 944_1, a key matrix K 944_2, and a value matrix V 944_3 corresponding to Patch1 are respectively output. Based on the multi-head attention mechanism, local features corresponding to Patch1 are obtained. In this way, local features corresponding to Patch1-PatchM can be obtained. Then, the M local features can be rearranged according to the division manner of the feature map 942, and then spliced to generate a global fusion feature. Next, the global fusion feature is input into the 3x3 conv 903, and the second difference information is output.

[0174] Next, the second difference information is merged with the target image 941_2, and an initial view point image 946 is output.

[0175] Finally, the foreground text is filled into the initial view point image 946 to generate a target view point image 947.

[0176] When filling the foreground text into the initial view point image 946, the position of the foreground text in the left view point image and the position of the foreground text in the right view point image are different. This difference is not only based on the disparity geometric relationship, but also includes the depth difference. Therefore, when determining the position of the foreground text in the right view point image, the depth information of the foreground text can also be considered.

[0177] According to an embodiment of the present disclosure, the depth information of the foreground text can be determined from the target depth map corresponding to the target frame based on the position; and the foreground text is filled into the initial view point image to generate the target view point image according to the depth information of the foreground text, the position, and the predetermined disparity relationship.

[0178] According to an embodiment of the present disclosure, the depth information of the foreground text can be the average depth information of all the characters corresponding to the foreground text region.

[0179] According to an embodiment of the present disclosure, the target position of the foreground text in the target view point image can be determined according to the depth information of the foreground text, the position, and the predetermined disparity relationship. Then, the foreground text is filled into the initial view point image to generate the target view point image based on the target position.

[0180] For example, the target disparity can be generated according to the depth information of the foreground text and the predetermined disparity relationship as formula (10):

[0181] Wherein, f represents the focal length of the virtual camera; B represents the baseline of the virtual camera; Z represents the depth information of the foreground text.

[0182] Then, the position (x L -d, y L ) of the foreground text in the target view image can be determined according to the position (x L , y L ) of the foreground text in the target frame and the target parallax.

[0183] According to the embodiment of the present disclosure, the foreground text is filled based on the depth information of the foreground text and the predetermined parallax relationship, which further improves the visibility and readability of the foreground text under different perspectives.

[0184] FIG. 10 is a flowchart of a video processing method according to another embodiment of the present disclosure.

[0185] As shown in FIG. 10, in this embodiment 1000, by performing foreground text recognition on a sequence of two-dimensional video frames to be processed 1010, a video frame including foreground text Vfa1011 and a video frame not including foreground text Vfb1012 are obtained.

[0186] For the video frame including foreground text Vfa1011: first, the depth map Dma1021 of this video frame can be generated by fusing the depth information of adjacent video frames. Then, the complete image Fra1031 can be generated by fusing the depth information of adjacent video frames and the image information of the non-foreground text region. Next, the right view image Rva1041 not including foreground text can be generated by fusing the depth information of adjacent video frames and the information of the complete image. Finally, the foreground text is filled into the right view image Rva1041 to generate the right view image Rva’ including foreground text.

[0187] For the video frame not including foreground text Vfb1012: first, the depth map Dma1022 of this video frame can be generated by fusing the depth information of adjacent video frames. Then, the right view image Rvb can be generated by fusing the depth information of adjacent video frames and the information of the initial image.

[0188] Finally, the initial image is taken as the left view image, and after splicing with the corresponding right view image, the three-dimensional video 1060 is output according to a predetermined format.

[0189] According to embodiments of this disclosure, when the initial image ratio is a predetermined ratio, for example, 16:9, the left and right viewpoint images can be horizontally stitched together to directly output a 3D video of 1060. When the initial image ratio is greater than the predetermined ratio, both the left and right viewpoint images need to be symmetrically filled with black on both sides to achieve the predetermined ratio before outputting the 3D video. When the initial image ratio is less than the predetermined ratio, both the left and right viewpoint images need to be symmetrically filled with black on both sides to achieve the predetermined ratio before outputting the 3D video.

[0190] Figure 11 is a block diagram of a video processing apparatus according to an embodiment of the present disclosure.

[0191] As shown in Figure 11, the device 1100 may include a determination module 1110, a depth estimation module 1120, a processing module 1130, and a generation module 1140.

[0192] The determination module 1110 is used to determine the target frame from the sequence of two-dimensional video frames to be processed; wherein the target frame includes foreground text.

[0193] The depth estimation module 1120 is used to perform depth estimation on the two-dimensional video frame sequence to be processed, and obtain a target depth map corresponding to each video frame. The depth information of the target depth map represents the fusion result of the depth information of the current video frame and the depth information of the adjacent video frames.

[0194] The processing module 1130 is used to process the target depth map and mask image corresponding to each of the adjacent video frames of the target frame in the two-dimensional video frame sequence to be processed, and generate a target image, wherein the target image includes the complete image information of the target frame; the mask image is obtained by masking the target frame based on the position of the foreground text in the target frame.

[0195] The generation module 1140 is used to process the target depth map and target image corresponding to each adjacent video frame based on the position and a predetermined parallax relationship, generate a target viewpoint image including foreground text, and output a three-dimensional video based on the target viewpoint image.

[0196] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the method described above.

[0197] According to an embodiment of this disclosure, a computer program product includes a computer program that, when executed by a processor, implements the method described above.

[0198] FIG. 12 shows a schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0199] As shown in FIG. 12, the device 1200 includes a computing unit 1201 that can perform various suitable actions and processes in accordance with computer programs stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from the storage unit 1208. Various programs and data required for operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other by a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0200] Various components in the device 1200 are connected to the I / O interface 1205, including an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a magnetic disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through computer networks, such as the Internet, and / or various telecommunication networks.

[0201] The computing unit 1201 can be various general purpose and / or special purpose processing components having processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs various methods and processes described above, such as the video processing methods. For example, in some embodiments, the video processing methods can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded onto the RAM 1203 and executed by the computing unit 1201, one or more steps of the video processing methods described above can be performed. Alternatively, in other embodiments, the computing unit 1201 can be configured to perform the video processing methods by any other suitable means, such as by means of firmware.

[0202] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0203] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0204] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0205] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0206] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0207] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0208] It should be understood that the various forms of flow shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure can be achieved, which is not limited herein.

[0209] The above detailed description does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing a video, comprising: determining a target frame from a sequence of two-dimensional video frames to be processed, wherein the target frame comprises foreground text; performing depth estimation on the sequence of two-dimensional video frames to be processed to obtain a target depth map corresponding to each video frame, wherein depth information of the target depth map represents a fusion result of depth information of a current video frame and depth information of a neighboring video frame; processing the target depth map corresponding to each neighboring video frame of the target frame and a mask image of the target frame to generate a target image, wherein the target image comprises complete image information of the target frame, the mask image is obtained by masking the target frame based on a position of the foreground text in the target frame, and the target image is generated based on the position and a predetermined parallax relationship; and processing the target depth map corresponding to each neighboring video frame of the target frame and the target image based on the position and the predetermined parallax relationship to generate a target view image comprising the foreground text, and outputting a three-dimensional video based on the target view image. The sequence of two-dimensional video frames to be processed comprises I video frames, and I is an integer greater than 1; and the method further comprises: performing foreground text recognition on the sequence of two-dimensional video frames to be processed to determine a foreground text region where the foreground text is located; and determining the i th video frame as the target frame in response to a difference between text content of a target region in a neighboring video frame of the i th video frame and the foreground text region in the i th video frame being less than a first predetermined threshold, wherein the target region has the same position in the neighboring video frame as the position of the foreground text region in the i th video frame, and i is an integer greater than 1 and less than or equal to I. 3.The method of claim 2, further comprising: obtaining positions of a plurality of characters in the foreground text region in the i th video frame; determining a character direction of the plurality of characters by fitting the positions of the plurality of characters; and determining the i th video frame as the target frame in response to the character direction being a predetermined direction. The method further comprises: detecting a character attribute of the plurality of characters in the foreground text region in the i th video frame; and determining the i th video frame as the target frame in response to the character attribute being a predetermined attribute. The method further comprises: determining the i th video frame as the target frame in response to a position of the foreground text region in the i th video frame belonging to a predetermined position range.

2. The method of claim 1, wherein, The method further comprises: performing speech recognition on audio corresponding to the i th video frame to obtain a speech recognition result; and determining the i th video frame as the target frame in response to a difference between the speech recognition result and text content in the foreground text region in the i th video frame being less than the first predetermined threshold. 7.The method of any one of claims 2-6, further comprising: performing binarization processing on the foreground text region based on the target depth map corresponding to the i th video frame to obtain first depth information corresponding to the text and second depth information corresponding to a background; and ​ ​ ​ ​ ​ 4. The method of claim 2 or 3, wherein, ​ ​ ​ 5. The method according to any one of claims 2-4, wherein, ​ ​ ​ 6. The method according to any one of claims 2-5, wherein, ​ ​ ​ ​ ​ ​ In response to a difference between the first depth information and the second depth information being greater than a second predetermined threshold, determining the ith video frame as the target frame.

8. The method of any one of claims 3-7, further comprising: determining respective weights of a plurality of target parameters; and determining the target frame by performing weighted processing on respective judgment results of the plurality of target parameters according to the weights; wherein the plurality of target parameters comprises at least two of the following: a text direction, a text attribute, a position of a foreground text region, a text content difference, and a depth difference between the text and the background.

9. The method of any one of claims 1-8, wherein, The depth estimation on the sequence of two-dimensional video frames to be processed to generate a target depth map corresponding to each video frame comprises: performing depth estimation on each video frame to obtain an initial depth map corresponding to each video frame; and processing, based on an attention mechanism, the initial depth map corresponding to the ith video frame, the initial depth map corresponding to the ith-1 video frame, and the initial depth map corresponding to the ith+1 video frame to generate a target depth map corresponding to the ith video frame; wherein i is an integer greater than 1 and less than or equal to I.

10. The method of claim 9, wherein, The depth estimation on each video frame to obtain an initial depth map corresponding to each video frame comprises: for each video frame, performing image compression on the video frame to obtain a predetermined resolution image; and processing, using a target network, image features of the predetermined resolution image to generate the initial depth map; wherein the target network is trained using depth information of each pixel in a predetermined resolution sample image as a label.

11. The method of claim 9 or 10, wherein, The processing, based on an attention mechanism, of the initial depth map corresponding to the ith video frame, the initial depth map corresponding to the ith-1 video frame, and the initial depth map corresponding to the ith+1 video frame to generate a target depth map corresponding to the ith video frame comprises: respectively extracting features from the initial depth map corresponding to the ith video frame, the initial depth map corresponding to the ith-1 video frame, and the initial depth map corresponding to the ith+1 video frame to generate respective first intermediate feature maps; and fusing, based on the attention mechanism, the respective first intermediate feature maps to generate the target depth map corresponding to the ith video frame.

12. The method of any one of claims 1-11, wherein, The processing of the target depth map corresponding to each of adjacent video frames of a target frame in the sequence of two-dimensional video frames to be processed and a mask image to generate a target image comprises: respectively extracting features from the target depth map corresponding to each of the adjacent video frames and the mask image to generate respective second intermediate feature maps; splicing the respective second intermediate feature maps to obtain a third intermediate feature map; processing, based on an attention mechanism, the third intermediate feature map to generate first difference information between the target image and the mask image; and processing, based on the first difference information, the mask image to generate the target image.

13. The method of claim 12, wherein, The processing, based on an attention mechanism, of the third intermediate feature map to generate first difference information between the target image and the mask image comprises: The third intermediate feature map is divided to obtain a plurality of feature blocks; The plurality of feature blocks are processed based on the attention mechanism to generate a plurality of local features corresponding to the plurality of pixel blocks respectively; The plurality of local features are spliced to obtain a global fusion feature; and The global fusion feature is processed to generate the first difference information.

14. The method of any one of claims 1-13, wherein, The target depth map corresponding to each of the adjacent video frames and the target image are processed based on the position and the predetermined parallax relationship to generate a target view image including the foreground text, including: The target depth map corresponding to each of the adjacent video frames and the target image are processed to generate an initial view image; wherein the initial view image has the same view direction as the target view image; and The foreground text is filled into the initial view image based on the position and the predetermined parallax relationship to generate the target view image.

15. The method of claim 14, wherein, The target depth map corresponding to each of the adjacent video frames and the target image are processed to generate an initial view image, including: The target depth map corresponding to each of the adjacent video frames and the target image are respectively feature extracted to generate a fourth intermediate feature map corresponding to each of the adjacent video frames and the target image; and The fourth intermediate feature map corresponding to each of the adjacent video frames and the target image are spliced to obtain a fifth intermediate feature map; The fifth intermediate feature map is processed based on the attention mechanism to generate second difference information between the initial view image and the target image; The target image is processed based on the second difference information to generate the initial view image.

16. The method of claim 14 or 15, wherein, The foreground text is filled into the initial view image based on the position and the predetermined parallax relationship to generate the target view image, including: The depth information of the foreground text is determined from the target depth map corresponding to the target frame based on the position; and The foreground text is filled into the initial view image according to the depth information of the foreground text, the position and the predetermined parallax relationship to generate the target view image.

17. The method of claim 16, wherein, The foreground text is filled into the initial view image according to the depth information of the foreground text, the position and the predetermined parallax relationship to generate the target view image, including: The target position of the foreground text in the target view image is determined according to the depth information of the foreground text, the position and the predetermined parallax relationship; and The foreground text is filled into the initial view image based on the target position to generate the target view image.

18. A video processing apparatus, comprising: a determination module configured to determine a target frame from a sequence of two-dimensional video frames to be processed; wherein the target frame includes foreground text; a depth estimation module configured to perform depth estimation on the sequence of two-dimensional video frames to be processed to obtain a target depth map corresponding to each video frame respectively, wherein the depth information of the target depth map represents the fusion result of the depth information of the current video frame and the depth information of the adjacent video frame; a determination module configured to determine a target frame from a sequence of two-dimensional video frames to be processed; wherein the target frame includes foreground text; a depth estimation module configured to perform depth estimation on the sequence of two-dimensional video frames to be processed to obtain a target depth map corresponding to each video frame respectively, wherein the depth information of the target depth map represents the fusion result of the depth information of the current video frame and the depth information of the adjacent video frame; a processing module, configured to process a target depth map and a mask image corresponding to each of adjacent video frames of a target frame in the to-be-processed two-dimensional video frame sequence, to generate a target image, wherein the target image comprises complete image information of the target frame; the mask image is obtained by masking the target frame based on a position of the foreground text in the target frame; and a generating module, configured to process the target depth map corresponding to each of the adjacent video frames and the target image based on the position and a predetermined parallax relationship, to generate a target viewpoint image comprising the foreground text, and output a three-dimensional video based on the target viewpoint image. 19.An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-17. The computer instructions are used to enable the computer to perform the method according to any one of claims 1-17.

20. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, 21.A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-17. ​

Images