Video image processing method, device, electronic device and storage medium

Abstract

The disclosed embodiments provide a video image processing method, device, electronic device and storage medium, the method comprising: determining a target depth view of each video frame in a target video, and determining a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view; for each target depth view, determining a target depth segmentation line corresponding to a current pixel point in the current target depth view, and determining a target pixel value of the current pixel point according to the pixel depth value of the current pixel point and the segmentation line depth value of the target depth segmentation line; determining a three-dimensional display video frame of each video frame in the target video according to the target pixel value of each pixel point in each target depth view. This technical solution realizes processing each pixel point in a video frame according to at least one predetermined depth segmentation line to obtain a three-dimensional display video frame of the corresponding video frame, thereby improving the technical effect of the convenience and universality of three-dimensional display.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a video image processing method, device, electronic device, and storage medium. Background Art

[0002] 3D special effects are currently a hot topic, and their purpose is to give users realistic 3D visual effects so that users have an immersive feeling when watching videos, thereby improving users' viewing experience.

[0003] At present, the realization of 3D special effects mainly relies on 3D special effects equipment, such as VR and AR glasses. This method can achieve better 3D visual effects, but it has problems of high cost and limited scenarios. For example, it cannot be realized on mobile terminals or PCs. Summary of the invention

[0004] The present disclosure provides a video image processing method, device, electronic device and storage medium to achieve the technical effect of displaying an image in three dimensions without the aid of a three-dimensional display device.

[0005] In a first aspect, an embodiment of the present disclosure provides a video image processing method, the method comprising:

[0006] Determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view;

[0007] For each target depth view, determine a target depth segmentation line corresponding to a current pixel in a current target depth view, and determine a target pixel value of the current pixel according to a pixel depth value of the current pixel and a segmentation line depth value of the target depth segmentation line;

[0008] According to the target pixel value of each pixel point in each target depth view, a three-dimensional display video frame of each video frame in the target video is determined.

[0009] In a second aspect, an embodiment of the present disclosure further provides a video image processing device, the device comprising:

[0010] A segmentation line determination module, configured to determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view;

[0011] a pixel value determination module, for determining, for each target depth view, a target depth segmentation line corresponding to a current pixel point in a current target depth view, and determining a target pixel value of the current pixel point according to a pixel depth value of the current pixel point and a segmentation line depth value of the target depth segmentation line;

[0012] The video display module is used to determine a three-dimensional display video frame of each video frame in the target video according to the target pixel value of each pixel point in each target depth view.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the image processing method as described in any of the embodiments of the present disclosure.

[0017] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium comprising computer executable instructions, wherein the computer executable instructions are used to execute any image processing method described in the embodiments of the present disclosure when executed by a computer processor.

[0018] The technical solution of the disclosed embodiment processes the target depth view of each video frame in the target video to obtain at least one depth segmentation line corresponding to the target video, and the depth segmentation line serves as the foreground-background segmentation line of each video frame in the target video. The target display information of each pixel in the video frame is determined based on the at least one depth segmentation line, and then the three-dimensional display video frame corresponding to the video frame is obtained based on the target display information. This solves the problem that when three-dimensional display is required in the prior art, a three-dimensional display device is required, and there is a high cost and poor universality of three-dimensional display. It realizes that the three-dimensional display video frame of the corresponding video frame can be obtained by processing each pixel in the video frame according to at least one predetermined depth segmentation line without the need for a three-dimensional display device, thereby improving the technical effect of the convenience and universality of three-dimensional display. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A schematic diagram of a video image processing method provided in the first embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of at least one depth segmentation line provided by an embodiment of the present disclosure;

[0022] Figure 3 A schematic diagram of a video image processing method provided in the second embodiment of the present disclosure;

[0023] Figure 4 The video frame and the depth map to be processed corresponding to the video frame provided by the embodiment of the present disclosure;

[0024] Figure 5 The video frame and the mask image to be processed corresponding to the video frame provided in the embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of a video image processing method provided in Embodiment 3 of the present disclosure;

[0026] Figure 7 A schematic diagram of a three-dimensional video frame provided by an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of the structure of a video image processing device provided in Embodiment 4 of the present disclosure;

[0028] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present disclosure. DETAILED DESCRIPTION

[0029] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0031] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0032] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] Embodiment 1

[0036] Figure 1 This is a flow chart of a video image processing method provided in the first embodiment of the present disclosure. The present embodiment of the present disclosure is applicable to any video display scene supported by the Internet, and is used to process each pixel of a video frame to obtain a three-dimensional display effect. The method can be executed by a video image processing device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC, or a server. The method provided in this embodiment can be executed by a server, a client, or a combination of a client and a server.

[0037] like Figure 1 As shown, the method includes:

[0038] S110: Determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view.

[0039] Among them, the video that needs to be processed currently is taken as the target video. The target video includes multiple video frames. The target depth view of each video frame can be determined separately. Usually, the video frame can be directly converted into a corresponding depth view, but at this time the depth information in the depth view is mostly an absolute depth information. In order to match the depth of the front and rear video frames, the depth view of the entire video frame after the absolute depth information is aligned can be used as the target depth view. The depth segmentation line can be a foreground and background segmentation line, that is, a segmentation line used to distinguish between the foreground and the background. The number of at least one depth segmentation line can be one or more. The user can mark multiple depth segmentation lines according to actual needs, and then determine the depth values ​​of the multiple depth segmentation lines according to the target depth view of each video frame. The depth value of the depth segmentation line is used as the segmentation line depth value.

[0040] Specifically, each video frame in the target video is extracted and processed to obtain a target depth view of each video frame. According to the depth view of each video frame, the pre-marked depth segmentation line can be processed to determine the segmentation line depth value of each depth segmentation line.

[0041] In this embodiment, before determining the target depth view of each video frame in the target video, it also includes: receiving the target video; setting at least one depth segmentation line corresponding to the target video, and determining the position and width of the at least one depth segmentation line to determine the target pixel value of the corresponding pixel point according to the position and width of the depth segmentation line.

[0042] Usually, the computing power of the server is much greater than that of the display device. For example, the display device can be a mobile terminal, PC, etc. Therefore, the target depth view of each video frame in the target video can be determined based on the server, and then the three-dimensional display video frame corresponding to each video frame can be determined. The three-dimensional display video frame can also be understood as a three-dimensional special effect video frame. That is, the target video is sent to the server, and the server processes each video frame to obtain a three-dimensional display video corresponding to the target video. At least one line is marked on the target video based on the marking tool. The line may be perpendicular to the edge line of the display device or not. The position can be understood as the position of the depth line in the target video. For example, see Figure 2 In mark 1, the position of the depth segmentation line is the position of the edge of the video, the width of the depth segmentation line matches the display parameters of the target video, and optionally, the width of the segmentation line is one twentieth of the length of the long side.

[0043] Specifically, the video to be processed by the server is taken as the target video. The depth line of the target video is marked as required to obtain at least one depth segmentation line. At the same time, when marking the depth segmentation line, the width and position of the depth segmentation line can be determined according to the display information of the target video. If there is one depth segmentation line, the depth segmentation line can be marked at any position; if there are two depth segmentation lines, in order to adapt to the user's viewing habits, it can be a position perpendicular to the long side of the display parameter when the video is playing normally. The width of the depth segmentation line is usually one twentieth of the width of the long side. Of course, the depth segmentation line can also be a segmentation line inside the target video that surrounds the target video, see Figure 2 Mark 2 in the annular depth dividing line.

[0044] It should be noted that Mark 1 and Mark 2 are used selectively, and the same target video usually does not correspond to both Mark 1 and Mark 2.

[0045] In this embodiment, determining the position and width of the at least one depth segmentation line includes: determining the position and width of the at least one depth segmentation line in the target video according to display parameters of the target video.

[0046] The display parameters are the display length and display width of the target video when it is displayed on the display interface. The position can be the relative position of the depth dividing line from the edge line of the target video. The width can be the width value of the depth dividing line corresponding to the display length in the target display video.

[0047] In this embodiment, the advantage of setting at least one depth segmentation line is that the segmentation line depth value of the at least one depth segmentation line can be determined, and then the target pixel value of each target video frame is determined according to the segmentation line depth value, thereby obtaining the technical effect of three-dimensional display of video frames.

[0048] S120 . For each target depth view, determine a target depth segmentation line corresponding to a current pixel in a current target depth view, and determine a target pixel value of the current pixel according to a pixel depth value of the current pixel and a segmentation line depth value of the target depth segmentation line.

[0049] Among them, the target video includes multiple video frames, and each video frame has a target depth view. Each pixel in the target depth view can be processed, and the pixel to be processed or being processed is taken as the current pixel. The depth segmentation line corresponding to the target video includes at least one, and the depth segmentation line where the current pixel is located can be taken as the target depth segmentation line. The pixel depth value can be the depth value of the current pixel in the target depth view. The segmentation line depth value can be the depth value of the predetermined segmentation line. The segmentation line depth value is used to determine the pixel value of the pixel in the video frame, thereby determining the corresponding three-dimensional display effect. The depth segmentation line has a certain width, and the corresponding segmentation line includes multiple pixels, and the depth values ​​of the multiple pixels are the same. According to the relationship between the pixel depth value and the segmentation line depth value, the target pixel value of the current pixel can be determined.

[0050] Specifically, for each target depth view of a video frame, the depth segmentation line corresponding to each pixel in the current video frame can be determined, and the corresponding relationship between the depth value of the current pixel and the depth value of the corresponding segmentation line can be determined. According to the corresponding relationship, the target pixel value of each pixel can be determined.

[0051] S130 . Determine a 3D display video frame of each video frame in the target video according to a target pixel value of each pixel point in each target depth view.

[0052] The target pixel value may be an RGB value of a pixel point, or may be a pixel value adjusted for a corresponding pixel point of a video frame. A three-dimensional display video frame may be a video frame that appears to a user as a three-dimensional special effect.

[0053] Specifically, the target pixel value of each pixel is obtained according to the pixel value of each pixel in each video frame re-determined in S120. The 3D display video frame corresponding to each video frame is determined based on the target pixel value of each pixel in each video frame. The target 3D video corresponding to the target video can be further determined according to each 3D display video frame.

[0054] It should be noted that the target three-dimensional video is a video that appears to be displayed in three dimensions from a visual perspective.

[0055] The technical solution of the disclosed embodiment processes the target depth view of each video frame in the target video to obtain at least one depth segmentation line corresponding to the target video, and the depth segmentation line serves as the foreground-background segmentation line of each video frame in the target video. The target display information of each pixel in the video frame is determined based on the at least one depth segmentation line, and then the three-dimensional display video frame corresponding to the video frame is obtained based on the target display information. This solves the problem that when three-dimensional display is required in the prior art, a three-dimensional display device is required, and there is a high cost and poor universality of three-dimensional display. It realizes that the three-dimensional display video frame of the corresponding video frame can be obtained by processing each pixel in the video frame according to at least one predetermined depth segmentation line without the need for a three-dimensional display device, thereby improving the technical effect of the convenience and universality of three-dimensional display.

[0056] Embodiment 2

[0057] Figure 3 This is a flow chart of a video image processing method provided in the second embodiment of the present disclosure. Based on the above embodiment, the depth view of each target in the target video frame can be further refined. The specific implementation method can refer to the detailed description of the technical solution. Among them, the technical terms that are the same or corresponding to the above embodiment are not repeated here.

[0058] like Figure 3 As shown, the method includes:

[0059] S210: Determine a depth view to be processed and feature points to be processed of each video frame.

[0060] The depth view directly converted from the video frame is used as the depth view to be processed. The feature points to be processed may be feature points of very stable local features in the video frame.

[0061] Specifically, a corresponding depth view processing algorithm may be used to convert each video frame into a depth view to be processed. Meanwhile, a feature point acquisition algorithm may be used to determine feature points to be processed in each video frame.

[0062] In this embodiment, determining the feature points to be processed of the depth view to be processed of each video frame includes: performing depth estimation on each video frame to obtain the depth view to be processed of each video frame; processing each video frame based on a feature point detection algorithm to determine the feature points to be processed of each video frame.

[0063] The depth value of each pixel in the depth view to be processed represents the distance value of each pixel in the video frame relative to the camera imaging plane. In other words, the depth view to be processed is a schematic diagram formed according to the distance of each pixel from the camera device plane. Optionally, objects farther away from the camera device are represented by darker colors. Figure 4 , the video frame can be Figure 4 (a), the depth view corresponding to the video frame is 4(b), that is, Figure 4 (b) is a simple depth view, in which the depth of the color represents the distance from the camera device. Dark colors indicate that the distance is farther from the camera, and light colors indicate that the distance is closer to the camera.

[0064] While determining the depth view to be processed, the feature points to be processed in each video frame can be determined. Optionally, a SIFI (scale invariant feature transform) feature point detection algorithm can be used to determine the feature points to be processed in each video frame. The SIFI feature point detection algorithm has the characteristic of remaining unchanged by rotation, scaling, brightness changes, etc. It is a very stable local feature and can be used as a unique representation of a smaller local area in a video frame. It can be understood that a feature point detection algorithm can be used to determine the feature points to be processed in a video frame.

[0065] S220 , sequentially process the feature points to be processed of two adjacent video frames to obtain a set of 3D feature point pairs of the two adjacent video frames.

[0066] Among them, the 3D feature point pair set includes multiple groups of feature point pairs. Each group of feature point pairs includes two feature points, and these two feature points are obtained after processing two adjacent video frames. For example, video frame 1 and video frame 2 are adjacent video frames, and the feature points to be processed in video frame 1 and video frame 2 can be determined respectively, and it is further determined which feature points to be processed in video frame 1 correspond to which feature points to be processed in video frame 2, and the two corresponding feature points are taken as a feature point pair. It should be noted that the feature point pair at this time may be a two-dimensional feature point pair, and the two-dimensional feature point pair can be further processed to obtain a 3D feature point pair.

[0067] Optionally, the processing of the feature points to be processed of two adjacent video frames in sequence to obtain a set of 3D feature point pairs of the two adjacent video frames includes: matching the feature points of the two adjacent video frames in sequence based on a feature point matching algorithm to obtain at least one group of 2D feature point pairs associated with the two adjacent video frames; reconstructing a 3D point cloud corresponding to the depth view to be processed of the two adjacent video frames, and obtaining an original 3D point cloud corresponding to the depth view to be processed, and at least one group of 3D feature point pairs corresponding to the at least one group of 2D feature point pairs; and determining the set of 3D feature point pairs of the two adjacent video frames based on the at least one group of 3D feature point pairs.

[0068] Among them, the feature point pairs that match in two adjacent video frames can be used as 2D feature point pairs. The number of 2D feature point pairs can be 8 groups. The number of at least one group of 2D feature point pairs is consistent with the number of feature point pairs matched in two adjacent video frames. If the number of feature point pairs determined based on two adjacent video frames is less than the preset number threshold, it may be that the video has transitioned, in which case no registration processing is required. 3D point cloud reconstruction can be performed on the depth view to be processed of the video frame to determine the 3D feature point pairs corresponding to the 2D feature point pairs in the two adjacent video frames. The number of groups of 2D feature point pairs in two adjacent video frames includes at least one, and accordingly, the number of groups of 3D feature point pairs also includes at least one group. At least one group of 3D feature point pairs can be used as a point pair in a 3D feature point pair set. Preferably, the number of at least one group of 3D feature point pairs is 8 groups.

[0069] Exemplarily, two adjacent video frames are represented by t and t-1. The feature points in the t frame and the t-1 frame can be determined based on the feature point processing algorithm, and based on the feature point matching algorithm, a one-to-one corresponding feature point pair (2D feature point pair) can be obtained. A feature point pair is a different projection of the same point on the t frame and the t-1 frame. The t frame and the t-1 frame are reconstructed into a 3D point cloud using the depth view to be processed, and the corresponding positions of the 2D feature point pairs in the 3D point cloud are determined to obtain 3D feature point pairs. It can be understood that the number of 3D feature point pairs is consistent with the number of 2D feature point pairs.

[0070] S230: Determine camera motion parameters of two adjacent video frames according to the multiple groups of 3D feature point pairs in the 3D feature point pair set, and use the camera motion parameters as camera motion parameters of a previous video frame of the two adjacent video frames.

[0071] Among them, the camera motion parameters can be solved by processing multiple groups of 3D feature point pairs in the 3D feature point pair set based on the camera motion parameter determination algorithm. The camera motion parameters include a rotation matrix and a displacement matrix. The rotation matrix and the displacement matrix represent the movement information of the camera in space when shooting two adjacent video frames. According to the camera motion parameters, it can be determined to process the point cloud other than the 3D feature point pairs in the two adjacent video frames. The obtained camera motion parameters can be used as the motion parameters of the previous video frame in the adjacent video frames.

[0072] Specifically, the RANSAC algorithm can be used to process the 3D feature point cloud in the set of 3D feature point pairs of two adjacent video frames, and the rotation matrix R and translation matrix T of the two adjacent video frames can be solved, and the rotation matrix and translation matrix can be used as the camera motion parameters of the previous video frame in the two adjacent video frames.

[0073] S240 : Determine a target depth view for each video frame according to the to-be-processed depth view of each video frame and corresponding camera motion parameters.

[0074] In this embodiment, the target depth view of each video frame is determined according to the depth view to be processed of each video frame and the corresponding camera motion parameters, including: for each depth view to be processed, according to the original 3D point cloud, rotation matrix and translation matrix of the current depth view to be processed, obtaining the 3D point cloud to be used of the current depth view to be processed; based on the original 3D point cloud of each depth view to be processed, the 3D point cloud to be used and the preset depth adjustment coefficient, obtaining the target depth view corresponding to each video frame.

[0075] Among them, the 3D point cloud directly reconstructed based on the depth view to be processed can be used as the original 3D point cloud. After the original 3D point cloud is processed by the rotation matrix and the translation matrix, the 3D point cloud obtained is used as the 3D point cloud to be used. In other words, the original 3D point cloud is an uncorrected point cloud, and the original 3D point cloud to be used is a point cloud after the camera parameters are corrected. The preset depth adjustment coefficient can be understood as a parameter adjustment coefficient, which is a coefficient used to re-process the original 3D point cloud and the 3D point cloud to be used. The point cloud processed by the preset depth adjustment coefficient is more consistent with the video frame.

[0076] Exemplarily, the rotation matrix in the camera motion parameters is represented by R, the translation matrix is ​​represented by T, and the depth value in the depth view to be processed in the corrected t frame can be:

[0077] P=P'*R+T

[0078] P"=P'*(1-a)+P*a

[0079] D=P”[:,:,2]

[0080] Wherein, P' is the point cloud of the t video frame before correction, P" is the point cloud of the t video frame after correction, P is an intermediate value, D is the depth of the 3D point cloud of the t video frame after correction, and a is a preset depth adjustment coefficient.

[0081] It can be understood that by processing the 3D point cloud of the video frame based on the camera motion parameters, the relative depth value between two adjacent video frames can be obtained, which solves the problem that the depth value in the depth view to be processed is an absolute value, resulting in inaccurate image registration, and realizes the alignment of the depth value of each pixel in the video frame, thereby obtaining a video frame with a relative depth value, which provides a reliability guarantee for the subsequent determination of the depth segmentation line.

[0082] Specifically, after performing depth registration on each video frame and obtaining the depth value of each pixel in each video frame, the depth view to be processed can be updated according to the depth value of each pixel, thereby obtaining a target depth view corresponding to each video frame, which is the view obtained after depth registration.

[0083] S250: Determine, according to each target depth view, a segmentation line depth value of at least one depth segmentation line corresponding to the target video.

[0084] In this embodiment, before determining the segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view, it also includes: determining a salient object in each video frame, and determining a mask map to be processed of the corresponding video frame based on the salient object, so as to determine the segmentation line depth value based on the mask map to be processed of each video frame and the target depth view.

[0085] Among them, the concept of salient objects comes from the study of users' visual systems. The object that first catches the user's eye in each video frame can be regarded as a salient object, that is, the object in the picture that is easy to pay attention to at first glance can be regarded as a salient object. Salient objects usually have the characteristics of being in the center of the picture, clear pixels, and applicable depth. A neural network for salient object segmentation can be pre-trained, and then the salient objects in each video frame are determined based on the neural network. After determining the salient object, the pixel points corresponding to the salient object can be set to a first preset pixel value, and the pixel points other than the salient object in the video frame can be set to a second preset pixel value. Exemplarily, the first preset pixel value can be 255, and the second preset pixel value can be 0. After determining the salient object in the current video frame based on the neural network, the color of the pixel points corresponding to the salient object can be set to white, and the color of the pixel points corresponding to the non-salient object can be set to black. The image obtained at this time is used as the mask image to be processed, see Figure 5 ,in, Figure 5 (a) in the figure represents a video frame. Figure 5(b) in the figure represents the mask image to be processed, and region 1 identifies the mask region of the salient object in the video frame. The mask image to be processed refers to the image obtained by setting the pixels of the salient region in the video frame to white and the pixels of the non-salient object to black. The depth segmentation line can be understood as the foreground-background segmentation line. The segmentation line depth value is used to determine the depth value of the corresponding pixel.

[0086] Specifically, each video frame in the target video is input into a pre-trained salient object segmentation model to obtain the salient objects in each video frame. The pixels corresponding to the salient objects are set to white, and the pixels other than the salient objects are set to black, so as to obtain a black and white schematic diagram including the outline of the salient objects, and the black and white schematic diagram is used as the mask image to be processed. Furthermore, the segmentation line depth value of at least one segmentation line can be determined by the target depth view of each video frame and the corresponding mask image to be processed.

[0087] In this embodiment, the method of determining the segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view includes: for each video frame, determining the average depth value of the mask area in the mask image to be processed according to the mask image to be processed of the current video frame and the target depth view; determining the segmentation line depth value of the at least one depth segmentation line according to the average depth value of each video frame and a preset segmentation line adjustment coefficient.

[0088] It should be noted that the 3D video perception of 2D video is mainly based on the user's visual illusion. Therefore, the depth values ​​of at least two depth segmentation lines can be predetermined, and then the pixel values ​​of corresponding pixels in the video frame can be adjusted according to the depth values, thereby achieving a 3D display effect.

[0089] The area corresponding to the salient object in the mask image to be processed is used as the mask area, that is, the white area in the mask image to be processed is the mask area. The average depth value is the value obtained after processing the depth values ​​of all pixels in the mask area. The preset segmentation line adjustment coefficient can be a coefficient pre-set based on experience, which can adjust the depth values ​​of at least two depth segmentation lines, thereby determining the pixel values ​​of corresponding pixels to achieve a 3D display effect.

[0090] Generally, if a salient object is highlighted on a display device, it is often understood as a three-dimensional special effect display, so the pixels on at least one depth segmentation line may be analyzed and processed to determine the target pixel value of the corresponding pixel.

[0091] Specifically, in order to clearly introduce how to determine the average depth value, we can take the determination of the average depth value of one of the video frames as an example. Obtain the mask image to be processed and the target depth view of the current video frame, and determine the depth values ​​corresponding to the pixels in the mask area of ​​the mask image to be processed in the target depth view. By summing the depth values ​​in the mask area, the total depth value of the mask area is obtained. At the same time, the depth values ​​of all pixels in the target depth view are summed to obtain the total depth value. Calculate the ratio of the total depth value and the total depth value of the mask area to obtain the average depth value of the current video frame. Based on this method, the average depth value of each video frame can be determined. After obtaining the average depth value of each video frame, the average depth value is processed according to the preset dividing line adjustment coefficient to obtain the dividing line depth value of at least one dividing line.

[0092] In this embodiment, the average depth value of each video frame can be determined by: if there is a mask image to be processed corresponding to the current video frame, then the depth value to be processed of each pixel to be processed in the mask area in the target depth view is determined; according to the depth value to be displayed of each pixel to be displayed in the target depth view and each depth value to be processed, the average depth value of the mask area is determined. Correspondingly, if there is no mask image to be processed corresponding to the current video frame, the average depth value of the current video frame is determined according to the average depth values ​​of the recorded video frames.

[0093] It can be understood that if the current video frame includes a salient object, then there is a mask image to be processed corresponding to the salient object, and the average depth value corresponding to the mask area can be determined in the above manner. If the current video frame does not include a salient object, the average depth value of the current video frame does not need to be calculated based on the target depth view and the mask image to be processed. The average depth values ​​of all video frames can be recorded, and the maximum depth value among all the average depth values ​​is used as the average depth value of the current video frame.

[0094] On the basis of the above technical solution, after determining the average depth value of each video frame, it also includes: determining the maximum and minimum values ​​of the average depth value according to the average depth value corresponding to each video frame; determining the dividing line depth value of the at least one depth dividing line according to the minimum value, the dividing line adjustment coefficient and the maximum value.

[0095] If the number of video frames includes N, then the average depth value of each video frame in the target video frame can be represented by a 1×N-order vector, and the value in the vector represents the average depth value corresponding to the video frame. The segmentation line adjustment coefficient is used to determine the final depth value of the depth segmentation line, and the final determined depth value is used as the segmentation line depth value.

[0096] Specifically, according to the average depth value in each video frame, the maximum depth value and the minimum depth value are selected therefrom. At the same time, according to the pre-set segmentation line adjustment coefficient, the segmentation line depth value of the depth segmentation line can be determined. At this time, the segmentation line depth value is a specific reference, that is, the obtained segmentation line depth value is relatively accurate.

[0097] On the basis of the above technical solution, the number of depth segmentation lines includes two, then the segmentation line depth values ​​of the two depth segmentation lines can be determined as follows: at least one depth segmentation line includes a first depth segmentation line and a second depth segmentation line, and according to the minimum value, the first segmentation line adjustment coefficient, the second segmentation line adjustment coefficient and the maximum value, the first segmentation line depth value of the first depth segmentation line and the second segmentation line depth value of the second depth segmentation line are determined.

[0098] For example, it is assumed that the mask image to be processed corresponding to each video frame is represented as: i |i=1,2,...,N}, the target depth view is represented as: {d i |i=1,2,...,N}; where i is the number of video frames in the target video frame, that is, there are N video frames in total. If there is a significant object in the video frame, the depth value of the mask area can be determined by using the following formula if∑mask i = > 0 corresponding to the function expression ∑d i / ∑mask i , where mask i Used to characterize the depth value corresponding to the pixel point of the salient object in the target depth view. If there is no salient object in the video frame, the depth value of the mask area can be determined by using the function expression max_depth corresponding to else in the following formula. That is, the maximum depth value of the mask area in each video frame. In this way, the depth value of the mask area of ​​each video frame can be obtained, which is the depth value of the salient object. If the first dividing line adjustment coefficient and the second dividing line adjustment coefficient are α1 and α2 respectively. Determining the depth value of the first dividing line and the second dividing line depth value can be: ref_depth1=d min +α1*(d max -d min );ref_depth2=d min +α2*(d max -d min ), where ref_depth1 represents the first segmentation line depth value and ref_depth2 represents the second segmentation line depth value, d max represents the maximum average depth value in all video frames, d minIndicates the minimum average depth value in all video frames. Using this calculation formula, the depth values ​​of the two segmentation lines can be determined. If the depth segmentation lines are distributed left and right, the depth value of the first segmentation line can usually be used as the depth value of the left segmentation line, and the depth value of the second segmentation line can be used as the depth value of the right segmentation line.

[0099] It should be noted that the above method of determining the depth value of the segmentation line can calculate the depth value of the segmentation line based on the dynamic change process of the depth of the salient object in the entire video, while taking into account the abnormal processing when there is no salient object in the video, and has stronger robustness. The values ​​of α1 and α2 can be set to 0.3 and 0.7 respectively.

[0100] S260. For each target depth view, determine a target depth segmentation line corresponding to a current pixel in a current target depth view, and determine a target pixel value of the current pixel according to a pixel depth value of the current pixel and a segmentation line depth value of the target depth segmentation line.

[0101] S270 . Determine a 3D display video frame of each video frame in the target video according to a target pixel value of each pixel point in each target depth view.

[0102] The technical solution of the embodiment of the present disclosure can determine the depth view to be processed of each video frame and the 3D feature point pairs of two adjacent video frames after determining the width and position of the depth segmentation line. The camera motion parameters between two adjacent video frames can be determined based on the 3D feature point pairs. The 3D point cloud and the corresponding depth value corresponding to each video frame can be determined based on the camera motion parameters, that is, the relative depth view corresponding to each video frame is obtained after aligning the depth view to be processed, that is, the target depth view. According to the target depth view and the mask map to be processed of each video frame, the average depth value of the salient object area of ​​each video frame can be determined, and the segmentation line depth value of the target video is obtained based on the average depth value. This method solves the problem that the prior art needs to rely on 3D display equipment for three-dimensional display, which has a high cost, and realizes the technical effect of determining the target pixel value of the corresponding pixel point based on the depth value of each pixel point of each video frame and the segmentation line depth value, and then realizing the three-dimensional display based on the target pixel value.

[0103] Embodiment 3

[0104] Figure 6 This is a flow chart of a video image processing method provided in the third embodiment of the present disclosure. Based on the above embodiments, the determination of the target depth segmentation line and the determination of the target display information can be further refined. The specific implementation methods can refer to the detailed description of this technical solution. The technical terms that are the same or corresponding to the above embodiments are not repeated here.

[0105] like Figure 6 As shown, the method includes:

[0106] S310: Determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view.

[0107] S320: according to the position information of the current pixel, the position and width of the at least one depth segmentation line, use the depth segmentation line including the current pixel as the target depth segmentation line.

[0108] The position information may be the horizontal and vertical coordinates of the pixel in the image. The position of the depth segmentation line may be the position information of the depth segmentation line in the target video. The width may be the width of the screen occupied by the depth segmentation line, that is, there are multiple pixels within the position and width.

[0109] Specifically, if the number of depth segmentation lines includes one, it can be determined whether the pixel is on the segmentation line based on the position information of the current pixel, and if so, the depth segmentation line is used as the target depth segmentation line. If the number of depth segmentation lines includes two, it is determined on which depth segmentation line the current pixel is located based on the position information of the current pixel and the position and width of each depth segmentation line, and the depth segmentation line on which the current pixel is located is used as the target depth segmentation line.

[0110] S330 , determining a target pixel value of the current pixel according to the pixel depth value of the current pixel, the segmentation line depth value, and the mask image to be processed of the video frame to which the current pixel belongs.

[0111] Optionally, if the pixel depth value of the current pixel is less than the segmentation line depth value, and the current pixel is located in the mask area in the mask image to be processed, the original pixel value of the current pixel is kept unchanged, and the original pixel value is used as the target pixel value; if the pixel depth value of the current pixel is greater than the segmentation line depth value, and the current pixel is located in the mask area in the mask image to be processed, the original pixel value of the current pixel is adjusted to a first preset pixel value, and the first preset pixel value is used as the target pixel value of the current pixel.

[0112] The depth value of the current pixel can be determined according to the target depth view of each video frame, and the depth value is used as the pixel depth value. The original pixel value is the pixel value of the pixel when each video frame is collected.

[0113] Specifically, if the pixel depth value of the current pixel is less than the segmentation line depth value of the target depth segmentation line, it is determined whether the current pixel is a pixel on a significant object. If it is also a pixel on a significant object, it means that the current pixel needs to be highlighted to obtain a corresponding three-dimensional display effect. The pixel value of the current pixel can be kept unchanged. If the pixel depth value of the current pixel is greater than the target depth segmentation line, it means that the pixel is far away from the camera device. At the same time, it can be determined that the pixel is located in the mask area, then the original pixel value of the current pixel can be set to the first preset pixel value, optionally, set to 0, or set to 255.

[0114] Optionally, if there is no target depth segmentation line corresponding to the current pixel point, the original pixel value of the current pixel point is used as the target pixel value.

[0115] It should also be noted that if the current pixel is not on the depth segmentation line, the original pixel value of the pixel is kept unchanged.

[0116] S340: Determine a 3D display video frame of each video frame in the target video according to a target pixel value of each pixel point in each target depth view.

[0117] Specifically, according to the depth value of each pixel in the target depth view and the depth value of the corresponding segmentation line, the target pixel value of each pixel in the video frame can be determined, and the 3D display video frame of the target video frame can be obtained based on the target pixel value of each pixel. The effect of the 3D display video frame of a certain video frame can be seen in Figure 7 Of course, the depth dividing line can be removed in actual display. This is just a schematic diagram and does not limit the technical solution accordingly. Figure 7 It can be seen that the video frame corresponds to a three-dimensional display effect.

[0118] The technical solution of the disclosed embodiment achieves the technical effect of three-dimensional display by determining the depth value of each pixel point and the corresponding depth dividing line in the target video, determining the target pixel value of each pixel point, and further displaying each pixel point according to the target pixel value, thereby solving the technical problem in the prior art that a three-dimensional display device must be used to achieve three-dimensional display.

[0119] Embodiment 4

[0120] Figure 8 A schematic diagram of the structure of a video image processing device provided in the fourth embodiment of the present disclosure is shown in FIG. Figure 8 As shown, the device includes: a segmentation line determination module 410 , a pixel value determination module 420 and a video display module 430 .

[0121] Among them, the segmentation line determination module 410 is used to determine the target depth view of each video frame in the target video, and determine the segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view; the pixel value determination module 420 is used to determine the target depth segmentation line corresponding to the current pixel point in the current target depth view for each target depth view, and determine the target pixel value of the current pixel point according to the pixel depth value of the current pixel point and the segmentation line depth value of the target depth segmentation line; the video display module 430 is used to determine the three-dimensional display video frame of each video frame in the target video according to the target pixel value of each pixel point in each target depth view.

[0122] On the basis of the above technical solutions, the device further includes:

[0123] A video receiving module, used for receiving the target video;

[0124] A dividing line setting module is used to set at least one depth dividing line corresponding to the target video, and determine the position and width of the at least one depth dividing line in the target video according to the display parameters of the target video, so as to determine the target pixel value of the corresponding pixel point according to the depth value corresponding to the position and width of the depth dividing line; wherein the display parameters are the display length and display width of the target video when it is displayed on the display interface.

[0125] On the basis of the above-mentioned technical solutions, the segmentation line determination module includes: a first information processing unit, which is used to determine the depth view to be processed and the feature points to be processed of each video frame; a feature point pair determination unit, which is used to process the feature points to be processed of two adjacent video frames in turn to obtain a set of 3D feature point pairs of two adjacent video frames; wherein the 3D feature point pair set includes multiple groups of 3D feature point pairs; a motion parameter determination unit, which is used to determine the camera motion parameters of two adjacent video frames according to the multiple groups of 3D feature point pairs in the 3D feature point pair set, and use the camera motion parameters as the camera motion parameters of the previous video frame in the two adjacent video frames; wherein the camera motion parameters include a rotation matrix and a displacement matrix; a depth view determination unit, which is used to determine the target depth view of each video frame according to the depth view to be processed of each video frame and the corresponding camera motion parameters.

[0126] On the basis of the above-mentioned technical solutions, the first information processing unit is also used to perform depth estimation on each video frame to obtain a depth view to be processed for each video frame; and process each video frame based on a feature point detection algorithm to determine the feature points to be processed for each video frame.

[0127] On the basis of the above-mentioned technical solutions, the feature point pair determination unit is further used to match the feature points to be processed of two adjacent video frames in sequence based on the feature point matching algorithm to obtain at least one group of 2D feature point pairs associated with the two adjacent video frames; by reconstructing the 3D point cloud of the depth view to be processed of the two adjacent video frames, an original 3D point cloud corresponding to the depth view to be processed and at least one group of 3D feature point pairs corresponding to the at least one group of 2D feature point pairs are obtained; and a set of 3D feature point pairs of the two adjacent video frames is determined based on the at least one group of 3D feature point pairs.

[0128] Based on the above technical solutions, the depth view determination unit is further used to:

[0129] For each depth view to be processed, the 3D point cloud to be used of the current depth view to be processed is obtained according to the original 3D point cloud, rotation matrix and translation matrix of the current depth view to be processed; based on the original 3D point cloud of each depth view to be processed, the 3D point cloud to be used and the preset depth adjustment coefficient, the target depth view corresponding to each video frame is obtained.

[0130] Based on the above technical solutions, the device also includes: a mask image determination module, which is used to determine the salient objects in each video frame, and determine the mask image to be processed of the corresponding video frame based on the salient objects, so as to determine the segmentation line depth value based on the mask image to be processed of each video frame and the target depth view.

[0131] On the basis of the above-mentioned technical solutions, the segmentation line determination module is used to determine, for each video frame, the average depth value of the mask area in the mask image to be processed according to the mask image to be processed and the target depth view of the current video frame; and determine the segmentation line depth value of the at least one depth segmentation line according to the average depth value of each video frame and the preset segmentation line adjustment coefficient.

[0132] On the basis of the above-mentioned technical solutions, the segmentation line determination module is used to determine the to-be-processed depth value of each to-be-processed pixel in the mask area in the target depth view if there is a to-be-processed mask image corresponding to the current video frame; determine the average depth value of the mask area according to the to-be-displayed depth value of each to-be-displayed pixel in the target depth view and each to-be-processed depth value; or, if there is no to-be-processed mask image corresponding to the current video frame, determine the average depth value of the current video frame according to the average depth value of each recorded video frame.

[0133] Based on the above technical solutions, the segmentation line determination module is used to determine the maximum and minimum values ​​of the average depth value according to the average depth value of each video frame; and determine the segmentation line depth value of the at least one depth segmentation line according to the minimum value, the segmentation line adjustment coefficient and the maximum value.

[0134] On the basis of the above-mentioned technical solutions, the at least one depth segmentation line includes a first depth segmentation line and a second depth segmentation line, the preset segmentation line adjustment coefficient includes a first segmentation line adjustment coefficient and a second segmentation line adjustment coefficient, the segmentation line determination module, the segmentation line depth value determination module, is also used to: determine the first segmentation line depth value of the first depth segmentation line, and the second segmentation line depth value of the second depth segmentation line according to the minimum value, the first segmentation line adjustment coefficient, the second segmentation line adjustment coefficient and the maximum value.

[0135] Based on the above technical solutions, the pixel value determination module is used to determine whether the current pixel is located on at least one depth segmentation line according to the position information of the current pixel, the position and width of the at least one depth segmentation line; if so, the depth segmentation line including the current pixel is used as the target depth segmentation line.

[0136] Based on the above technical solutions, the pixel value determination module is used to determine the target pixel value of the current pixel according to the pixel depth value of the current pixel, the segmentation line depth value, and the mask image to be processed of the video frame to which the current pixel belongs.

[0137] On the basis of the above-mentioned technical solutions, the pixel value determination module is used to keep the original pixel value of the current pixel unchanged and use the original pixel value as the target pixel value if the pixel depth value of the current pixel is less than the segmentation line depth value and the current pixel is located in the mask area in the mask image to be processed; if the pixel depth value of the current pixel is greater than the segmentation line depth value and the current pixel is located in the mask area in the mask image to be processed, adjust the original pixel value of the current pixel to a first preset pixel value and use the first preset pixel value as the target pixel value of the current pixel.

[0138] The technical solution of the disclosed embodiment processes the target depth view of each video frame in the target video to obtain at least one depth segmentation line corresponding to the target video, and the depth segmentation line serves as the foreground-background segmentation line of each video frame in the target video. The target display information of each pixel in the video frame is determined based on the at least one depth segmentation line, and then the three-dimensional display video frame corresponding to the video frame is obtained based on the target display information. This solves the problem that when three-dimensional display is required in the prior art, a three-dimensional display device is required, and there is a high cost and poor universality of three-dimensional display. It realizes that the three-dimensional display video frame of the corresponding video frame can be obtained by processing each pixel in the video frame according to at least one predetermined depth segmentation line without the need for a three-dimensional display device, thereby improving the technical effect of the convenience and universality of three-dimensional display.

[0139] The video image processing device provided in the embodiments of the present disclosure can execute the video image processing method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0141] Embodiment 5

[0142] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present disclosure. Fig. 9 , which shows an electronic device (eg, Fig. 9 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0143] like Fig. 9As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0144] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0145] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0146] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0147] The electronic device provided by the embodiment of the present disclosure and the video image processing method provided by the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0148] Embodiment 6

[0149] The embodiments of the present disclosure provide a computer storage medium on which a computer program is stored. When the program is executed by a processor, the video image processing method provided by the above embodiments is implemented.

[0150] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0151] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0152] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0153] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0154] Determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view;

[0155] For each target depth view, determine a target depth segmentation line corresponding to a current pixel in a current target depth view, and determine a target pixel value of the current pixel according to a pixel depth value of the current pixel and a segmentation line depth value of the target depth segmentation line;

[0156] According to the target pixel value of each pixel point in each target depth view, a three-dimensional display video frame of each video frame in the target video is determined.

[0157] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0158] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0159] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0160] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0161] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0162] According to one or more embodiments of the present disclosure, [Example 1] provides an image video processing method, the method comprising:

[0163] Determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view;

[0164] For each target depth view, determine a target depth segmentation line corresponding to a current pixel in a current target depth view, and determine a target pixel value of the current pixel according to a pixel depth value of the current pixel and a segmentation line depth value of the target depth segmentation line;

[0165] According to the target pixel value of each pixel point in each target depth view, a three-dimensional display video frame of each video frame in the target video is determined.

[0166] According to one or more embodiments of the present disclosure, [Example 2] provides an image video processing method, the method comprising:

[0167] Optionally, before determining the target depth view of each video frame in the target video, it also includes: receiving the target video; setting at least one depth segmentation line corresponding to the target video, and determining the position and width of the at least one depth segmentation line to determine the target pixel value of the corresponding pixel point according to the position and width of the depth segmentation line.

[0168] According to one or more embodiments of the present disclosure, [Example 3] provides an image video processing method, the method comprising:

[0169] Optionally, determining the position and width of the at least one depth dividing line includes: determining the position and width of the at least one depth dividing line in the target video according to display parameters of the target video; wherein the display parameters are the display length and display height of the playback interface when playing the target video.

[0170] According to one or more embodiments of the present disclosure, [Example 4] provides an image video processing method, the method comprising:

[0171] Optionally, determining the target depth view of each video frame in the target video includes: determining a to-be-processed depth view and to-be-processed feature points of each video frame; processing the to-be-processed feature points of two adjacent video frames in sequence to obtain a set of 3D feature point pairs of two adjacent video frames; wherein the 3D feature point pair set includes multiple groups of 3D feature point pairs; determining camera motion parameters of two adjacent video frames based on the multiple groups of 3D feature point pairs in the 3D feature point pair set, and using the camera motion parameters as the camera motion parameters of a previous video frame in the two adjacent video frames; determining the target depth view of each video frame based on the to-be-processed depth view of each video frame and the corresponding camera motion parameters.

[0172] According to one or more embodiments of the present disclosure, [Example 5] provides an image video processing method, the method comprising:

[0173] Optionally, determining the feature points to be processed of the depth view to be processed of each video frame includes: performing depth estimation on each video frame to obtain the depth view to be processed of each video frame; and processing each video frame based on a feature point detection algorithm to determine the feature points to be processed of each video frame.

[0174] According to one or more embodiments of the present disclosure, [Example 6] provides an image video processing method, the method comprising:

[0175] Optionally, the processing of the feature points to be processed of two adjacent video frames in sequence to obtain a set of 3D feature point pairs of the two adjacent video frames includes: matching the feature points of the two adjacent video frames in sequence based on a feature point matching algorithm to obtain at least one group of 2D feature point pairs associated with the two adjacent video frames; reconstructing a 3D point cloud corresponding to the depth view to be processed of the two adjacent video frames, and obtaining an original 3D point cloud corresponding to the depth view to be processed, and at least one group of 3D feature point pairs corresponding to the at least one group of 2D feature point pairs; and determining the set of 3D feature point pairs of the two adjacent video frames based on the at least one group of 3D feature point pairs.

[0176] According to one or more embodiments of the present disclosure, [Example 7] provides an image video processing method, the method comprising:

[0177] Optionally, determining the camera motion parameters of two adjacent video frames based on multiple groups of 3D feature point pairs in the 3D feature point pair set, and using the camera motion parameters as the camera motion parameters of the previous video frame among the two adjacent video frames, includes: processing the position information of each group of 3D feature point pairs in each 3D feature point pair set to obtain a rotation matrix and a displacement matrix in each camera motion parameter, and using the rotation matrix and the displacement matrix as the camera motion parameters of the previous video frame among the two adjacent video frames.

[0178] According to one or more embodiments of the present disclosure, [Example 8] provides an image video processing method, the method comprising:

[0179] Optionally, determining the target depth view of each video frame according to the depth view to be processed of each video frame and the corresponding camera motion parameters includes: for each depth view to be processed, obtaining the 3D point cloud to be used corresponding to the current depth view to be processed according to the original 3D point cloud and camera motion parameters of the current depth view to be processed; and obtaining the target depth view corresponding to each video frame based on the original 3D point cloud corresponding to each depth view to be processed, the 3D point cloud to be used and a preset depth adjustment coefficient.

[0180] According to one or more embodiments of the present disclosure, [Example 9] provides an image video processing method, the method comprising:

[0181] Optionally, before determining the segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view, it also includes: determining the salient objects in each video frame, and determining the mask map to be processed of the corresponding video frame based on the salient objects, so as to determine the segmentation line depth value based on the mask map to be processed of each video frame and the target depth view.

[0182] According to one or more embodiments of the present disclosure, [Example 10] provides an image video processing method, the method comprising:

[0183] Optionally, determining the segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view includes: for each video frame, determining the average depth value of the mask area in the mask image to be processed according to the mask image to be processed of the current video frame and the target depth view; determining the segmentation line depth value of the at least one depth segmentation line according to the average depth value of each video frame and a preset segmentation line adjustment coefficient.

[0184] According to one or more embodiments of the present disclosure, [Example 11] provides an image video processing method, the method comprising:

[0185] Optionally, determining the average depth value of the mask area in the mask image to be processed according to the mask image to be processed and the target depth view of the current video frame includes: if there is a mask image to be processed corresponding to the current video frame, determining the to-be-processed depth value of each to-be-processed pixel in the mask area in the target depth view; and determining the average depth value of the mask area according to the to-be-displayed depth value of each to-be-displayed pixel in the target depth view and each to-be-processed depth value.

[0186] According to one or more embodiments of the present disclosure, [Example 12] provides an image video processing method, the method comprising:

[0187] Optionally, determining the average depth value of the mask area in the mask map to be processed based on the mask map to be processed and the target depth view of the current video frame includes: if there is no mask map to be processed corresponding to the current video frame, determining the average depth value of the current video frame based on the average depth values ​​of the recorded video frames.

[0188] According to one or more embodiments of the present disclosure, [Example 13] provides an image video processing method, the method comprising:

[0189] Optionally, determining the dividing line depth value of the at least one depth dividing line according to the average depth value of each video frame and a preset dividing line adjustment coefficient includes: determining the maximum and minimum values ​​of the average depth value according to the average depth value corresponding to each video frame; and determining the dividing line depth value of the at least one depth dividing line according to the minimum value, the dividing line adjustment coefficient and the maximum value.

[0190] According to one or more embodiments of the present disclosure, [Example 14] provides an image video processing method, the method comprising:

[0191] Optionally, the at least one depth segmentation line includes a first depth segmentation line and a second depth segmentation line, the preset segmentation line adjustment coefficient includes a first segmentation line adjustment coefficient and a second segmentation line adjustment coefficient, and determining the segmentation line depth value of the at least one depth segmentation line according to the minimum value, the segmentation line adjustment coefficient and the maximum value includes: determining the first segmentation line depth value of the first depth segmentation line, and the second segmentation line depth value of the second depth segmentation line according to the minimum value, the first segmentation line adjustment coefficient, the second segmentation line adjustment coefficient and the maximum value.

[0192] According to one or more embodiments of the present disclosure, [Example 15] provides an image video processing method, the method comprising:

[0193] Optionally, determining the target depth segmentation line corresponding to the current pixel in the current target depth view includes: determining whether the current pixel is located on the at least one depth segmentation line according to the position information of the current pixel and the position and width of the at least one depth segmentation line;

[0194] If so, the depth segmentation line including the current pixel point is used as the target depth segmentation line.

[0195] According to one or more embodiments of the present disclosure, [Example 16] provides an image video processing method, the method comprising:

[0196] Optionally, determining the target pixel value of the current pixel based on the pixel depth value of the current pixel and the segmentation line depth value of the target depth segmentation line includes: determining the target pixel value of the current pixel based on the pixel depth value of the current pixel, the segmentation line depth value, and a mask image to be processed of the video frame to which the current pixel belongs.

[0197] According to one or more embodiments of the present disclosure, [Example 17] provides an image video processing method, the method comprising:

[0198] Optionally, the target pixel value of the current pixel is determined according to the pixel depth value of the current pixel, the dividing line depth value, and the mask image to be processed of the video frame to which the current pixel belongs, including: if the pixel depth value of the current pixel is less than the dividing line depth value, and the current pixel is located in the mask area in the mask image to be processed, the original pixel value of the current pixel is kept unchanged, and the original pixel value is used as the target pixel value; if the pixel depth value of the current pixel is greater than the dividing line depth value, and the current pixel is located in the mask area in the mask image to be processed, the original pixel value of the current pixel is adjusted to a first preset pixel value, and the first preset pixel value is used as the target pixel value of the current pixel.

[0199] According to one or more embodiments of the present disclosure, [Example 18] provides an image video processing method, the method comprising:

[0200] Optionally, if there is no target depth segmentation line corresponding to the current pixel point, the original pixel value of the current pixel point is used as the target pixel value.

[0201] According to one or more embodiments of the present disclosure, [Example 19] provides an image and video processing device, the device comprising:

[0202] A segmentation line determination module, configured to determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view;

[0203] a pixel value determination module, for determining, for each target depth view, a target depth segmentation line corresponding to a current pixel point in a current target depth view, and determining a target pixel value of the current pixel point according to a pixel depth value of the current pixel point and a segmentation line depth value of the target depth segmentation line;

[0204] The video display module is used to determine a three-dimensional display video frame of each video frame in the target video according to the target pixel value of each pixel point in each target depth view.

[0205] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0206] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0207] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video image processing method, characterized in that: include: Determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view; wherein the at least one depth segmentation line is preset; For each target depth view, determine a target depth segmentation line corresponding to a current pixel in a current target depth view, and determine a target pixel value of the current pixel according to a pixel depth value of the current pixel and a segmentation line depth value of the target depth segmentation line; According to the target pixel value of each pixel point in each target depth view, a three-dimensional display video frame of each video frame in the target video is determined.

2. The method according to claim 1, characterized in that Before determining the target depth view of each video frame in the target video, the method further includes: Receiving the target video; Setting at least one depth segmentation line corresponding to the target video, and determining a position and a width of the at least one depth segmentation line in the target video according to display parameters of the target video, so as to determine a target pixel value of a corresponding pixel point according to a depth value corresponding to the position and the width of the depth segmentation line; The display parameters are the display length and display width of the target video when it is displayed on the display interface.

3. The method according to claim 1, characterized in that The determining of the target depth view of each video frame in the target video includes: Determine the depth view to be processed and the feature points to be processed of each video frame; Processing the feature points to be processed of two adjacent video frames in sequence to obtain a set of 3D feature point pairs of the two adjacent video frames; wherein the set of 3D feature point pairs includes multiple groups of 3D feature point pairs; Determine the camera motion parameters of two adjacent video frames according to the multiple groups of 3D feature point pairs in the 3D feature point pair set, and use the camera motion parameters as the camera motion parameters of the previous video frame in the two adjacent video frames; wherein the camera motion parameters include a rotation matrix and a displacement matrix; According to the to-be-processed depth view of each video frame and the corresponding camera motion parameters, a target depth view of each video frame is determined.

4. The method according to claim 3, characterized in that The determining of the to-be-processed feature points of the to-be-processed depth view of each video frame includes: Performing depth estimation on each video frame to obtain a depth view to be processed for each video frame; Each video frame is processed based on a feature point detection algorithm to determine the feature points to be processed of each video frame.

5. The method according to claim 3, characterized in that: The step of sequentially processing the feature points to be processed of two adjacent video frames to obtain a set of 3D feature point pairs of the two adjacent video frames includes: Based on the feature point matching algorithm, the feature points to be processed of two adjacent video frames are matched in sequence to obtain at least one set of 2D feature point pairs associated with the two adjacent video frames; Reconstructing 3D point clouds of the depth views to be processed of the two adjacent video frames to obtain an original 3D point cloud corresponding to the depth view to be processed and at least one group of 3D feature point pairs corresponding to the at least one group of 2D feature point pairs; A set of 3D feature point pairs of the two adjacent video frames is determined based on the at least one set of 3D feature point pairs.

6. The method according to claim 4, characterized in that The step of determining a target depth view of each video frame according to the to-be-processed depth view of each video frame and the corresponding camera motion parameter comprises: For each depth view to be processed, according to the original 3D point cloud, the rotation matrix and the translation matrix of the current depth view to be processed, obtain the 3D point cloud to be used of the current depth view to be processed; Based on the original 3D point cloud of each depth view to be processed, the 3D point cloud to be used, and a preset depth adjustment coefficient, a target depth view corresponding to each video frame is obtained.

7. The method according to claim 1, characterized in that Before determining the segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view, the method further includes: A notable object in each video frame is determined, and a mask map to be processed of the corresponding video frame is determined based on the notable object, so as to determine the segmentation line depth value based on the mask map to be processed of each video frame and a target depth view.

8. The method according to claim 7, characterized in that The determining, according to each target depth view, a segmentation line depth value of at least one depth segmentation line corresponding to the target video comprises: For each video frame, according to the mask image to be processed and the target depth view of the current video frame, determine the average depth value of the mask area in the mask image to be processed; The segmentation line depth value of the at least one depth segmentation line is determined according to the average depth value of each video frame and a preset segmentation line adjustment coefficient.

9. The method according to claim 8, characterized in that The step of determining an average depth value of a mask area in the mask image to be processed according to the mask image to be processed and the target depth view of the current video frame includes: If there is a mask image to be processed corresponding to the current video frame, determining a depth value to be processed of each pixel point to be processed in the mask area in the target depth view; Determine the average depth value of the mask area according to the depth values ​​to be displayed of each pixel to be displayed in the target depth view and the depth values ​​to be processed; or, if there is no mask map to be processed corresponding to the current video frame, determine the average depth value of the current video frame according to the average depth values ​​of the recorded video frames.

10. The method according to claim 8, characterized in that The step of determining the segmentation line depth value of the at least one depth segmentation line according to the average depth value of each video frame and the preset segmentation line adjustment coefficient includes: According to the average depth value of each video frame, determine the maximum value and the minimum value of the average depth value; A segmentation line depth value of the at least one depth segmentation line is determined according to the minimum value, the segmentation line adjustment coefficient, and the maximum value.

11. The method according to claim 10, characterized in that The at least one depth segmentation line includes a first depth segmentation line and a second depth segmentation line, the preset segmentation line adjustment coefficient includes a first segmentation line adjustment coefficient and a second segmentation line adjustment coefficient, and the determining of the segmentation line depth value of the at least one depth segmentation line according to the minimum value, the segmentation line adjustment coefficient and the maximum value includes: A first cutting line depth value of the first depth cutting line and a second cutting line depth value of the second depth cutting line are determined according to the minimum value, the first cutting line adjustment coefficient, the second cutting line adjustment coefficient and the maximum value.

12. The method according to claim 1, characterized in that The determining of the target depth segmentation line corresponding to the current pixel point in the current target depth view includes: Determine whether the current pixel is located on at least one depth segmentation line according to the position information of the current pixel, the position and width of the at least one depth segmentation line; If so, the depth segmentation line including the current pixel point is used as the target depth segmentation line.

13. The method according to claim 1, characterized in that The step of determining a target pixel value of the current pixel point according to the pixel depth value of the current pixel point and the segmentation line depth value of the target depth segmentation line comprises: The target pixel value of the current pixel is determined according to the pixel depth value of the current pixel, the segmentation line depth value, and the mask image to be processed of the video frame to which the current pixel belongs.

14. The method according to claim 13, characterized in that The step of determining the target pixel value of the current pixel point according to the pixel depth value of the current pixel point, the segmentation line depth value, and the mask image to be processed of the video frame to which the current pixel point belongs includes: If the pixel depth value of the current pixel point is less than the segmentation line depth value, and the current pixel point is located in the mask area in the mask image to be processed, the original pixel value of the current pixel point is kept unchanged, and the original pixel value is used as the target pixel value; If the pixel depth value of the current pixel is greater than the segmentation line depth value, and the current pixel is located in the mask area in the mask image to be processed, the original pixel value of the current pixel is adjusted to a first preset pixel value, and the first preset pixel value is used as the target pixel value of the current pixel.

15. A video image processing device, characterized in that: include: A segmentation line determination module, used to determine a target depth view of each video frame in a target video, and determine a segmentation line depth value of at least one depth segmentation line corresponding to the target video according to each target depth view; wherein the at least one depth segmentation line is preset; a pixel value determination module, for determining, for each target depth view, a target depth segmentation line corresponding to a current pixel point in a current target depth view, and determining a target pixel value of the current pixel point according to a pixel depth value of the current pixel point and a segmentation line depth value of the target depth segmentation line; The video display module is used to determine a three-dimensional display video frame of each video frame in the target video according to the target pixel value of each pixel point in each target depth view.

16. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video image processing method as described in any one of claims 1-14.

17. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the video image processing method according to any one of claims 1 to 14 when executed by a computer processor.

Images