Method for processing 3D video stream, and related device

Abstract

Provided in the embodiments of the present disclosure are a method for processing a 3D video stream, a computer-readable storage medium and a device. The method comprises: obtaining a 3D image frame from a 3D video stream; segmenting the 3D image frame into a plurality of segmented regions; determining a region to be processed in the 3D image frame and a plurality of pixels to be adjusted in said region, said pixels being pixels adjacent to edges of the plurality of segmented regions in a region where the region to be processed overlaps with the plurality of segmented regions; adjusting depth values of the plurality of pixels to be adjusted, so as to obtain a processed 3D image frame; and generating a processed 3D video stream on the basis of the processed 3D image frame. The embodiments described in the present disclosure can process depth values of the 3D video stream at defect positions, such that the processed 3D video stream can be adapted to display screens of various sizes.

Description

Method for processing 3D video stream and related device TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and more particularly, to a method, a computer readable storage medium and a device for processing 3D video stream. BACKGROUND

[0002] Three-dimensional (3D) content is usually produced by professional designers, which takes a long period and costs a high price. For different sizes of display screens, the 3D content needs to be modeled and re-produced according to real scenes.

[0003] SUMMARY

[0004] Embodiments of the present disclosure provide a method, a computer readable storage medium and a device for processing 3D video stream.

[0005] In a first aspect of the present disclosure, a method for processing 3D video stream is provided. The method comprises: obtaining a 3D image frame from a 3D video stream; segmenting the 3D image frame into a plurality of segmented regions; determining a to-be-processed region in the 3D image frame and a plurality of to-be-adjusted pixels in the to-be-processed region, the to-be-adjusted pixels being pixels adjacent to edges of the plurality of segmented regions in a region where the to-be-processed region overlaps with the plurality of segmented regions; adjusting depth values of the plurality of to-be-adjusted pixels to obtain a processed 3D image frame; and generating a processed 3D video stream based on the processed 3D image frame.

[0006] In embodiments of the present disclosure, adjusting the depth values of the plurality of to-be-adjusted pixels comprises: calculating an average depth value based on the depth values of the plurality of to-be-adjusted pixels; and for each of the plurality of to-be-adjusted pixels, modifying the depth value of the to-be-adjusted pixel based on depth values of a plurality of neighborhood pixels around the to-be-adjusted pixel and the average depth value.

[0007] In embodiments of the present disclosure, modifying the depth value of the to-be-adjusted pixel based on the depth values of the plurality of neighborhood pixels around the to-be-adjusted pixel and the average depth value comprises: determining a maximum depth value among the depth values of the plurality of neighborhood pixels around the to-be-adjusted pixel; and modifying the depth value of the to-be-adjusted pixel to the maximum depth value.

[0008] In embodiments of the present disclosure, the modification of the depth value of each of the plurality of to-be-adjusted pixels is repeated a predetermined number of times.

[0009] In embodiments of the present disclosure, generating the processed 3D video stream comprises: determining a parallax 3D image frame similar to the 3D image frame; adjusting depth values of pixels of the parallax 3D image frame to be the same as depth values of corresponding pixels of the processed 3D image frame; and replacing the 3D image frame and the parallax 3D image frame in the 3D video stream with a processed 3D image frame and a processed parallax 3D image frame respectively to generate the processed 3D video stream.

[0010] In embodiments of the present disclosure, adjusting the depth values of the pixels of the parallax 3D image frame comprises: adjusting depth values of pixels in the parallax 3D image frame corresponding to the plurality of to-be-adjusted pixels of the 3D image frame.

[0011] In embodiments of the present disclosure, the parallax 3D image frame is determined according to image similarity.

[0012] In embodiments of the present disclosure, determining the parallax 3D image frame comprises: determining, among the plurality of to-be-adjusted pixels of the 3D image frame, a minimum first horizontal pixel coordinate, a maximum second horizontal pixel coordinate, a minimum first vertical pixel coordinate, and a maximum second vertical pixel coordinate; determining a minimum first bounding rectangle based on the minimum first horizontal pixel coordinate, the maximum second horizontal pixel coordinate, the minimum first vertical pixel coordinate, and the maximum second vertical pixel coordinate; expanding a size of the minimum first bounding rectangle to obtain an expanded rectangle; determining similarity of images of the 3D image frame and a 3D image frame adjacent to the 3D image frame in the 3D video stream in the expanded rectangle; and in response to the similarity being greater than a first threshold value, determining that the adjacent 3D image frame is the parallax 3D image frame.

[0013] In embodiments of the present disclosure, the first threshold value is 0.85.

[0014] In embodiments of the present disclosure, the parallax 3D image frame is determined according to a coincidence ratio of areas.

[0015] In embodiments of the present disclosure, determining the same-view 3D image frame comprises: segmenting a 3D image frame adjacent to the 3D image frame in the 3D video stream into a plurality of segmentation regions; determining a largest segmentation region of the 3D image frame and a largest segmentation region of the adjacent 3D image frame; determining a minimum third horizontal pixel coordinate, a maximum fourth horizontal pixel coordinate, a minimum third vertical pixel coordinate, and a maximum fourth vertical pixel coordinate in pixels of adjacent edges in the largest segmentation region of the 3D image frame; determining a minimum second bounding rectangle based on the minimum third horizontal pixel coordinate, the maximum fourth horizontal pixel coordinate, the minimum third vertical pixel coordinate, and the maximum fourth vertical pixel coordinate; determining a minimum fifth horizontal pixel coordinate, a maximum sixth horizontal pixel coordinate, a minimum fifth vertical pixel coordinate, and a maximum sixth vertical pixel coordinate in pixels of adjacent edges in the largest segmentation region of the adjacent 3D image frame; determining a minimum third bounding rectangle based on the minimum fifth horizontal pixel coordinate, the maximum sixth horizontal pixel coordinate, the minimum fifth vertical pixel coordinate, and the maximum sixth vertical pixel coordinate; calculating a coincidence ratio of an area of an overlapping region of the minimum second bounding rectangle and the minimum third bounding rectangle and an area of a non-overlapping region; and determining the adjacent 3D image frame as the same-view 3D image frame in response to the coincidence ratio being greater than a second threshold.

[0016] In embodiments of the present disclosure, the second threshold is 0.5.

[0017] In embodiments of the present disclosure, segmenting the 3D image frame is based on an image segmentation model.

[0018] In embodiments of the present disclosure, determining the region to be processed in the 3D image frame comprises: determining the region to be processed based on a pixel selected by a user.

[0019] In embodiments of the present disclosure, adjusting the depth values of the plurality of pixels to be adjusted comprises: adjusting the depth values of the plurality of pixels to be adjusted based on a plurality of depth values input by a user.

[0020] According to a second aspect of the present disclosure, there is provided a device for processing a 3D video stream. The device comprises: an obtaining module configured to obtain a 3D image frame from a 3D video stream; a segmenting module configured to segment the 3D image frame into a plurality of segmentation regions; a determining module configured to determine a region to be processed in the 3D image frame and a plurality of pixels to be adjusted in the region to be processed, the pixels to be adjusted being pixels adjacent to edges of the plurality of segmentation regions in a region where the region to be processed overlaps with the plurality of segmentation regions; an adjusting module configured to adjust depth values of the plurality of pixels to be adjusted to obtain a processed 3D image frame; and a generating module configured to generate a processed 3D video stream based on the processed 3D image frame.

[0021] According to a third aspect of the present disclosure, a computer readable storage medium is provided. The computer readable storage medium has stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0022] According to a fourth aspect of the present disclosure, a device for processing a 3D video stream is provided. The device comprises a processor; and a memory storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method according to the first aspect of the present disclosure.

[0023] According to a fifth aspect of the present disclosure, a computer program product is provided. The computer program product comprises computer program instructions, wherein the computer program instructions, when executed by a processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0024] Further aspects and ranges are apparent from the description provided herein. It should be understood that the various aspects of the present application can be practiced alone or in combination with one or more other aspects. It should also be understood that the description and specific examples herein are intended to be illustrative only and are not intended to limit the scope of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0025] The drawings described herein are for the purpose of illustrating selected embodiments only and are not intended to limit the scope of the present application, wherein:

[0026] FIG. 1 shows an exemplary 3D image frame displayed on a display screen;

[0027] FIG. 2 shows a schematic flowchart of a method for processing a 3D video stream according to an embodiment of the present application;

[0028] FIG. 3A shows an exemplary 3D video stream according to an embodiment of the present application;

[0029] FIG. 3B shows an exemplary 3D image frame;

[0030] FIG. 4A shows a segmented exemplary 3D image frame;

[0031] FIG. 4B shows a pixel to be adjusted in FIG. 4A;

[0032] FIG. 5 shows a schematic flowchart for adjusting depth values of a plurality of pixels to be adjusted according to an embodiment of the present application;

[0033] FIG. 6 shows an exemplary depth map;

[0034] FIG. 7 shows a schematic flow chart for determining a parallax 3D image frame according to an embodiment of the present application;

[0035] FIG. 8A shows an exemplary minimum first circumscribed rectangle of a 3D image frame shown in FIG. 4B;

[0036] FIG. 8B shows an exemplary adjacent 3D image frame;

[0037] FIG. 9 shows a schematic flow chart for determining a parallax 3D image frame according to another embodiment of the present application;

[0038] FIGS. 10A and 10B respectively show an exemplary segmented 3D image frame and an exemplary segmented adjacent 3D image frame;

[0039] FIG. 11 shows an exemplary overlapping region and a non-overlapping region of a minimum second circumscribed rectangle and a minimum third circumscribed rectangle;

[0040] FIG. 12 shows an exemplary structural block diagram of a device for processing a 3D video stream according to an embodiment of the present disclosure;

[0041] FIG. 13 shows a schematic block diagram of a device for processing a 3D video stream according to an embodiment of the present disclosure;

[0042] FIG. 14 shows a schematic diagram of a load balancing module according to an embodiment of the present disclosure; and

[0043] FIG. 15 shows a schematic diagram of a log storage module according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present disclosure. The embodiments of the present disclosure will be described in detail below with reference to the drawings and in combination with the embodiments. It should be noted that the features in the embodiments of the present disclosure can be combined with each other without conflict.

[0045] Generally, it takes 10 to 30 days to produce a 3D content (e.g., a 3D movie), and the production process includes the following steps:

[0046] 1. Use modeling software (such as Autodesk-Maya, Mudbox, and Zbrush, etc.) to model a real scene in three dimensions. Specifically, three-dimensional modeling of a real scene includes the following three main steps. First, use a camera and measuring equipment to take and measure a real scene to obtain accurate data. Then, process the obtained data. Next, convert the processed data into a three-dimensional scene or three-dimensional model.

[0047] 2. Bind the subject, and the following is an example of how to bind the subject as a person. First, establish the skeleton of the person. Then, skin the skeleton and various control points, and draw the degree of influence of each control point on the muscles and bones on the model.

[0048] 3. Make animation and adjust the action. First, shoot the action performed by a real person. Then, make animation according to the real action. Finally, adjust the action frame by frame.

[0049] 4. Simulate materials and lighting. First, simulate the real materials of the picture to make the whole picture credible. Then, like a film lighting designer, create atmosphere and picture, and simulate real light and shadow effects by controlling the type, intensity, shape, color, reflection method, shadow, etc. of the light.

[0050] 5. Synthesize the final footage. First, synthesize the rendered picture. Then, adjust the light and material in the picture. Next, synthesize the background and special effects with the picture.

[0051] 6. Edit the synthesized footage. Splice the synthesized shots together, adjust the color tone, dubbing, music, and add the title, subtitles, and staff list.

[0052] The above process of making 3D content usually needs very professional personnel to complete, the process is complex and time-consuming, and is not suitable for large-scale generation of 3D content, especially the process cannot be implemented on a mobile device.

[0053] Embodiments of the present application can use artificial intelligence (AI) technology to generate 3D content to improve the efficiency of generating 3D content and reduce the cost of making 3D content.

[0054] However, the generated 3D content (for example, 3D video stream or 3D image frame) cannot be adapted to any size of naked-eye 3D display screen. For example, on a 10.95-inch naked-eye 3D display screen, the 3D content has good display effect. However, on a 98-inch naked-eye 3D display screen, since the 3D content is enlarged for display, it cannot achieve good display effect.

[0055] FIG. 1 shows displaying a 3D image frame generated by using an AI technology on a large-sized display screen. The 3D image frame presents problems such as a wire drawing and a blur at a position shown in a black square in FIG. 1. This is because an error is generated when estimating a depth value at the position. Therefore, it is necessary to manually repair the depth value at the position. Generally, when a 3D movie is made, a video stream is 25 frames per second or 60 frames per second, and it is very laborious to manually repair 3D image frames frame by frame by a person.

[0056] FIG. 2 shows a schematic flowchart of a method for processing a three-dimensional (3D) video stream according to an embodiment of the present application. As shown in FIG. 2, in block 202, a 3D image frame is obtained from a 3D video stream. The 3D video stream includes a plurality of 3D image frames.

[0057] FIG. 3A shows an exemplary 3D video stream according to an embodiment of the present application. FIG. 3B shows an exemplary 3D image frame. In an embodiment of the present application, the 3D image frame can be obtained from the 3D video stream shown in FIG. 3A based on a selection from a user. For example, the 3D image frame can be a 3D image frame having a defect (e.g., a wire drawing, a blur, etc.).

[0058] In an embodiment of the present application, the 3D image frame in the 3D video stream can be generated based on an Artificial Intelligence Generated Content (AIGC) technology. Specifically, generating the 3D image frame can include the following three steps. Step 1, obtaining a two-dimensional (2D) image. Step 2, generating a plurality of viewpoint images corresponding to the 2D image based on the 2D image and a preset viewpoint condition. The viewpoint condition is determined based on display parameters of a display screen, and the viewpoint condition includes a number of viewpoints and a viewpoint angle. The viewpoint image represents a parallax image at different viewpoint positions. Step 3, performing stitching processing and viewpoint interleaving processing on the plurality of viewpoint images to generate the 3D image frame.

[0059] In the above step 2, a depth map is obtained based on the 2D image. A foreground image and a background image are extracted from the depth map, and a relative position relationship between the foreground image and the background image is determined. Based on the number of viewpoints and the viewpoint angle, a plurality of target viewpoints corresponding to the foreground image are determined. Based on the plurality of target viewpoints, a plurality of first viewpoint images corresponding to the foreground image are determined, and a plurality of second viewpoint images corresponding to the background image are determined based on the relative position relationship. The plurality of first viewpoint images and the plurality of second viewpoint images are fused to generate a plurality of viewpoint images corresponding to the 2D image.

[0060] It should be noted that the 2D image can be obtained by an image acquisition device (for example, a camera, a video camera), or can be obtained from the photo album of a terminal device (for example, a mobile phone, a computer). In addition, a 2D image that can reflect the text content in the prompt word can also be generated based on the prompt word input by the user through a text-to-image algorithm.

[0061] With reference back to FIG. 2, in block 204, the 3D image frame is segmented into a plurality of segmentation regions. FIG. 4A shows an example segmented 3D image frame, in which the numbers in the boxes represent pixel numbers. As shown in FIG. 4A, the 3D image frame is segmented into segmentation regions a, b and c, wherein segmentation region a includes pixels 1-3, 8-10 and 15-17, segmentation region b includes pixels 22-24, 29-33 and 36-49, and segmentation region c includes pixels 4-7, 11-14, 18-21, 25-28 and 34-35.

[0062] In embodiments of the present application, segmenting the 3D image frame can be based on a conventional image segmentation model, such as a Segment Anything Model (SAM).

[0063] It should be noted that embodiments of the present application take the 3D image frame shown in FIG. 4A as an example for illustration. It should be understood that the 3D image frame can include more or fewer pixels according to the actual needs of the resolution of the display screen. In addition, the 3D image frame can be segmented into more or fewer segmentation regions according to the number of objects included in the 3D image frame.

[0064] With reference back to FIG. 2, in block 206, a to-be-processed region in the 3D image frame is determined. In embodiments of the present application, the to-be-processed region can be determined based on the pixels selected by the user. As shown by the black dashed box in FIG. 4A, the user selects pixels 9-11, 16-18, 23-25, 30-32 and 37-39 at the defect location, and therefore the to-be-processed region is determined as region d enclosed by the black dashed box.

[0065] It should be noted that embodiments of the present application take the to-be-processed region d shown in FIG. 4A as an example for illustration. It should be understood that a person skilled in the art can determine the to-be-processed region according to the actual position of the defect in the 3D image frame.

[0066] In block 206, a plurality of pixels to be adjusted in the region to be processed is also determined. In an embodiment of the present application, the pixels to be adjusted are the pixels adjacent to the edges of the segmented regions in the region to be processed that overlaps with the plurality of segmented regions. As shown in FIG. 4A, the pixels to be adjusted are pixels 10, 16 and 17 in segmented region a, pixels 23, 24 and 32 in segmented region b, and pixels 11, 18 and 25 in segmented region c. FIG. 4B shows these pixels to be adjusted in FIG. 4A in detail. As shown in FIG. 4B, the pixels enclosed by the dashed boxes are the pixels to be adjusted.

[0067] In block 208, the depth values of the plurality of pixels to be adjusted are adjusted. In an embodiment of the present application, the depth values of the plurality of pixels to be adjusted can be adjusted based on a plurality of depth values inputted by a user. With continued reference to FIG. 4B, for pixel 10 to be adjusted, a depth value dep 10 is inputted by a user, the depth value of pixel 10 to be adjusted is modified to dep 10 . Similarly, for other pixels to be adjusted, depth values inputted by a user are received to modify the depth values of the corresponding pixels to be adjusted to the inputted depth values.

[0068] FIG. 5 shows a schematic flowchart for adjusting the depth values of a plurality of pixels to be adjusted according to another embodiment of the present application.

[0069] As shown in FIG. 5, in block 502, an average depth value is calculated based on the depth values of the plurality of pixels to be adjusted. With continued reference to FIG. 4B, an arithmetic mean of the depth values of pixels 10, 16, 17, 23, 24, 32, 11, 18 and 25 to be adjusted can be calculated, and the arithmetic mean is determined as the average depth value.

[0070] In an embodiment of the present application, the depth value of each of the plurality of pixels to be adjusted can be changed to the average depth value. FIG. 6 shows a depth map of the 3D image frame in FIG. 4B after the depth values of the pixels are changed, in which the numbers in the boxes represent the depth values. The depth map refers to an image in which the distances (depths) from the image capturing device to the points in the scene are taken as the pixel values. As shown in FIG. 6, for pixels 10, 16, 17, 23, 24, 32, 11, 18 and 25 to be adjusted in FIG. 4B, their depth values are all changed to the same depth value 5.

[0071] It is noted that the depth values of the pixels shown in FIG. 6 are for illustrative purposes only, and the depth values of the pixels depend on the actual distances from the image capturing device to the points in the scene.

[0072] In block 504, for each of the plurality of pixels to be adjusted, the depth value of the pixel to be adjusted is modified based on the depth values of the plurality of neighboring pixels around the pixel to be adjusted and the corresponding average depth value. How to modify the depth value of the pixel to be adjusted is explained below in connection with FIG. 4B and FIG. 6. For the pixel 10 to be adjusted in FIG. 4B, the pixels 2-4, 9, 11 and 16-18 around it are the neighboring pixels. As shown in the depth map of FIG. 6, among the depth values of these neighboring pixels 2-4, 9, 11 and 16-18, the maximum depth value is 9. The depth value of the pixel 10 to be adjusted is modified from the value 5 to the value 9.

[0073] Similarly, for the remaining pixels to be adjusted 16, 17, 23, 24, 32, 11, 18 and 25, the same process of modifying the depth value of the pixel 10 to be adjusted is applied to modify the remaining pixels to be adjusted, which is not repeated here.

[0074] In the embodiments of the present application, for each of the steps shown in FIG. 5, after being performed once, it can be repeatedly performed, for example, 5 times, so that the depth values at the region to be processed are smoother. Thus, the processed 3D image frame is obtained.

[0075] With reference back to FIG. 2, in block 210, a processed 3D video stream is generated based on the processed 3D image frame. In the embodiments of the present application, generating the processed 3D video stream includes the following steps. First, a parallax 3D image frame similar to the 3D image frame is determined. Then, the depth values of the pixels of the parallax 3D image frame are adjusted to be the same as the depth values of the corresponding pixels of the processed 3D image frame. Next, the 3D image frame and the parallax 3D image frame in the 3D video stream are respectively replaced by the processed 3D image frame and the processed parallax 3D image frame to generate the processed 3D video stream.

[0076] Generally, in the case of a 3D video stream being a movie, a scene can last for 1 second. Within the 1 second, there are 25-30 3D image frames, and the scenes of these 3D image frames are basically the same. In the present application, these 3D image frames can be referred to as parallax 3D image frames.

[0077] It should be noted that an image frame is composed of pixels, and the pixel coordinates refer to the position of the pixel in the image frame. To determine the coordinates of a pixel, the coordinate system of the image frame must be determined first. As shown in FIG. 4B, the lower left corner of the image frame is taken as the origin O, and a coordinate system x-y in pixels is established. The pixel horizontal coordinate x and the pixel vertical coordinate y of a pixel are the row number and the column number where the pixel is located, respectively.

[0078] In embodiments of the present application, the parallax-free 3D image frame can be determined according to image similarity. FIG. 7 shows a schematic flowchart for determining a parallax-free 3D image frame according to embodiments of the present application. In block 702, continuing to refer to FIG. 4B, among the plurality of to-be-adjusted pixels 10, 16, 17, 23, 24, 32, 11, 18 and 25, the minimum first horizontal pixel coordinate x min (corresponding to the horizontal pixel coordinate of the to-be-adjusted pixel 16 or 23), the maximum second horizontal pixel coordinate x max (corresponding to the horizontal pixel coordinate of the to-be-adjusted pixel 33), the minimum first vertical pixel coordinate y min (corresponding to the vertical pixel coordinate of the to-be-adjusted pixel 32 or 33), and the maximum second vertical pixel coordinate y max (corresponding to the vertical pixel coordinate of the to-be-adjusted pixel 10 or 11) are determined.

[0079] In block 704, based on the minimum first horizontal pixel coordinate x min , the maximum second horizontal pixel coordinate x max , the minimum first vertical pixel coordinate y min , and the maximum second vertical pixel coordinate y max , a minimum first circumscribed rectangle is determined. FIG. 8A shows an exemplary minimum first circumscribed rectangle R1 of the 3D image frame shown in FIG. 4B. The pixel coordinates at the four endpoints of the minimum first circumscribed rectangle R1 are (x min , y max ), (x min , y min ), (x max , y min ), and (x max , y max ), respectively.

[0080] In block 706, the size of the minimum first circumscribed rectangle R1 is expanded to obtain an expanded rectangle. In embodiments of the present application, the pixel coordinates at the four endpoints of the minimum first circumscribed rectangle R1 can be respectively expanded to (x min *(1+T1), y max *(1+T1)), (x min *(1+T1), y min *(1+T1)), (x max *(1+T1), y min *(1+T1)), and (x max *(1+T1), y max *(1+T1)) to obtain the expanded rectangle R1’. In this embodiment, the area of R1’ is (1+T1) 2 times the area of R1. In one example, the value of T1 can be 20%, so that the area of R1’ is 1.44 times the area of R1.

[0081] In block 708, a similarity of images in the extended rectangle between the 3D image frame and a 3D image frame adjacent to the 3D image frame in the 3D video stream is determined. In this application, the adjacent 3D image frame refers to the 3D image frame adjacent in time to the 3D image frame, for example, the time interval of the two image frames in the video stream is less than one second. FIG. 8B shows a 3D image frame adjacent to the 3D image frame shown in FIG. 8A. In an embodiment of this application, the similarity of the 3D image frame shown in FIG. 8A and the adjacent 3D image frame shown in FIG. 8B in the same extended rectangle R1’ is determined based on a structural similarity (SSIM) algorithm.

[0082] It should be noted that the SSIM (Structural Similarity) algorithm is used to measure the similarity of two image frames, and the value of the similarity of SSIM is between 0 and 1. The greater the value of the similarity, the higher the similarity of the two image frames.

[0083] Specifically, given two image frames X and Y, the structural similarity of the two image frames can be obtained according to the following formula:

[0084] wherein μ x and μ y respectively represent the average of X and Y, and respectively represent the variance of X and Y, δ xy is the covariance of X and Y. C1 and C2 are constants to avoid instability when the denominator is 0.

[0085] As an implementation of the structural similarity theory, the structural similarity defines the structural information as an attribute independent of brightness, contrast, and reflecting the structure of objects in the scene, and models the distortion as a combination of three different factors of brightness, contrast and structure. The mean value is used as the estimate of brightness, the standard deviation is used as the estimate of contrast, and the covariance is used as the measure of structural similarity.

[0086] In block 710, in response to the similarity being greater than a first threshold, it is determined that the adjacent 3D image frame shown in FIG. 8B is a same-lens 3D image frame. In an embodiment, the first threshold can be 0.85.

[0087] In embodiments of the present application, the parallax-free 3D image frame can be determined according to the coincidence ratio of areas. FIG. 9 shows a schematic flowchart for determining a parallax-free 3D image frame according to another embodiment of the present application. In block 902, a 3D image frame adjacent to the 3D image frame in the 3D video stream is segmented into a plurality of segmented areas. FIGS. 10A and 10B show a segmented schematic 3D image frame and a schematic adjacent 3D image frame, respectively.

[0088] In block 904, the largest segmented area of the 3D image frame and the largest segmented area of the adjacent 3D image frame are determined. As shown in FIG. 10A, the largest segmented area of the 3D image frame is area b. As shown in FIG. 10B, the largest segmented area of the adjacent 3D image frame is area b’.

[0089] In block 906, referring to the 3D image frame shown in FIG. 10A, the minimum third horizontal pixel coordinate x min (corresponding to the horizontal pixel coordinate of pixel 22, 29, 36 or 43), the maximum fourth horizontal pixel coordinate x max (corresponding to the horizontal pixel coordinate of pixel 42 or 49), the minimum third vertical pixel coordinate y min (i.e., the vertical pixel coordinate of pixel 43, 44, 45, 46, 47, 48 or 49), and the maximum fourth vertical pixel coordinate y max (corresponding to the vertical pixel coordinate of pixel 22, 23 or 24) are determined in the pixels of the adjacent edges in the largest segmented area b. Based on the minimum third horizontal pixel coordinate x min , the maximum fourth horizontal pixel coordinate x max , the minimum third vertical pixel coordinate y min , and the maximum fourth vertical pixel coordinate y max , the minimum second circumscribed rectangle R2 is determined. As shown in FIG. 10A, the pixel coordinates at the four endpoints of the minimum second circumscribed rectangle R2 are (x min , y max ), (x min , y min ), (x max , y min ), and (x max , y max ), respectively.

[0090] In block 908, referring to the adjacent 3D image frame shown in FIG. 10B, the minimum fifth horizontal pixel coordinate x min (corresponding to the horizontal pixel coordinate of pixel 15’, 22’, 29’, 36’ or 43’), the maximum sixth horizontal pixel coordinate x max(corresponding to the horizontal pixel coordinate of pixel 27', 34', 41', or 48'), a minimum fifth vertical pixel coordinate y min (corresponding to the vertical pixel coordinate of pixel 43', 44', 45', 46', 47', or 48'), and a maximum sixth vertical pixel coordinate y max (corresponding to the vertical pixel coordinate of pixel 15', 16', or 17'). Based on the minimum fifth horizontal pixel coordinate x min , the maximum sixth horizontal pixel coordinate x max , the minimum fifth vertical pixel coordinate y min , and the maximum sixth vertical pixel coordinate y max , a minimum third circumscribed rectangle R3 is determined. As shown in FIG. 10B, the pixel coordinates at the four endpoints of the minimum third circumscribed rectangle R3 are (x min , y max ), (x min , y min ), (x max , y min ), and (x max , y max ), respectively.

[0091] In block 908, a coincidence ratio of an area of an overlapping region of the minimum second circumscribed rectangle R2 and the minimum third circumscribed rectangle R3 to an area of a non-overlapping region is calculated. FIG. 11 shows a schematic overlapping region and a non-overlapping region of the minimum second circumscribed rectangle and the minimum third circumscribed rectangle. As shown in FIG. 11, if the area of the overlapping region of the minimum second circumscribed rectangle R2 and the minimum third circumscribed rectangle R3 is S1, and the area of the non-overlapping region is S2, then the coincidence ratio is S1 / S2.

[0092] In block 910, in response to the coincidence ratio being greater than a second threshold value, it is determined that the adjacent 3D image frames shown in FIG. 10B are the same-camera 3D image frames. In one example, the second threshold value can be 0.5.

[0093] In embodiments of the present application, adjusting the depth values of the pixels of the same-camera 3D image frames can include adjusting the depth values of the pixels in the same-camera 3D image frames that correspond to the plurality of to-be-adjusted pixels of the 3D image frames.

[0094] With reference to FIGS. 8A and 8B, if it is determined that the 3D image frames in FIG. 8B are the same-camera 3D image frames, the depth values of pixels 10', 16', 17', 23', 24', 32', 11', 18', and 25' in FIG. 8B are modified to the depth values of to-be-adjusted pixels 10, 16, 17, 23, 24, 32, 11, 18, and 25 in FIG. 8A, respectively, to obtain a processed same-camera 3D image frame.

[0095] Similarly, if it is determined that the 3D image frame in FIG. 10B is a same-mirror 3D image frame, the depth values of the pixels 10', 16', 17', 23', 24', 32', 11', 18', and 25' in FIG. 10B are modified to the depth values of the to-be-adjusted pixels 10, 16, 17, 23, 24, 32, 11, 18, and 25 in FIG. 10A respectively, so as to obtain a processed same-mirror 3D image frame.

[0096] It should be noted that the 3D image frames shown in FIG. 8A and FIG. 10A are the same as the 3D image frame shown in FIG. 4A.

[0097] In the embodiments of the present disclosure, the original 3D image frame and the original same-mirror 3D image frame in the original 3D video stream can be replaced by the processed 3D image frame and the processed same-mirror 3D image frame respectively to generate a processed 3D video stream.

[0098] As can be seen from the above description, by using the method according to the embodiments of the present disclosure, the depth values of the 3D video stream at the defect position can be processed, so that the processed 3D video stream can be adapted to various sizes of display screens, avoiding the problem that the 3D video stream needs to be re-produced when the display screen is replaced.

[0099] FIG. 12 shows an exemplary structural block diagram of a device 1200 for processing a three-dimensional (3D) video stream according to an embodiment of the present disclosure. The device 1200 can include an obtaining module 1210, a dividing module 1220, a determining module 1230, an adjusting module 1240, and a generating module 1250.

[0100] In the embodiments of the present disclosure, the obtaining module 1210 is configured to obtain a 3D image frame from a 3D video stream. The 3D image frame can be a 3D image frame with defects (e.g., streaks, blurring, etc.).

[0101] In the embodiments of the present disclosure, the dividing module 1220 is configured to divide the 3D image frame into a plurality of divided regions. Dividing the 3D image frame can be based on a conventional image segmentation model, such as a divide-and-conquer model.

[0102] In the embodiments of the present disclosure, the determining module 1230 is configured to determine a to-be-processed region in the 3D image frame and a plurality of to-be-adjusted pixels in the to-be-processed region. The to-be-adjusted pixels are pixels adjacent to edges of the plurality of divided regions in a region where the to-be-processed region overlaps with the plurality of divided regions. Determining the to-be-processed region in the 3D image frame includes determining the to-be-processed region based on a pixel selected by a user.

[0103] In embodiments of the present disclosure, the adjusting module 1240 is configured to adjust the depth values of the plurality of to-be-adjusted pixels. The adjusting the depth values of the plurality of to-be-adjusted pixels comprises: calculating an average depth value based on the depth values of the plurality of to-be-adjusted pixels; and for each of the plurality of to-be-adjusted pixels, modifying the depth value of the to-be-adjusted pixel based on the depth values of the plurality of neighboring pixels around the to-be-adjusted pixel and the average depth value, to obtain a processed 3D image frame.

[0104] In embodiments of the present disclosure, the generating module 1250 is configured to generate a processed 3D video stream based on the processed 3D image frame. The generating the processed 3D video stream comprises: determining a parallax 3D image frame similar to the 3D image frame; adjusting the depth values of the pixels of the parallax 3D image frame to be the same as the depth values of the corresponding pixels of the processed 3D image frame; and replacing the 3D image frame and the parallax 3D image frame in the 3D video stream with the processed 3D image frame and the processed parallax 3D image frame respectively, to generate the processed 3D video stream.

[0105] [According to Rule 91 Correction 05.12.2024] The various units of the device 1200 can also further complete the functions and methods introduced in FIGS. 1-11, which are not repeated here.

[0106] FIG. 13 shows a schematic block diagram of a device 1300 for processing a three-dimensional 3D video stream, according to embodiments of the present disclosure. The device 1300 includes one or more processors 1302 and a memory 1304. The memory 1304 is coupled to the processor(s) 1302 via a bus and an I / O interface 1306, and stores instructions executable by the processor(s) 1302. When the instructions are executed by the processor(s) 1302, the device 1300 can perform the steps of the method for processing a three-dimensional 3D video stream in any one of the embodiments described above.

[0107] The memory 1304 can include a readable medium in the form of volatile memory, such as a random access memory (RAM) and / or a cache memory, and can further include a read-only memory (ROM).

[0108] The memory 1304 can also include program / utility 1311 having a set of programs / modules 1312, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, which can include implementation of a network environment.

[0109] The bus can be representative of one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus structures, and the like.

[0110] Device 1300 can also communicate with one or more external devices such as a keyboard or a pointing device, through I / O interface 1306. Additionally, device 1300 can communicate with one or more devices such as a display, a printer, or a plotter, also through I / O interface 1306. In some embodiments, device 1300 can further include a communication interface 1307 that can be used to enable device 1300 to communicate with one or more other devices over a network, such as a local area network (LAN), a wide area network (WAN), and / or the Internet, for example. As depicted, communication interface 1307 can include a network adapter coupled to bus 1301 and used to send and receive communications over a network. It should be appreciated that communication interface 1307 can be any wired or wireless communication device, including a modem, a wireless transceiver, a wireless network interface card, a LAN port, a telephonic modem, or the like. It should be appreciated that other modules and / or other devices can be used in conjunction with device 1300 in place of, or in addition to, those depicted in FIG. 13. Such devices include, but are not limited to, mobile computing devices such as cellular telephone, computer platforms, and other devices that can execute one or more instructions. Further, it should be appreciated that devices of system 1300 can be configured to operate as a portable media player or other media device.

[0111] In embodiments of the present disclosure, a computer readable storage medium is also provided, having stored thereon computer program instructions which, when executed by, for example, a processor, can implement the steps of the method for processing a three-dimensional (3D) video stream in any one of the embodiments described above. In some possible implementation manners, various aspects of the present disclosure can also be implemented in the form of a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the method for processing a three-dimensional (3D) video stream according to various example embodiments of the present disclosure.

[0112] In embodiments of the present disclosure, a computer program product is also provided. The computer program product includes computer program instructions, wherein the computer program instructions, when executed by a processor, can implement the steps of the method for processing a three-dimensional (3D) video stream in any one of the embodiments described above.

[0113] The program product for implementing the above-described method according to embodiments of the present disclosure can take a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in the present document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0114] The program product can take any combination of one or more computer-readable media to implement the program instructions. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] The computer-readable storage medium can include a data signal embodied in a carrier wave, sent over a communication medium such as a metal, optical fiber, or wireless medium. Such a data signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. A computer-readable storage medium can be any medium that can be read by a machine, including but not limited to read-only memory (ROM), random access memory (RAM), flash memory, etc. The program code embodied on the computer-readable storage medium can also reside on a computer-readable storage medium associated with one or more components of the computer system 1000, such as random access memory (RAM), magnetic storage, optical storage, or any suitable combination thereof. The computer-readable storage medium can be loaded into the computer system 1000 via, for example, the interface 1020.

[0116] The program code can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device (for example, through the Internet using an Internet Service Provider). The computer system 1000 can also include a communications interface 1020.

[0117] Those skilled in the art will understand that the various aspects of the application can be implemented as a system, method or program product. Therefore, aspects of the application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system." Furthermore, aspects of the application can take the form of program code embodied in any tangible medium of expression, which can be executed by a program execution device or system.

[0118] In embodiments of the present disclosure, the method for processing 3D image frames in a 3D video stream can be based on the load balancing processing method of Nginx. FIG. 14 shows a schematic diagram of a load balancing module according to an embodiment of the present disclosure. Specifically, the role of Nginx load balancing between GPU and CPU includes the following: when the access volume per unit time of a server is larger, the server pressure is greater, and when it is too large to exceed its own bearing capacity, the server will crash. In order to avoid server crashes and provide users with a better experience, the server pressure is shared through load balancing. By establishing many servers, a server cluster is formed, when a user accesses a website, an intermediate server is accessed first, and then the intermediate server selects a server with less pressure in the server cluster, and then introduces the access request to the server. Through this method, each server in the server cluster will be guaranteed to have balanced pressure, sharing the server pressure and avoiding server crashes. Shared storage is used to store code data of algorithms and model files called by algorithms. The update of models and image data is realized through interaction with the Redis cluster / Minio service. In addition to the Redis cluster, the Minio service can also be used for data interaction. Taking image data as an example, how to interact is explained. Upload the image to the storage server through the minio protocol or the redis protocol, and get the link of the image on minio. Through the link, the picture can be downloaded and displayed

[0119] FIG. 15 shows a schematic diagram of a log storage module according to an embodiment of the present disclosure. The storage of logs is realized through kafka cluster interaction. The kafka information delivery method includes: receiving a request message, sending the message to the kafka cluster, and at the same time, the message storage server, listening to the kafka message, receiving the log message, and storing it to the database on the storage server. Subsequently, data viewing, management and analysis can be performed on the storage server. Through the operation feedback and question request of the user in the front end, through the log storage module, the feedback of the user and the user ID are bundled and stored through kafka cluster interaction to complete.

[0120] The Nginx load balancing function is to increase the concurrency and fully utilize the GPU computing resources, not to reduce the server pressure. The function of load balancing is not only limited to different docker containers on the same server, but also can be used for scheduling between multiple servers. Different algorithms are placed on different servers, and when the single task request is large, it can be loaded to multiple servers to process the task at the same time. Nginx is a tool for implementing the load balancing scheme, and the load balancing method is achieved by the following methods: HTTP and reverse proxy method. HTTP redirection realizes load balancing. When the user initiates a request to the server, the request is first intercepted by the cluster dispatcher; the dispatcher selects a server according to a certain allocation strategy, encapsulates the IP address of the selected server in the Location field of the HTTP response message header, sets the status code of the response message to 302, and finally returns the response message to the browser. When the browser receives the response message, it parses the Location field and initiates a request to the URL, and then the specified server processes the user's request and finally returns the result to the user. In the process of using HTTP redirection to realize server cluster load balancing, a server needs to be used as a request dispatcher. A user's operation needs to initiate two HTTP requests, one to send a request to the dispatch server to obtain the IP of the backend server, and the second to send a request to the backend server to obtain the processing result. The reverse proxy server is a server located in front of the actual server, and all requests sent to the server first pass through the reverse proxy server. The server calls the AI server to process the user's request to generate a result and returns it to the user or directly returns it to the user.

[0121] It should be noted that in addition to Nginx, other tools can also achieve load balancing, for example, HAProxy (HTTP method).

[0122] The above only describes the preferred embodiments of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art can make various modifications and changes to the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing a 3D video stream, comprising: obtaining a 3D image frame from the 3D video stream; segmenting the 3D image frame into a plurality of segmented regions; determining a to-be-processed region in the 3D image frame and a plurality of to-be-adjusted pixels in the to-be-processed region, the to-be-adjusted pixels being pixels adjacent to edges of the plurality of segmented regions in a region where the to-be-processed region overlaps with the plurality of segmented regions; adjusting depth values of the plurality of to-be-adjusted pixels to obtain a processed 3D image frame; and generating a processed 3D video stream based on the processed 3D image frame. The adjusting of the depth values of the plurality of to-be-adjusted pixels comprises:

2. The method of claim 1, wherein, calculating an average depth value based on the depth values of the plurality of to-be-adjusted pixels; and for each of the plurality of to-be-adjusted pixels, modifying the depth value of the to-be-adjusted pixel based on depth values of a plurality of neighborhood pixels around the to-be-adjusted pixel and the average depth value. The modifying of the depth value of the to-be-adjusted pixel based on the depth values of the plurality of neighborhood pixels around the to-be-adjusted pixel and the average depth value comprises:

3. The method of claim 2, wherein, determining a maximum depth value among the depth values of the plurality of neighborhood pixels around the to-be-adjusted pixel; and modifying the depth value of the to-be-adjusted pixel to the maximum depth value. The modification of the depth value of each of the plurality of to-be-adjusted pixels is repeatedly performed a predetermined number of times.

4. The method of claim 2 or 3, wherein, The generating of the processed 3D video stream comprises:

5. The method of any one of claims 1 to 4, wherein, determining a same-view 3D image frame similar to the 3D image frame; adjusting depth values of pixels of the same-view 3D image frame to be the same as depth values of corresponding pixels of the processed 3D image frame; and replacing the 3D image frame and the same-view 3D image frame in the 3D video stream with a processed 3D image frame and a processed same-view 3D image frame, respectively, to generate the processed 3D video stream. The adjusting of the depth values of the pixels of the same-view 3D image frame comprises adjusting depth values of pixels in the same-view 3D image frame corresponding to the plurality of to-be-adjusted pixels of the 3D image frame.

6. The method of claim 5, wherein, The same-view 3D image frame is determined according to image similarity.

7. The method of claim 5, wherein, The determining of the same-view 3D image frame according to image similarity comprises:

8. The method of claim 7, wherein, determining, among the plurality of to-be-adjusted pixels of the 3D image frame, a minimum first horizontal pixel coordinate, a maximum second horizontal pixel coordinate, a minimum first vertical pixel coordinate, and a maximum second vertical pixel coordinate; determining a minimum first bounding rectangle based on the minimum first horizontal pixel coordinate, the maximum second horizontal pixel coordinate, the minimum first vertical pixel coordinate, and the maximum second vertical pixel coordinate; extending a size of the minimum first bounding rectangle to obtain an extended rectangle; determining similarity of images of the 3D image frame and a 3D image frame adjacent to the 3D image frame in the 3D video stream in the extended rectangle; and in response to the similarity being greater than a first threshold value, determining that the adjacent 3D image frame is the same-view 3D image frame. The same-view 3D image frame is determined according to a coincidence ratio of areas.

9. The method of claim 5, wherein, The determining of the same-view 3D image frame according to a coincidence ratio of areas comprises:

10. The method of claim 9, wherein, ​ segmenting a 3D image frame adjacent to the 3D image frame in the 3D video stream into a plurality of segmentation regions; determining a maximum segmentation region of the 3D image frame and a maximum segmentation region of the adjacent 3D image frame; determining, among pixels of adjacent edges in the maximum segmentation region of the 3D image frame, a minimum third horizontal pixel coordinate, a maximum fourth horizontal pixel coordinate, a minimum third vertical pixel coordinate, and a maximum fourth vertical pixel coordinate; determining a minimum second bounding rectangle based on the minimum third horizontal pixel coordinate, the maximum fourth horizontal pixel coordinate, the minimum third vertical pixel coordinate, and the maximum fourth vertical pixel coordinate; determining, among pixels of adjacent edges in the maximum segmentation region of the adjacent 3D image frame, a minimum fifth horizontal pixel coordinate, a maximum sixth horizontal pixel coordinate, a minimum fifth vertical pixel coordinate, and a maximum sixth vertical pixel coordinate; determining a minimum third bounding rectangle based on the minimum fifth horizontal pixel coordinate, the maximum sixth horizontal pixel coordinate, the minimum fifth vertical pixel coordinate, and the maximum sixth vertical pixel coordinate; calculating an overlap ratio of an area of an overlapping region of the minimum second bounding rectangle and the minimum third bounding rectangle and an area of a non-overlapping region; and in response to the overlap ratio being greater than a second threshold value, determining that the adjacent 3D image frame is the same-mirror 3D image frame.

11. The method of any one of claims 1 to 10, wherein, Segmenting the 3D image frame is based on an image segmentation model.

12. The method of claim 1, wherein, Determining the to-be-processed region in the 3D image frame includes: Determining the to-be-processed region based on a user-selected pixel.

13. The method of claim 1, wherein, Adjusting the depth values of the plurality of to-be-adjusted pixels includes: Adjusting the depth values of the plurality of to-be-adjusted pixels based on a plurality of depth values input by a user.

14. An apparatus for processing a 3D video stream, comprising: an obtaining module configured to obtain a 3D image frame from a 3D video stream; a segmenting module configured to segment the 3D image frame into a plurality of segmentation regions; a determining module configured to determine a to-be-processed region in the 3D image frame and a plurality of to-be-adjusted pixels in the to-be-processed region, the to-be-adjusted pixels being pixels adjacent to edges of the plurality of segmentation regions in a region where the to-be-processed region overlaps with the plurality of segmentation regions; an adjusting module configured to adjust depth values of the plurality of to-be-adjusted pixels; and a generating module configured to generate a processed 3D video stream based on the processed 3D image frame. The computer program instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 13.

15. A computer readable storage medium having stored thereon computer program instructions, wherein, 16. An apparatus for processing a 3D video stream, comprising: a processor; and a memory storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method according to any one of claims 1 to 13. The computer program instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 13.

17. A computer program product comprising computer program instructions, wherein, The computer program instructions, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 13.

Images