3D video generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a 3D video generation method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the image main body extraction of an image frame of an original video, and obtaining a sub-image; performing zooming processing on the sub-image to obtain a target sub-image; a background image required by the 3D video to be generated is acquired, at least one auxiliary background area is generated on the background image, and the image content of the auxiliary background area is different from that of the background image; the target subimage is superposed on the background image to obtain a 3D video frame, and the region where the target subimage is located and the auxiliary background region have an overlapping region; and generating a 3D video based on all the acquired 3D video frames. Therefore, the target subimage is superposed on the background image, the target subimage is ensured to be always located on the auxiliary background area, and the perspective effect of the target subimage and the background image is enhanced, so that the 3D video watching effect is created, the 3D video generation efficiency is improved, and the manufacturing cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a 3D video generation method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of science and technology and the improvement of people's living standards, more and more users are willing to experience new technological products, and naked-eye 3D technology and corresponding products are one of the technologies that users are willing to experience.

[0003] Among them, naked-eye 3D technology has been developed for a long time since 2013, but it is less popularized and its development is more reflected in the development of hardware. The development of ordinary 2D video to 3D video is extremely slow. Currently, the generation method of 3D video on the market is mainly manual processing. The generation method of 3D video is to manually cut out the video frames in the 2D video one by one, resulting in high labor costs and low production efficiency. Summary of the invention

[0004] The embodiments of the present application provide a 3D video generation method, device, electronic device and storage medium, which can improve the efficiency of 3D video generation and reduce manufacturing costs.

[0005] In a first aspect, an embodiment of the present application provides a 3D video generation method, comprising:

[0006] Extract the main image of the image frame of the original video to obtain a sub-image;

[0007] Performing scaling processing on the sub-image to obtain a target sub-image;

[0008] Acquire a background image required for a 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein image content of the auxiliary background area is different from image content of the background image;

[0009] The target sub-image is superimposed on the background image to obtain a 3D video frame, wherein the area where the target sub-image is located overlaps with the auxiliary background area;

[0010] A 3D video is generated based on all the acquired 3D video frames.

[0011] In a second aspect, an embodiment of the present application further provides a 3D video generating device, including:

[0012] A sub-image determination module is used to extract the image main body from the image frame of the original video to obtain a sub-image;

[0013] A target sub-image determination module, used for performing scaling processing on the sub-image to obtain a target sub-image;

[0014] A background image acquisition module, used to acquire a background image required for the 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein the image content of the auxiliary background area is different from the image content of the background image;

[0015] A 3D video frame determination module, used for superimposing the target sub-image onto the background image to obtain a 3D video frame, wherein an area where the target sub-image is located overlaps with the auxiliary background area;

[0016] The 3D video generation module is used to generate a 3D video based on all the acquired 3D video frames. In a third aspect, the embodiment of the present application also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute the steps of any one of the 3D video generation methods provided in the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for a processor to load to execute the steps of any one of the 3D video generation methods provided in the embodiments of the present application.

[0018] In a fifth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in any one of the 3D video generation methods provided in the embodiments of the present application.

[0019] By adopting the scheme of the embodiment of the application, the image frame of the original video can be extracted to obtain a sub-image; the sub-image can be scaled to obtain a target sub-image; the background image required for the 3D video to be generated is obtained, and at least one auxiliary background area is generated on the background image, and the image content of the auxiliary background area is different from the image content of the background image; the target sub-image is superimposed on the background image to obtain a 3D video frame, and the area where the target sub-image is located overlaps with the auxiliary background area; based on all the acquired 3D video frames, a 3D video is generated. By superimposing the target sub-image on the background image, it is ensured that the target sub-image is always located above the auxiliary background area, and the perspective effect of the target sub-image and the background image is enhanced, thereby creating a 3D video viewing effect, improving the efficiency of 3D video generation, and reducing manufacturing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of a system architecture of the 3D video generation method provided in the embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of an embodiment of a 3D video generation method provided in an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of determining a sub-image provided in an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of generating an auxiliary background area in an embodiment of the present application;

[0025] Figure 5 is another schematic diagram of generating an auxiliary background area in an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of the structure of a 3D video generating device provided in an embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0029] At the same time, in the description of the embodiments of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0030] Embodiments of the present application provide a 3D video generation method, device, electronic device, and computer-readable storage medium.

[0031] The 3D video generation method provided in the embodiment of the present application can be applied to the system architecture 100. Figure 1 ,like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0032] Users can use terminal devices 101 , 102 , 103 to interact with server 105 via network 104 to receive or send messages, etc.

[0033] The terminal devices 101, 102, 103 may be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Eperts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Eperts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. The terminal may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. The terminal and the server may perform two-way communication via a network.

[0034] The server 105 may be a server that provides various services, such as a background server that supports the pages displayed on the terminal devices 101, 102, and 103. The server may be an independent server, or a server network or server cluster composed of servers, including but not limited to a computer, a network host, a single network server, a plurality of network server sets, or a cloud server composed of a plurality of servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0035] It should be noted that the 3D video generation method provided in the embodiment of the present application is executed by a server, and accordingly, the 3D video generation device is disposed in the server.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. The terminal devices 101, 102, 103 in the embodiment of the present application may specifically correspond to application systems in actual production.

[0037] The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is an electronic device that can call a deduplication algorithm. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.

[0038] Please refer to Figure 2 The specific process of the 3D video generation method may be as follows: Step 201 to Step 205.

[0039] in:

[0040] Step 201: extract the main body of the image frame of the original video to obtain a sub-image.

[0041] The 3D video generation method of this embodiment can be applied to any scenario where the image frames of an original video are converted into 3D video frames by an electronic device, for example, an AIGC (AI Generated Content) scenario.

[0042] In this embodiment, the original video is a 2D video, and the image frames in the original video are image video frames with 2D effects. The image subject refers to the specific content in the image frame, and the specific content is the subject. The above sub-image is the image subject.

[0043] It is understandable that the image subject includes but is not limited to one subject or multiple subjects. For example, the image subject is a human subject in the image frame of the original video, or the image subject is multiple human subjects in the image frame of the original video, or the image subject is a human subject in the image frame of the original video and objects related to the human subject, such as skateboards, roller skates, etc. The number of subjects of the image subject is not limited in this embodiment.

[0044] It is understandable that the sub-image may be an image that includes only the main body of the image. The sub-image may also be an image of the same size as the video frame of the original video, and the area where the main body of the image is located in the sub-image is the same as the area where the main body of the image is located in the image frame of the original video. Preferably, the sub-image in this embodiment is an image that includes only the main body of the image.

[0045] It is understandable that the original video may be composed of multiple image frames containing the same image subject, or the original video may be composed of multiple image frames containing different image subjects. According to different types of original videos, different methods may be adopted to select the image subject of the image frames of the original video. When the original video frame includes multiple image frames of the same image subject, the image subject may be extracted simultaneously from the multiple image frames containing the same image subject in the original video, or the image subject may be extracted one by one from the multiple image frames containing the same image subject in the original video. It is preferred to extract the image subject simultaneously to improve the efficiency of image subject extraction.

[0046] In this example, if Figure 3 As shown, Figure 3 301 in the figure is an image frame of the original video. The image main body is extracted from the image frame of the original video to obtain a sub-image, such as Figure 3 303 in , and a background image example, such as Figure 3 302 in the.

[0047] In this embodiment, the method of acquiring the image frames of the original video includes but is not limited to acquiring all video frames of the original video, or acquiring video frames within a preset time period in the original video, or performing frame extraction processing on the original video. The video frames within a preset time period in the original video are acquired, such as acquiring all video frames from the 10th second to the 30th second in the original video. The frame extraction processing is performed on the original video, such as determining the extracted video frames from all video frames from the 10th second to the 30th second in the original video according to the rule of extracting one frame every 5 frames.

[0048] In this embodiment, the process of extracting the image body from the image frames of the original video can be parallel processing or serial processing. Parallel processing refers to extracting the image body from the image frames of the original video at the same time to obtain a sub-image sequence. Serial processing refers to extracting the image body from the image frames of the original video one by one to obtain a sub-image.

[0049] In an optional example, performing image body extraction on the image frame of the original video to obtain a sub-image may include: performing edge detection processing on the image frame of the original video based on an edge detection operator to obtain an image after edge detection; generating a detection sliding window, and using the detection sliding window to perform sliding window detection in the image after edge detection; performing image body extraction processing in the image after edge detection according to the number of pixels in the detection sliding window and an image body recognition rule, and using the extracted image, i.e., the sub-image, as the main part of the image frame of the original video.

[0050] Among them, the formats of the image frames of the original video may include: bmp, jpeg, png, etc. A variety of edge detection operators may be used, such as the Laplace operator, the Canny operator, etc. The detection sliding window includes a horizontal sliding window and a vertical sliding window. For example, four sliding windows of fixed size are respectively taken at the four edges of the image after edge detection. The horizontal sliding window is a sliding window that slides in the horizontal direction, and the vertical sliding window is a sliding window that slides in the vertical direction. The height of the horizontal sliding window is rounded to the height of the image, and the width is a fixed size. The number of pixels of the height of the horizontal sliding window is the number of pixels of the width of the image after edge detection. The height of the vertical sliding window is rounded to the width of the image, and the height is a fixed size. The number of pixels of the width of the vertical sliding window is the number of pixels of the length of the image after edge detection.

[0051] Optionally, in order to ensure the reliability of edge detection processing, before edge detection processing is performed on the image frames of the original video based on the edge detection operator, a smoothing filter is used to filter the image frames of the original video. A variety of smoothing filters can be used, such as a Gaussian filter, a Gaussian filter template is generated based on a Gaussian function and a filter scale, and a convolution operation is performed on the image frames of the original video using the Gaussian filter template, which can make the edges of the image frames of the original video sharper while suppressing image noise on the surface of the image to be processed.

[0052] In this embodiment, edge detection processing is performed on the image based on the edge detection operator and the image body is extracted by sliding window detection. There is no need to accurately set the points of the foreground area and the background area, which improves the processing efficiency of detecting and extracting sub-images.

[0053] In an optional example, extracting the image body from the image frame of the original video to obtain the sub-image may include:

[0054] Generate an image subject mask map according to the image frame of the original video;

[0055] Based on the image main body mask image, the image main body is extracted from the image frame of the original video to obtain a sub-image.

[0056] The image subject mask is used to represent the mask of the area where the image subject is located in the image frame of the original video. The image frame of the original video is masked by the image subject mask so that only the image subject can be displayed in the image frame, and the image subject is extracted to obtain a sub-image.

[0057] Optionally, the image subject mask map may be implemented by binarizing the image frame of the original video, and determining the image subject mask map by representing the area where the image subject is located with 255 and representing other areas with 0.

[0058] Optionally, the above-mentioned image subject mask map can be implemented by performing target detection on the image subject in the image frame of the original video, determining the outline of the image subject based on the detected target, and using the outline as the boundary line to distinguish the features of the area where the image subject is located from other areas to generate an image subject mask map.

[0059] Through the image subject mask map, the image subject of the image frame of the original video can be quickly extracted to obtain a sub-image, thereby improving the processing efficiency of detecting and extracting sub-images.

[0060] Step 202: Scaling the sub-image to obtain a target sub-image.

[0061] In this embodiment, the target sub-image is a sub-image that has been scaled.

[0062] It can be understood that the target sub-image is an image that only includes the main body of the image after the scaling process. Scaling includes reduction and enlargement, that is, when the scaling process is reduction, the scaling process is to reduce the sub-image according to the preset reduction ratio to obtain the target sub-image; when the scaling process is enlargement, the scaling process is to enlarge the sub-image according to the preset enlargement ratio to obtain the target sub-image, wherein the preset reduction ratio can be a percentage, a multiple, etc. For example, the preset reduction ratio is 10%, or the preset reduction ratio is 1 times reduction. The preset enlargement ratio can be a percentage, a multiple, etc. For example, the preset enlargement ratio is 10%, or the preset enlargement ratio is 1 times enlargement.

[0063] It is understandable that reducing the sub-image can also achieve the 3D visual effect of the video frame where the sub-image is located, but when the sub-image is reduced, the 3D visual effect achieved by the video frame where the sub-image is located is not as good as the 3D visual effect achieved by the video frame where the sub-image is located when the sub-image is enlarged. Preferably, the embodiment of the present application adopts a preset magnification ratio of 1.2 times to scale the sub-image to obtain the target sub-image.

[0064] In a specific implementation, performing scaling processing on the sub-image to obtain a target sub-image may include:

[0065] In the sub-image, calculating a bounding box of the image body;

[0066] Determine the center of the bounding box as the scaling control point of the sub-image;

[0067] The sub-image is scaled based on the scaling control point to obtain a target sub-image.

[0068] In an optional example, the method for determining the scaling control point of the sub-image may include determining a bounding box corresponding to the sub-image, and determining the scaling control point of the sub-image according to the bounding box corresponding to the sub-image.

[0069] The bounding box is used to represent the optimal bounding space of the image subject in the sub-image. The bounding box algorithm is used to determine the bounding box corresponding to the sub-image. The bounding box algorithm refers to an algorithm for solving the optimal bounding space of a discrete point set. The bounding box includes but is not limited to a bounding sphere, a directional bounding box, and a fixed-direction convex bounding box.

[0070] Optionally, a rectangular bounding box corresponding to the sub-image is determined, and the length and width of the rectangular bounding box corresponding to the sub-image are determined according to the extension direction of the sub-image. If the extension direction of the sub-image is horizontal arrangement, the horizontal length of the sub-image is used as the length of the rectangular bounding box corresponding to the sub-image, and the vertical length of the sub-image is used as the width of the rectangular bounding box corresponding to the sub-image.

[0071] For example, when the present application adopts a preset magnification ratio of 1.2 times, the scaling process of the sub-image specifically includes scaling the sub-image with a scaling control point determined from the bounding box as the center and adopting a preset magnification ratio of 1.2 times to obtain a target sub-image.

[0072] In an optional example, the method for determining the scaling control point of a sub-image may include arbitrarily selecting a point from the sub-image as the scaling control point of the sub-image, scaling the sub-image with the scaling control point as a fixed point, and obtaining a target sub-image.

[0073] For example, when the present application adopts a preset magnification ratio of 1.2 times, the scaling processing of the sub-image specifically includes taking the scaling control point as a fixed point and expanding the distance between other points in the sub-image to 1.2 times to achieve the scaling processing of the sub-image and obtain the target sub-image.

[0074] Further, in one embodiment, if the video frame of the original video includes multiple image bodies, multiple sub-images are included in the same video frame, and the sub-images are scaled to obtain a target sub-image, which may include:

[0075] determining a primary sub-image from among all sub-images;

[0076] In the primary sub-image, calculating a bounding box of the image body;

[0077] Determining the center of the bounding box as a scaling control point of the main sub-image;

[0078] All the sub-images are scaled based on the scaling control point to obtain a target sub-image.

[0079] In this embodiment, the method for determining the main sub-image may be to determine the area ratio of the image subject corresponding to each sub-image in the video frame of the original video, and to use the sub-image with the largest area ratio as the main sub-image. The method for determining the main sub-image may also be to specify the main sub-image from all sub-images according to a user request.

[0080] In one embodiment, if the video frame of the original video includes multiple image subjects, multiple sub-images are included in the same video frame, and the sub-images are scaled to obtain a target sub-image, which may include:

[0081] In all the sub-images, a bounding box including the image bodies corresponding to all the sub-images is calculated;

[0082] Determine the center of the bounding box as a scaling control point;

[0083] All the sub-images are scaled based on the scaling control point to obtain a target sub-image.

[0084] The method for determining the common scaling control point of all sub-images may include determining a bounding box that can contain each sub-image, and determining the common scaling control point of all sub-images according to the bounding box.

[0085] Step 203: Acquire a background image required for the 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein the image content of the auxiliary background area is different from the image content of the background image.

[0086] In this embodiment, the background image is used to represent the background image of the 3D video to be generated. The background image may be the background image of the original video or a user-defined background image.

[0087] In this embodiment, the auxiliary background area is located above the image layer where the background image is located. Generating at least one auxiliary background area on the background image may be to first generate the auxiliary background area, establish a transparent layer on the image layer where the background image is located, and render the auxiliary background area onto the transparent layer. Generating at least one auxiliary background area on the background image may also be to directly generate the auxiliary background area on the background image according to relevant parameters of the auxiliary background area, and the image content of the auxiliary background area is different from the image content of the background image. The image content of the auxiliary background area includes, but is not limited to, pure color image content and non-pure color image content. For example, the image content of the auxiliary background area is black image content. The image content of the auxiliary background area can be specifically adjusted according to actual conditions, and this embodiment does not impose specific restrictions.

[0088] In this embodiment, generating at least one auxiliary background area on the background image may also be directly generating the auxiliary background area on the background image to highlight an area different from the background image.

[0089] In an optional example, the method for obtaining the background image required for the 3D video to be generated may be: obtaining a background image preset by a user. The background image preset by the user may be the same image as the background image of the original video, or the background image preset by the user may be different from the background image of the original video. When the background image preset by the user is different from the background image of the original video, the background switching function of the original video can be easily implemented.

[0090] For example, the background image of the original video is the first background image, which is the seaside, and the background image preset by the user is the second background image, which is the land. The image content of the first background image is different from the image content of the second background image. In the subsequent image superposition process, the background switching function of the original video is easily realized, which improves the personalized experience of users in making videos.

[0091] In another optional example, a method for obtaining a background image required for a 3D video to be generated may be: obtaining an image frame in which the image subject does not exist after image subject extraction is performed on an image frame of the original video; performing image restoration on an area in the image frame where the image subject is located based on image content in a background area of ​​the image frame, to obtain the background image required for the 3D video to be generated, wherein the background area is an area excluding an area where the image subject is located.

[0092] Among them, the image restoration method includes but is not limited to restoring the image based on an image restoration model and restoring the image based on image content.

[0093] In an optional example, a method for repairing an image based on an image repair model may include pre-training an image repair model, optimizing parameters of a generator and a discriminator in the image repair model, inputting the image to be repaired into the generator of the trained image repair model, and generating a first image in which the area to be repaired has semantic information but is blurred; searching for similar blocks from images in a database of the pre-trained model based on image blocks of the area to be repaired and surrounding areas in the first image; replacing image blocks of the area to be repaired in the first image with image blocks corresponding to the area to be repaired in the similar blocks to obtain a second image, and fusing the repaired boundaries of the second image.

[0094] In an optional example, a method for repairing an image based on image content may include mean interpolation, that is, for the pixel points corresponding to the area where the image subject is located in the image frame, a repair frame of a preset size with each pixel point as the center redetermines the pixel value of the pixel point from the image content of other areas of the image frame. For example, a repair frame of a preset size is constructed with a pixel point as the center and the other eight pixels around the pixel point. Then, for the pixel point corresponding to the area where the image subject is located in the image frame, the pixel values ​​corresponding to the eight pixels around the pixel point are evenly distributed to the pixel point and updated to the repaired image. The image is iteratively updated multiple times until the result is satisfactory, ensuring the effect and overall performance of the image repair. The number of iterations can be dynamically adjusted according to the size of the area where the image subject is located to ensure the effect and overall performance of the image repair.

[0095] In a specific implementation, generating at least one auxiliary background area on the background image may include:

[0096] Acquire a mask image, wherein the mask image includes at least one mask area;

[0097] Based on the mask image, generating an auxiliary background area corresponding to the mask area on the background image;

[0098] or,

[0099] The position information corresponding to the auxiliary background area in the background image is determined, and the pixel points on the position information are replaced with preset background pixel points.

[0100] In this embodiment, the mask image is used to represent a mask image that highlights the main body of the image. The image content of the auxiliary background area is different from the image content of the background image. The above-mentioned method of obtaining the mask image includes but is not limited to user setting and random generation.

[0101] In this embodiment, the method for obtaining the mask layer may be to obtain mask parameters and generate a mask image according to the mask parameters. The mask parameters include but are not limited to the size of the mask image, the size of the mask area, and the mask content of the mask area. Figure 4 As shown, in Figure 4 middle, Figure 4 302 in the figure is a background image example. Figure 4 304 in FIG. 1 is a mask image, and the mask image has the same size as the background image. Figure 4 As shown in 304, there are two mask areas on the mask image, which are the top 1 / 4 area of ​​the mask image and the bottom 1 / 4 area of ​​the mask image. The mask content is black, that is, the mask area is the black area corresponding to the top 1 / 4 area of ​​the mask image and the black area corresponding to the bottom 1 / 4 area of ​​the mask image. Figure 4As shown in 305, the mask image is superimposed on the background image to generate an auxiliary background area corresponding to the mask area.

[0102] In this embodiment, a mask image may be superimposed on the background image to generate an auxiliary background area corresponding to the mask area on the background image. Alternatively, mask image parameters may be determined and an auxiliary background area corresponding to the mask area may be generated at a position on the background image corresponding to the mask image parameters.

[0103] In this embodiment, through the mask image, it is possible to control the generation of a mask area on the background image to ensure that the target sub-image is always located above the auxiliary background area, thereby enhancing the perspective effect of the target sub-image and the background image, thereby creating a 3D video viewing effect, improving the efficiency of 3D video generation, and reducing manufacturing costs.

[0104] In the present embodiment, the position information includes but is not limited to coordinate position information and row and column position information. The coordinate position information refers to the coordinate position information of the auxiliary background area on the background image. For example, taking the lower left corner of the background image as the midpoint of the coordinate system, the coordinate position information corresponding to the auxiliary background area is that the ordinate is less than 0.25 and the ordinate is greater than 0.75, then the pixel points in the area of ​​the background image with the ordinate less than 0.25 are replaced with preset background pixel points, and the pixel points in the area of ​​the background image with the ordinate greater than 0.75 are replaced with preset background pixel points. The row and column position information corresponding to the auxiliary background area refers to the position information of the auxiliary background area in the row and column diagram corresponding to the background image. For example, a row and column diagram corresponding to the background image is generated, assuming that the row and column position information corresponding to the auxiliary background area is from the 1st row to the 3rd row, then the pixel points in the area corresponding to the 1st row to the 3rd row in the row and column diagram corresponding to the background image are replaced with preset background pixel points.

[0105] The preset background pixel points include but are not limited to background pixel points with preset pixel values, for example, background pixel points with a preset pixel value of 0.

[0106] In a specific implementation, generating at least one auxiliary background area on the background image may include:

[0107] Acquire an extension direction of the image subject on the image frame;

[0108] Determine a target image edge of the extended pixels in the background image that intersects the extension direction;

[0109] Preset background pixels are extended outward from the edge of the target image to generate at least one auxiliary background area on the background image.

[0110] In this embodiment, the extension direction refers to the arrangement of the sub-images in the horizontal or vertical direction. The extension direction of the sub-images can be obtained by determining the pixel ratio of the sub-images in the horizontal or vertical direction, and taking the direction with the larger pixel ratio as the extension direction of the sub-image.

[0111] In this embodiment, if the extension direction of the sub-image is horizontal arrangement, the target image edge of the extended pixels in the background image is determined to be arranged vertically, and the preset background pixels are extended outwardly from the target image edge to generate an auxiliary background area on the background image. If the extension direction of the sub-image is vertical arrangement, the target image edge of the extended pixels in the background image is determined to be arranged horizontally, and the preset background pixels are extended outwardly from the target image edge to generate an auxiliary background area on the background image.

[0112] In this embodiment, if Figure 5 As shown, according to the longitudinal extension direction of the sub-image, the extension direction of the target image edge of the extended pixel in the background image is determined as the horizontal extension direction, and the preset background pixel points are extended horizontally outward from the target image edge to generate an auxiliary background area on the background image, and the Figure 5 307 of them.

[0113] In a specific implementation, generating at least one auxiliary background area on the background image may include:

[0114] Get auxiliary background area parameters;

[0115] At least one auxiliary background region is generated on the background image based on the auxiliary background region parameters.

[0116] In this embodiment, the auxiliary background region parameter refers to the region parameter used in the rendering system, that is, according to the region parameter used in the rendering system, an auxiliary background region corresponding to the region parameter is generated on the background image.

[0117] In this embodiment, the auxiliary background area parameters include but are not limited to area coordinates and area fill color. For example, if the area coordinates are uv.y<0.25 and uv.y>0.75, and the area fill color is black, that is, the top 1 / 4 area of ​​the background image and the bottom 1 / 4 area of ​​the background image are filled with black, and the two areas filled with black are both auxiliary background areas.

[0118] Step 204: superimpose the target sub-image onto the background image to obtain a 3D video frame, wherein the area where the target sub-image is located overlaps with the auxiliary background area.

[0119] In this embodiment, by constructing an image layer corresponding to the target sub-image, in the image layer corresponding to the target sub-image, except for the target sub-image, other areas are invisible areas, and the target sub-image is superimposed on the background image to obtain a 3D video frame. When the target sub-image is superimposed on the background image, the target sub-image can be superimposed on the position where the image main body is located on the background image, or the target sub-image can be superimposed on other positions of the background image. The superimposed position can be specifically set according to actual conditions.

[0120] In this embodiment, if Figure 4 306 or Figure 5 As shown in 308, the Figure 4 306 or Figure 5 308 in all of them are to superimpose the target sub-image on the background image to obtain a 3D video frame. By superimposing the target sub-image on the background image, it is ensured that the target sub-image is always located above the auxiliary background area, which enhances the perspective effect of the target sub-image and the background image, thereby creating a 3D video viewing effect, improving the efficiency of 3D video generation, and reducing manufacturing costs. It can be understood that the layer order of the 3D video frame of this embodiment is the layer corresponding to the background image, the layer corresponding to the auxiliary background area, and the layer corresponding to the target sub-image. The auxiliary background area is used to highlight the subject and create a 3D effect.

[0121] In an optional example, before superimposing the target sub-image onto the background image to obtain the 3D video frame, the following steps may also be included:

[0122] Get fuzzy parameters;

[0123] Based on the blur parameter, blur processing is performed on the background image.

[0124] In this embodiment, the blur parameters include but are not limited to standard deviation and transparency. The method of obtaining the blur parameters can be user-defined settings or system default settings. The blur parameters are used to control the degree and effect of blurring the background image.

[0125] Optionally, the blurring method includes but is not limited to Gaussian blur, motion blur, etc.

[0126] In an optional example, Gaussian blur is used to take the average value of the surrounding pixels of each pixel in the background image, and the pixel value corresponding to the pixel is updated according to the obtained average value. For example, there is an intermediate pixel point, the pixel value of the pixel point is 2, and the average value of the pixels around the pixel point is 1. The pixel value corresponding to the pixel point is updated by the average value, and the pixel value of the pixel point after the update is 1. In terms of numerical value, the pixel points are smoothed, and in terms of graphics, a blurring effect is produced, causing the intermediate pixel points to lose details. Through Gaussian blur, noise in the image is removed, edges are smoothed, etc.

[0127] In another optional example, by adjusting the size and standard deviation of the blur kernel in the Gaussian blur, each pixel in the background image is updated, thereby blurring the background image, and the degree and effect of the blur can be controlled.

[0128] Step 205: Generate a 3D video based on all the acquired 3D video frames.

[0129] In this embodiment, the 3D video may be generated by splicing the 3D video frames based on the timestamps of all the 3D video frames obtained to obtain the 3D video. The 3D video may also be generated by constructing a 3D video frame sequence based on the obtained 3D video frames, and inputting the 3D video frame sequence into a rendering system for rendering to generate the 3D video.

[0130] It is understandable that the 3D video is only a video with a 3D visual effect. The 3D video enhances the perspective effect of the target sub-image and the background image.

[0131] By adopting the solution of the application embodiment, by superimposing the target sub-image on the background image, it is ensured that the target sub-image is always located above the auxiliary background area, thereby enhancing the perspective effect of the target sub-image and the background image, thereby creating a 3D video viewing effect, improving the efficiency of 3D video generation, and reducing manufacturing costs.

[0132] This embodiment further provides a 3D video generating device, which can be integrated into an electronic device, such as a computer device. The computer device can be a terminal, a server, or other device, and this embodiment has no limitation on this.

[0133] For example, Figure 6 As shown, the 3D video generating device may include:

[0134] The sub-image determination module 401 is used to extract the image main body from the image frame of the original video to obtain a sub-image;

[0135] A target sub-image determination module 402 is used to perform scaling processing on the sub-image to obtain a target sub-image;

[0136] A background image acquisition module 403 is used to acquire a background image required for the 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein the image content of the auxiliary background area is different from the image content of the background image;

[0137] A 3D video frame determination module 404 is used to superimpose the target sub-image onto the background image to obtain a 3D video frame, wherein the area where the target sub-image is located overlaps with the auxiliary background area;

[0138] The 3D video generation module 405 is configured to generate a 3D video based on all the acquired 3D video frames.

[0139] Optionally, in the device of the embodiment of the present application, generating at least one auxiliary background area on the background image includes:

[0140] A mask image acquisition unit, used to acquire a mask image, wherein the mask image includes at least one mask area;

[0141] The first auxiliary background region generating unit is used to generate an auxiliary background region corresponding to the mask region on the background image based on the mask image.

[0142] Optionally, in the device of the embodiment of the present application, generating at least one auxiliary background area on the background image includes:

[0143] An extension direction acquisition unit, used to acquire the extension direction of the sub-image;

[0144] a target image edge determination unit, configured to determine a target image edge of the extended pixels in the background image according to the extension direction;

[0145] The second auxiliary background region generating unit is used to expand preset background pixels outward from the edge of the target image to generate at least one auxiliary background region on the background image.

[0146] Optionally, in the device of the embodiment of the present application, the sub-image determination module 401 includes:

[0147] An image subject mask map generating unit, used for generating an image subject mask map according to an image frame of an original video;

[0148] The sub-image determining unit is used to extract the image body of the image frame of the original video based on the image body mask map to obtain a sub-image.

[0149] Optionally, in the device of the embodiment of the present application, the step of obtaining a background image required for the 3D video to be generated includes:

[0150] An image frame determining unit, configured to obtain an image frame in which the image subject does not exist after image subject extraction is performed on the image frame of the original video;

[0151] The image restoration unit is used to perform image restoration on the area where the image subject is located in the image frame based on the image content of the background area of ​​the image frame to obtain the background image required for the 3D video to be generated, wherein the background area is the area other than the area where the image subject is located.

[0152] Optionally, in the device of the embodiment of the present application, the target sub-image determination module 402 includes:

[0153] A zoom control point determination unit, used to determine the zoom control point of the sub-image;

[0154] The target sub-image determining unit is used to perform scaling processing on the sub-image with the scaling control point as the center to obtain the target sub-image.

[0155] Optionally, in the apparatus of the embodiment of the present application, before the 3D video frame determination module 404, the apparatus further includes:

[0156] A fuzzy parameter determination unit, used for obtaining fuzzy parameters;

[0157] The blurring unit is used to blur the background image based on the blurring parameter. Through the 3D video generation device, the target sub-image is superimposed on the background image to ensure that the target sub-image is always located above the auxiliary background area, thereby enhancing the perspective effect of the target sub-image and the background image, thereby creating a 3D video viewing effect, improving the efficiency of 3D video generation, and reducing manufacturing costs.

[0158] Accordingly, an embodiment of the present application further provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and other terminal devices. Alternatively, the electronic device may be a server.

[0159] like Figure 7 As shown, Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or arrange components differently.

[0160] The processor 501 is the control center of the electronic device 500, and uses various interfaces and lines to connect various parts of the entire electronic device 500, and executes various functions and processes data of the electronic device 500 by running or loading software programs and / or units stored in the memory 502, and calling data stored in the memory 502. The processor 501 can be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0161] In the embodiment of the present application, the processor 501 in the electronic device 500 will load instructions corresponding to the processes of one or more application programs into the memory 502 according to the following steps, and the processor 501 will run the application programs stored in the memory 502 to implement various functions, such as:

[0162] Extract the main image of the image frame of the original video to obtain a sub-image;

[0163] Performing scaling processing on the sub-image to obtain a target sub-image;

[0164] Acquire a background image required for a 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein image content of the auxiliary background area is different from image content of the background image;

[0165] The target sub-image is superimposed on the background image to obtain a 3D video frame, wherein the area where the target sub-image is located overlaps with the auxiliary background area;

[0166] A 3D video is generated based on all the acquired 3D video frames.

[0167] Furthermore, various functions implemented by running the application stored in the memory 502 can also be found in the description of the aforementioned embodiments, which will not be repeated here.

[0168] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0169] Optional, such as Figure 7 As shown, the electronic device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. The processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507, respectively. Those skilled in the art can understand that Figure 7The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0170] The touch display screen 503 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize the input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 503 can also be used as a part of the input unit 506 to realize the input function.

[0171] The radio frequency circuit 504 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals between the network device or other electronic devices.

[0172] The audio circuit 505 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 505 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 505 and converted into audio data, and then the audio data is output to the processor 501 for processing, and then sent to another electronic device through the radio frequency circuit 504, or the audio data is output to the memory 502 for further processing. The audio circuit 505 may also include an earplug jack to provide communication between an external headset and an electronic device.

[0173] The input unit 506 may be configured to receive input video data information (eg, original video) and control information (eg, frame extraction information, sub-image scaling control information, etc.).

[0174] The power supply 507 is used to supply power to various components of the electronic device 500. Optionally, the power supply 507 can be logically connected to the processor 501 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 507 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0175] although Figure 7 Not shown, the electronic device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0176] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0177] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0178] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute any of the 3D video generation methods provided in the embodiments of the present application. For example, the computer program can execute the following steps of the 3D video generation method:

[0179] Extract the main image of the image frame of the original video to obtain a sub-image;

[0180] Performing scaling processing on the sub-image to obtain a target sub-image;

[0181] Acquire a background image required for a 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein image content of the auxiliary background area is different from image content of the background image;

[0182] The target sub-image is superimposed on the background image to obtain a 3D video frame, wherein the area where the target sub-image is located overlaps with the auxiliary background area;

[0183] Based on all the acquired 3D video frames, a 3D video is generated. Further, the detailed steps of the above method steps can also refer to the description in the above embodiments, which will not be repeated here.

[0184] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0185] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0186] Since the computer program stored in the computer-readable storage medium can execute any of the 3D video generation methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the 3D video generation methods provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0187] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in various optional implementations of the above embodiments.

[0188] In the above-mentioned 3D video generation device, computer-readable storage medium, electronic device, and computer program product embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the above-mentioned 3D video generation device, computer-readable storage medium, computer program product, electronic device and its corresponding units can refer to the description of the 3D video generation method in the above embodiment, and will not be repeated here.

[0189] The above is a detailed introduction to a 3D video generation method, device, electronic device, computer-readable storage medium and computer program product provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A 3D video generation method, characterized in that: include: Extract the main image of the image frame of the original video to obtain a sub-image; Performing scaling processing on the sub-image to obtain a target sub-image; Acquire a background image required for a 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein image content of the auxiliary background area is different from image content of the background image; The target sub-image is superimposed on the background image to obtain a 3D video frame, wherein the area where the target sub-image is located overlaps with the auxiliary background area; A 3D video is generated based on all the acquired 3D video frames.

2. The 3D video generation method according to claim 1, characterized in that: The step of generating at least one auxiliary background area on the background image comprises: Acquire a mask image, wherein the mask image includes at least one mask area; Based on the mask image, generating an auxiliary background area corresponding to the mask area on the background image; or, The position information corresponding to the auxiliary background area in the background image is determined, and the pixel points on the position information are replaced with preset background pixel points.

3. The 3D video generation method according to claim 1, characterized in that: The step of generating at least one auxiliary background area on the background image comprises: Acquire an extension direction of the image subject on the image frame; Determining a target image edge in the background image that intersects with the extension direction; Preset background pixels are extended outward from the edge of the target image to generate at least one auxiliary background area on the background image.

4. The 3D video generation method according to claim 1, characterized in that: The step of performing scaling processing on the sub-image to obtain a target sub-image includes: In the sub-image, calculating a bounding box of the image body; Determine the center of the bounding box as the scaling control point of the sub-image; The sub-image is scaled based on the scaling control point to obtain a target sub-image.

5. The 3D video generation method according to claim 1, characterized in that: The extracting the main body of the image frame of the original video to obtain a sub-image includes: Generate an image subject mask map according to the image frame of the original video; Based on the image main body mask image, the image main body is extracted from the image frame of the original video to obtain a sub-image.

6. The 3D video generation method according to claim 4, characterized in that: Before generating at least one auxiliary background area on the background image, the method includes: After extracting the image subject from the image frame of the original video, an image frame where the image subject does not exist is obtained; Image restoration is performed on the area where the image subject is located in the image frame based on the image content of the background area of ​​the image frame to obtain a background image required for the 3D video to be generated, wherein the background area is an area other than the area where the image subject is located.

7. The 3D video generation method according to any one of claims 1 to 6, characterized in that: Before superimposing the target sub-image onto the background image to obtain a 3D video frame, the method further includes: Get fuzzy parameters; Based on the blur parameter, blur processing is performed on the background image.

8. A 3D video generation device, characterized in that: include: A sub-image determination module is used to extract the image main body from the image frame of the original video to obtain a sub-image; A target sub-image determination module, used for performing scaling processing on the sub-image to obtain a target sub-image; A background image acquisition module, used to acquire a background image required for the 3D video to be generated, and generate at least one auxiliary background area on the background image, wherein the image content of the auxiliary background area is different from the image content of the background image; A 3D video frame determination module, used for superimposing the target sub-image onto the background image to obtain a 3D video frame, wherein an area where the target sub-image is located overlaps with the auxiliary background area; The 3D video generation module is used to generate a 3D video based on all the acquired 3D video frames.

9. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps of the 3D video generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps of the 3D video generation method according to any one of claims 1 to 7.

Images