Method and device for creating 3D lens movement based on picture depth of field, and storage medium

Abstract

This invention relates to a method, apparatus, and storage medium for creating 3D lens motion based on image depth of field. The method includes: editing and preprocessing different depth regions of an input image to obtain a depth information map; acquiring lens motion information; and calculating the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region of the depth information map to obtain a 3D image of the input image. This application, without establishing a 3D model, calculates a 3D image of the input image based on the lens motion information and the depth information of pixels in each depth region. Depth fault regions are offset by pixel offset and not displayed. Since there are relatively few depth fault regions in the entire image, displaying them as a 3D image will not affect the overall visual effect, thus avoiding the problem in existing technologies where the transition of depth fault regions is stiff and unnatural, making it unsuitable for images with large depth of field.

Description

Technical Field

[0001] This invention relates to the field of image and video processing technology, and more specifically to a method, apparatus and storage medium for creating 3D camera motion based on image depth of field. Background Technology

[0002] Currently, most methods for reconstructing camera motion using depth information maps involve actual modeling. This means using depth information to create a 3D model and then moving the camera in 3D space to achieve perspective transformations. The main problem with this method is that the transitions in depth fault areas are stiff and unnatural, making it unsuitable for images with large depth of field. In Adobe Effect, there is a Volumax landscape template that uses multi-layer stacking and displacement map technology to achieve 3D perspective transformations without relying on modeling. However, this template is more resource-intensive and not suitable for mobile devices. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method, apparatus and storage medium for creating 3D camera motion based on image depth of field, so as to solve the problem in the prior art that the method of actually building a 3D model using depth of field information and obtaining the perspective effect by moving the camera in three-dimensional space has a stiff and unnatural transition in the depth fault area, and is not suitable for images with large depth of field.

[0004] According to a first aspect of the present invention, a method for creating 3D camera motion based on image depth of field is provided, comprising:

[0005] The depth-of-field map is obtained by editing different depth regions of the input image;

[0006] Preprocessing the depth-of-field segmentation map yields a depth-of-field information map;

[0007] Acquire camera motion information;

[0008] Based on the motion information of the lens and the depth information of the pixels in each depth region in the depth information map, the offset information of the pixels in each depth region is calculated to obtain the 3D image of the input image.

[0009] By tilting the canvas, 3D images can be obtained from different angles;

[0010] The display area is cropped to obtain the final output image.

[0011] Preferably,

[0012] The motion information of the lens includes the lens's advance distance, pitch angle, and yaw angle.

[0013] Preferably,

[0014] The step of calculating the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region of the depth information map includes:

[0015] Based on the depth information and advance distance information of each depth region in the depth information map, the offset information of each pixel in the depth region from the center point is calculated by a preset translation formula.

[0016] Preferably, it further includes:

[0017] Based on the depth information and pitch angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the X-axis is calculated by a preset deflection formula.

[0018] Based on the depth information and yaw angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the Y-axis is calculated by using a preset yaw formula.

[0019] Preferably,

[0020] The preprocessing of the depth-of-field segmentation map to obtain the depth-of-field information map includes:

[0021] Feathering, blurring, and overlay operations are performed on each depth region in the depth field segmentation map to obtain a depth information map. The gray value of each pixel in the depth information map represents the depth value corresponding to that pixel.

[0022] Preferably,

[0023] The process of editing different depth regions of the input image to obtain a depth-of-field map includes:

[0024] Based on the different depth regions of the input image, it is divided into foreground, midground, and background regions to obtain a depth-of-field map.

[0025] According to a second aspect of the present invention, an apparatus for creating 3D camera motion based on image depth of field is provided, comprising:

[0026] Depth of field generation module: used to edit different depth regions of the input image to obtain a depth of field map, and to preprocess the depth of field map to obtain a depth information map;

[0027] Lens motion module: Used to acquire lens motion information, calculate the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region in the depth information map, and obtain the 3D image of the input image;

[0028] It is also used to deflect the canvas to obtain 3D images from different angles;

[0029] Output module: Used to crop the display area to obtain the final output image.

[0030] According to a third aspect of the present invention, a storage medium is provided, the storage medium storing a computer program, which, when executed by a host controller, implements the steps of the above-described method.

[0031] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0032] This application obtains a depth-of-field segmentation map by editing different depth regions of the input image; preprocesses the depth-of-field segmentation map to obtain a depth information map; acquires the lens motion information; and calculates the offset information of the pixels in each depth region based on the lens motion information and the depth information of the pixels in each depth region to obtain a 3D image of the input image. This application, without establishing a 3D model, calculates a 3D transformed image of the input image based on the lens motion information and the depth information of the pixels in each depth region. The depth fault regions are offset by the pixel offsets and are not displayed. Since the depth fault regions are relatively few in the entire image, displaying them as a 3D image will not affect the overall visual effect, thus avoiding the problem in existing technologies where the transition of depth fault regions is stiff and unnatural, making it unsuitable for images with large depth of field.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0035] Figure 1 This is a flowchart illustrating a method for creating 3D camera motion based on image depth of field according to an exemplary embodiment;

[0036] Figure 2 This is a schematic diagram of a system for creating 3D camera motion based on image depth of field, according to an exemplary embodiment.

[0037] Figure 3 This is a schematic diagram illustrating an image in a camera according to another exemplary embodiment;

[0038] Figure 4 This is a top view of a 3D scene illustrated according to another exemplary embodiment;

[0039] Figure 5 This is a schematic diagram of a new 3D scene in an offset scene, as shown according to another exemplary embodiment;

[0040] Figure 6 This is a schematic diagram of the imaging of a new 3D scene in an offset scene in a camera, according to another exemplary embodiment.

[0041] Figure 7 This is a schematic diagram of a new 3D scene in a translational scene, according to another exemplary embodiment;

[0042] Figure 8 This is a schematic diagram of the imaging of a new 3D scene in a panning scene in a camera, according to another exemplary embodiment.

[0043] In the attached diagram: 1-Depth of field generation module, 2-Lens motion module, 3-Output module. Detailed Implementation

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0045] Example 1

[0046] Figure 1 This is a flowchart illustrating a method for creating 3D camera motion based on image depth of field according to an exemplary embodiment, such as... Figure 1 As shown, the method includes:

[0047] S1, edit different depth regions of the input image to obtain a depth-of-field map;

[0048] S2, preprocess the depth-of-field segmentation map to obtain the depth-of-field information map;

[0049] S3, acquires lens motion information;

[0050] S4. Based on the motion information of the lens and the depth information of the pixels in each depth region in the depth information map, calculate the offset information of the pixels in each depth region to obtain the 3D image of the input image.

[0051] S5, rotate the canvas to obtain 3D images from different angles;

[0052] S6, perform a cropping operation on the display area to obtain the final output image;

[0053] Understandably, in this application, an input image is edited according to depth regions to obtain a depth-of-field map. This depth-of-field map is then preprocessed to obtain a depth information map. Lens motion information is acquired, and based on the lens motion information and the depth information of pixels in each depth region of the depth information map, pixel offset information is calculated to construct a 3D image of the input image. The canvas is then deflected; this deflection can be understood as the camera observing the 3D image from different angles to enhance the 3D perspective effect. Finally, the 3D image is cropped to obtain the final output image. In this application, without establishing a 3D model, a 3D transformed image of the input image is calculated based on the lens motion information and the depth information of pixels in each depth region. Depth fault regions are offset by pixel offset and not displayed. Since there are relatively few depth fault regions in the entire image, displaying them through a 3D image will not affect the overall visual effect, thus avoiding the problem in existing technologies where the transition of depth fault regions is stiff and unnatural, making it unsuitable for images with large depth of field.

[0054] Preferably,

[0055] The motion information of the lens includes the lens's advance distance, pitch angle, and yaw angle.

[0056] It is understandable that the advance distance information is the forward and backward movement of the camera, and the pitch angle information and yaw angle information are the turning of the camera. By obtaining the motion information of the camera, the image can be offset, and the offset information can be calculated based on the offset operation.

[0057] Preferably,

[0058] The step of calculating the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region of the depth information map includes:

[0059] Based on the depth information and advance distance information of each depth region in the depth information map, the offset information of each pixel in the depth region from the center point is calculated by a preset translation formula.

[0060] Understandably, as shown in the attached document Figure 4 As shown, a top-down view of a 3D scene is displayed, and its image in the camera is attached. Figure 3 As shown, by moving the view frustum forward, mimicking the forward movement of the camera, a new 3D scene is obtained, as shown in the attached image. Figure 7 As shown, its image in the camera at this time will be as shown in the attached image. Figure 8As shown, the size change of distant objects is smaller than that of nearby objects. Analyzing this process, we can see that during the camera's forward and backward movement, the offset of any pixel from the center point on the imaging screen is inversely proportional to the pixel's depth. For a pixel with depth d (0 <= d <= 1), assuming the forward and backward movement is p, the offset of this pixel from the center point is k2*p*(1.0-d), where k2 is a translation constant. Based on this conclusion, we can offset the position of any point on the input image according to the camera's movement.

[0061] Preferably, it further includes:

[0062] Based on the depth information and pitch angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the X-axis is calculated by a preset deflection formula.

[0063] Based on the depth information and yaw angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the Y axis is calculated by using a preset yaw formula.

[0064] Understandably, taking the horizontal plane offset (yaw angle rotation) as an example, [the following text is incomplete and requires further context]. Figure 4 A top-down view of a 3D scene is shown, as illustrated in the attached image. Figure 3 As shown, here we rotate the view frustum to the left, mimicking the process of the camera rotating to the left, to obtain a new 3D scene (as shown in the attached diagram). Figure 5 At this time, its image in the camera will be as shown in the attached image. Figure 6 As shown, distant objects appear to move a greater distance than nearby objects. Analyzing this process, we can see that during lens rotation, the distance any pixel moves on the imaging screen is directly proportional to the pixel's depth. For a pixel with depth d (0 <= d <= 1), assuming a rotation angle of θ, the pixel's movement distance is k1 * d * θ, where k1 is the yaw constant. Based on this conclusion, we can offset the position of any point on the input image according to the yaw and pitch angles. The pitch angle determines the pixel's offset along the X-axis, and the yaw angle determines the pixel's offset along the Y-axis.

[0065] Preferably,

[0066] The preprocessing of the depth-of-field segmentation map to obtain the depth-of-field information map includes:

[0067] Feathering, blurring, and overlay operations are performed on each depth region in the depth field segmentation map to obtain a depth information map. The gray value of each pixel in the depth information map represents the depth value corresponding to that pixel.

[0068] Understandably, feathering, blurring, and overlay operations are performed on various depth regions to obtain a depth information map with a more natural transition of depth information. The grayscale value of each pixel represents the depth value corresponding to that pixel. Further adjustments can be made to the generated depth information map to obtain a more accurate depth information map through operations such as dodging and dodge.

[0069] Preferably,

[0070] The process of editing different depth regions of the input image to obtain a depth-of-field map includes:

[0071] Based on the different depth regions of the input image, it is divided into near-field region, mid-field region and far-field region to obtain a depth-of-field map.

[0072] Understandably, for different depth regions of the input image, brushes symbolizing the foreground, middle ground, and background are used to roughly define the area, resulting in a depth-of-field map.

[0073] Example 2

[0074] The apparatus for creating 3D lens motion based on image depth of field as shown in this embodiment is illustrated in the attached diagram. Figure 2 As shown, the device includes:

[0075] Depth of field generation module 1: used to edit different depth regions of the input image to obtain a depth of field segmentation map, and to preprocess the depth of field segmentation map to obtain a depth of field information map;

[0076] Lens motion module 2: Used to acquire lens motion information, calculate the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region in the depth information map, and obtain the 3D image of the input image;

[0077] It is also used to deflect the canvas to obtain 3D images from different angles;

[0078] Output module 3: Used to crop the display area to obtain the final output image;

[0079] Understandably, the depth-of-field generation module 1 edits an input image according to its depth regions to obtain a depth-of-field map. This map is then preprocessed to obtain a depth information map. The lens motion module 2 acquires the lens's motion information. Based on this motion and the depth information of pixels in each depth region of the depth information map, pixel offset information is calculated to construct a 3D image of the input image. The canvas is then deflected; this deflection can be understood as the camera observing the 3D image from different angles to enhance the 3D perspective effect. Finally, the output module 3 crops the 3D image to obtain the final output image. In this application, without establishing a 3D model, a 3D transformed image of the input image is calculated based on the lens motion and the depth information of pixels in each depth region. Depth fault regions are offset by pixel offset and not displayed. Since there are relatively few depth fault regions in the entire image, displaying them as a 3D image will not affect the overall visual effect, thus avoiding the problem in existing technologies where the transition of depth fault regions is abrupt and unnatural, making it unsuitable for images with large depth of field.

[0080] Example 3:

[0081] This embodiment provides a storage medium storing a computer program, which, when executed by a host controller, implements the various steps in the above method.

[0082] It is understood that the storage medium mentioned above can be a read-only memory, a hard disk, or an optical disk, etc.

[0083] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0084] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0085] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0086] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0087] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0088] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0089] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0090] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0091] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for creating 3D camera motion based on image depth of field, characterized in that, include: The depth-of-field map is obtained by editing different depth regions of the input image; Preprocessing the depth-of-field segmentation map yields a depth-of-field information map; Acquire camera motion information; The motion information of the lens includes the lens's advance distance, pitch angle, and yaw angle. Based on the motion information of the lens and the depth information of the pixels in each depth region in the depth information map, the offset information of the pixels in each depth region is calculated to obtain the 3D image of the input image. The step of calculating the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region of the depth information map includes: Based on the depth information and advance distance information of each depth region in the depth information map, the offset information of each pixel in the depth region from the center point is calculated by a preset translation formula. The preset translation formula is: For a pixel of depth d (0 <= d <= 1), assuming the forward and backward movement distance is p, the offset of this pixel from the center point is k2. p (1.0-d), where k2 is the translation constant; Based on the depth information and pitch angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the X-axis is calculated by a preset deflection formula. Based on the depth information and yaw angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the Y axis is calculated by using a preset yaw formula. The preset deflection formula is: For a pixel with depth d (0 <= d <= 1), assuming a rotation angle of θ, the distance the pixel moves is k1. d θ, where k1 is the deflection constant; By tilting the canvas, 3D images can be obtained from different angles; The display area is cropped to obtain the final output image.

2. The method according to claim 1, characterized in that, The preprocessing of the depth-of-field segmentation map to obtain the depth-of-field information map includes: Feathering, blurring, and overlay operations are performed on each depth region in the depth field segmentation map to obtain a depth information map. The gray value of each pixel in the depth information map represents the depth value corresponding to that pixel.

3. The method according to claim 2, characterized in that, The process of editing different depth regions of the input image to obtain a depth-of-field map includes: Based on the different depth regions of the input image, it is divided into foreground, midground, and background regions to obtain a depth-of-field map.

4. A device for creating 3D camera motion based on image depth of field, characterized in that, The device includes: Depth of field generation module: used to edit different depth regions of the input image to obtain a depth of field map, and to preprocess the depth of field map to obtain a depth information map; Lens motion module: Used to acquire lens motion information, calculate the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region in the depth information map, and obtain the 3D image of the input image; The motion information of the lens includes the lens's advance distance, pitch angle, and yaw angle. The step of calculating the offset information of pixels in each depth region based on the lens motion information and the depth information of pixels in each depth region of the depth information map includes: Based on the depth information and advance distance information of each depth region in the depth information map, the offset information of each pixel in the depth region from the center point is calculated by a preset translation formula. The preset translation formula is: For a pixel of depth d (0 <= d <= 1), assuming the forward and backward movement distance is p, the offset of this pixel from the center point is k2. p (1.0-d), where k2 is the translation constant; Based on the depth information and pitch angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the X-axis is calculated by a preset deflection formula. Based on the depth information and yaw angle information of each depth region in the depth information map, the offset information of each pixel in the depth region from its original position on the Y axis is calculated by using a preset yaw formula. The preset deflection formula is: For a pixel with depth d (0 <= d <= 1), assuming a rotation angle of θ, the distance the pixel moves is k1. d θ, where k1 is the deflection constant; It is also used to deflect the canvas to obtain 3D images from different angles; Output module: Used to crop the display area to obtain the final output image.

5. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by the main controller, implements the steps of the method for creating 3D lens motion based on image depth of field as described in any one of claims 1-3.

Images