Image processing method, electronic equipment and computer readable storage medium

Abstract

The invention provides an image processing method, electronic equipment and a computer readable storage medium, and relates to the technical field of terminals, the method is applied to the electronic equipment, and the method comprises the following steps: obtaining a first video, and processing the first video to obtain a target video, the target video being a video with a Hash effect. Wherein the processing of the first video comprises the following steps: generating a corresponding three-dimensional expression according to multiple frames of images in the first video, obtaining corresponding camera parameters, and calculating an equivalent focal length corresponding to the images, thereby rendering to obtain a Hash region Coke effect image corresponding to each frame of image, and finally synthesizing a target video. In this way, even if a video is obtained in which people and / or objects are moving in the foreground and / or background, a Hash effect video can be generated. The generated Hash region Coke effect video not only can retain the dynamic effect of the foreground object, but also can show the motion change in the background object, so that the display effect is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Since the Hitchcock zoom effect video requires high photography skills, it is currently difficult for ordinary users to directly shoot a video with the Hitchcock zoom effect using an electronic device. In addition, it is also difficult for electronic devices to process the video that has been shot into a Hitchcock zoom effect video. Summary of the Invention

[0003] In view of this, the present application provides an image processing method, an electronic device and a computer-readable storage medium, which can improve the display effect of the image, while retaining the dynamic effect of the foreground object, showing the motion changes in the background object, thereby enhancing the authenticity and display effect of the Hitchcock zoom effect video.

[0004] In a first aspect, the present application provides an image processing method, which is applied to an electronic device. The method includes: obtaining a first video, the first video including N frames of images, N being a positive integer greater than or equal to 2; processing the first video to obtain a target video, the target video including the i-th target image and the i+1-th target image, i being a positive integer, the i-th target image including the first target object and the first background, the i+1-th target image including the second target object and the second background; the image field size of the first target object and the image field size of the second target object are the same, the screen ratio occupied by the first target object and the screen ratio occupied by the second target object are the same, the image field sizes of the first background and the second background are different, and the screen ratio occupied by the first background and the screen ratio occupied by the second background are the same. Wherein, the screen ratio occupied by the first target object or the second target object refers to the relative area of ​​the first target object or the second target object in the image (that is, the ratio of the screen to the entire screen). The screen in the screen ratio refers to the image where the first target object is located, that is, the i-th target image, or the image where the second target object is located, such as the i+1-th target image.

[0005] In a possible implementation of the first aspect, the method includes: obtaining a first video, the first video including N frames of images, where N is a positive integer greater than or equal to 2; processing the first video to obtain a target video, the target video including an i-th target image and an i+1-th target image, where i is a positive integer, the i-th target image including a first target object and a first background, and the i+1-th target image including a second target object and a second background; the image field size of the first target object is the same as the image field size of the second target object, the number of pixels occupied by the first target object is the same as the number of pixels occupied by the second target object, the image field sizes of the first background and the second background are different, and the number of pixels occupied by the first background is the same as the number of pixels occupied by the second background.

[0006] In addition, the first video is a video to be processed as described later. The first video can be a multi-frame image, a video containing a multi-frame image, or a file containing a multi-frame image. For ease of understanding, in the description of the embodiments below, the first video is referred to as a video to be processed. The electronic device can process the first video to obtain a target video. The target video includes at least the i-th target image and the i+1-th target image. The first target object included in the i-th target image and the second target object included in the i+1-th target image can be the representation of the same object in different images. For example, the first target object and the second target object can be the images described later. Figure 1 The first background included in the i-th target image and the second background included in the i+1-th target image can be the same background in different images. For example, the first background and the second background can be the following description Figure 1 Background 102.

[0007] In the above implementation, the target object's image field size and screen ratio remain unchanged in the two target frames included in the target video (i.e., the number of pixels remains unchanged when the resolution remains unchanged); while the background's screen ratio remains unchanged (i.e., the number of pixels remains unchanged when the resolution remains unchanged), the image field size changes. As a result, the target object's area ratio in the video screen remains unchanged, while the area ratios of the remaining captured objects (i.e., the background) in the video screen increase or decrease in sequence. Thus, by processing the first video, the electronic device obtains a target video with a Hitchcock effect.

[0008] In a possible implementation of the first aspect, processing the first video to obtain a target video includes processing each of N frames of images to obtain N frames of target images, thereby obtaining the target video. Each of the N frames of images may include the i-th frame of image described below, and the N frames of target images may include the i-th frame of target image described below.

[0009] In the above implementation, each frame of the first video is processed to obtain the corresponding target image, which means that each frame of the target image in the target video has been processed. Therefore, by combining these target images in either sequential or reverse order, the target video can be made smooth, vivid, and expressive.

[0010] In another possible implementation of the first aspect, processing the first video to obtain a target video includes processing M frames of images among N frames of images to obtain M frames of target images, where M is a positive integer less than N, to obtain the target video. Each of the N frames of images may include the i-th frame of image described below, and the M frames of target images may include the i-th frame of target image described below.

[0011] In the above implementation, instead of processing every frame of the first video to obtain the corresponding target image, M frames are selected from N frames for processing. This means that the electronic device does not need to process every frame. This reduces the computational load and power consumption of the electronic device, speeds up the generation of the target video, and makes the process of obtaining the target video more efficient and timely.

[0012] In another possible implementation of the first aspect, M frames of images from N frames of images are processed to obtain M frames of target images, including: selecting one frame of image from every x frames of image from the N frames of image to obtain M frames of image, where x is a positive integer less than N; and processing each frame of image in the M frames of image to obtain M frames of target image.

[0013] In another possible implementation of the first aspect, processing M frames of images among N frames of images to obtain M frames of target images includes: the M frames of images are obtained by processing one frame of image for every x frames of image among the N frames of images.

[0014] In the above implementation, by selecting one frame for processing every x frames, the processed images can be distributed as evenly as possible in the first video. This allows the target video to be obtained more efficiently and promptly while preserving the coherence and smoothness of the image as much as possible.

[0015] In another possible implementation of the first aspect, N frames of images include an i-th image, and processing each of the N frames of images to obtain N target images includes: obtaining camera parameters and depth information of the i-th image; obtaining a three-dimensional representation of the i-th image based on pixel values ​​of the i-th image and the depth information of the i-th image; and obtaining the i-th target image based on the camera parameters of the i-th image and the three-dimensional representation of the i-th image. The camera parameters of the i-th image may be camera intrinsic parameters and camera extrinsic parameters corresponding to the i-th image, and the three-dimensional representation of the i-th image may be an LDI representation or an MPI representation.

[0016] In the above implementation, the electronic device first obtains the camera parameters and depth information of the i-th image. The camera parameters can be directly read by the electronic device; the depth information can be calculated by the electronic device or directly read. The pixel value of the i-th image is the original image information that the electronic device can obtain based on the i-th image. The electronic device then obtains a three-dimensional representation of the i-th image based on the pixel value of the i-th image and the depth information of the i-th image. In this step, the electronic device further processes the read / calculated information to obtain a three-dimensional representation of the i-th image. Using the three-dimensional representation of the i-th image, the electronic device can further process and ultimately render the i-th target image.

[0017] In another possible implementation of the first aspect, N frames of images include an (i-1)th image and an (i)th image, where (i) is a positive integer greater than or equal to 2. Each of the N frames of images is processed to obtain N frames of target images, including: obtaining camera parameters of the (i)th image and depth information of the (i-1)th image; obtaining a three-dimensional representation of the (i)th image based on pixel values ​​of the (i)th image and the depth information of the (i-1)th image; and obtaining the (i)th target image based on the camera parameters of the (i)th image and the three-dimensional representation of the (i)th image.

[0018] In the above implementation, the electronic device processes each frame, but not each frame is processed using the depth information and camera parameters of the current frame. Specifically, the electronic device obtains depth information and camera parameters once every 2 frames, and the current frame image (i-th image) uses the depth information of the previous frame image (i-1-th image) to calculate the current frame image (i-th image) three-dimensional representation. In addition to the above method, the electronic device may also obtain depth information and camera parameters once every x frames, for example: the current frame image (i-th image) uses the depth information of the previous x frames (ix-th image) to calculate the current frame image (i-th image) three-dimensional representation. In this way, the changing details of the video picture can be retained as much as possible, while also reducing some of the performance power consumption and computational complexity, thereby speeding up the generation of the target video.

[0019] In another possible implementation of the first aspect, M frames of images include an i-th image, where i is a positive integer, and each frame of the M frames of images is processed to obtain M frames of target images, including: obtaining camera parameters of the i-th image and depth information of the i-th image; obtaining a three-dimensional representation of the i-th image based on pixel values ​​of the i-th image and the depth information of the i-th image; and obtaining an i-th target image based on the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

[0020] In the above implementation, the electronic device does not process every frame of the image, but processes it once every x frames, and each frame is processed using its own depth information and camera parameters. In this way, the amount of calculation and power consumption can be reduced and the processing speed can be improved. Specifically, the electronic device originally needed to process every frame of the image, but by processing it once every x frames, the amount of calculation and computing time originally required for the unprocessed frames are saved. Therefore, the electronic device can generate the target video faster and more efficiently.

[0021] In another possible implementation of the first aspect, M frames of images include an iy-th image and an i-th image, where i is a positive integer greater than y, and each frame of the M frames of images is processed to obtain M frames of target images, including: obtaining camera parameters of the i-th image and depth information of the iy-th image; obtaining a three-dimensional representation of the i-th image based on pixel values ​​of the i-th image and the depth information of the iy-th image; and obtaining an i-th target image based on the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

[0022] In the above implementation, the electronic device not only processes once every x frames, but also does not process each frame using its own depth information and camera parameters. The electronic device obtains depth information and camera parameters once every y frames, and the current frame image (i-th image) uses the depth information of the previous y frames (iy-th image) to calculate the three-dimensional representation of the current frame image (i-th image). In this way, while reducing the amount of calculation and power consumption and improving the rendering speed, the changing details of the video picture can be retained as much as possible, and the amount of calculation and power consumption reduced by this method will be greater than processing once every x frames but using its own depth information and camera parameters for each frame.

[0023] In another possible implementation of the first aspect, the camera parameters of the i-th image are camera intrinsic parameters and camera extrinsic parameters corresponding to the i-th image, where the camera intrinsic parameters include: focal length, principal point coordinates, radial distortion coefficient and / or tangential distortion coefficient, and the camera extrinsic parameters include: rotation parameters and / or translation parameters.

[0024] In the above implementation, by using camera intrinsic parameters and extrinsic parameters, image processing can be performed more accurately.

[0025] In another possible implementation manner of the first aspect, the three-dimensional representation of the i-th image is a layered depth image LDI representation or a multi-plane image MPI representation.

[0026] In the above implementation, by using three-dimensional representation, three-dimensional scene information can be efficiently stored and processed, thereby better handling occlusion relationships and perspective changes in the image and generating a target image that is more in line with the expectations.

[0027] In another possible implementation of the first aspect, obtaining an i-th target image based on camera parameters of an i-th image and a three-dimensional representation of the i-th image includes: calculating an i-th equivalent focal length based on a frame number of the i-th image and the camera parameters of the i-th image; and rendering the i-th image based on the i-th equivalent focal length and the three-dimensional representation of the i-th image to obtain the i-th target image.

[0028] In the above implementation, camera parameters are used to adjust the image's viewing angle and perspective, ensuring that the generated target image conforms to a specific viewing angle. Using a 3D representation allows for more accurate rendering of objects in the image, preserving their 3D structure and generating a more realistic image. By combining camera parameters and a 3D representation, a more realistic target image can be rendered.

[0029] In another possible implementation of the first aspect, the viewpoint corresponding to the first target object is a first viewpoint, the viewpoint corresponding to the second target object is a second viewpoint, and the first viewpoint and the second viewpoint are located at different positions. A viewpoint generally refers to the position and angle of a camera or sensor when capturing an image or video, and the camera or sensor referred to herein may be virtual.

[0030] In the above implementation, the difference in viewpoints affects how objects in the image are projected onto a two-dimensional plane, thereby affecting the computer vision algorithm's ability to recognize and understand the objects. The different positions of the first and second viewpoints mean that the virtual camera or sensor is at different positions and / or angles when capturing the first and second target objects.

[0031] In another possible implementation of the first aspect, obtaining the first video includes obtaining the first video in response to a user's shooting operation. The shooting operation can take various forms. For example, the user can initiate shooting by clicking a button, using a voice command, or in other ways.

[0032] In the above implementation, the first video is captured by the user's shooting operation. This allows the user to decide when to start and end shooting based on their needs and intentions, providing greater flexibility and autonomy. Furthermore, video capture is real-time; that is, the electronic device begins recording immediately after the user performs the shooting operation. This approach provides greater user control and real-time performance, making it suitable for a variety of video recording scenarios requiring active user participation.

[0033] In another possible implementation of the first aspect, in response to a user's shooting operation, a first video is obtained, including: displaying a first interface, the first interface including a first control; in response to a first operation of the user on the first control, the electronic device starts shooting; in response to a second operation of the user on the first control, the electronic device ends shooting, and the first video is obtained.

[0034] In the above implementation, the electronic device displays a user interface, referred to as the first interface, on which there is a control, referred to as the first control. This control may be a button, an icon, or other interactive element. When the user performs a first operation on the first control, the electronic device starts shooting a video. The first operation may be a click, a long press, or other forms of interaction. When the user performs a second operation on the first control, the electronic device ends shooting and generates a first video. The second operation may be clicking again, releasing the button, etc. Using the above method, through the interaction between the user interface and the control, the user can conveniently and accurately control the shooting of the video, providing a good user experience and real-time feedback.

[0035] In another possible implementation of the first aspect, in the above implementation, processing each frame of N frames of images to obtain N frames of target images also includes: in response to a third operation of the user on the second control, processing each frame of the N frames of images to obtain N frames of target images.

[0036] In the above implementation, the user can, through a third operation on the second control, cause the electronic device to process each of the N frames of image. This approach provides the user with more options. The second control can also be used to instruct the generation of a high-quality and / or high-frame-rate Hitchcock-effect video.

[0037] In another possible implementation of the first aspect, in response to a fourth operation of the user on the third control, M frames of the N frames of images are processed to obtain M frames of target images, where M is a positive integer less than N, to obtain the target video.

[0038] In the above implementation, the user can make the electronic device process M frames of N frames through the fourth operation of the third control. This method gives the user more options and allows them to choose to process in a frame skipping manner. The third control can also be used to indicate the real-time generation of a Hitchcock effect video. In a second aspect, the present application provides an electronic device, which includes a display screen, a memory, and one or more processors; the display screen, the memory, and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, and the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method described in the first aspect and any possible design method thereof.

[0039] In a third aspect, the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method described in the first aspect and any possible design thereof.

[0040] In a fourth aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, enables the electronic device to execute the method described in the first aspect above and any possible design method thereof.

[0041] In a fifth aspect, the present application provides a device, which is included in an electronic device and has the function of implementing the electronic device behavior in any of the above aspects and possible implementation methods. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes at least one module or unit corresponding to the above function. For example, a display module or unit, a computing module or unit, and a storage module or unit, etc.

[0042] In a sixth aspect, an embodiment of the present application provides a chip system, which includes a processor and may also include a memory, for implementing any one of the methods provided in the first aspect. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0043] It can be understood that the electronic device described in the second aspect and any possible design method provided above, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a Hitchcock effect video provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of the software architecture of an electronic device provided in an embodiment of the present application;

[0047] Figure 4 A flowchart of an image processing method provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of a scene for generating a Hitchcock effect video provided in an embodiment of the present application Figure 1 ;

[0049] Figure 6 A schematic diagram of a scene for generating a Hitchcock effect video provided in an embodiment of the present application Figure 2 ;

[0050] Figure 7 A schematic diagram of a scene for generating a Hitchcock effect video provided in an embodiment of the present application Figure 3 ;

[0051] Figure 8 A schematic diagram of LDI estimation provided in an embodiment of the present application Figure 1 ;

[0052] Figure 9 A schematic diagram of LDI estimation provided in an embodiment of the present application Figure 2 ;

[0053] Figure 10 A schematic diagram of an MPI estimation provided in an embodiment of the present application;

[0054] Figure 11 A schematic diagram of an equivalent focal length calculation provided in an embodiment of the present application;

[0055] Figure 12 A schematic diagram of frame skipping processing provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0057] It should be noted that in the embodiments of the present application, words such as "exemplary," "optional," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary," "optional," or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary," "optional," or "for example" is intended to present the relevant concepts in a concrete manner.

[0058] Before introducing the embodiments of the present application, the technologies involved in the embodiments of the present application are first introduced in detail.

[0059] 1. Hitchcock Zoom

[0060] Hitchcock zoom, which can also be called "push-pull lens" or "zoom slide", is a shooting method. Among them, zoom refers to changing the focal length (FL) of the lens. Specifically, during the shooting process, the photographer changes the focal length of the lens while moving the lens. For example, the photographer moves the lens so that the lens moves toward the subject, which reduces the focal length of the lens; or, the photographer moves the lens so that the lens moves away from the subject, which increases the focal length of the lens. Through such a shooting method, the area ratio of the target object being photographed in the shooting picture remains unchanged, while the proportion of other objects being photographed in the shooting picture changes, thereby creating a visual effect of spatial distortion. It should be noted that in the above-mentioned shooting process, the photographer moving the lens refers to the real-world movement of the lens caused by the photographer changing the position of the lens in the real world.

[0061] Among them, the lens can also be called a camera, a camera, a video camera, a camera module, etc., or it can also be an electronic device that includes a camera, a camera and / or a camera module.

[0062] 2. Hitchcock Effect

[0063] In the embodiment of the present application, the Hitchcock effect is achieved by processing the video with an algorithm so that the display effect of the video is the same as or similar to that of the video obtained by shooting with the Hitchcock zoom method. This display effect is called the Hitchcock zoom effect, or the Hitchcock camera effect, and is referred to as the Hitchcock effect in this application. Specifically, this display effect is as follows: Figure 1 As shown, Figure 1 The four frames of images shown are images selected from a dynamic image or video with a Hitchcock effect. It can be seen that in the four frames of images, the area ratio of the portrait 101 in the shooting picture in each frame of image remains unchanged or remains basically unchanged. That is, in this video, the portrait 101 is the target object being photographed. In the four frames of images, the area ratio of the background 102 (i.e., the buildings and trees in the picture) behind the portrait 101 in the shooting picture in each frame of image decreases successively. That is, in this video, the background 102 (i.e., the buildings and trees in the picture) are other objects being photographed, and these other objects being photographed can be collectively referred to as background. With such Figure 1 The dynamic image or video with the image change characteristics shown can be considered as a video with Hitchcock effect. The Hitchcock effect video described below refers to a dynamic image or video with Hitchcock effect, which can also be called a target video.

[0064] It should be noted that Figure 1 What is shown is a situation where the proportion of the background in the shooting picture becomes smaller, but the embodiments of the present application are not limited to this situation. In a continuous multi-frame image, if the area ratio of the target object being photographed in the shooting picture remains unchanged, and the area ratios of the remaining objects being photographed in the shooting picture increase or decrease successively, then the video or dynamic image in which the continuous multi-frame picture is located also has a Hitchcock effect. Among them, the change ratio of the remaining objects being photographed behind the target object is opposite to that of the remaining objects being photographed in front of the target object. For example, the area ratio of the building behind the target object in the shooting picture increases successively, and the area ratio of the trees in front of the target object in the shooting picture decreases successively.

[0065] 3. Videos to be processed

[0066] In the embodiment of the present application, the video to be processed refers to the video or dynamic image to be processed. The video to be processed may include multiple frames of images, or, in some cases, may include only one frame of image.

[0067] 4. Frame Number

[0068] A frame number is the sequence number of each frame in a video or continuous image. Frame numbers can be used to identify and track specific frames in a video. When processing video or multi-frame image data, frame numbers can help locate specific moments in time or events.

[0069] 5. Camera parameters

[0070] Camera parameters refer to camera intrinsic parameters and camera extrinsic parameters. These two sets of parameters together define the mapping process from 3D space coordinates to 2D image coordinates.

[0071] 6. Camera Intrinsics

[0072] Camera intrinsic parameters are parameters related to the camera's inherent characteristics. These parameters describe the geometric and optical properties of the camera lens. These parameters can include focal length, principal point coordinates (usually the image center), and distortion information (e.g., radial and tangential distortion coefficients). These parameters are typically obtained through a calibration process and are known constants when the camera settings remain unchanged.

[0073] 7. Camera Extrinsics

[0074] Camera extrinsics are parameters that describe the position and orientation of a camera in a global coordinate system. They are used to represent the camera's specific position and orientation in three-dimensional space. They include rotation parameters and / or translation parameters. Specifically, camera extrinsics typically consist of a rotation matrix and a translation vector. The rotation matrix is ​​a 3x3 matrix that represents the rotation of the camera coordinate system relative to the world coordinate system. The translation vector is a three-dimensional vector that represents the straight-line distance and direction from the origin of the world coordinate system to the origin of the camera coordinate system.

[0075] 8. Exchangeable Image File Format (EXIF) information

[0076] EXIF is a metadata format used to store digital photos and audio files. EXIF ​​information can be a type of metadata included in an image, recording various settings and conditions when the photo was taken, such as shooting time, location, camera model, focal length, exposure time, ISO speed, etc. EXIF ​​information can also include some information related to camera internal parameters, such as focal length and distortion information. Electronic devices can obtain camera internal parameters by obtaining EXIF ​​information. EXIF ​​information is usually embedded in file formats such as JPEG, TIFF, and RIFF, and contains various detailed information about the image or audio file.

[0077] 9. Image Field

[0078] The image field refers to the collection of all points on the imaging plane when an optical system (e.g., a camera lens, a camera, a sensor, etc.) forms an image. It describes the extent and quality of the image formed by the optical system on the imaging plane. In an optical system, the size and quality of the image field are affected by many factors, including lens design, optical component quality, and aperture size.

[0079] The main characteristics of the image field include: image field size, image field flatness, image field distortion, image field illumination, etc. Among them, the image field size refers to the physical size of the imaging plane, which can be used to express the range of the imaging area that the optical system can cover, usually in millimeters (mm).

[0080] For the same object in an image, the difference in Image Field Size is primarily reflected in the field of view. A larger Image Field Size means the optical system has a larger imaging range and can capture a wider field of view. This means the image includes not only the target object but also potentially more background (or surrounding environment). A smaller Image Field Size, on the other hand, means the optical system has a smaller imaging range and a narrower field of view. This means the image primarily focuses on the target object, with less background (or surrounding environment).

[0081] In the prior art, the photographer needs to use the Hitchcock zoom shooting method to obtain the following image: Figure 1 During the shooting process, the photographer needs to move the lens and change the focal length simultaneously. However, for non-professional photographers, it is difficult for users to achieve the zoom that meets the Hitchcock effect requirements while shooting and moving simultaneously. Therefore, the photographer needs to shoot repeatedly to ensure that the zoom meets the Hitchcock effect requirements, that is, the speed of moving the lens matches the speed of changing the lens focal length.

[0082] As can be seen from the above, the current Hitchcock zoom shooting method requires high shooting skills from the photographer. It is difficult for non-professional photographers to shoot Hitchcock effect videos without professional shooting equipment. Therefore, it is necessary to use algorithms to generate Hitchcock effect videos. However, the current technology for generating Hitchcock effect videos often uses algorithms to cut out or split the foreground and background of an image or multiple images, and then merge them after algorithmic processing. Although this method can simulate the Hitchcock effect to a certain extent, it has obvious limitations: the processed Hitchcock effect videos are prone to poor fusion of the foreground and background or artifacts. Specifically, when there are moving elements in the background, or the light and shadow relationship between the foreground and background changes, the existing technology cannot accurately reflect these dynamic changes, affecting the display effect.

[0083] To this end, an embodiment of the present application provides an image processing method. When an electronic device executes the image processing method, it can obtain a first video, and process the first video to obtain a target video, which is a video with a Hitchcock effect. The processing of the first video includes: generating a corresponding three-dimensional expression based on multiple frames of images in the first video, obtaining corresponding camera parameters, and calculating the equivalent focal length corresponding to the image, thereby rendering a Hitchcock effect image corresponding to each frame of the image to obtain the target video. In this way, a Hitchcock effect video can be output when a video with people and / or objects moving in the foreground and / or background is input. In other words, the image processing method provided in the embodiment of the present application can not only retain the dynamic effect of the foreground object, but also show the motion changes in the background object, thereby greatly improving the authenticity, dynamic expressiveness and display effect of the generated Hitchcock effect video.

[0084] In order to better understand the embodiments of the present application, the electronic device provided by the embodiments of the present application is first introduced.

[0085] The image processing method provided in the embodiment of the present application can be applied to the electronic device 100. For example, the electronic device 100 can be a mobile phone, a tablet computer, a smart screen, a laptop computer, a vehicle-mounted device, a wearable device (such as a smart watch), an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), an artificial intelligence (AI) device, and other terminal devices with image processing functions. The embodiment of the present application does not limit the specific type of the electronic device 100 or the installed operating system. It should be noted that the electronic device provided in the embodiment of the present application also includes a terminal.

[0086] Figure 2 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0087] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0088] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0089] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0090] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0091] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the instruction or data again, it can directly call it from this memory, thereby improving system efficiency.

[0092] In some embodiments, the processor 110 may include one or more interfaces. It should be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely illustrative and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0093] The charging management module 140 is configured to receive charging input from a charger. While charging the battery 142, the charging management module 140 can also power the electronic device through the power management module 141. The power management module 141 is configured to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160.

[0094] The wireless communication function of electronic device 100 is implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Mobile communication module 150 can provide wireless communication solutions for electronic device 100, including 2G / 3G / 4G / 5G.

[0095] The wireless communication module 160 can provide wireless communication solutions for application on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0096] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0097] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0098] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device can include 1 or N display screens 194, where N is a positive integer greater than 1. For example, the electronic device can be a foldable mobile phone, a tablet, or a computer.

[0099] In the embodiment of the present application, the electronic device 100 displays the original image (RAW image) captured by the camera, and Figure 1 、 Figure 5 、 Figure 6 、 Figure 7 The capabilities of the user interface shown rely on the display functionality provided by the GPU, display 194, and application processor described above.

[0100] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0101] The ISP can be used to process data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0102] The camera 193 is used to capture still images or videos. The object / subject generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0103] In some embodiments, the camera 193 can be an ordinary camera. In this case, the electronic device uses the photos or videos taken by the camera 193 as the video to be processed. Afterwards, the electronic device can generate a Hitchcock effect video based on the video to be processed. In other embodiments, when the electronic device has multiple cameras 193, or has a depth camera 193 such as time-of-flight (TOF) or structured light, the electronic device obtains the video to be processed with depth information. Afterwards, the electronic device can generate a Hitchcock effect video based on the video to be processed according to the depth information of the video to be processed and camera parameters. For specific embodiments, please refer to the subsequent Figure 4 、 Figures 8 to 11 In some embodiments, the image processing method provided in the embodiments of the present application can also be applied to electronic devices without cameras. The electronic device can use a saved image or video as a video to be processed, and generate a Hitchcock effect video based on the video to be processed according to the depth information of the video to be processed or the zoom ratio set by the user.

[0104] The DSP is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 is selecting a frequency point, the DSP is used to perform Fourier transform on the frequency point energy.

[0105] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0106] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0107] In the embodiment of the present application, the electronic device 100 implements the shooting method provided in the embodiment of the present application, firstly relying on the ISP and the images captured by the camera 193, and secondly relying on the video codec and the image computing and processing capabilities provided by the GPU. Specifically, the electronic device 100 can implement neural network algorithms such as face recognition, body recognition, and re-identification through the computing and processing capabilities provided by the NPU.

[0108] The internal memory 121 may include one or more random access memories (RAMs) and one or more non-volatile memories (NVMs). RAMs may include static random access memories (SRAMs), dynamic random access memories (DRAMs), synchronous dynamic random access memories (SDRAMs), and double data rate synchronous dynamic random access memories (DDR SDRAMs, such as the fifth generation DDR SDRAM, generally referred to as DDR5 SDRAMs). Non-volatile memories may include disk storage devices and flash memories.

[0109] The random access memory can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into the random access memory for direct reading and writing by the processor 110.

[0110] In the embodiment of the present application, the code for implementing the shooting method described in the embodiment of the present application may be stored in a non-volatile memory. When running the camera application, the electronic device 100 may load the executable code stored in the non-volatile memory into the random access memory.

[0111] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0112] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0113] Audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Receiver 170B, also known as a "handset," is used to convert audio electrical signals into sound signals. Microphone 170C, also known as a "microphone," is used to convert sound signals into electrical signals. Headphone jack 170D is used to connect wired headphones.

[0114] In the embodiment of the present application, when the electronic device 100 activates the camera to capture images, it can also activate the microphone 170C to capture sound signals, and convert the sound signals into electrical signals for storage. In this way, the user can obtain video with sound.

[0115] The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194 .

[0116] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (e.g., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake.

[0117] The pressure sensor 180C is used to measure air pressure. The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing. The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The light-emitting diode can be an infrared light-emitting diode. The electronic device 100 emits infrared light outward through the light-emitting diode. The electronic device 100 uses the photodiode to detect infrared reflected light from nearby objects / subjects. When sufficient reflected light is detected, it can be determined that there is an object / object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object / object near the electronic device 100. The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, accessing application locks, fingerprint photography, fingerprint answering calls, etc. The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to implement temperature processing strategies.

[0118] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0119] In the embodiment of the present application, the electronic device 100 can use the touch sensor 180K to detect the user's click, slide and other operations on the display screen 194 to achieve Figure 1 、 Figure 5 、 Figure 6 、 Figure 7 The shooting or picture editing method shown.

[0120] Bone conduction sensor 180M can acquire vibration signals. Buttons 190 include a power button, volume button, and the like. Electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. Indicator 192 can be an indicator light. SIM card interface 195 is used to connect a SIM card. Electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than one.

[0121] Figure 3 A schematic diagram of the software architecture of the electronic device 100 is shown.

[0122] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the software structure of the electronic device 100 is exemplified by taking an operating system of a layered architecture as an example.

[0123] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the operating system is divided into four layers: the application layer, the application framework layer, the hardware abstraction layer (HAL), and the driver layer.

[0124] Figure 3 1 is a schematic diagram of the software structure of the electronic device 100 according to an embodiment of the present application. Figure 3 The hardware layer is also shown in the figure to explain the coordination relationship between related hardware and software.

[0125] It should be noted that the embodiment of the present application is illustrated by taking a common operating system as an example. In other operating systems, as long as the functions implemented by each functional module are similar to those of the embodiment of the present application, the solution of the present application can also be implemented.

[0126] The application layer can include a series of application packages. Figure 3 As shown, the application package may include applications such as a camera, an image editing application, etc.

[0127] The camera is used to provide a shooting function. The image editing application is used to provide an image editing function. For example, common image editing applications include photo albums, photo galleries, and image editors. In an embodiment of the present application, in response to a user operation, the electronic device can generate a Hitchcock effect video based on the video to be processed using the camera, image editing application, or other application.

[0128] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0129] like Figure 3 As shown, the application framework layer may include a camera service. In an embodiment of the present application, the application framework layer may include a camera access interface, wherein the camera access interface may include camera management and camera devices. The camera access interface is used to provide an application programming interface and programming framework for camera applications. It is understandable that the application framework layer may also include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, etc.

[0130] The HAL layer is located above the driver layer. The HAL layer can abstract the hardware of the hardware layer, hide the differences between different hardware, and provide a unified hardware interface for the upper layer. The HAL layer can provide a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer can include a camera hardware abstraction layer and a camera algorithm library.

[0131] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. The camera algorithm library can include the operating code and data for implementing the shooting method provided in the embodiments of the present application. For example, the camera algorithm library can include an image processing module that can be used to perform processing operations such as scaling and splicing images.

[0132] The driver layer is the layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer can include camera device drivers, DSP drivers, and image processor drivers. Figure 3 As shown, the driver layer includes a camera driver, which is used to control the camera of the hardware layer to perform shooting actions.

[0133] The camera device driver drives the camera's sensor to capture images and the image signal processor to pre-process them. The DSP driver drives the DSP to process images. The image processor driver drives the graphics processor to process images.

[0134] It is understandable that the driver layer may also include display drivers, audio drivers, sensor drivers, etc., and these drivers will not be described in detail here.

[0135] The following combination Figure 3 In addition to the description of the above system structure, the image processing method in the embodiment of the present application is specifically described in the scene where the image has been captured and the scene where the image is taken:

[0136] In the shooting scenario, in response to the user's operation of opening the camera application (for example, clicking the camera application icon), the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera by calling the camera device (camera device 1, camera device 2 and / or other camera devices) in the camera hardware abstraction layer.

[0137] The camera hardware abstraction layer sends this instruction to the camera device driver in the kernel layer. This camera device driver activates the corresponding camera sensor and uses it to collect image light signals. A camera device in the camera hardware abstraction layer corresponds to a camera sensor in the hardware layer. The image obtained by the camera's photosensitive element converting the captured light signal into an electrical signal is called a raw image (RAW image). This process is also called camera image acquisition.

[0138] It should be noted that a first cache with a preset storage space size can be set in the electronic device 100. For example, the first cache can be a storage area of ​​the memory of the electronic device 100. During the process of the camera capturing images, the first cache is used to store the captured images. The electronic device 100 can store the images captured by the camera and the identification information of the images in the first cache. The multi-frame cache images in the first cache are the multi-frame images captured by the camera. The identification information includes but is not limited to at least one of the image capture timestamp and the image frame number. The multi-frame cache images can be stored in the first cache in the form of a cache queue. The cache images in the cache queue can be sorted according to the capture timestamp or the image frame number. The capture timestamp can be obtained based on the camera capture.

[0139] Then, the camera sensor can transmit the collected image light signal to the image signal processor for preprocessing to obtain an image electrical signal (which can be a RAW image), and transmit the above RAW image to the camera hardware abstraction layer through the camera device driver.

[0140] The camera hardware abstraction layer can send RAW images to the camera algorithm library. The process of the camera sending the captured image to the camera algorithm library is called reporting. The camera algorithm library stores the program code that implements the shooting method provided in the embodiments of this application. Based on the digital signal processor and image processor, the camera algorithm library executes this code to achieve capabilities such as generating Hitchcock effect videos.

[0141] The camera algorithm library can send the RAW image captured by the camera to the camera hardware abstraction layer. Then, the camera hardware abstraction layer can send it for display. The electronic device displays the shooting interface, and the user can operate the shooting controls in the shooting interface. The shooting controls may include: photo control, video control and / or Hitchcock effect video shooting mode control. Similar to the above scenario, the touch sensor receives touch and / or click operations, and after a series of processing by the driver layer and the application framework layer, it can be identified that the control corresponding to the operation is a shooting control. Then, the camera calls the interface provided by the camera service of the framework layer. After the interface provided by the camera service is called, the camera HAL in the HAL layer can call the camera driver of the driver layer, and the camera driver can capture RAW images through the camera.

[0142] The camera driver then sends the RAW image to the image processing module in the HAL layer. The image processing module processes the RAW image into a Hitchcock effect image. The image processing module can then send the generated Hitchcock effect image to the gallery application for display or store it in the electronic device's memory.

[0143] In a scenario where an image has been captured, an image editing application located in the application layer (e.g., a camera application, a gallery application, or other application with image editing functionality) displays an image editing interface in the electronic device. The image editing interface includes the video to be processed and a video editing control (e.g., a control for generating a Hitchcock effect video). When the touch sensor receives a touch and / or click operation, a corresponding hardware interrupt is sent to the driver layer. The driver layer processes the touch operation into a raw input event (including information such as touch coordinates and a timestamp of the touch operation). The raw input event is stored in the driver layer. The framework layer obtains the raw input event from the driver layer and can identify that the control corresponding to the input event is a control for generating a Hitchcock effect video. The electronic device 100 can first obtain a video to be processed, wherein the video to be processed includes N frames of images, where N is a positive integer greater than or equal to 1, and the images in the video to be processed are labeled 1, 2, ..., N in sequence according to the order of the N frames in the video to be processed. The label can also be called a frame number, and the number of video images included in the image can also be called the number of frames. The image editing application then calls the interface provided by the framework layer's camera service. After the interface is called, the image processing module in the HAL layer processes the N frames of video to be processed into a Hitchcock-effect video. The image processing module then sends the generated Hitchcock-effect video to the image editing application for display.

[0144] The following describes an image processing method provided by an embodiment of the present application.

[0145] like Figure 4 This is a flow chart of an image processing method provided in an embodiment of the present application. The method is performed by an electronic device and includes the following steps:

[0146] S401, obtain the i-th frame image

[0147] The video to be processed includes multiple frames of images (for example, N frames of images), and these images include at least one unprocessed image. The electronic device can determine an unprocessed image from the at least one unprocessed image as the i-th frame of image, and obtain the i-th frame of image, where i is a positive integer greater than or equal to 1. It should be noted that the video to be processed herein may also be referred to as the first video, and the i-th frame of image may also be referred to as the i-th image.

[0148] Optionally, the electronic device may also directly obtain the i-th frame image, where the i-th frame image is an unprocessed image in the video to be processed.

[0149] Optionally, the process of the electronic device acquiring the i-th frame image can also be divided into the following steps S4011-S4013:

[0150] S4011, get the video to be processed

[0151] The method for acquiring the video to be processed corresponds to different application scenarios. For example, in a scenario where a Hitchcock effect video is being captured, in response to a user operating a capture control, the electronic device may use the image captured at the moment the user operates the capture control as the video to be processed. For another example, in a scenario where an image has already been captured, in response to a user selecting an image, the electronic device may use the image corresponding to the selected image as the video to be processed. The capture control may include a photo control, a video recording control, and / or a Hitchcock effect video capture mode control.

[0152] The following describes the specific implementation of S4011 in the scenario of shooting a Hitchcock effect video.

[0153] In some embodiments, in a scene where a Hitchcock effect video is captured, the electronic device may use the image captured by the camera at the moment the user operates the shooting control as the video to be processed. Alternatively, in a scene where a Hitchcock effect shooting mode is enabled, the electronic device may use the image (and / or video) captured by the camera at the moment the user operates the shooting control as the video to be processed.

[0154] Among them, the electronic device turning on the Hitchcock effect video shooting mode may mean that the electronic device can execute the image processing method provided in the embodiment of the present application to generate a Hitchcock effect video based on the video to be processed.

[0155] For example, when the electronic device starts the Hitchcock effect video shooting mode to shoot, the display screen of the electronic device displays the following: Figure 5 The shooting preview interface shown in Figure (a) of FIG. The shooting preview interface includes a photo control 5011, a thumbnail icon 502 for opening the gallery, a viewfinder 503, a shooting mode selection bar 504, a switching icon 505 for switching between front and rear cameras, etc.

[0156] The image displayed in the viewfinder 503 is the preview image captured by the current camera in real time. Figure 5 As shown in Figure (a) in the figure, without any user operation, it is displayed in the area that the user may be interested in (for example, a portrait or face area); or Figure 5 As shown in FIG. 5( c ), the viewfinder 503 may be displayed in a corresponding area according to a user operation (eg, a click or long press operation).

[0157] The shooting mode selection bar 504 includes multiple shooting modes, such as night mode, portrait mode, photo mode, video mode, professional mode, aperture mode or more other modes. Figure 5 As shown in FIG. 5A , the user can operate the shooting mode selection bar 504 (eg, click, slide left or right, click multiple times, or long press, etc.).

[0158] like Figure 5 As shown in Figure (b) in the figure, after the user selects the video mode, the display form of the shooting control may not change, or may change, for example: Figure 5 The photo control 5011 in Figure (a) changes to Figure 5 (b) in FIG. 5012 shows the video start control. Figure 5 As shown in FIG. 5( b ), when the user operates the video recording start control 5012 , the electronic device starts recording from the moment the user operates the video recording start control 5012 .

[0159] like Figure 5 As shown in Figure (c), during the video recording process, the display form of the shooting control changes from Figure 5 The video start control 5012 in Figure (b) changes to Figure 5 The video recording process control 506 in FIG (c) includes two sub-controls: a video recording end control 5061 and a video recording pause control 5062. In addition, during the video recording process, the thumbnail icon 502 for opening the gallery disappears (is not displayed). Figure 5As shown in Figure (c), during the video recording process, the user can click or touch the display screen of the electronic device to display the viewfinder 503 in the corresponding portrait area. When the video end control 5061 is operated, the electronic device ends the video recording. The electronic device uses the video captured by the camera during this period as the video to be processed. After the electronic device ends the video recording, Figure 5 As shown in FIG. 5( d ) , a thumbnail of a frame of image in the video to be processed is displayed on a thumbnail icon 502 for opening the gallery.

[0160] In some examples, the electronic device displays Figure 5 The shooting preview interface shown in Figure (a) of FIG. 1 can also be called the first interface, and the thumbnail icon 502 displayed on the interface can also be called the first thumbnail. Afterwards, in response to the user's operation on the shooting mode selection bar 504, the electronic device enters the video recording mode and can display the same Figure 5 In response to the user's operation on the video start control 5012, the electronic device starts recording and may display the video preview interface similar to FIG (b). Figure 5 The video recording process interface is similar to that in FIG (c), wherein the time display area 508 can display the duration of the current video recording. Afterwards, in response to the user's operation of the video recording end control 5061, the electronic device ends the video recording and can display the same Figure 5 At this time, the thumbnail icon 502 for opening the gallery in the interface is updated to a thumbnail of a frame image in the video to be processed (also referred to as the second thumbnail). Afterwards, in response to the user's operation on the video end control 5061, the electronic device displays the same Figure 5 The second video preview interface is similar to Figure (d) in FIG. 1 , in which the thumbnail icon 502 for opening the gallery is a thumbnail of a frame in the video to be processed, which is the first thumbnail in this application, and the video to be processed is the first video in this application. It should be noted that the video start control 5012 and the video end control 5061 can be considered as two different controls, or as two different forms of the same control. When they are considered as two different forms of the same control, the video start control 5012 and the video end control 5061 can both be referred to as the first control.

[0161] like Figure 6 Figure (a) to Figure 6 As shown in FIG. 5(c), in response to the user's operation on the thumbnail icon 502 (or the second thumbnail), the electronic device displays the first image display interface. Optionally, the first image display interface can be as follows: Figure 6 The figure (a) in FIG. 1 only includes the Hitchcock effect video, and the electronic device can directly display the video on the first image display interface. Figure 6(b) shows the Hitchcock effect video (target video). Optionally, the first image display interface can also be as shown in FIG. Figure 6 As shown in Figure (c), it includes not only the Hitchcock effect video (target video) but also the original video (first video). The electronic device can simultaneously display the Hitchcock effect video and the original video on the first image display interface. Specifically, the two videos can be arranged top to bottom or left to right. The top to bottom arrangement is used as an example in the figure. By displaying the two videos simultaneously on a single interface, users can better compare and observe the Hitchcock video generation effect, thereby obtaining a better interactive experience and display effect.

[0162] It should be noted that when an electronic device displays Figure 5 In the case of the recording process interface shown in Figure (c), it means that the camera application is running. At this time, the camera starts to capture images continuously and outputs these images in the order of capturing images. These images are sent to the preview stream processing module and the data stream processing module respectively, wherein the image sequence processed by the preview stream processing module is sequentially displayed on the display screen of the electronic device, providing the user with a preview screen during the shooting process. The electronic device selects the image corresponding to the user's operation of the recording start control 5012 and the recording end control 5061 from the image sequence processed by the data stream processing module as the captured image, that is, the video to be processed. In the above process, only the preview stream processing process and the data stream processing process during the recording process are briefly introduced. For more detailed processing process, please refer to the relevant technology, which will not be repeated here.

[0163] Furthermore, it should be noted that the aforementioned shooting control may also be a Hitchcock effect video shooting mode control. Specifically, after the user selects the Hitchcock effect shooting mode, the electronic device may display the Hitchcock effect video shooting mode control, which may function similarly to the start recording control 5012. For example, in response to the user operating the Hitchcock effect video shooting mode control, the electronic device may start recording and directly generate a Hitchcock effect video after the recording is completed. For details, please refer to the relevant description above and will not be repeated here.

[0164] The following describes the specific implementation of S4011 in a scene where an image has been captured.

[0165] The term "captured image" refers to a scenario where an electronic device has already captured an image through capture, downloading, or communication with another electronic device, and the image has been stored in the electronic device. In this scenario, the user selects an image through touch, long press, or click, and the electronic device can use the selected image as the video to be processed. The following describes various implementations of this scenario with reference to the accompanying drawings. It should be noted that the captured image here can be a video or a single-frame image.

[0166] In some embodiments, as Figure 7 As shown in FIG. 1 (a), the electronic device displays an album display interface, which includes zoom icons of multiple stored videos. After the user operates the zoom icon 701 of one of the videos, the electronic device can display a second image display interface corresponding to the zoom icon 701, such as Figure 7 As shown in FIG. 5( b ), the second image display interface includes a control 702 for generating a Hitchcock effect video. In response to a user's operation on the control 702 for generating a Hitchcock effect video, the electronic device may use the video as a video to be processed.

[0167] In the above embodiment, Figure 7 The album display interface shown in Figure (a) can be the first interface in this application. The album display interface includes zoom icons (i.e., thumbnails) of multiple stored images. The zoom icon 701 operated by the user is the first thumbnail in this application. The user's operation on the zoom icon 701 is the first operation in this application, and the video corresponding to the zoom icon 701 is the first video in this application.

[0168] S4012: Determine whether there is any unprocessed image in the video to be processed.

[0169] The video to be processed contains multiple frames of images, and each frame of the image can be transformed into a Hitchcock effect image after being processed by the image processing solution of the embodiment of the present application. The unprocessed image in the video to be processed refers to an image that has not yet been processed.

[0170] After the electronic device determines that there are still unprocessed images in the video to be processed, the following steps can be performed to generate a Hitchcock effect image. Otherwise, after the electronic device determines that there are no unprocessed images in the video to be processed, it means that the video to be processed has been processed and the process ends.

[0171] S4013: Determine the i-th frame image in the video to be processed.

[0172] Optionally, the electronic device may determine the i-th frame image in chronological order, for example, in chronological order of shooting from early to late, or from late to early, to determine an unprocessed image as the i-th frame image.

[0173] Optionally, the electronic device may determine the i-th frame image according to the frame number size, for example, by determining an unprocessed image as the i-th frame image in order of frame number size from small to large, or from large to small. Exemplarily, the frame number of the N-th RAW image reported by the camera may be N.

[0174] S402: Obtain the i-th camera parameter corresponding to the i-th frame image.

[0175] Optionally, the electronic device may obtain the i-th camera parameter according to the i-th frame number of the i-th frame image.

[0176] The i-th camera parameters include the i-th camera intrinsic parameters and the i-th camera extrinsic parameters. The electronic device can determine the i-th camera intrinsic parameters and the i-th camera extrinsic parameters corresponding to the i-th frame image through the i-th frame number. Among them, the i-th camera intrinsic parameters can be obtained based on the EXIF ​​information of the electronic device. For example, the EXIF ​​information usually includes the focal length of the camera lens, and in the EXIF ​​data, it is usually marked as "Focal Length". In this solution, it is assumed that the user basically keeps the electronic device in a stationary state when shooting the video, and the camera extrinsic parameters obtained are the camera extrinsic parameters corresponding to the preset trajectory. In this case, the electronic device will calculate the corresponding camera extrinsic parameters based on the preset trajectory in combination with the timestamp of each frame of the image.

[0177] Specifically, before shooting a video, the electronic device is pre-set with one or more virtual camera movement trajectories. The preset trajectory can be a straight line, a curve, or other complex path. In an embodiment of the present application, the preset trajectory can be a path for the virtual camera to move from front to back or from back to front.

[0178] The i-th frame number can be used to represent the timestamp corresponding to the i-th frame image, that is, the shooting time of the frame. The electronic device can match the preset trajectory with the timestamp corresponding to the i-th frame image according to the i-th frame number. For example, assuming that the preset trajectory is a straight line for the virtual camera to move from front to back, the system will calculate the position and direction of the virtual camera at that point in time based on the timestamp of each frame image. Based on the correspondence between the trajectory and time, the system can calculate the camera external parameters corresponding to each frame image. These parameters include the position (e.g., coordinates) and direction (e.g., orientation) of the virtual camera in space.

[0179] S403: Obtain an i-th depth map corresponding to the i-th frame image.

[0180] Optionally, the electronic device may perform depth estimation on the i-th frame image to obtain an i-th depth map.

[0181] The present application embodiment does not limit the depth estimation method, including monocular depth estimation, binocular depth estimation and TOF-based depth estimation. The following describes these methods and their implementation methods:

[0182] Monocular depth estimation uses only the i-th frame image to estimate depth information. A pre-trained depth estimation model (such as MiDaS, MonoDepth, etc.) can be used to directly process the i-th frame image to obtain the i-th depth map.

[0183] Binocular depth estimation relies on comparing the positional differences (disparity) between the i-th frame image and another corresponding image taken from a slightly different perspective. Depth information is calculated by comparing the positional differences (disparity) of the same feature points in the i-th frame image and the corresponding image. The basic principle of the binocular vision system is based on triangulation. This involves measuring the positional differences (disparity) of the same scene point in two images with different perspectives. Combining this with the known distance between the cameras (baseline distance) and the camera's intrinsic parameters, the distance of each point in the scene relative to the camera can be calculated.

[0184] Time-of-flight (TOF)-based depth estimation involves a TOF camera directly measuring depth information by sending light pulses and measuring the time it takes for the light to reflect back from an object. When using a TOF camera, depth information is typically provided directly by the camera's hardware and driver software, eliminating the need for additional depth estimation algorithms. Electronic devices can obtain depth maps using the camera's API or SDK.

[0185] When the electronic device only includes a single camera, the i-th depth map can be obtained by monocular depth estimation. When the electronic device includes at least two cameras, the i-th depth map can be obtained by binocular depth estimation. When the electronic device includes a TOF device, the i-th depth map can also be obtained by TOF-based depth estimation. When the electronic device does not include a TOF device, or when only a video to be processed without any depth information is obtained, the electronic device can estimate the depth information of the video to be processed based on the above-mentioned monocular or binocular depth estimation method or other depth estimation technology. For specific depth estimation technology, please refer to the relevant technology, which will not be repeated here.

[0186] S404: Obtain a three-dimensional representation of the i-th frame image.

[0187] Optionally, the electronic device may obtain a three-dimensional representation of the i-th frame image according to the i-th frame image and the i-th depth map.

[0188] Optionally, the electronic device may obtain a three-dimensional representation of the i-th frame image by performing layered depth image (LDI) estimation or multiplane image (MPI) estimation on the i-th frame image and the i-th depth map. The embodiments of the present application are not limited to the use of LDI estimation or MPI estimation, and both methods can obtain a three-dimensional representation of the image. These methods and their implementation methods are described below:

[0189] LDI estimates can be used to represent and render three-dimensional scenes. LDI is an image format that adds depth information, and potentially multiple layers of pixel information, to traditional two-dimensional images. Specifically, LDI consists of a regular grid of integer coordinates, but each location can have 0, 1, or multiple pixels. Each LDI pixel stores an RGB color and a depth value. This format is suitable for scenes with occlusion, enabling more complete capture and reconstruction of the 3D scene.

[0190] The specific steps of performing LDI estimation on the i-th frame image and the i-th depth map to obtain the three-dimensional representation of the i-th frame image are as follows:

[0191] In step S405, the electronic device Figure 8 The i-th frame image shown in Figure (a) is obtained as follows Figure 8 The i-th depth map is shown in Figure (b) in the figure. Since the purpose of this step is to repair the occluded part of the scene, it is necessary to find the depth jump area (also called depth discontinuity area) and expand its content. However, as Figure 8 As shown in Figure (c) in the figure, the areas with depth jumps (discontinuities) in the i-th depth map are blurred across multiple pixels, making it difficult to accurately locate them. Therefore, preprocessing is required to sharpen the depth jump areas. Specifically, a bidirectional median filter can be used to sharpen the i-th depth map, and the following is obtained: Figure 8 The preprocessed i-th depth map is shown in Figure (d) of FIG. This preprocessing step can effectively reduce the blurred areas in the depth map and make the depth jump areas clearer. The specific preprocessing technology can be found in the related art and will not be repeated here.

[0192] However, the depth jump area found after such processing is not accurate enough. In the pre-processed depth map, the electronic device can detect the depth jump area by calculating the parallax difference between adjacent pixels. Specifically, the depth jump area can be determined by setting a parallax difference threshold, such as Figure 8 As shown in Figure (e), the electronic device can detect the presence of some isolated fragments or noise points. In order to improve the accuracy of the depth jump area, the electronic device needs to clean up and connect the detected depth jump area. Therefore, the depth jump area also needs to be processed as follows: Figure 9 The operations shown are:

[0193] First, the electronic device performs binary processing on the detected depth jump area, marking the depth jump area as 1 and other areas as 0. Then, the adjacent depth jump areas are connected by the connected domain analysis method, and isolated fragments and noise points are removed. Specifically, the above method can be used to obtain the following: Figure 9As shown in Figure (a), area A and area B are depth jump areas. It can be seen that the depths of area A and area B are different, so they are not in the same plane, and the edges of the jump between area A and area B are connected by electronic devices to form a continuous LDI representation. The LDI representation formed in this way is the original LDI representation. However, such an LDI representation cannot achieve the repair of the occluded area, such as Figure 9 As shown in Figure (a), there is no corresponding LDI information for area C below area B. If the viewing angle changes and area C below area B leaks out and needs to be displayed, there will be no corresponding depth and color information, resulting in an incomplete visual effect.

[0194] Therefore, context-aware depth and color inpainting of the original LDI information is also required. Specifically, Figure 9 As shown in Figure (b) in the figure, the electronic device can divide area A and area B and remove the continuous area formed by the connection. Figure 9 As shown in Figure (c), the electronic device can use a context-aware depth and color patching method to patch the occluded area (for example, area C below area B is the area that needs to be patched). Specifically, a learning-based patching model can be used to use at least a part of area A as a reference area, generate the color and depth content of area C based on the information of area A, and fuse it into the original LDI representation. The patching model can generate color and depth values ​​consistent with the surrounding environment based on local context information, thereby achieving patching of the occluded area. Finally, as Figure 9 As shown in Figure (d) in the figure, the electronic device synthesizes the repaired depth map and color map into the final layered depth image (LDI) representation. Through the above steps, the electronic device can detect, clean and repair the depth jump area, thereby generating a high-quality LDI representation for subsequent image rendering and perspective synthesis. For example, the processed depth jump area can be seen in Figure 8 (f) in the figure. LDI can adapt to any number of deep layers to express sparseness, has the advantages of good generalization, low storage, and can be converted into lightweight 3D meshes to speed up rendering.

[0195] MPI estimation can be used to represent and render three-dimensional scenes by decomposing the scene into a series of parallel image planes. Figure 10 Figure (a) and Figure 10In Figure (b), each plane (plane A, plane B, and plane C) is located at a different depth. The depth between two adjacent planes is the same, and each plane contains the color and transparency information of the scene at that depth. Specifically, each plane can be an RGBA image. RGBA images are an image format in which each pixel contains four components: red (R), green (G), blue (B), and transparency (Alpha, A).

[0196] The steps of performing MPI estimation on the i-th frame image and the i-th depth map to obtain a three-dimensional representation of the i-th frame image are as follows:

[0197] (1) Determine a series of planes (e.g., plane A, plane B, plane C), each plane corresponding to a specific depth in the scene.

[0198] (2)Reference Figure 10 , project the i-th frame image onto each depth plane and adjust the position of the pixels according to the depth map to simulate the parallax effect of different depths.

[0199] (3) Estimate the transparency (i.e., alpha value) of pixels on each plane to handle occlusion and translucency effects.

[0200] (4) Generate MPI: Combine multiple depth planes with transparency information to form the final MPI representation.

[0201] MPI can simulate occlusion relationships between objects by adjusting the transparency of each plane, thereby adding a sense of depth and three-dimensionality to a two-dimensional image. When the observer's perspective (virtual camera) changes, these planes can be rearranged and rendered accordingly to generate a new scene image, thus achieving free perspective transformation.

[0202] Specifically, if Figure 10 Figure (a) and Figure 10 As shown in Figure (b), the occlusion relationship, position, and size of the same object in the two images corresponding to the reference viewpoint and the target viewpoint are different. The relationship between the reference viewpoint and the target viewpoint is the distance t pushed back from the virtual camera.

[0203] As described above, the process of obtaining a 3D representation of the i-th frame image, by processing each frame of the processed video using the aforementioned LDI or MPI estimation methods, generates its corresponding 3D representation, thus providing the basis for the subsequent virtual lens push operation. However, to produce a Hitchcock effect image, simply changing the position of the virtual lens is not sufficient; the corresponding focal length must also be changed to achieve the effect of maintaining a relatively constant foreground object's frame rate while producing a "pushing and pulling" effect.

[0204] S405: Obtain the equivalent focal length corresponding to the i-th frame image

[0205] Optionally, the electronic device may obtain the equivalent focal length corresponding to the i-th frame image according to the i-th frame number and the i-th camera parameter.

[0206] In an embodiment of the present application, after obtaining the three-dimensional representation corresponding to the image, the virtual lens advances, which simulates the effect of changing the shooting angle and viewing angle by physically moving the lens in actual shooting. Since the screen ratio of the foreground object in the Hitchcock effect video remains basically unchanged, and the screen ratio of the background object gradually decreases over time, a visual compression effect is achieved. In order to achieve this effect of the foreground object screen ratio remaining basically unchanged, the focal length needs to be adjusted. To this end, the embodiment of the present application calculates the equivalent focal length corresponding to each frame of the image, that is, the focal length that needs to be adjusted, and the following is combined with Figure 11 , the calculation process is introduced taking the i-th frame image as an example.

[0207] Figure 11 Figure 2 shows a scene with a virtual lens shift, which includes foreground objects, background objects, and an imaging plane. The imaging plane refers to the location of the image sensor, which is located inside the camera and is where the lens focuses light. The camera gathers light through the lens, and the imaging plane captures this light and converts it into an electronic image.

[0208] For convenience, in Figure 11 Only two planes corresponding to foreground objects and background objects are shown. In actual situations, since there are many objects to be photographed, foreground objects or background objects will correspond to multiple planes. This embodiment of the present application does not limit this. The following is an introduction using the example that the foreground object corresponds to only one plane. Among them, the distance between the object corresponding to the foreground object and the camera is farther, and the distance between the object corresponding to the background object and the camera is closer. Exemplarily, the foreground object refers to the object whose area ratio of the shooting picture in the Hitchcock effect video remains unchanged, and may also be referred to as the target shooting object. Correspondingly, the background object refers to the object whose area ratio of the shooting picture of multiple frames of images in the Hitchcock effect video changes, that is, other shooting objects except the target shooting object. Exemplarily, in the example Figure 1 In the Hitchcock effect video shown, a portrait 101 is a background object, and buildings and trees 102 behind the portrait are foreground objects.

[0209] Specifically, if Figure 11In the virtual lens position translation scene shown, the plane corresponding to the foreground object is the foreground plane S, the plane corresponding to the background object is the background plane S', the original imaging plane is called the first imaging plane O, the distance between the foreground plane S and the first imaging plane O is the foreground depth Z0, and the first focal length of the first imaging plane O relative to the foreground plane S is f0. In this captured scene, after the camera translates a translation distance t, the moved imaging plane is called the second imaging plane O', the distance between the first imaging plane O and the second imaging plane O' is the translation distance t, and the equivalent focal length of the second imaging plane O' relative to the i-th frame image of the foreground plane S is f n The foreground depth Z0 is obtained from the i-th depth map obtained in step S406.

[0210] According to the principle of similar triangles, the first focal length f0 and the equivalent focal length f corresponding to the i-th frame image n The relationship can be expressed as:

[0211]

[0212] Among them, f0 is the first focal length, Z0 is the foreground depth, f n is the equivalent focal length corresponding to the i-th frame image, and t is the translation distance.

[0213] Thus, the electronic device can calculate the equivalent focal length corresponding to the i-th frame image based on the i-th frame number and the i-th camera parameters, wherein the i-th camera parameters include: the first focal length f0. Exemplarily, the calculation process can be shown as the following formula:

[0214]

[0215] Among them, f0 is the first focal length, Z0 is the foreground depth, f n is the equivalent focal length corresponding to the i-th frame image, and t is the translation distance.

[0216] Thus, the electronic device can obtain the equivalent focal length f corresponding to the i-th frame image n The electronic device can then calculate the equivalent focal length of each frame image to generate a Hitchcock effect image. Please refer to the following steps for details, which will not be repeated here.

[0217] Optionally, in other embodiments, the electronic device may further obtain a first scaling factor for the i-th frame image based on the i-th frame number and the depth map of the i-th frame image, and determine the equivalent focal length corresponding to the i-th frame image using the first scaling factor. This is described using the i-th frame image as an example. Exemplarily, the calculation process may be as shown in the following formula:

[0218]

[0219] Wherein, d is the mean depth d of the foreground in the i-th frame image. The specific size of d can be obtained through the i-th depth map. Step is the distance forward of the camera of the adjacent frame. It is preset in the embodiment of the present application and can be considered as a constant. n is the i-th frame number.

[0220] The first scaling factor s can be calculated by the above formula. The electronic device can calculate the first scaling factor s corresponding to each frame image, and then calculate the equivalent focal length f corresponding to the i-th frame image by the following formula: n .

[0221]

[0222] Thus, the electronic device can also obtain the equivalent focal length f corresponding to the i-th frame image n The electronic device can then calculate the equivalent focal length of each frame image to generate a Hitchcock effect image. Please refer to the following steps for details, which will not be repeated here.

[0223] S406: Generate the i-th frame target image

[0224] Optionally, the electronic device may generate a Hitchcock effect image corresponding to the i-th frame image based on the equivalent focal length corresponding to the i-th frame image and the three-dimensional representation of the i-th frame image. The Hitchcock effect image may also be referred to as the i-th frame target image.

[0225] An electronic device can generate the i-th target image corresponding to the i-th image frame using a projection rendering method based on the equivalent focal length corresponding to the i-th image frame and the i-th image frame's 3D representation. Projection rendering refers to a method of reprojecting the image using new camera parameters to generate a new image. The following describes the specific implementation of generating the i-th target image using MPI as an example. The processing method for LDI is similar to MPI and is not further described here.

[0226] In projection rendering, a ray originates from the camera's viewpoint and traverses multiple planes. For each plane, the intersection of the ray and the plane is calculated, resulting in the coordinates of the corresponding pixel. At each intersection, the color value and transparency are obtained. Specifically, the electronic device needs to sample along the ray at multiple depth planes to obtain the color and density information for each sampling point.

[0227] Specifically, for a given target viewpoint V t , an image can be rendered from the MPI of the i-th frame image in a differentiable way by using Planar Inverse Warping. Specifically, each pixel [u t ,v t] is mapped to the pixel [u on the i-th source view MPI plane through the homography function s ,v s ], refer to the following formula for details:

[0228]

[0229] Where R is the rotation matrix, which is a 3×3 matrix used to describe the rotation of the camera in three-dimensional space. It rotates points in the camera coordinate system to the world coordinate system. t is the translation matrix, which is a 3×1 vector used to describe the translation of the camera in three-dimensional space. It translates points in the camera coordinate system to the world coordinate system.

[0230] n is the normal vector of the plane in the source view, which is a 3×1 vector used to describe the normal direction of the plane. In the source view (i.e., the i-th frame image), the plane can be considered to be parallel to the camera optical axis, so the normal vector can be expressed as n = [0,0,1] T .

[0231] R and t are both obtained based on the external parameters of the i-th camera. s and K t are the camera internal parameter matrices, K s Represents the camera intrinsic parameter matrix of the source view (i.e., the i-th frame image), K t The camera intrinsic parameter matrix represents the target view. The camera intrinsic parameter matrix K is a 3×3 matrix used to describe the internal parameters of the camera, including focal length, principal point coordinates, etc. The following takes K as an example (K s and K t Similar, not repeated), its specific form is as follows:

[0232]

[0233] Among them, f x and f y are the focal lengths of the camera in the x and y directions, c x and c y are the coordinates of the optical center.

[0234] Afterwards, the color c′ is obtained by bilinear sampling i and transparency α′ i , assign the accumulated color values ​​to the corresponding pixels and synthesize the rendered image Thus, the target image of the i-th frame is generated, specifically referring to the following formula:

[0235]

[0236] Among them, color c′ iIt includes three components: red (R), green (G) and blue (B). The value range of each component is 0 to 255. It can be directly sampled or calculated according to a preset algorithm or rule. i It can be calculated according to the principle of classical volume rendering using the following formula:

[0237] α i =exp(-δ i σ i )

[0238] where δ i is the density of spatial points, σ i is the distance between the i-th plane and the i+1-th plane, where the i-th plane and the i+1-th plane refer to the depth planes in the MPI estimation process in step S406.

[0239] Using this formula, electronic devices can transform pixel positions between different perspectives based on the 3D representation of each frame, providing a foundation for subsequent image synthesis. The perspective change trajectory can be pre-set, for example, where the virtual lens position moves away from the subject at equal distances from frame to frame.

[0240] Next, we will introduce how to use the results obtained from the above formula to generate a complete image. Specifically, it involves the process of generating a complete image on a blank RGB image. Assume that the resolution of the blank image is 640×640 pixels, and each pixel has a corresponding [u t ,v t ] coordinates. By traversing all [u t ,v t ] coordinates, a complete RGB image can be generated. This method is also applicable to other image formats, depending on the required image type. The embodiment of this application does not limit the output format. The resolution of the blank image can also be other resolution sizes instead of 640×640 pixels, and this embodiment of the application does not limit this. Next, take the output resolution of 640×640 pixels as an example to introduce the specific steps:

[0241] First, initialize a blank RGB image with a resolution of 640×640 pixels. Each pixel in this image is initially set to zero, or black. Define a two-dimensional coordinate system on this image, where u and v represent the horizontal and vertical coordinates of the image, respectively. The value of u ranges from 0 to 639, and the value of v ranges from 0 to 639.

[0242] After that, traverse every pixel on the image, that is, traverse all [u t ,v t ] coordinates. For each coordinate point [u t,v t ], calculate the color value of that pixel according to the formula mentioned above, and assign the calculated color value to that pixel. After the traversal is complete, the color values ​​of all pixels have been determined, and the resulting image is a complete RGB image. This image can be used for display, storage, or further processing.

[0243] Through the above steps, this technical solution achieves the process of generating a complete RGB image from a blank image. This method is versatile and applicable to image generation requirements of different resolutions and image formats. It can also be applied to other image formats, such as grayscale images and YUV images. The specific image format depends on actual needs. When generating an image, different output formats can be selected as needed.

[0244] S407: Generate target video

[0245] The above steps describe the process of generating the i-th target image from the i-th frame image in the video to be processed. The following steps describe the steps of generating the Hitchcock effect video (i.e., the target video) based on the obtained target image:

[0246] For each frame in the video, rendering is performed using the corresponding virtual lens position based on its order (i.e., frame number) in the video (i.e., the video being processed). For example, if the video has 100 frames and 100 virtual lens advances (or pulls) are performed, the i-th frame of the video will be rendered using the i-th virtual lens position. In other words, each frame in the video being processed can be processed to obtain N target images, thus obtaining the target video. Furthermore, each frame is processed using its corresponding information before rendering.

[0247] Specifically, its own information may include: camera intrinsic parameters, camera extrinsic parameters, depth information and / or three-dimensional representation, etc. Taking depth information as an example, the electronic device may obtain camera parameters and depth information of the i-th image, obtain a three-dimensional representation of the i-th image based on the information (e.g., pixel values) of the i-th image and the depth information of the i-th image, and obtain the i-th target image based on the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

[0248] Optionally, the electronic device can also perform frame skipping rendering, that is, not rendering every frame of the image, but rendering once every few frames. Figure 12For example, an electronic device can render one frame every three frames (for example, the first frame of the three frames is rendered in the figure, but other frames within the three frames can also be rendered) to obtain a Hitchcock effect image and display the Hitchcock effect image for a certain duration (for example, the duration originally required to display the three frames). For example, the video to be processed includes N frames, and M frames of the N frames are processed to obtain M target images to obtain the target video, where M is a positive integer less than N. This approach can reduce computational complexity, power consumption, and improve rendering speed. Specifically, while the electronic device previously needed to render each frame, frame skipping can save the computational complexity and time required for the skipped frames. As a result, the electronic device can generate Hitchcock effect videos faster and better. For example, if the user chooses to generate a low-frame-rate video, or the electronic device is in low-power mode, or the user chooses to capture / record a Hitchcock effect video using instant capture / real-time recording, the electronic device can perform frame skipping to shorten the time required to generate the Hitchcock effect video.

[0249] Optionally, the electronic device may also render each frame, but not use the depth information and camera parameters of the current frame for rendering each frame. Instead, the depth information and camera parameters are obtained once every x frames for rendering these x frames. That is, although each frame of the image is obtained and a Hitchcock image is generated, the corresponding information is not necessarily used to generate the Hitchcock image. For example: the electronic device may obtain the camera parameters of the i-th image and the depth information of the i-1-th image; obtain the three-dimensional representation of the i-th image based on the pixel values ​​of the i-th image and the depth information of the i-1-th image; and obtain the i-th target image based on the camera parameters of the i-th image and the three-dimensional representation of the i-th image. It should be noted that the example shown is that the depth information and camera parameters are obtained once every 2 frames, and the current frame image (i-th image) uses the depth information of the previous frame image (i-1-th image) to calculate the three-dimensional representation of the current frame image (i-th image). However, the depth information and camera parameters can also be obtained once every x frames. For example, the current frame image (the i-th image) uses the depth information of the previous x frames (the ixth image) to calculate the three-dimensional representation of the current frame image (the i-th image).

[0250] This approach can also preserve as much detail as possible while reducing computational complexity and power consumption, improving rendering speed, and preserving the changing details of the video image. For example, when the electronic device is in low-power mode, or when the user chooses to shoot and record / record a Hitchcock-effect video in real time, the electronic device can use this approach to process and display the Hitchcock-effect video relatively quickly and in real time, while also preserving as much detail and display effect as possible.

[0251] When generating a Hitchcock effect video, the trajectory of the virtual lens position change can be preset. For example, the virtual lens position can be moved away from the subject at equal distances frame by frame to form a trajectory of virtual lens position change. Specifically, for each frame of the video, the system calculates the corresponding virtual lens position based on the preset trajectory and uses the aforementioned method to render the corresponding Hitchcock effect image based on this position.

[0252] By connecting these rendering results in the order or reverse order of the original video sequence, a video that not only shows the dynamic changes of the background but also maintains the dynamic effect of the foreground is finally generated, that is, the Hitchcock effect video (target video) is obtained. The optional scheme of skipping frame rendering or skipping frame information acquisition can further save power consumption and / or improve rendering speed. For example, Figure 5 As shown, the electronic device can perform image processing synchronously during the shooting process.

[0253] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step in the above-mentioned method embodiment.

[0254] An embodiment of the present application further provides a computer program product, including a computer program. When the computer program runs on an electronic device, the electronic device executes each function or step in the above method embodiment.

[0255] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0256] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0257] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0258] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0259] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0260] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method includes: Obtain a first video, where the first video includes N frames of images, where N is a positive integer greater than or equal to 2; Processing the first video to obtain a target video, the target video including an i-th target image and an i+1-th target image, where i is a positive integer, the i-th target image including a first target object and a first background, and the i+1-th target image including a second target object and a second background; The image field size of the first target object is the same as that of the second target object, the proportion of the screen occupied by the first target object is the same as that of the second target object, the image field size of the first background is different from that of the second background, and the proportion of the screen occupied by the first background is the same as that of the second background.

2. The method according to claim 1, characterized in that The processing of the first video to obtain a target video includes: Each frame of the N frames of images is processed to obtain N frames of target images, so as to obtain the target video.

3. The method according to claim 1, characterized in that The processing of the first video to obtain a target video includes: M frames of images among the N frames of images are processed to obtain M frames of target images, where M is a positive integer less than N, to obtain the target video.

4. The method according to claim 3, characterized in that The processing of M frames of images among the N frames of images to obtain M frames of target images includes: Selecting one frame of image for every x frames of image from the N frames of image to obtain the M frames of image, where x is a positive integer less than N; Each frame of the M frames of images is processed to obtain M frames of target images.

5. The method according to claim 2, characterized in that The N frames of images include an i-th image, and processing each of the N frames of images to obtain N frames of target images includes: Get the camera parameters and depth information of the i-th image; Obtaining a three-dimensional representation of the i-th image according to the pixel values ​​of the i-th image and the depth information of the i-th image; The i-th target image is obtained according to the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

6. The method according to claim 2, characterized in that The N frames of images include an ixth image and an i-th image, where i is a positive integer greater than x, and processing each of the N frames of images to obtain N frames of target images includes: Obtaining camera parameters of the i-th image and depth information of the ix-th image; Obtaining a three-dimensional representation of the i-th image according to the pixel values ​​of the i-th image and the depth information of the ixth image; The i-th target image is obtained according to the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

7. The method according to claim 4, characterized in that The M frames of images include an i-th image, where i is a positive integer, and processing each frame of the M frames of images to obtain M frames of target images includes: Obtaining camera parameters and depth information of the i-th image; Obtaining a three-dimensional representation of the i-th image according to the pixel values ​​of the i-th image and the depth information of the i-th image; The i-th target image is obtained according to the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

8. The method according to claim 4, characterized in that The M frames of images include an iy-th image and an i-th image, where i is a positive integer greater than y, and processing each of the M frames of images to obtain M frames of target images includes: Obtaining the camera parameters of the i-th image and the depth information of the iy-th image; Obtaining a three-dimensional representation of the i-th image according to the pixel values ​​of the i-th image and the depth information of the iy-th image; The i-th target image is obtained according to the camera parameters of the i-th image and the three-dimensional representation of the i-th image.

9. The method according to any one of claims 5 to 8, characterized in that The camera parameters of the i-th image are the camera intrinsic parameters and camera extrinsic parameters corresponding to the i-th image, the camera intrinsic parameters include: focal length, principal point coordinates, radial distortion coefficient and / or tangential distortion coefficient, and the camera extrinsic parameters include: rotation parameters and / or translation parameters.

10. The method according to any one of claims 5 to 9, characterized in that The three-dimensional representation of the i-th image is a layered depth image LDI representation or a multi-plane image MPI representation.

11. The method according to any one of claims 5 to 10, characterized in that The obtaining the i-th target image according to the camera parameters of the i-th image and the three-dimensional representation of the i-th image includes: The i-th equivalent focal length is calculated according to the frame number of the i-th image and the camera parameters of the i-th image; The i-th image is rendered according to the i-th equivalent focal length and the three-dimensional representation of the i-th image to obtain the i-th target image.

12. The method according to any one of claims 1 to 11, characterized in that The viewpoint corresponding to the first target object is a first viewpoint, the viewpoint corresponding to the second target object is a second viewpoint, and positions of the first viewpoint and the second viewpoint are different.

13. The method according to any one of claims 1 to 12, characterized in that Obtaining the first video includes: In response to a shooting operation of the user, a first video is obtained.

14. The method according to claim 13, wherein: The step of obtaining a first video in response to a shooting operation of a user includes: Displaying a first interface, wherein the first interface includes a first control; In response to a first operation of a first control by a user, the electronic device starts shooting; In response to the user's second operation on the first control, the electronic device ends shooting and obtains the first video.

15. An electronic device, characterized in that: The electronic device includes a display screen, a memory, and one or more processors; the display screen, the memory, and the processor are coupled; the display screen is used to display an image generated by the processor, and the memory is used to store computer program code, wherein the computer program code includes computer instructions; when the processor executes the computer instructions, the electronic device executes the method as described in any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 14.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the electronic device is caused to perform the method according to any one of claims 1 to 14.

Images