Shooting method, device, equipment, medium and product

Abstract

The invention discloses a shooting method, device and equipment, a medium and a product, and belongs to the technical field of shooting. The shooting method is applied to the electronic equipment, the electronic equipment comprises a first camera and a second camera, and the field angle of the first camera is smaller than that of the second camera. The shooting method comprises the following steps: receiving shooting input; in response to the shooting input, carrying out shooting through the first camera to obtain a first shot image, and carrying out shooting through the second camera to obtain a second shot image; image expansion is carried out on the first shot image according to the second shot image to obtain an expanded image, and the resolution of the expanded image is the same as that of the second shot image; and displaying the extended image and the second shot image in the horizontal direction.

Description

Technical Field

[0001] The present application belongs to the field of photographing technology, and specifically relates to a photographing method, device, equipment, medium and product. Background Art

[0002] An immersive image is an image that can cover the human eye's field of view of at least 120 (horizontally) × 70 (vertically) and can form a 3D effect. In other words, an immersive image is an image with a large field of view (FOV) and can form a 3D effect. This type of image can give users an immersive experience and has been widely used in many fields such as games and movies.

[0003] Currently, there are several ways to shoot immersive images: 1. Shooting with professional shooting equipment; 2. Shooting with multiple electronic devices at the same time using a large FOV; 3. Shooting by manually moving an electronic device.

[0004] Among them, using professional filming equipment for shooting is very expensive; using multiple electronic devices to shoot at the same time or manually moving an electronic device for shooting is cumbersome and easily introduces many errors, and cannot form an effective 3D effect. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a shooting method, device, equipment, medium and product, which can solve the technical problems of high cost or cumbersome operation and inability to form effective 3D effects when shooting immersive images in related technologies.

[0006] In a first aspect, an embodiment of the present application provides a shooting method, which is applied to an electronic device, wherein the electronic device includes a first camera and a second camera, and the field of view of the first camera is smaller than the field of view of the second camera; the method includes:

[0007] Receive shooting input;

[0008] In response to a shooting input, shooting with a first camera to obtain a first shot image, and shooting with a second camera to obtain a second shot image;

[0009] Expanding the first captured image according to the second captured image to obtain an expanded image, wherein the expanded image has the same resolution as the second captured image;

[0010] The expanded image and the second captured image are displayed in the horizontal direction.

[0011] In a second aspect, an embodiment of the present application provides a shooting device, which is applied to an electronic device, wherein the electronic device includes a first camera and a second camera, and the field of view of the first camera is smaller than the field of view of the second camera; the device includes:

[0012] A receiving module, used for receiving shooting input;

[0013] A shooting module, configured to, in response to a shooting input, shoot through a first camera to obtain a first shot image, and shoot through a second camera to obtain a second shot image;

[0014] a processing module, configured to perform image expansion on the first captured image according to the second captured image to obtain an expanded image, wherein the expanded image has the same resolution as the second captured image;

[0015] The display module is used to display the extended image and the second captured image in a horizontal direction.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0018] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the method described in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0020] In an embodiment of the present application, the electronic device includes a first camera and a second camera, wherein the field of view of the first camera is smaller than the field of view of the second camera, and when receiving a shooting input, the electronic device can obtain a first shot image by shooting through the first camera, and obtain a second shot image by shooting through the second camera, that is, through one electronic device, a large FOV image and a small FOV image can be obtained at the same time by shooting once, and on this basis, the small FOV image is expanded using the large FOV image to obtain an expanded image with the same resolution as the large FOV image, and then two images of large FOV are obtained, which correspond to the left and right views of the human eye, respectively, so as to form an effective 3D effect. The embodiment of the present application can obtain an immersive image of a large FOV through one electronic device and one shooting input, without the need to use professional shooting equipment, thus reducing costs, and without the need to use multiple electronic devices or manually move electronic devices, thus reducing operations, eliminating errors, and improving the quality of immersive images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of a shooting training method provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of another shooting training method provided in an embodiment of the present application;

[0023] Figure 3 A flowchart of another shooting training method provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of another shooting training method provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of the structure of a photographing device provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually a class, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] Immersive images are images with a large FOV and can form a good 3D effect, allowing users to have an immersive feeling. They have been widely used in games, movies and other fields.

[0031] At present, immersive images are mainly obtained through professional equipment, or by shooting with multiple large FOV devices at the same time, or by moving small FOV devices multiple times to shoot. For the first method, the cost is very high. For the latter two methods, not only are the operations cumbersome, but also large errors will occur, making it difficult to form an effective 3D effect.

[0032] To this end, the embodiments of the present application provide a shooting method, device, equipment, medium and product, which can solve the technical problems of high cost or cumbersome operation and inability to form effective 3D effects when shooting immersive images in related technologies.

[0033] The following describes in detail the shooting method, device, equipment, medium and product provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0034] The shooting method provided in the embodiment of the present application can be applied to the scenes of shooting immersive images and immersive videos. When it is necessary to shoot an immersive video, each frame of the shot image can be processed according to the scheme provided in the embodiment of the present application to obtain an immersive video.

[0035] The embodiment of the present application can obtain an immersive image with a large FOV through one electronic device and one shooting input, without the need to use professional shooting equipment, thus reducing costs. There is also no need to use multiple electronic devices or manually move electronic devices, thus reducing operations, eliminating errors, and improving the quality of the immersive image.

[0036] Figure 1A flowchart of a shooting method provided for an embodiment of the present application, the shooting method can be applied to an electronic device with a shooting function, the electronic device can include at least two cameras, taking a first camera and a second camera as an example, the first camera and the second camera have different field of view angles, for example, the field of view angle of the first camera can be smaller than the field of view angle of the second camera, that is, the first camera is a small FOV camera, and the second camera is a large FOV camera. Of course, the field of view angle of the first camera can also be greater than the field of view angle of the second camera, that is, the first camera is a large FOV camera, and the second camera is a small FOV camera, which does not affect the essence of the solution. In some embodiments, the optical centers of the first camera and the second camera can be at the same level, which can simplify the amount of calculation involved in the subsequent image processing process and improve the image processing efficiency.

[0037] In the embodiment of the present application, the field of view angle of the first camera is smaller than the field of view angle of the second camera. In some embodiments, the field of view angle of the first camera can be approximately equal to 80°, and the field of view angle of the second camera can be greater than or equal to 120°. It can be understood that the above 80° is only an exemplary description. In actual application, an angle less than 80° or greater than 80° but less than the field of view angle of the second camera can also be selected.

[0038] Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, etc.

[0039] like Figure 1 As shown, the shooting method may include the following steps:

[0040] S110: Receive shooting input.

[0041] S120: In response to a shooting input, shooting is performed by a first camera to obtain a first shot image, and shooting is performed by a second camera to obtain a second shot image.

[0042] S130: Perform image expansion on the first captured image according to the second captured image to obtain an expanded image.

[0043] The extended image has the same resolution as the second captured image.

[0044] S140 , display the expanded image and the second captured image in a horizontal direction.

[0045] In the embodiment of the present application, a large FOV image and a small FOV image can be obtained simultaneously by shooting once with an electronic device. On this basis, the small FOV image can be expanded using the large FOV image to obtain an expanded image with the same resolution as the large FOV image, and then two images of large FOV can be obtained, corresponding to the left and right views of the human eye, respectively, so as to form an effective 3D effect. The embodiment of the present application can obtain an immersive image of a large FOV through an electronic device and a single shooting input, without the need to use professional shooting equipment, reducing costs, and without the need to use multiple electronic devices or manually move electronic devices, reducing operations, eliminating errors, and improving the quality of immersive images.

[0046] The above steps are described in detail below:

[0047] In S110, the shooting input is used to make the first camera and the second camera shoot at the same time. Exemplarily, the above-mentioned shooting input includes but is not limited to: the user's touch input to the shooting shutter through a touch device such as a finger or a stylus, or a voice command input by the user, or a specific gesture input by the user, or other feasible inputs, which can be determined according to actual use requirements and are not limited in the embodiments of the present application. The specific gesture in the embodiments of the present application can be any one of a single-click gesture, a sliding gesture, a drag gesture, a pressure recognition gesture, a long press gesture, an area change gesture, a double-press gesture, and a double-click gesture; the click input in the embodiments of the present application can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long press input or a short press input.

[0048] For example, the above-mentioned shooting input may be: a user's click input on a shooting shutter.

[0049] In S120, the first captured image is an image captured by the first camera, also known as a small FOV image. The second captured image is an image captured by the second camera, also known as a large FOV image. Both the small FOV image and the large FOV image are in RGB format.

[0050] In an embodiment of the present application, after the electronic device receives the user's shooting input, it can control the first camera and the second camera to shoot at the same time, so that a small FOV image and a large FOV image can be obtained at the same time. This can effectively avoid the problem of inconsistent shooting time of the two images, and further avoid introducing more errors and affecting the 3D effect of the image.

[0051] In S130, the expanded image here is the processed small FOV image. It can be understood that the small FOV image has a smaller field of view, so the details of its central area are richer than those of the large FOV image. Therefore, the large FOV image can be used to expand the small FOV image to retain important information of the small FOV image and improve the quality of the immersive image.

[0052] The expanded small FOV image is a large FOV image, and the resolution of the large FOV image is the same as the large FOV image directly captured by the second camera, that is, the two images finally obtained are large FOV images with the same resolution, thereby forming an effective 3D effect.

[0053] Exemplarily, the overlapping area of ​​the large FOV image and the small FOV image can be determined, and then the surrounding area of ​​the small FOV image can be spliced ​​using the area of ​​the large FOV image except the overlapping area to expand the small FOV image and obtain an extended image.

[0054] Exemplarily, an image fusion algorithm may be used to fuse a large FOV image and a small FOV image to obtain an extended image.

[0055] Exemplarily, the large FOV image and the small FOV image may also be input into the image extension model, and the image extension model may extend the small FOV image according to the large FOV image and output the extended image.

[0056] Of course, other expansion methods may also be used, which are not limited in the embodiments of the present application. By expanding the small FOV image, an immersive image with a 3D sense can be obtained by shooting once.

[0057] In S140, after the electronic device obtains the extended image, it can display the extended image and the large FOV image captured by the second camera in the horizontal direction. The two images correspond to the left and right views of the human eye respectively. In this way, the user can see the 3D effect by watching the two images displayed by the electronic device through a 3D device.

[0058] The embodiment of the present application can obtain an immersive image with a large FOV through a traditional electronic device and a single shooting input, without the need to use professional large-scale shooting equipment, thus reducing costs. There is also no need to use multiple electronic devices or manually move electronic devices, thus reducing operations, eliminating errors, and improving the quality of the immersive image.

[0059] Figure 2 A flowchart of another shooting method provided in an embodiment of the present application, Figure 2 and Figure 1 The difference is that Figure 1 The S130 in can be refined into Figure 2 S210-S230 in.

[0060] S210: Convert the second captured image to the plane where the first captured image is located to obtain a converted image.

[0061] The converted image here is the converted second captured image, that is, the converted second captured image and the first captured image are in the same plane.

[0062] The embodiments of the present application do not limit the specific conversion process. For example, the second captured image can be converted to the plane where the first captured image is located through a homography matrix, or the second captured image can be converted to the plane where the first captured image is located through projection.

[0063] S220 , performing feature extraction on the first captured image to obtain first feature points of the first captured image, and performing feature extraction on the converted image to obtain second feature points of the converted image.

[0064] The first feature point is a feature point extracted from the first captured image that can characterize the characteristics of the first captured image, and the first feature point may include multiple features. Similarly, the second feature point is a feature point extracted from the second captured image that can characterize the characteristics of the second captured image, and the second feature point may include multiple features.

[0065] The embodiment of the present application does not limit the method for extracting feature points. For example, the feature points of the first captured image and the second captured image can be extracted respectively through a deep learning model, or the feature points of the first captured image and the second captured image can be extracted respectively through a machine learning algorithm.

[0066] By extracting feature points of the first shot image and the second shot image, it is helpful to subsequently expand the first shot image according to the second shot image.

[0067] S230 , performing image expansion on the first captured image according to the matching relationship between the first feature point and the second feature point and the converted image to obtain an expanded image.

[0068] The matching relationship between the first feature point and the second feature point is used to characterize which first feature point matches which second feature point. The matching here means that the two feature points correspond to the same pixel point, that is, the point seen by the user is the same point in the two images.

[0069] Due to different viewing angles, the first feature point and the second feature point actually obtained are not in a one-to-one correspondence. Therefore, after obtaining the first feature point and the second feature point, the matching relationship between the first feature point and the second feature point can be determined by a feature matching algorithm to obtain a feature point pair. Each feature point pair includes a first feature point and a second feature point, and the first feature point and the second feature point correspond to the same pixel point.

[0070] According to the matching relationship between the first feature point and the second feature point and the converted image, the first captured image can be expanded from a small FOV image to a large FOV image, and the resolution of the expanded large FOV image is the same as that of the large FOV image directly captured by the second camera, thereby effectively forming a 3D effect.

[0071] Exemplarily, the above S1303 may include the following steps:

[0072] Determining an overlapping area of ​​the first captured image and the converted image according to a matching relationship between the first feature point and the second feature point;

[0073] An image area other than the overlapping area in the converted image is synthesized with the first captured image to obtain an extended image.

[0074] The overlapping area is the area where the two images contain the same pixels. Exemplarily, based on the feature point matching pairs, it is possible to determine which pixels in the first captured image overlap with pixels in the second captured image, thereby obtaining the overlapping area of ​​the first captured image and the second captured image.

[0075] After the overlapping area is determined, an image area other than the overlapping area can be determined from the second captured image, and the image area can be synthesized with the first captured image to obtain an extended image, thereby achieving the expansion of the first captured image. Exemplarily, the non-overlapping area of ​​the second captured image can be spliced ​​to the periphery of the first captured image in combination with an adaptive sparse method, so that the details of the first captured image can be retained and the image quality can be improved.

[0076] In order to further improve the quality of the extended image, after the extended image is initially obtained, the extended image can be post-processed such as color correction, contrast adjustment, smoothing of splicing, etc.

[0077] The embodiment of the present application determines the overlapping area of ​​the two images based on the matching relationship between the feature points of the two images in the same plane, and then expands the first captured image based on the overlapping area using the peripheral area of ​​the second captured image, thereby retaining the detailed content of the first captured image and improving the quality of the expanded image.

[0078] Figure 3 A flowchart of another shooting method provided in an embodiment of the present application, Figure 3 and Figure 2 The difference is that Figure 2 The S210 in can be refined into Figure 3 S310-S320 in.

[0079] S310, performing distortion correction on the first captured image according to the first calibration parameter of the first camera to obtain a corrected first captured image, and performing distortion correction on the second captured image according to the second calibration parameter of the second camera to obtain a corrected second captured image.

[0080] The calibration parameters of the camera are the parameters obtained by calibrating the camera. The calibration parameters may include internal parameters and external parameters. The internal parameters may be parameters describing the internal properties of the camera, such as the focal length, optical center coordinates, and distortion parameters of the camera. The external parameters may be parameters describing the position and posture of the camera in the world coordinate system, usually including a rotation matrix and a translation vector.

[0081] The first calibration parameter here is a parameter obtained by calibrating the first camera, and the first calibration parameter may include the internal parameter and external parameter of the first camera. The second calibration parameter here is a parameter obtained by calibrating the second camera, and the second calibration parameter may include the internal parameter and external parameter of the second camera. The specific calibration process is not described in detail in the embodiment of the present application. According to the external parameter of the first camera and the external parameter of the second camera, the relative position and direction of the first camera and the second camera can be further obtained.

[0082] For example, before shooting, the first camera and the second camera can be calibrated to obtain first calibration parameters and second calibration parameters. Subsequently, after obtaining the first captured image and the second captured image, the first calibration parameters and the second calibration parameters can be directly used to perform image processing on the first captured image and the second captured image, thereby improving image processing efficiency.

[0083] Distortion correction is used to correct the geometric distortion of an image and improve the geometric quality of the image. Geometric distortion may include, for example, scale distortion, skew distortion, center shift distortion, etc. of the image.

[0084] For example, the first captured image may be subjected to distortion correction according to the distortion parameters of the first camera, and the second captured image may be subjected to distortion correction according to the distortion parameters of the second camera, so as to improve the quality of the first captured image and the second captured image. The specific correction process is not limited in the embodiments of the present application.

[0085] S320: Convert the corrected second captured image to the plane where the corrected first captured image is located to obtain a converted image.

[0086] The plane where the corrected first captured image is located is the plane where the first captured image is located before correction. The plane where the corrected second captured image is located is the plane where the second captured image is located before correction. After the image is corrected, the corrected second captured image can be converted to the plane where the first captured image is located. By image correction, the quality of the image can be improved. When the first captured image is subsequently expanded, the quality of the expanded image can be improved, thereby forming a more realistic 3D sense.

[0087] Exemplarily, the above S320 may include the following steps:

[0088] Determining a homography matrix according to the first calibration parameter and the second calibration parameter;

[0089] According to the homography matrix, the corrected second captured image is converted to the plane where the corrected first captured image is located to obtain a converted image.

[0090] The homography matrix is ​​a conversion matrix between two planes, that is, it can convert an image from one plane to another plane. For example, in the embodiment of the present application, the second captured image can be converted to the plane where the first captured image is located through the homography matrix, so that the converted second captured image and the first captured image are in the same plane, which is convenient for subsequent image expansion and can improve the quality of the expanded image.

[0091] Exemplarily, the homography matrix can be determined according to the internal and external parameters of the two cameras, and the specific determination process is not described in detail here. Exemplarily, the homography matrix obtained above can be further optimized by an iterative method such as the Levenberg-Marquardt algorithm to obtain a homography matrix with the minimum reprojection error.

[0092] According to the homography matrix obtained above, the corrected second captured image can be converted to the plane where the corrected first captured image is located to obtain a converted image.

[0093] For example, A=H*B, where A is the pixel coordinate of the converted image, H is the homography matrix with a size of 3*3, and B is the pixel coordinate of the second captured image before conversion, B=[x, y, 1] T , T represents transpose.

[0094] The embodiment of the present application utilizes the calibration parameters of the camera to correct the geometric distortion of the image and obtain the homography matrix according to the calibration parameters, and then converts the second captured image to the plane where the first captured image is located according to the homography matrix, providing a more accurate basis for subsequent image expansion.

[0095] Taking the case where the optical center of the first camera and the optical center of the second camera are located on the same horizontal line, Figure 4A flowchart of another shooting method provided in an embodiment of the present application, Figure 4 and Figure 2 The difference is that Figure 2 The S210 in can be refined into Figure 4 S410-S420 in.

[0096] S410: Perform monocular depth estimation on the second captured image to obtain a depth image of the second captured image.

[0097] Monocular depth estimation is used to obtain a depth image of the second captured image. The depth image is a three-dimensional image. Each pixel in the depth image has a pixel coordinate and a depth value of the image. At the same time, each point corresponds to the RGB image one-to-one, so each point also contains RGB color information.

[0098] For example, a depth image with accurate depth and high resolution can be obtained through processes such as optimal resolution search, dual depth estimation, and local enhancement of the depth map.

[0099] S420 . According to the conversion matrix and the projection matrix, convert the depth image to the plane where the first captured image is located to obtain a converted image.

[0100] The transformation matrix is ​​used to transform the depth image from the world coordinate system to the camera coordinate system, and the projection matrix is ​​used to project the depth image in the camera coordinate system to the clipping space. The two work together to achieve the conversion of 3D image to 2D image. The specific operation process is not limited in the embodiment of this application. The size of the transformation matrix is ​​4*4. The transformation matrix includes a rotation part and a displacement part.

[0101] Because the first image and the second image are taken by cameras with aligned optical centers, there is no rotation in the depth image conversion process, only translation, that is, the rotation part of the conversion matrix is ​​the unit matrix, and the displacement part is the movement distance of the first camera relative to the second camera. The unit matrix here is a matrix with 1 on the main diagonal and 0 on the rest of the elements.

[0102] Exemplarily, the transformation matrix can be expressed as: Wherein, R represents the rotation part, the size of which is 3*3. In the embodiment of the present application, R is a 3*3 unit matrix. P represents the displacement part, the size is 3*1, for example, P = [dx, dy, dz] T, dx, dy, dz represent the movement distance of the first camera relative to the second camera in the x, y, z directions, respectively. It should be noted that when the optical centers of the first camera and the second camera are not on the same horizontal line, the rotation part in the above transformation matrix is ​​no longer a unit matrix, that is, when the optical centers of the first camera and the second camera are not on the same horizontal line, it is necessary not only to translate the depth image but also to rotate the depth image.

[0103] For example, according to the transformation matrix and the projection matrix, the 3D depth image can be transformed into the 2D plane where the first captured image is located, providing a basis for subsequent image expansion. When the depth image is transformed into the 2D plane, each point still has RGB information, so the transformed RGB image, that is, the transformed image, can be obtained.

[0104] The embodiment of the present application obtains a depth image of the second captured image by means of monocular depth estimation, and then realizes conversion between images of different FOVs according to the depth image, thereby eliminating the dependence on camera calibration and simplifying the operation.

[0105] Considering that there may be missing parts when converting a 3D image to a 2D plane, in order to improve the quality of the immersive image, in some embodiments, the above S420 may include the following steps:

[0106] According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain a transformed initial image;

[0107] In the case where the converted initial image is missing, determining a method for repairing the missing area according to the distance between the first camera and the second camera and the shooting distance between the first camera and the second camera;

[0108] The missing area is repaired according to the repair method, and the repaired initial image is determined as the converted image.

[0109] The initial image here is the image directly obtained by converting the depth image to the plane where the first captured image is located. The shooting distance of the first camera is the distance between the first camera and the captured object, and the shooting distance of the second camera is the distance between the second camera and the captured object. The missing area is the missing part in the initial image, which is usually manifested as a black hole. The repair method is used to repair the missing part in the converted image. The size of the missing area is different, and the corresponding repair method can be different. Therefore, the appropriate repair method can be adaptively selected according to the size of the missing area to improve the image repair quality.

[0110] Exemplarily, when the initial image is missing, the method for repairing the missing area can be determined based on the distance between the first camera and the second camera and the shooting distance between the first camera and the second camera. For example, the smaller the distance between the two cameras, the longer the shooting distance, the smaller the resulting hole, and the larger the distance between the two cameras, the closer the shooting distance, the larger the resulting hole. In actual application, different thresholds can be set. When the distance between the two cameras is less than the first threshold and the shooting distance is greater than the second threshold, it is considered that the distance between the two cameras is small, the shooting distance is far, and the resulting hole is correspondingly smaller. Conversely, when the distance between the two cameras is greater than or equal to the first threshold and the shooting distance is less than the second threshold, it is considered that the distance between the two cameras is large, the shooting distance is close, and the resulting hole is correspondingly larger.

[0111] Exemplarily, when the hole is small, it can be repaired by bilateral filtering, and when the hole is large, it can be repaired by using Inpainting technology. Of course, the above two methods can also be used in combination to improve the repair quality.

[0112] The embodiment of the present application can verify the converted initial image. When the initial image is incomplete, it can adaptively select a suitable repair method according to the distance between the two cameras and the shooting distance to repair the missing area, thereby improving the flexibility of image repair and ensuring the quality of image repair.

[0113] The embodiment of the present application installs two cameras with different field of view angles in an electronic device, and can simultaneously obtain a large FOV image and a small FOV image through one shooting, and expand the small FOV image into a large FOV image through image expansion technology. That is, the embodiment of the present application can obtain an immersive image with a large FOV through one shooting with an electronic device, and can form a good 3D effect, without the need to use professional equipment, multiple devices, or multiple shooting, which not only reduces costs but also reduces user operations.

[0114] It should be noted that the shooting method provided in the embodiment of the present application can be executed by a shooting device or a processing module in the shooting device for executing the shooting method. In the embodiment of the present application, the shooting device provided in the embodiment of the present application is described by taking the shooting method executed by the shooting device as an example.

[0115] Figure 5 A schematic diagram of the structure of a photographing device provided in an embodiment of the present application.

[0116] like Figure 5 As shown, the photographing device 500 may include:

[0117] Receiving module 501, used for receiving shooting input;

[0118] The shooting module 502 is used for shooting a first image through a first camera and shooting a second image through a second camera in response to a shooting input;

[0119] The processing module 503 is used to perform image expansion on the first captured image according to the second captured image to obtain an expanded image, wherein the expanded image has the same resolution as the second captured image;

[0120] The display module 504 is configured to display the extended image and the second captured image in a horizontal direction.

[0121] In an embodiment of the present application, the electronic device includes a first camera and a second camera, wherein the field of view of the first camera is smaller than the field of view of the second camera. When receiving a shooting input, the electronic device can obtain a first shot image by shooting with the first camera, and obtain a second shot image by shooting with the second camera, that is, through one electronic device, a large FOV image and a small FOV image can be obtained at the same time by shooting once. On this basis, the small FOV image is expanded using the large FOV image to obtain an expanded image with the same resolution as the large FOV image, and then two images of large FOV are obtained, which correspond to the left and right views of the human eye, respectively, so as to form an effective 3D effect. The embodiment of the present application can obtain an immersive image of a large FOV through one electronic device and one shooting input, without the need to use professional shooting equipment, thus reducing costs, and without the need to use multiple electronic devices or manually move electronic devices, thus reducing operations, eliminating errors, and improving the quality of immersive images.

[0122] In some possible implementations of the embodiments of the present application, the processing module 503 is specifically configured to:

[0123] Converting the second captured image to the plane where the first captured image is located to obtain a converted image;

[0124] Performing feature extraction on the first captured image to obtain first feature points of the first captured image, and performing feature extraction on the converted image to obtain second feature points of the converted image;

[0125] According to the matching relationship between the first feature point and the second feature point and the conversion image, the first captured image is expanded to obtain an expanded image.

[0126] In some possible implementations of the embodiments of the present application, the processing module 503 is specifically configured to:

[0127] Performing distortion correction on the first captured image according to the first calibration parameter of the first camera to obtain a corrected first captured image, and performing distortion correction on the second captured image according to the second calibration parameter of the second camera to obtain a corrected second captured image;

[0128] The corrected second captured image is converted to the plane where the corrected first captured image is located to obtain a converted image.

[0129] In some possible implementations of the embodiments of the present application, the processing module 503 is specifically configured to:

[0130] Determining a homography matrix according to the first calibration parameter and the second calibration parameter;

[0131] According to the homography matrix, the corrected second captured image is converted to the plane where the corrected first captured image is located to obtain a converted image.

[0132] In some possible implementations of the embodiments of the present application, the optical center of the first camera and the optical center of the second camera are located on the same horizontal line;

[0133] The processing module 503 is specifically used for:

[0134] Performing monocular depth estimation on the second captured image to obtain a depth image of the second captured image;

[0135] According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain a transformed image;

[0136] The conversion matrix includes a rotation part and a displacement part, the rotation part includes a unit matrix, and the displacement part includes a moving distance of the first camera relative to the second camera.

[0137] In some possible implementations of the embodiments of the present application, the processing module 503 is specifically configured to:

[0138] According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain a transformed initial image;

[0139] In the case where the converted initial image is missing, determining a method for repairing the missing area according to the distance between the first camera and the second camera and the shooting distance between the first camera and the second camera;

[0140] The missing area is repaired according to the repair method, and the repaired initial image is determined as the converted image.

[0141] In some possible implementations of the embodiments of the present application, the processing module 503 is specifically configured to:

[0142] Determining an overlapping area of ​​the first captured image and the converted image according to a matching relationship between the first feature point and the second feature point;

[0143] An image area other than the overlapping area in the converted image is synthesized with the first captured image to obtain an extended image.

[0144] The embodiment of the present application installs two cameras with different field of view angles in an electronic device, and can simultaneously obtain a large FOV image and a small FOV image through one shooting, and expand the small FOV image into a large FOV image through image expansion technology. That is, the embodiment of the present application can obtain an immersive image with a large FOV through one shooting with an electronic device, and can form a good 3D effect, without the need to use professional equipment, multiple devices, or multiple shooting, which not only reduces costs but also reduces user operations.

[0145] The shooting device in the embodiment of the present application can be a device, or a component in an electronic device, such as an integrated circuit or a chip. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0146] The electronic device in the embodiment of the present application may be a terminal with an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0147] The shooting device provided in the embodiment of the present application can achieve Figures 1 to 4 The various processes in the shooting method embodiment can achieve the same technical effect. To avoid repetition, they will not be described here.

[0148] like Figure 6As shown, an embodiment of the present application also provides an electronic device 600, including a processor 601 and a memory 602, wherein the memory 602 stores programs or instructions that can be executed on the processor 601, and when the program or instructions are executed by the processor 601, the various steps of the above-mentioned shooting method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0149] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile terminals and non-mobile terminals.

[0150] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0151] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0152] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The structure of the electronic device 700 shown in the figure does not constitute a limitation on the electronic device 700. The electronic device 700 may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0153] The user input unit 707 is used to receive shooting input;

[0154] The processor 710 is configured to, in response to a shooting input, shoot through the first camera to obtain a first shot image, and shoot through the second camera to obtain a second shot image; perform image expansion on the first shot image according to the second shot image to obtain an expanded image, wherein the expanded image has the same resolution as the second shot image;

[0155] The display unit 706 is configured to display the extended image and the second captured image in a horizontal direction.

[0156] The embodiment of the present application can obtain an immersive image with a large FOV through one electronic device and one shooting input, without the need to use professional shooting equipment, thus reducing costs. There is also no need to use multiple electronic devices or manually move electronic devices, thus reducing operations, eliminating errors, and improving the quality of the immersive image.

[0157] In some possible implementations of the embodiments of the present application, the processor 710 is specifically configured to:

[0158] Converting the second captured image to the plane where the first captured image is located to obtain a converted image;

[0159] Performing feature extraction on the first captured image to obtain first feature points of the first captured image, and performing feature extraction on the converted image to obtain second feature points of the converted image;

[0160] According to the matching relationship between the first feature point and the second feature point and the conversion image, the first captured image is expanded to obtain an expanded image.

[0161] In some possible implementations of the embodiments of the present application, the processor 710 is specifically configured to:

[0162] Performing distortion correction on the first captured image according to the first calibration parameter of the first camera to obtain a corrected first captured image, and performing distortion correction on the second captured image according to the second calibration parameter of the second camera to obtain a corrected second captured image;

[0163] The corrected second captured image is converted to the plane where the corrected first captured image is located to obtain a converted image.

[0164] In some possible implementations of the embodiments of the present application, the processor 710 is specifically configured to:

[0165] Determining a homography matrix according to the first calibration parameter and the second calibration parameter;

[0166] According to the homography matrix, the corrected second captured image is converted to the plane where the corrected first captured image is located to obtain a converted image.

[0167] In some possible implementations of the embodiments of the present application, the optical center of the first camera and the optical center of the second camera are located on the same horizontal line;

[0168] The processor 710 is specifically configured to:

[0169] Performing monocular depth estimation on the second captured image to obtain a depth image of the second captured image;

[0170] According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain a transformed image;

[0171] The conversion matrix includes a rotation part and a displacement part, the rotation part includes a unit matrix, and the displacement part includes a moving distance of the first camera relative to the second camera.

[0172] In some possible implementations of the embodiments of the present application, the processor 710 is specifically configured to:

[0173] According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain a transformed initial image;

[0174] In the case where the converted initial image is missing, determining a method for repairing the missing area according to the distance between the first camera and the second camera, and the shooting distance between the first camera and the second camera;

[0175] The missing area is repaired according to the repair method, and the repaired initial image is determined as the converted image.

[0176] In some possible implementations of the embodiments of the present application, the processor 710 is specifically configured to:

[0177] Determining an overlapping area of ​​the first captured image and the converted image according to a matching relationship between the first feature point and the second feature point;

[0178] An image area other than the overlapping area in the converted image is synthesized with the first captured image to obtain an extended image.

[0179] The embodiment of the present application installs two cameras with different field of view angles in an electronic device, and can simultaneously obtain a large FOV image and a small FOV image through one shooting, and expand the small FOV image into a large FOV image through image expansion technology. That is, the embodiment of the present application can obtain an immersive image with a large FOV through one shooting with an electronic device, and can form a good 3D effect, without the need to use professional equipment, multiple devices, or multiple shooting, which not only reduces costs but also reduces user operations.

[0180] It should be understood that in the embodiment of the present application, the input unit 704 may include a graphics processor (Graphics Processing Unit, GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and at least one of other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0181] The memory 709 can be used to store software programs and various data. The memory 709 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 709 may include a volatile memory or a non-volatile memory, or the memory 709 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 709 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0182] The processor 710 may include one or more processing units; optionally, the processor 710 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 710.

[0183] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned shooting method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0184] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0185] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned shooting method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0186] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0187] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned shooting method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0188] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0190] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A shooting method, characterized in that: Applied to an electronic device, the electronic device includes a first camera and a second camera, the field of view angle of the first camera is smaller than the field of view angle of the second camera; the method includes: Receive shooting input; In response to the shooting input, shooting with the first camera to obtain a first shot image, and shooting with the second camera to obtain a second shot image; performing image expansion on the first captured image according to the second captured image to obtain an expanded image, wherein the expanded image has the same resolution as the second captured image; The extended image and the second photographed image are displayed in a horizontal direction.

2. The method according to claim 1, characterized in that: The step of performing image expansion on the first captured image according to the second captured image to obtain an expanded image includes: Converting the second captured image to the plane where the first captured image is located to obtain a converted image; performing feature extraction on the first captured image to obtain first feature points of the first captured image, and performing feature extraction on the converted image to obtain second feature points of the converted image; The first captured image is expanded according to the matching relationship between the first feature point and the second feature point and the converted image to obtain the expanded image.

3. The method according to claim 2, characterized in that The converting the second captured image to the plane where the first captured image is located to obtain a converted image includes: Performing distortion correction on the first captured image according to first calibration parameters of the first camera to obtain a corrected first captured image, and performing distortion correction on the second captured image according to second calibration parameters of the second camera to obtain a corrected second captured image; The corrected second captured image is converted to the plane where the corrected first captured image is located to obtain a converted image.

4. The method according to claim 3, characterized in that The step of converting the corrected second captured image to a plane where the corrected first captured image is located to obtain a converted image comprises: Determining a homography matrix according to the first calibration parameter and the second calibration parameter; According to the homography matrix, the corrected second captured image is converted to the plane where the corrected first captured image is located to obtain the converted image.

5. The method according to claim 2, characterized in that: The optical center of the first camera and the optical center of the second camera are located on the same horizontal line; The converting the second captured image to the plane where the first captured image is located to obtain a converted image includes: Performing monocular depth estimation on the second captured image to obtain a depth image of the second captured image; According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain the transformed image; The conversion matrix includes a rotation part and a displacement part, the rotation part includes a unit matrix, and the displacement part includes a moving distance of the first camera relative to the second camera.

6. The method according to claim 5, characterized in that The converting the depth image to the plane where the first captured image is located according to the conversion matrix and the projection matrix to obtain the converted image includes: According to the transformation matrix and the projection matrix, the depth image is transformed to the plane where the first captured image is located to obtain a transformed initial image; In the case where the converted initial image is missing, determining a method for repairing the missing area according to the distance between the first camera and the second camera, and the shooting distance between the first camera and the second camera; The missing area is repaired according to the repairing method, and the repaired initial image is determined as the converted image.

7. The method according to any one of claims 2 to 6, characterized in that: The step of performing image expansion on the first captured image according to the matching relationship between the first feature point and the second feature point and the converted image to obtain the expanded image includes: determining an overlapping area between the first captured image and the converted image according to a matching relationship between the first feature point and the second feature point; An image area in the converted image except the overlapping area is synthesized with the first captured image to obtain the extended image.

8. A photographing device, characterized in that: Applied to an electronic device, the electronic device comprises a first camera and a second camera, the field of view angle of the first camera is smaller than the field of view angle of the second camera; the device comprises: A receiving module, used for receiving shooting input; a shooting module, configured to, in response to the shooting input, shoot through the first camera to obtain a first shot image, and shoot through the second camera to obtain a second shot image; a processing module, configured to perform image expansion on the first captured image according to the second captured image to obtain an expanded image, wherein the expanded image has the same resolution as the second captured image; A display module is used to display the extended image and the second captured image in a horizontal direction.

9. The device according to claim 8, characterized in that The processing module is specifically used for: Converting the second captured image to the plane where the first captured image is located to obtain a converted image; performing feature extraction on the first captured image to obtain first feature points of the first captured image, and performing feature extraction on the converted image to obtain second feature points of the converted image; The first captured image is expanded according to the matching relationship between the first feature point and the second feature point and the converted image to obtain the expanded image.

10. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. An electronically readable storage medium, characterized in that The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

12. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 7.

Images