Panoramic image generation method, device, electronic device and medium

Abstract

The present application discloses a method, device, electronic device and medium for generating panoramic images, which belongs to the field of image processing technology. The method comprises: performing three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the front-view image; adjusting the first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle, the shooting object corresponding to the first image is within the shooting range of the panoramic image generating device; generating a target panoramic image based on the second local three-dimensional model and the front-view image.

Description

Technical Field

[0001] The embodiments of the present invention belong to the field of image processing technology, and in particular, relate to a panoramic image generation method, device, electronic device and medium. Background Art

[0002] With the rapid development of terminal technology, more and more users tend to use cameras in more portable terminal devices to take photos and record. When users want to record a scene that is not fully covered by the camera's field of view, the camera's panoramic shooting mode provides users with a better choice.

[0003] However, in the process of shooting a panoramic image, due to the different relative angles between different shooting objects and the camera, it is easy to cause obvious deformation of the finally generated panoramic image. Summary of the invention

[0004] The embodiments of the present application provide a panoramic image generation method, device, electronic device and medium, which can solve the problem of image deformation that easily occurs when taking panoramic images.

[0005] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating a panoramic image, the method comprising:

[0007] Performing three-dimensional reconstruction based on a first image of the shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the orthographic image;

[0008] Adjusting the first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle, the photographed object corresponding to the first image is within the photographing range of the panoramic image generating device;

[0009] A target panoramic image is generated based on the second local 3D model and the orthographic image.

[0010] In a second aspect, an embodiment of the present application provides a panoramic image generation device, including:

[0011] a model building unit, configured to perform three-dimensional reconstruction based on a first image of the shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the orthographic image;

[0012] an adjusting unit, configured to adjust the first observation angle of the first local three-dimensional model to a second observation angle of view to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle of view, the photographed object corresponding to the first image is within the photographing range of the panoramic image generating device;

[0013] An image generating unit is used to generate a target panoramic image based on the second local three-dimensional model and the orthographic image.

[0014] In a third aspect, an embodiment of the present application provides a terminal device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the panoramic image generation method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the panoramic image generation method described in the first aspect are implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method for generating a panoramic image as described in the first aspect.

[0017] In the embodiment of the present application, when shooting a panoramic image, a three-dimensional reconstruction can be performed based on a first image of the shooting scene and the depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the front view image, and then the first observation angle of the first local three-dimensional model is adjusted to a second observation angle to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle, the shooting object corresponding to the first image is within the shooting range of the panoramic image generating device, and finally, based on the second local three-dimensional model and the front view image, a target panoramic image is generated. On the one hand, the depth information of the image of the shooting scene can be obtained, and when establishing the local three-dimensional model, the establishment of the three-dimensional model of the front view image is omitted, thereby improving the establishment efficiency of the three-dimensional model; on the other hand, the observation angle of the local three-dimensional model is adjusted, so that the panoramic image generating device can include all the shooting objects in the target panoramic image generated at the corresponding position of the adjusted observation angle, thereby making the finally generated panoramic image not visually deformed, thereby improving the quality of the panoramic image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a specific implementation process of a panoramic image generation method provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of opening a camera of a terminal device to enter a depth panoramic image and collect depth information of an image in a panoramic image generation method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of images from multiple perspectives collected in the panoramic image generation method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of a local three-dimensional model established in the panoramic image generation method provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of adjusting the view cone vertices in the panoramic image generation method provided in an embodiment of the present application so that the first observation angle of the first local three-dimensional model is adjusted to the second observation angle;

[0023] Figure 6 A schematic diagram of the camera space and the world space in the panoramic image generation method provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a panoramic image generated before the panoramic image generation method provided by the embodiment of the present application is adopted and a schematic diagram of a panoramic image generated after the panoramic image generation method provided by the embodiment of the present application is adopted;

[0025] Figure 8 A schematic diagram of the structure of a panoramic image generation device provided in an embodiment of the present application;

[0026] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0027] Fig.10 A schematic diagram of the hardware structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] The following describes in detail the method for generating a panoramic image provided by the embodiment of the present application through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0031] In order to solve the problem in existing panoramic image generation methods that when the relative angle between the photographed object and the camera changes, the generated panoramic image is easily caused to be obviously deformed, the present application provides a panoramic image generation method, and the execution subject of the method can be but not limited to a mobile phone, a tablet computer, etc., which can be configured to execute at least one of the user terminals of the method provided in an embodiment of the present invention.

[0032] For ease of description, the following takes the method's execution subject as a terminal device capable of executing the method as an example to introduce the implementation of the method. It can be understood that the method's execution subject as a terminal device is only an exemplary description and should not be understood as a limitation on the method.

[0033] At present, when generating panoramic images, it is necessary to stitch the sub-images collected from multiple perspectives. When the viewing angle range of the panoramic image is large, the stitched panoramic image often appears deformed. When the subject is a crowd, the crowd on the same plane in the stitched panoramic image often has a small field of view in the middle and large people, while the fields of view on both sides are large and people appear small. When the subject is a building complex on the same plane, the stitched panoramic image will also have the phenomenon that the building in the middle is high and the buildings on both sides gradually become lower, which will mislead the observer of the panoramic image.

[0034] Existing methods for correcting deformed panoramic images include stretching methods such as affine transformation, which can perform targeted corrections on specific scenes to restore the actual effect. However, this method requires calibration of specific shooting scenes, so this method is only effective for some specific shooting scenes. When the shooting scene is not one of these specific shooting scenes, the image correction method using affine transformation cannot eliminate the deformation caused by the panoramic image.

[0035] To this end, the method for generating a panoramic image provided by the present invention includes: first, performing three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the front-view image; then, adjusting the first observation perspective of the first local three-dimensional model to a second observation perspective to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation perspective, the shooting object corresponding to the first image is within the shooting range of a panoramic image generating device; finally, generating a target panoramic image based on the second local three-dimensional model and the front-view image.

[0036] On the one hand, it is possible to obtain depth information of the image of the captured scene, and when establishing a local three-dimensional model, it is possible to omit the establishment of a three-dimensional model of the orthographic image, thereby improving the efficiency of establishing the three-dimensional model; on the other hand, the observation angle of the local three-dimensional model is adjusted, so that the panoramic image generating device can include all captured objects at the corresponding positions of the adjusted observation angle, thereby ensuring that the finally generated panoramic image will not be obviously deformed visually, thereby improving the quality of the panoramic image.

[0037] Combine the following Figure 1 The specific implementation flow diagram of the panoramic image generation method shown in the figure provides a detailed introduction to the implementation process of the method, including:

[0038] S110, performing three-dimensional reconstruction based on a first image of the shooting scene and depth information of the first image to obtain a first local three-dimensional model, where the first image is an image of the shooting scene other than the front-view image.

[0039] Among them, three-dimensional reconstruction is performed based on the first image of the shooting scene and the depth information of the first image. Specifically, three-dimensional reconstruction can be performed according to the grayscale value of each pixel in the first image corresponding to the depth information of the first image, and the depth information of the first image to obtain a first local three-dimensional model.

[0040] Optionally, before performing three-dimensional reconstruction based on the first image of the captured scene and the depth information of the first image to obtain the first local three-dimensional model, the method provided in the embodiment of this specification further includes:

[0041] The depth information of the first image is acquired through a TOF lens or multi-eye vision technology.

[0042] Figure 2 A schematic diagram of opening the camera of a terminal device to enter a depth panoramic image and collect depth information of an image in the panoramic image generation method provided in an embodiment of the present application. Figure 2 When the user wants to take a panoramic image, he can open the camera of the terminal device and click Figure 2, and select the "Depth Panorama" mode to capture images from multiple perspectives of the subject for which you want to generate a panoramic image. In this "Depth Panorama" mode, the user can fix the position of the camera, change the camera's shooting angle clockwise or counterclockwise, and capture images within the required viewing angle range, which includes all the photographed objects. At this time, in this shooting mode, the camera of the terminal device, i.e., the panoramic image generation device, will automatically obtain depth information about the images of multiple perspectives of the panoramic image.

[0043] Optionally, the number of first images may also be multiple, and three-dimensional reconstruction is performed based on the first image of the shooting scene and the depth information of the first image to obtain a first local three-dimensional model, including:

[0044] Three-dimensional reconstruction is performed based on at least two first images of the shooting scene and depth information of the at least two first images to obtain a first local three-dimensional model.

[0045] Figure 3 A schematic diagram of images from multiple perspectives collected in the panoramic image generation method provided in an embodiment of the present application. Figure 3 In the above example, the images collected from multiple perspectives include an image from a normal perspective (i.e. Figure 3 The middle image (b) and the first image (i.e. Figure 3 Image (a) and image (c) in the figure.

[0046] The orthographic image is an image whose angle with the plane projected by all the photographed objects in the target panoramic image is zero. Since the observation angle of the generated target panoramic image is consistent with the direction of the orthographic image, the establishment of the three-dimensional model of the orthographic image can be omitted during the three-dimensional model establishment process, thereby saving computing resources and improving the modeling efficiency of the first local three-dimensional model.

[0047] Figure 4 This is a schematic diagram of a local three-dimensional model established in the panoramic image generation method provided in the embodiment of the present application. Figure 4 In order to improve the efficiency of building a three-dimensional model and save computing resources, the orthographic image may not be involved in the building of the local three-dimensional model.

[0048] In addition, the specific three-dimensional model building process of performing three-dimensional reconstruction based on the first image of the shooting scene and the depth information of the first image to obtain the first local three-dimensional model can refer to the existing three-dimensional model building process. As long as it involves the method of building a three-dimensional model based on the depth information of the image and the two-dimensional image captured by the camera, it is applicable to the present application, and the embodiments of the present application do not make specific limitations on this.

[0049] S120: Adjust the first observation perspective of the first partial three-dimensional model to a second observation perspective to obtain a second partial three-dimensional model.

[0050] Wherein, at a corresponding position of the second observation angle of view, the photographing object corresponding to the first image is within the photographing range of the panoramic image generating device.

[0051] Optionally, after the first local 3D model is converted into a 2D image by means of projection transformation, in order to avoid obvious deformation problems in the target panoramic image generated after the converted 2D images are spliced, the embodiment of the present application may adjust the observation angle of the first local 3D model before converting the first local 3D model into a 2D image by means of projection transformation, so that the panoramic image generation device can include all the objects in the target panoramic image finally generated at the corresponding position of the adjusted observation angle. Specifically, the first observation angle of the first local 3D model is adjusted to the second observation angle to obtain the second local 3D model, including:

[0052] The position of the camera of the panoramic image generating device that captures the first image is taken as the vertex of the viewing cone;

[0053] Move the vertex of the viewing cone away from the photographic object corresponding to the first image, that is, move the panoramic image generating device away from the photographic object corresponding to the first image, so as to adjust the observation angle of the first local three-dimensional model to the second observation angle, so that the panoramic image generating device can capture all the photographic objects in the target panoramic image at the corresponding position of the second observation angle.

[0054] Figure 5 A schematic diagram of adjusting the view cone vertices in the panoramic image generation method provided in an embodiment of the present application so that the first observation angle of the first local three-dimensional model is adjusted to the second observation angle. Figure 5 In the present invention, when the panoramic image generating device, i.e., the camera of the terminal device, takes a photo, the camera is close to the subject corresponding to the first image in the initial state, so the single field of view of the camera cannot cover all the subjects corresponding to the first image and the orthographic image. However, the field of view of the camera is usually constant. Taking this into consideration, the embodiment of the present application chooses to change the scene range captured by the camera when the object distance is changed, that is, to increase the object distance to expand the scene range captured by the camera.

[0055] Specifically, the panoramic image generating device can be the vertex of the viewing cone. In order to obtain a larger field of view, the vertex of the viewing cone is translated backward until the shooting field of view of the panoramic image generating device can cover all objects in the target panoramic image, so that the target panoramic image finally presented avoids visual deformation, thereby improving the image quality of the generated panoramic image.

[0056] S130: Generate a target panoramic image based on the second local 3D model and the orthographic image.

[0057] Optionally, since the second local 3D model is 3D and the target panoramic image to be generated is a 2D image, the second local 3D model can be converted into a 2D image by projection transformation. Specifically, based on the second local 3D model and the orthographic image, generating the target panoramic image includes:

[0058] Obtain a first two-dimensional image according to the first image and the second local three-dimensional model;

[0059] The first two-dimensional image and the orthographic image are synthesized to obtain a target panoramic image.

[0060] Optionally, obtaining a first two-dimensional image according to the first image and the second local three-dimensional model includes:

[0061] Acquire a first distance between a vertex of the viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generating device, a second distance between a vertex of the viewing cone at the second observation angle and a photographed object, and a horizontal observation angle and a vertical observation angle in the second observation angle;

[0062] A first two-dimensional image is obtained based on the first distance, the second distance, the horizontal observation angle and the vertical observation angle in the second observation angle, the first image, and the second local three-dimensional model.

[0063] Among them, projection transformation is the process of transforming a three-dimensional object into a two-dimensional image. Figure 6 A schematic diagram of the camera space and world space in the panoramic image generation method provided in the embodiment of the present application. Figure 6 In , all objects in the 3D scene will participate in rendering only when they are within the viewing cone. Parts outside the viewing cone will not participate in rendering. Figure 6 In the above figure, the specific geometric shape of the "view cone" is limited to the two angles of the horizontal observation angle (FovH) and the vertical observation angle (FovV). In addition, the positions of the top and bottom surfaces of the pyramid can also be determined. These two surfaces can also be called "clipping planes". Among them, the plane where the camera is located is usually recorded as the top surface of the pyramid (near plane), and the plane where the object is located is usually recorded as the bottom surface of the pyramid (far plane). The distance from the top surface of the pyramid to the camera is n, and the distance from the bottom surface of the pyramid to the camera is f.

[0064] Objects in three-dimensional space are usually represented by world coordinates, and when a camera takes a picture of an object, the world coordinate system can be converted into the camera coordinate system. This requires conversion between the two coordinate systems, and the conversion from world coordinates to camera coordinates can be achieved through translation and rotation.

[0065] The coordinate system in the camera space takes the direction of the camera lens as the z-axis (which can be positive or negative) to calculate the "Z value / depth value" of objects in the three-dimensional space. When there are multiple objects in the three-dimensional scene, from the perspective of the viewing cone, multiple objects may be mapped to the same pixel. At this time, the color of the object with the smallest absolute value of the "Z value / depth value" can be selected as the color of this pixel, that is, the color of the object closest to the camera is selected as the color of this pixel.

[0066] Among them, the matrix formula of projection transformation is as follows:

[0067]

[0068] Among them, (x, y, z) are the coordinates of the object in the second local three-dimensional model, n is the first distance between the vertex of the viewing cone at the second observation perspective and the panoramic image generating device, and f is the second distance between the vertex of the viewing cone at the second observation perspective and all the objects in the generated target panoramic image.

[0069] The results after normalizing the above calculation results are as follows:

[0070]

[0071] The final coordinates in two-dimensional space That is, it corresponds to the coordinates (x, y, z) in the second local three-dimensional model.

[0072] Optionally, in order to improve the efficiency of generating panoramic images, the orthographic image can be retained, and after the three-dimensional model is converted into a two-dimensional image by means of projection transformation, the orthographic image is spliced ​​into the two-dimensional image according to the complete shape of the object in the image, so as to generate a target panoramic image. Specifically, the first two-dimensional image and the orthographic image are synthesized to obtain the target panoramic image, including:

[0073] The first two-dimensional image and the orthographic image are stitched together to generate a target panoramic image.

[0074] Figure 7 It is a schematic diagram of a panoramic image generated before adopting the panoramic image generation method provided in an embodiment of the present application and a panoramic image generated after adopting the panoramic image generation method provided in an embodiment of the present application. Figure 7The left figure (a) is a schematic diagram of a panoramic image generated without adjusting the observation angle of the first local 3D model. Figure 7 The right figure (b) in FIG. 1 is a schematic diagram of a panoramic image generated by adjusting the first observation angle of the first local 3D model to the second observation angle. Obviously, the panoramic image generated by adjusting the first observation angle of the first local 3D model to the second observation angle can avoid obvious image deformation, thereby improving the image quality of the panoramic image.

[0075] In the embodiment of the present application, when shooting a panoramic image, a three-dimensional reconstruction can be performed based on a first image of the shooting scene and the depth information of the first image to obtain a first local three-dimensional model, where the first image is an image of the shooting scene other than the front view image, and then the first observation angle of the first local three-dimensional model is adjusted to a second observation angle to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle, the shooting object corresponding to the first image is within the shooting range of the panoramic image generating device, and finally, based on the second local three-dimensional model and the front view image, a target panoramic image is generated. On the one hand, the depth information of the image of the shooting scene can be obtained, and when establishing the local three-dimensional model, the establishment of the three-dimensional model of the front view image is omitted, thereby improving the establishment efficiency of the three-dimensional model; on the other hand, the observation angle of the local three-dimensional model is adjusted, so that the panoramic image generating device can include all the shooting objects at the corresponding position of the adjusted observation angle, thereby making the finally generated panoramic image not visually deformed, thereby improving the quality of the panoramic image.

[0076] It should be noted that the panoramic image generation method provided in the embodiment of the present application can be executed by a panoramic image generation device, or a control module in the panoramic image generation device for executing the panoramic image generation method. In the embodiment of the present application, the panoramic image generation device executing the panoramic image generation method is taken as an example to illustrate the panoramic image generation device provided in the embodiment of the present application.

[0077] See also Figure 8 , is a schematic diagram of the structure of a panoramic image generating device 800 provided in an embodiment of the present invention, such as Figure 8 As shown, it includes a model building unit 802, an adjustment unit 803 and an image generating unit 804, wherein:

[0078] A model building unit 802 is used to perform three-dimensional reconstruction based on a first image of the shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the orthographic image;

[0079] An adjusting unit 803 is configured to adjust the first observation angle of the first local three-dimensional model to a second observation angle to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle, the photographed object corresponding to the first image is within the photographing range of the panoramic image generating device;

[0080] The image generating unit 804 is configured to generate a target panoramic image based on the second local 3D model and the orthographic image.

[0081] Optionally, in one implementation, the image generating unit 804 is configured to:

[0082] Obtaining a first two-dimensional image according to the first image and the second local three-dimensional model;

[0083] The first two-dimensional image and the orthographic image are synthesized to obtain the target panoramic image.

[0084] Optionally, in one implementation, the image generating unit 804 is configured to:

[0085] Acquire a first distance between a vertex of the viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generating device, a second distance between a vertex of the viewing cone at the second observation angle and a photographed object, and a horizontal observation angle and a vertical observation angle in the second observation angle;

[0086] The first two-dimensional image is obtained based on the first distance, the second distance, the horizontal observation angle and the vertical observation angle in the second observation angle, the first image, and the second local three-dimensional model.

[0087] Optionally, in one implementation, before the model building unit 802 performs three-dimensional reconstruction based on the first image of the captured scene and the depth information of the first image, the device further includes:

[0088] The information acquisition unit 801 is used to acquire the depth information of the first image through a TOF lens or multi-eye vision technology.

[0089] Optionally, in one implementation, the model building unit 802 is used to:

[0090] Three-dimensional reconstruction is performed based on at least two first images of the shooting scene and depth information of the at least two first images to obtain the first local three-dimensional model.

[0091] The panoramic image generation device 800 provided in the embodiment of the present application can perform three-dimensional reconstruction based on the first image of the shooting scene and the depth information of the first image when shooting a panoramic image, so as to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the front view image, and then adjust the first observation angle of the first local three-dimensional model to the second observation angle to obtain a second local three-dimensional model, wherein at the corresponding position of the second observation angle, the shooting object corresponding to the first image is within the shooting range of the panoramic image generation device, and finally generate a target panoramic image based on the second local three-dimensional model and the front view image. On the one hand, the depth information of the image of the shooting scene can be obtained, and when establishing the local three-dimensional model, the establishment of the three-dimensional model of the front view image is omitted, thereby improving the establishment efficiency of the three-dimensional model; on the other hand, the observation angle of the local three-dimensional model is adjusted, so that the panoramic image generation device can include all the shooting objects at the corresponding position of the adjusted observation angle, thereby making the finally generated panoramic image not visually deformed, thereby improving the quality of the panoramic image.

[0092] The panoramic image generation device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0093] The panoramic image generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0094] The panoramic image generation device provided in the embodiment of the present application can achieve Figures 1 to 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0095] Optional, such as Fig. 9As shown, an embodiment of the present application also provides an electronic device M09, including a memory M91, a processor M92, and a program or instruction stored in the memory M91 and executable on the processor M92. When the program or instruction is executed by the processor M92, each process of the above-mentioned panoramic image generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0096] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0097] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0098] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0099] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0100] Among them, the processor 1010 performs three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the front-view image; adjusts the first observation perspective of the first local three-dimensional model to a second observation perspective to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation perspective, the shooting object corresponding to the first image is within the shooting range of the panoramic image generating device; and generates a target panoramic image based on the second local three-dimensional model and the front-view image.

[0101] In the embodiment of the present application, when shooting a panoramic image, a three-dimensional reconstruction can be performed based on a first image of the shooting scene and the depth information of the first image to obtain a first local three-dimensional model, where the first image is an image of the shooting scene other than the front view image, and then the first observation angle of the first local three-dimensional model is adjusted to a second observation angle to obtain a second local three-dimensional model, wherein at a corresponding position of the second observation angle, the shooting object corresponding to the first image is within the shooting range of the panoramic image generating device, and finally, based on the second local three-dimensional model and the front view image, a target panoramic image is generated. On the one hand, the depth information of the image of the shooting scene can be obtained, and when establishing the local three-dimensional model, the establishment of the three-dimensional model of the front view image is omitted, thereby improving the establishment efficiency of the three-dimensional model; on the other hand, the observation angle of the local three-dimensional model is adjusted, so that the panoramic image generating device can include all the shooting objects at the corresponding position of the adjusted observation angle, thereby making the finally generated panoramic image not visually deformed, thereby improving the quality of the panoramic image.

[0102] Optionally, the processor 1010 is further configured to obtain a first two-dimensional image according to the first image and the second local three-dimensional model;

[0103] The first two-dimensional image and the orthographic image are synthesized to obtain the target panoramic image.

[0104] Optionally, the processor 1010 is further configured to obtain a first distance between a vertex of the viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generating device, a second distance between a vertex of the viewing cone at the second observation angle and a photographed object, and a horizontal observation angle and a vertical observation angle in the second observation angle;

[0105] The first two-dimensional image is obtained based on the first distance, the second distance, the horizontal observation angle and the vertical observation angle in the second observation angle, the first image, and the second local three-dimensional model.

[0106] Optionally, the processor 1010 is further configured to obtain depth information of the first image through a TOF lens or multi-eye vision technology.

[0107] Optionally, the processor 1010 is further configured to perform three-dimensional reconstruction based on at least two first images of the captured scene and depth information of the at least two first images to obtain the first local three-dimensional model.

[0108] By adopting the method provided in the present application, on the one hand, the depth information of the image of the captured scene can be obtained, and when establishing a local three-dimensional model, the three-dimensional model establishment of the orthographic image is omitted, thereby improving the efficiency of establishing the three-dimensional model; on the other hand, the observation angle of the local three-dimensional model projection is adjusted, so that the panoramic image generation device can include all the captured objects at the corresponding positions of the adjusted observation angle, thereby ensuring that the finally generated panoramic image will not be obviously deformed visually, thereby improving the quality of the panoramic image.

[0109] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch processor. Other input devices 10072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.

[0110] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned panoramic image generation method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0111] The processor is a processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0112] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned panoramic image generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0113] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0114] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0115] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0116] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A panoramic image generation method, applied to a panoramic image generation device, characterized in that: include: Performing three-dimensional reconstruction based on a first image of the shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the orthographic image; The first observation angle of the first local three-dimensional model is adjusted to a second observation angle to obtain a second local three-dimensional model, wherein, at a corresponding position of the second observation angle, the photographed object corresponding to the first image is within the photographing range of the panoramic image generating device; and compared with the corresponding position of the first observation angle, the corresponding position of the second observation angle is farther away from the photographed object; A target panoramic image is generated based on the second local 3D model and the orthographic image.

2. The method according to claim 1, characterized in that Generating a target panoramic image based on the second local 3D model and the orthographic image includes: Obtaining a first two-dimensional image according to the first image and the second local three-dimensional model; The first two-dimensional image and the orthographic image are synthesized to obtain the target panoramic image.

3. The method according to claim 2, characterized in that Obtaining a first two-dimensional image according to the first image and the second local three-dimensional model includes: Acquire a first distance between a vertex of the viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generating device, a second distance between a vertex of the viewing cone at the second observation angle and a photographed object, and a horizontal observation angle and a vertical observation angle in the second observation angle; The first two-dimensional image is obtained based on the first distance, the second distance, the horizontal observation angle and the vertical observation angle in the second observation angle, the first image, and the second local three-dimensional model.

4. The method according to claim 1, characterized in that Before performing three-dimensional reconstruction based on the first image of the captured scene and the depth information of the first image, the method further includes: The depth information of the first image is obtained through a TOF lens or multi-eye vision technology.

5. The method according to any one of claims 1 to 4, characterized in that: Performing three-dimensional reconstruction based on a first image of a shooting scene and depth information of the first image to obtain a first local three-dimensional model includes: Three-dimensional reconstruction is performed based on at least two first images of the shooting scene and depth information of the at least two first images to obtain the first local three-dimensional model.

6. A panoramic image generation device, characterized in that: include: a model building unit, configured to perform three-dimensional reconstruction based on a first image of the shooting scene and depth information of the first image to obtain a first local three-dimensional model, wherein the first image is an image of the shooting scene other than the orthographic image; an adjusting unit, configured to adjust the first observation angle of the first local three-dimensional model to a second observation angle of view, so as to obtain a second local three-dimensional model, wherein, at a corresponding position of the second observation angle of view, the photographed object corresponding to the first image is within the photographing range of the panoramic image generating device; and compared with the corresponding position of the first observation angle of view, the corresponding position of the second observation angle of view is farther away from the photographed object; An image generating unit is used to generate a target panoramic image based on the second local three-dimensional model and the orthographic image.

7. The device according to claim 6, characterized in that The image generating unit is used for: Obtaining a first two-dimensional image according to the first image and the second local three-dimensional model; The first two-dimensional image and the orthographic image are synthesized to obtain the target panoramic image.

8. The device according to claim 7, characterized in that The image generating unit is used for: Acquire a first distance between a vertex of the viewing cone at a second observation angle in the second local three-dimensional model and a camera in the panoramic image generating device, a second distance between a vertex of the viewing cone at the second observation angle and a photographed object, and a horizontal observation angle and a vertical observation angle in the second observation angle; The first two-dimensional image is obtained based on the first distance, the second distance, the horizontal observation angle and the vertical observation angle in the second observation angle, the first image, and the second local three-dimensional model.

9. The device according to claim 6, characterized in that Before the model building unit performs three-dimensional reconstruction based on the first image of the captured scene and the depth information of the first image, the device further includes: An information acquisition unit is used to acquire depth information of the first image through a TOF lens or multi-eye vision technology.

10. The device according to any one of claims 6 to 9, characterized in that: The model building unit is used to: Three-dimensional reconstruction is performed based on at least two first images of the shooting scene and depth information of the at least two first images to obtain the first local three-dimensional model.

11. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the panoramic image generation method as described in any one of claims 1 to 5.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the panoramic image generation method according to any one of claims 1 to 5 are implemented.

Images